473,836 Members | 1,563 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Sqlite3 textfactory and user-defined function

I've run into a problem with text encoding in the Sqlite3 module. I
think it may be a bug. By default sqlite3 converts strings in the
database from UTF-8 to unicode. This conversion can be controlled by
changing the connection's text_factory.

I have a database that stores strings in 8-bit ISO-8859. So, I set
the text_factory to do no conversion. In my database I use user
defined functions. I noticed that even when I set text_factory =
lambda x:x, it appears to do UTF-8 to unicode conversion on strings
that are passed to my user defined function.

I've included a small program that illustrates the problem. It
creates a database and table in memory and then populates 2 rows. One
row contains an ASCII string. The other row contains a string with
the non-ascii string, "Tést".

Then, the program does an SQL select which calls the user-defined
function, my_func(). The resulting row tuples contain 8-bit strings.
But, my_func() is passed unicode strings. Notice, my_func is called
with None instead of "Tést". I suspect this is because the binary
representation of "Tést" is not valid UTF-8.

Is there a way to turn off the UTF-8 to unicode when my_func() is
called? Is this a bug or intended behavior?

import sqlite3

def create_table(db ase):
#dbase.execute( r"""PRAGMA encoding = "UTF-16le";""")
dbase.execute(r """CREATE TABLE `my_table` ( 'id' INTEGER, 'column'
BLOB); """)

def add_rows(dbase) :
c = dbase.cursor()
string1 = "Test"
string2 = "T\xe9st"
try:
print string1
c.execute(r"""I NSERT INTO `my_table` ('id', 'column') VALUES
(?,?)""", (1,string1))
print string2
c.execute(r"""I NSERT INTO `my_table` ('id', 'column') VALUES
(?,?)""", (2,string2,))
finally:
c.close()

def select_rows(dba se):
c = dbase.cursor()
try:
c.execute(r"""S ELECT *, my_func(`column `) FROM `my_table`""")
for row in c:
print row
finally:
c.close()

def factory(x):
print 'x =', x
return x

def my_func(p):
print 'my_func(%r) type = %s' % (p,type(p))

def my_test():
db_path = ":memory:"

try:
os.remove(db_pa th)
except:
pass

dbase = sqlite3.connect (db_path)
dbase.text_fact ory = lambda x:x
dbase.create_fu nction('my_func ', 1, my_func)
try:
create_table(db ase)
add_rows(dbase)
select_rows(dba se)
finally:
dbase.commit()
dbase.close()

my_test()

Jun 27 '08 #1
1 3110
je***********@h otmail.com wrote:
I've run into a problem with text encoding in the Sqlite3 module. I
think it may be a bug. By default sqlite3 converts strings in the
database from UTF-8 to unicode. This conversion can be controlled by
changing the connection's text_factory.

I have a database that stores strings in 8-bit ISO-8859. So, I set
the text_factory to do no conversion. In my database I use user
defined functions. I noticed that even when I set text_factory =
lambda x:x, it appears to do UTF-8 to unicode conversion on strings
that are passed to my user defined function. [...]
I've answered the same question on the pysqlite mailing list a few weeks
back:

Thread "Trouble with create_function interface to sqlite"

http://itsystementwicklung.de/piperm...ay/000062.html

-- Gerhard
Jun 27 '08 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
1980
by: John Machin | last post by:
Apologies in advance if this is a bit bloggy, but I'd like to get comments on whether I've lost the plot (or, more likely, failed to acquire it) before I start reporting bugs etc. From "What's new ...": """ # Create table c.execute('''create table stocks (date timestamp, trans varchar, symbol varchar, qty decimal, price decimal)''')
66
7100
by: mensanator | last post by:
Probably just me. I've only been using Access and SQL Server for 12 years, so I'm sure my opinions don't count for anything. I was, nevertheless, looking forward to Sqlite3. And now that gmpy has been upgraded, I can go ahead and install Python 2.5. So I open the manual to Section 13.13 where I find the first example of how to use Sqlite3:
2
4957
by: Josh | last post by:
Hi, I'm running into a problem when trying to create a view in my sqlite database in python. I think its a bug in the sqlite3 api that comes with python 2.5. This works as expected: conn = sqlite3.connect(':memory:') conn.execute("create table foo (a int,b int)") conn.execute('create view bar as select * from foo')
4
8927
by: Simon | last post by:
I installed the source code on unix for python 2.5.1. The install went mainly okay, except for some failures regarding: _ssl, _hashlib, _curses, _curses_panel. No errors regarding sqlite3. However, when I start python and do an import sqlite3 I get: /ptmp/bin/python Python 2.5.1 (r251:54863, May 29 2007, 05:19:30) on sunos5
0
1458
by: Josh Ritter | last post by:
A number of our Windows customers have an issue with the sqlite3 module included with Python 2.5.1 We've tracked the problem down to the sqlite3.dll included with the Python 2.5.1 distrubtion. It is stripped and thus cannot be relocated. This causes the following exception on computers where something is already loaded into the address the sqlite3.dll wants to use: File "sqlite3\__init__.pyc", line 24, in <module> File...
33
2578
by: Stef Mientki | last post by:
hello, I discovered that boolean evaluation in Python is done "fast" (as soon as the condition is ok, the rest of the expression is ignored). Is this standard behavior or is there a compiler switch to turn it on/off ? thanks, Stef Mientki
0
4136
by: David | last post by:
- Are there any peculiarities with using curs.executemany(...) vs. multiple How many times are you calling execute vs a single executemany? The python call overhead will add up for thousands of calls. The relevant source code is here if you're interested: http://svn.python.org/projects/python/trunk/Modules/_sqlite/cursor.c
3
2980
by: milan_sanremo | last post by:
I have sqlite installed, but when I try to import sqlite3 I receive: Python 2.5.1 (r251:54863, Nov 3 2007, 02:54:36) on sunos5 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named sqlite3 Yet:
0
1766
by: Ben Lee | last post by:
hi folks -- a quick python and sqlite3 performance question. i find that inserting a million rows of in-memory data into an in-memory database via a single executemany() is about 30% slower than using the sqlite3 CLI and the .import command (reading the same data from a disk file, even.) i find this surprising, executemany() i assume is using a prepared statement and this is exactly what the .import command does (based on my quick...
15
14815
by: Kurda Yon | last post by:
Hi, I try to "build" and "install" pysqlite? After I type "python setup.py build" I get a lot of error messages? The first error is "src/ connection.h:33:21: error: sqlite3.h: No such file or directory". So, I assume that the absence of the "sqlite3.h" is the origin of the problem. I found on the web, that this file should be either in "/usr/local/ include" or in "/usr/local/lib". I check this directories and I really
0
9827
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9677
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10862
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10607
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10262
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9389
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
6986
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5662
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
3
3119
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.