473,573 Members | 2,491 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

sqlite utf8 encoding error

I have an application that uses sqlite3 to store job/error data. When
I log in as a German user the error codes generated are translated into
German. The error code text is then stored in the db. When I use the
fetchall() to retrieve the data to generate a report I get the
following error:

Traceback (most recent call last):
File "c:\Pest3\Gloss er\baseApp\repo rtGen.py", line 199, in
OnGenerateButto nNow
self.OnGenerate Button(event)
File "c:\Pest3\Gloss er\baseApp\repo rtGen.py", line 243, in
OnGenerateButto n
warningresult = messagecursor1. fetchall()
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'

I thought that all strings were stored in unicode in sqlite.

Greg Miller

Nov 22 '05 #1
16 8475
Greg Miller wrote:
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'
$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text für Übereinstimmung sfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
File "test.py", line 4, in ?
u = s.decode("utf-8") # <-- this gives an error
File "lib/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range
I thought that all strings were stored in unicode in sqlite.


did you pass in a Unicode string or an 8-bit string when you stored the text ?

</F>

Nov 22 '05 #2
Greg Miller wrote:
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'
$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text für Übereinstimmung sfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
File "test.py", line 4, in ?
u = s.decode("utf-8") # <-- this gives an error
File "lib/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range
I thought that all strings were stored in unicode in sqlite.


did you pass in a Unicode string or an 8-bit string when you stored the text ?

</F>

Nov 22 '05 #3
Greg Miller enlightened us with:
'Keinen Text für Übereinstimmung sfehler gefunden'
You posted it as "Keinen Text f<FC>r ...", which is Latin-1, not
UTF-8.
I thought that all strings were stored in unicode in sqlite.


Only if you put them into the DB as such. Make sure you're inserting
UTF-8 text, since the DB won't do character conversion for you.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Nov 22 '05 #4
Greg Miller enlightened us with:
'Keinen Text für Übereinstimmung sfehler gefunden'
You posted it as "Keinen Text f<FC>r ...", which is Latin-1, not
UTF-8.
I thought that all strings were stored in unicode in sqlite.


Only if you put them into the DB as such. Make sure you're inserting
UTF-8 text, since the DB won't do character conversion for you.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Nov 22 '05 #5
Fredrik Lundh napisa³(a):
UnicodeDecode Error: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'


$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text für Übereinstimmung sfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
File "test.py", line 4, in ?
u = s.decode("utf-8") # <-- this gives an error
File "lib/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range


I cann't wait for the moment when encoded strings go away from Python.
The more I program in this language, the more confusion this difference
is causing. Now most of functions and various object's methods accept
strings and unicode, making it hard to find sources of Unicode*Errors.

--
Jarek Zgoda
http://jpa.berlios.de/
Nov 22 '05 #6
Fredrik Lundh napisa³(a):
UnicodeDecode Error: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'


$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text für Übereinstimmung sfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
File "test.py", line 4, in ?
u = s.decode("utf-8") # <-- this gives an error
File "lib/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range


I cann't wait for the moment when encoded strings go away from Python.
The more I program in this language, the more confusion this difference
is causing. Now most of functions and various object's methods accept
strings and unicode, making it hard to find sources of Unicode*Errors.

--
Jarek Zgoda
http://jpa.berlios.de/
Nov 22 '05 #7
Jarek Zgoda wrote:
Fredrik Lundh napisa³(a):
UnicodeDecode Error: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'


$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text für Übereinstimmung sfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
File "test.py", line 4, in ?
u = s.decode("utf-8") # <-- this gives an error
File "lib/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range


I cann't wait for the moment when encoded strings go away from Python.
The more I program in this language, the more confusion this difference
is causing. Now most of functions and various object's methods accept
strings and unicode, making it hard to find sources of Unicode*Errors.


Library writers can speed up the transition by hiding 8bit interface,
for example:

import sqlite
sqlite.I_promis e_to_pass_8bit_ string_only_in_ utf8_encoding(m y_signature="si g.gif")

if you don't call this function 8bit strings will not be accepted :)
IMHO if libraries keep on excepting both str and unicode till python
3.0, it will just prolong the confusion of unicode newbies instead of
guiding them in the right direction _right now_.

Nov 22 '05 #8
Jarek Zgoda wrote:
Fredrik Lundh napisa³(a):
UnicodeDecode Error: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'


$ more test.py
# -*- coding: iso-8859-1 -*-
u = u'Keinen Text für Übereinstimmung sfehler gefunden'
s = u.encode("iso-8859-1")
u = s.decode("utf-8") # <-- this gives an error

$ python test.py
Traceback (most recent call last):
File "test.py", line 4, in ?
u = s.decode("utf-8") # <-- this gives an error
File "lib/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_de code(input, errors, True)
UnicodeDecodeEr ror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range


I cann't wait for the moment when encoded strings go away from Python.
The more I program in this language, the more confusion this difference
is causing. Now most of functions and various object's methods accept
strings and unicode, making it hard to find sources of Unicode*Errors.


Library writers can speed up the transition by hiding 8bit interface,
for example:

import sqlite
sqlite.I_promis e_to_pass_8bit_ string_only_in_ utf8_encoding(m y_signature="si g.gif")

if you don't call this function 8bit strings will not be accepted :)
IMHO if libraries keep on excepting both str and unicode till python
3.0, it will just prolong the confusion of unicode newbies instead of
guiding them in the right direction _right now_.

Nov 22 '05 #9
On 17 Nov 2005 03:47:00 -0800, "Greg Miller" <et**********@g mail.com>
wrote:
I have an application that uses sqlite3 to store job/error data. When
I log in as a German user the error codes generated are translated into
German. The error code text is then stored in the db. When I use the
fetchall() to retrieve the data to generate a report I get the
following error:

Traceback (most recent call last):
File "c:\Pest3\Gloss er\baseApp\repo rtGen.py", line 199, in
OnGenerateButt onNow
self.OnGenerate Button(event)
File "c:\Pest3\Gloss er\baseApp\repo rtGen.py", line 243, in
OnGenerateButt on
warningresult = messagecursor1. fetchall()
UnicodeDecodeE rror: 'utf8' codec can't decode bytes in position 13-18:
unsupported Unicode code range

does anyone have any idea on what could be going wrong? The string
that I store in the database table is:

'Keinen Text für Übereinstimmung sfehler gefunden'

I thought that all strings were stored in unicode in sqlite.

No, they are stored as UTF-8 in sqlite and pysqlite has no way to make
sure the string you insert into the database is really encoded in
UTF-8 (the only secure way is to use Unicode strings).

How did you insert that string?

As a partial solution, try to disable automatic conversion of text
fields in Unicode strings:
def convert_text(s) :
# XXX do not use Unicode
return s
# Register the converter with SQLite
sqlite.register _converter("TEX T", convert_text)
....connect(".. .",
detect_types=sq lite.PARSE_DECL TYPES|sqlite.PA RSE_COLNAMES
)


Regards Manlio Perillo
Nov 22 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
6891
by: Richard Lewis | last post by:
Hi there, I'm having a problem with unicode files and ftplib (using Python 2.3.5). I've got this code: xml_source = codecs.open("foo.xml", 'w+b', "utf8") #xml_source = file("foo.xml", 'w+b') ftp.retrbinary("RETR foo.xml", xml_source.write)
1
27687
by: Vlajko Knezic | last post by:
Not so sure what is going on here but is something to do with the way UTF8 is handled in Perl and/or LibXML The sctript below: - accepts a value from a form text field; - builds XML document around it,
3
7743
by: hunterb | last post by:
I have a file which has no BOM and contains mostly single byte chars. There are numerous double byte chars (Japanese) which appear throughout. I need to take the resulting Unicode and store it in a DB and display it onscreen. No matter which way I open the file, convert it to Unicode/leave it as is or what ever, I see all single bytes ok, but...
4
11497
by: H Lee | last post by:
Hi, I'm an XML newbie, and not sure if this is the appropriate newsgroup to post my question, so feel free to suggest other newgroups where I should post this message if this is the case. I'm having issues using XmlTextWriter, saving it out to a file with UTF8 encoding, and seeing "dirty", or "human unreadable" characters show up *right...
0
362
by: Greg Miller | last post by:
I have an application that uses sqlite3 to store job/error data. When I log in as a German user the error codes generated are translated into German. The error code text is then stored in the db. When I use the fetchall() to retrieve the data to generate a report I get the following error: Traceback (most recent call last): File...
4
4878
by: EmeraldShield | last post by:
(Dot Net 2 C# application - using Encoding.UTF8 with a StreamReader) I have a very strange problem that I cannot explain with a UTF8 Readline() although this could exist in other types of encoding, I have not tried them. Our application wrote this sequence to a UTF8 file. Now I am loading it back and the text is not coming back in the same...
7
13047
by: amygdala | last post by:
Hi, I'm trying to let PHP write a 'sitemap.xml' sitemap for Google and other searchengines. It's working, except that the content in the XML file doesn't seem to be UTF8. (Which it should be, judging by the information given on Google's webmaster helpcenter). The way I test to see if the content is UTF8, is by opening the XML file in...
4
4472
by: weheh | last post by:
I'm developing a cgi-bin application that must be unicode sensitive. I'm striving for a UTF8 implementation. I'm running python 2.3 on a development machine (windows xp) and a server (windows xp server). Both environments are running Apache 2.2 with the same configuration file. The problem is this. On my development machine I get the...
39
5840
by: alex | last post by:
I've converted a latin1 database I have to utf8. The process has been: # mysqldump -u root -p --default-character-set=latin1 -c --insert-ignore --skip-set-charset mydb mydb.sql # iconv -f ISO-8859-1 -t UTF-8 mydb.sql mydb_utf8.sql mysqlCREATE DATABASE mydb_utf8 CHARACTER SET utf8 COLLATE utf8_general_ci;
0
7747
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7987
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8180
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7740
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8036
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5557
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5263
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3706
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3703
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.