473,847 Members | 1,459 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Encoding and norwegian (non ASCII) characters.

Hello,

I am having great problems writing norwegian characters to file
from a python application. My (simplified) scenario is as follows:

1. I have a web form where the user can enter his name.

2. I use the cgi module module to get to the input from the user:
....
name = form["name"].value

3. The name is stored in a file

fileH = open(namefile , "a")
fileH.write("na me:%s \n" % name)
fileH.close()

Now, this works very well indeed as long the users have 'ascii' names,
however when someone enters a name with one of the norwegian characters
- it breaks at the write() statement.

UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x8f in position
.....

Now - I understand that the ascii codec can't be used to decode the
particular characters, however my attempts of specifying an alternative
encoding have all failed.

I have tried variants along the line:

fileH = codecs.open(nam efile , "a" , "latin-1") / fileH =
open(namefile , "a")
fileH.write(nam e) / fileH.write(nam e.encode("latin-1"))

It seems *whatever* I do the Python interpreter fails to see my pledge
for an alternative encoding, and fails with the dreaded
UnicodeDecodeEr ror.

Any tips on this would be *highly* appreciated.
Joakim

Oct 7 '06 #1
2 6728
jo*********@gma il.com wrote:
Hello,

I am having great problems writing norwegian characters æøå to file
from a python application. My (simplified) scenario is as follows:

1. I have a web form where the user can enter his name.

2. I use the cgi module module to get to the input from the user:
....
name = form["name"].value

3. The name is stored in a file

fileH = open(namefile , "a")
fileH.write("na me:%s \n" % name)
fileH.close()

Now, this works very well indeed as long the users have 'ascii' names,
however when someone enters a name with one of the norwegian characters
æøå - it breaks at the write() statement.

UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x8f in position
....

Now - I understand that the ascii codec can't be used to decode the
particular characters, however my attempts of specifying an alternative
encoding have all failed.

I have tried variants along the line:

fileH = codecs.open(nam efile , "a" , "latin-1") / fileH =
open(namefile , "a")
fileH.write(nam e) / fileH.write(nam e.encode("latin-1"))

It seems *whatever* I do the Python interpreter fails to see my pledge
for an alternative encoding, and fails with the dreaded
UnicodeDecodeEr ror.

Any tips on this would be *highly* appreciated.
The approach with codecs.open() should succeed
>>out = codecs.open("tm p.txt", "a", "latin1")
out.write(u" øå")
out.write("ab c")
out.write("æ øå")
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/local/lib/python2.4/codecs.py", line 501, in write
return self.writer.wri te(data)
File "/usr/local/lib/python2.4/codecs.py", line 178, in write
data, consumed = self.encode(obj ect, self.errors)
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc3 in position 0:
ordinal not in range(128)

provided that you write only unicode strings with characters in the range
unichr(0)...uni chr(255) and normal strs in the range chr(0)...chr(12 7).

You have to decode non-ascii strs before feeding them to write() with the
appropriate encoding (that only you know)
>>out.write(uni code("\xe6\xf8\ xe5", "latin1"))
If there are unicode code points beyond unichr(255) you have to change the
encoding in codecs.open(), typically to UTF-8.

# raises UnicodeEncodeEr ror
codecs.open("tm p.txt", "a", "latin1").write (u"\u1234")

# works
codecs.open("tm p.txt", "a", "utf8").write(u "\u1234")

Peter

Oct 7 '06 #2
jo*********@gma il.com wrote:
>
I am having great problems writing norwegian characters to file
from a python application. My (simplified) scenario is as follows:

1. I have a web form where the user can enter his name.

2. I use the cgi module module to get to the input from the user:
....
name = form["name"].value
The cgi module should produce plain strings, not Unicode objects, which
makes some of the later behaviour quite "interestin g".
3. The name is stored in a file

fileH = open(namefile , "a")
fileH.write("na me:%s \n" % name)
fileH.close()

Now, this works very well indeed as long the users have 'ascii' names,
however when someone enters a name with one of the norwegian characters
- it breaks at the write() statement.

UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x8f in position
This is odd, since writing plain strings to files shouldn't involve any
Unicode conversions. If you received a plain string from the cgi
module, the text you write to the file should still be a plain string.
This is like obtaining a sequence of bytes and just passing them
around. Perhaps your Python configuration is different in some
non-standard way, although I wouldn't want to point the finger at
anything in particular (although sys.getdefaulte ncoding might suggest
something).
Now - I understand that the ascii codec can't be used to decode the
particular characters, however my attempts of specifying an alternative
encoding have all failed.

I have tried variants along the line:

fileH = codecs.open(nam efile , "a" , "latin-1") / fileH = open(namefile , "a")
fileH.write(nam e) / fileH.write(nam e.encode("latin-1"))

It seems *whatever* I do the Python interpreter fails to see my pledge
for an alternative encoding, and fails with the dreaded
UnicodeDecodeEr ror.
To use a file opened through codecs.open, you really should present
Unicode objects to the write method. Otherwise, I imagine that the
method will try and automatically convert to Unicode the plain string
that the name object supposedly is, and this conversion will assume
that the string only contains ASCII characters (as is Python's default
behaviour) and thus cause the error you are seeing. Only after getting
the text as a Unicode object will the method then try to encode the
text in the specified encoding in order to write it to the file.

In other words, you'll see this behaviour:

name (plain string) -Unicode object -encoded text (written to
file)

Or rather, in the failure case:

name (plain string) -error! (couldn't produce the Unicode object)

As Peter Otten suggests, you could first make the Unicode object
yourself, stating explicitly that the name object contains "latin-1"
characters. In other words:

name (plain string) -Unicode object

Then, the write method has an easier time:

Unicode object -encoded text (written to file)

All this seems unnecessary for your application, I suppose, since you
know (or believe) that the form values only contain "latin-1"
characters. However, as is the standard advice on such matters, you may
wish to embrace Unicode more eagerly, converting plain strings to
Unicode as soon as possible and only converting them to text in various
encodings when writing them out.

In some Web framework APIs, the values of form fields are immediately
available as Unicode without any (or much) additional work. WebStack
returns Unicode objects for form fields, as does the Java Servlet API,
but I'm not particularly aware of many other Python frameworks which
enforce or promote such semantics.

Paul

Oct 8 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
2819
by: janeaustine50 | last post by:
Python's InteractiveInterpreter uses the built-in compile function. According to the ref. manual, it doesn't seem to concern about the encoding of the source string. When I hand in an unicode object, it is encoded in utf-8 automatically. It can be a problem when I'm building an interactive environment using "compile", with a different encoding from utf-8. IDLE itself has the same problem. ( '<a string with non-ascii-encoding>' is...
48
4662
by: Zenobia | last post by:
Recently I was editing a document in GoLive 6. I like GoLive because it has some nice features such as: * rewrite source code * check syntax * global search & replace (through several files at once) * regular expression search & replace. Normally my documents are encoded with the ISO setting. Recently I was writing an XHTML document. After changing the encoding to UTF-8 I used the
37
10185
by: chandy | last post by:
Hi, I have an Html document that declares that it uses the utf-8 character set. As this document is editable via a web interface I need to make sure than high-ascii characters that may be accidentally entered are properly represented when the document is served. My programming language allows me to get the ascii value for any individual character so what I am doing when a change is saved is to look at each character in the content and...
13
6151
by: Dan V. | last post by:
How do I create a one line text file with these control codes? e.g.: 144 = 0x90 and 147 = 0x93? I am trying to create a one line text file with these characters all one one row with no spaces. 1. 144 = 0x90 2. 147 = 0x93 3. STX = (^B = 2 = 0x2) 4. NUL = (^@ = 0 = 0x0)
9
23721
by: Mark | last post by:
I've run a few simple tests looking at how query string encoding/decoding gets handled in asp.net, and it seems like the situation is even messier than it was in asp... Can't say I think much of the "improvements", but maybe someone here can point me in the right direction... First, it looks like asp.net will automatically read and recognize query strings encoded in utf8 and 16-bit unicode, only the latter is some mutant, non-standard...
19
3347
by: Thomas W | last post by:
I'm getting really annoyed with python in regards to unicode/ascii-encoding problems. The string below is the encoding of the norwegian word "fdselsdag". I stored the string as "fdselsdag" but somewhere in my code it got translated into the mess above and I cannot get the original string back. It cannot be printed in the console or written a plain text-file. I've tried to convert it using
4
17606
by: Ciuin | last post by:
Hi all, I need to mail() emails with user input that does contain non-ascii (umlauts, accents) and non-latin (cyrillic) characters in the "Subject:" and "From:" headers. I understand that they are typically encoded in UTF8 like this: =?UTF-8?B?w5Z0emkg0J/RgNC40LLQtdGC?= but I cannot find a PHP function to encode the input string in this
8
2689
by: Erwin Moller | last post by:
Hi group, I could use a bit of guidance on the following matter. I am starting a new project now and must make some decisions regarding encoding. Environment: PHP4.3, Postgres7.4.3 I must be able to receive forminformation and store that in a database and later produce it on screen on the client (just plain HTML).
9
4153
by: =?Utf-8?B?RGFu?= | last post by:
I have the following code section that I thought would strip out all the non-ascii characters from a string after decoding it. Unfortunately the non-ascii characters are still in the string. What am I doing wrong? Dim plainText As String plainText = "t═e" Dim plainTextBytes() As Byte Dim enc As Encoding = Encoding.ASCII plainTextBytes = enc.GetBytes(plainText)
0
9734
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10995
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10654
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10718
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10347
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
5729
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5915
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4540
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3168
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.