Problems with unicode

[James Laamnna]

I'm trying to write out a XML document using a StringIO class, however
I always run into the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x92 in position
4: ordinal
not in range(128)

Apparently in the batch that I'm encoding there is one string with
non-ascii characters in it.
Is there any way to just have it encode everything as unicode and not
ascii?
Or should I just strip out non-ascii characters (a last resort which I
do not want to do)
Thanks.

[David Opstad]

In article <a0**************************@posting.google.com >,
ja****@appliedminds.com (James Laamnna) wrote:

Apparently in the batch that I'm encoding there is one string with
non-ascii characters in it. Is there any way to just have it encode
everything as unicode and not ascii?

A better question to ask is this: where did the supposed ASCII data come
from in the first place? If, for instance, it came from a Windows
machine, then there's a chance it's actually ISO-8859-1 encoding, in
which case you can preserve the 0x92 by encoding using that codec,
instead of the 'ascii' one. Similarly, if the original text came from a
Mac, then it's likely in Mac Roman, so if you use the 'mac-roman' codec
you'll be able to preserve the correct character in your resulting
Unicode.

Dave

[Jarek Zgoda]

David Opstad <op****@batnet.com> pisze:

If, for instance, it came from a Windows
machine, then there's a chance it's actually ISO-8859-1 encoding

If it came from Windows, it's actually CP-1252, not Latin-1.

http://www.effbot.org/zone/unicode-gremlins.htm

--
Jarek Zgoda
http://jpa.berlios.de/