469,133 Members | 1,239 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,133 developers. It's quick & easy.

Unicode list

Hello,

I have this little grep-like program:

++++++++++snip++++++++++
#!/usr/bin/python

import sys
import re

pattern = sys.argv[1]
inputfile = file(sys.argv[2], 'r')

for line in inputfile:
matches = re.findall(pattern, line)
if matches:
print matches
++++++++++snip++++++++++

Like this, the program prints some characters as strange escape
sequences, which is due to the input file being encoded in utf-8: When I
convert "re.findall..." to a string and wrap an "unicode()" around it,
the matches get printed correctly. Is it possible to make "matches"
unicode without saving it as a single string first? The function "unicode
()" seems only to work for strings. Or is there a general way of telling
Python to abandon the ancient and evil land of iso-8859 for good and use
utf-8 only?

Regards,
Rehceb
Apr 1 '07 #1
4 2282
Rehceb Rotkiv wrote:
Hello,

I have this little grep-like program:

++++++++++snip++++++++++
#!/usr/bin/python

import sys
import re

pattern = sys.argv[1]
inputfile = file(sys.argv[2], 'r')

for line in inputfile:
matches = re.findall(pattern, line)
if matches:
print matches
++++++++++snip++++++++++

Like this, the program prints some characters as strange escape
sequences, which is due to the input file being encoded in utf-8:
So the UTF-8 data gets printed to your terminal which isn't configured
for UTF-8, right?
When I convert "re.findall..." to a string and wrap an "unicode()" around it,
the matches get printed correctly.
How do you meaningfully convert it to a string? The matches variable
refers to a list, but you surely don't want to be dealing with the
list's string representation.
Is it possible to make "matches" unicode without saving it as a single string first?
Why not convert your input into Unicode and then, for the benefit of
certain kinds of character classes, use re.findall in Unicode mode (by
specifying re.U as a flag)? Then, each match will be produced as a
Unicode object.
The function "unicode()" seems only to work for strings. Or is there a general way of telling
Python to abandon the ancient and evil land of iso-8859 for good and use utf-8 only?
The only refuge from ancient and evil lands is found by climbing the
mountain of Unicode: convert from encoded text as soon as you can,
work only with Unicode objects, produce encoded text only when
necessary.

Paul

Apr 1 '07 #2
Like this, the program prints some characters as strange escape
sequences, which is due to the input file being encoded in utf-8: When I
convert "re.findall..." to a string and wrap an "unicode()" around it,
the matches get printed correctly. Is it possible to make "matches"
unicode without saving it as a single string first? The function "unicode
()" seems only to work for strings. Or is there a general way of telling
Python to abandon the ancient and evil land of iso-8859 for good and use
utf-8 only?
Python does not live in the ancient and evi land of iso-8859; it lives
in the ancient and evil land of ASCII.

When printing a list, the individual elements are converted with repr(),
not with str(). For a string object, repr() adds escape codes for all
bytes that are not printable ASCII characters. To avoid this call to
repr, you need to iterate over the list yourself, and print it:

if matches:
for m in matches:
print m,
print

HTH,
Martin
Apr 1 '07 #3
Rehceb Rotkiv schrieb:
Hello,

I have this little grep-like program:

++++++++++snip++++++++++
#!/usr/bin/python

import sys
import re

pattern = sys.argv[1]
inputfile = file(sys.argv[2], 'r')

for line in inputfile:
matches = re.findall(pattern, line)
if matches:
print matches
++++++++++snip++++++++++

Like this, the program prints some characters as strange escape
sequences, which is due to the input file being encoded in utf-8
As Paul said, your terminal is likely set to iso-8859 encoding, which
is why it doesn't display UTF-8 correctly. The above program produces
correct UTF-8 output.

What you could do is:
1. read the file in as unicode
2. print the unicode to the terminal (will use the terminal encoding) or
convert the unicode to strings with an explicit encoding before printing

codecs.open() is very helpful for step 1, BTW.

Georg

Apr 1 '07 #4
When printing a list, the individual elements are converted with repr(),
not with str(). For a string object, repr() adds escape codes for all
bytes that are not printable ASCII characters.
Thanks Martin, you're right, it were the repr() calls that messed up the
output. Iterating the array like you proposed is even 1/100s faster ;)

Regards,
Rehceb
Apr 1 '07 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by John Sidney-Woollett | last post: by
9 posts views Thread by Gerry | last post: by
17 posts views Thread by Adam Olsen | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.