471,353 Members | 1,759 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,353 software developers and data experts.

Interpreting non-ascii characters.


Hello everybody,

I want to create a script which reads files in a
current directory and renames them according to some
scheme. The file names are in Russian - sometimes
the names encoded as win-1251, sometimes as koi8-r etc.
I want to read in file name and convert it to list for
further processing. The problem is that Python treats
non-ascii characters as multibyte characters - for
example, hex code for "Small Character A" in koi8-r is
0xc1, but Python interprets it as a sequence of
\xd0, \xb1 bytes.

What can I do so that Python interprets non-ascii
characters correctly?
Jul 17 '07 #1
3 5324
On 18/07/2007 4:11 AM, ddtl wrote:
Hello everybody,

I want to create a script which reads files in a
current directory and renames them according to some
scheme. The file names are in Russian - sometimes
the names encoded as win-1251, sometimes as koi8-r etc.
You have a file system with 8-bit file names with no indication of
'codepage' or 'encoding', either globally or per file? Which operating
system are you using?
I want to read in file name and convert it to list for
further processing.
Read file name from a text file? Or do you mean using e.g. glob.glob()
or os.listdir()

What do you mean by "convert it to list"? Do you mean 'foo.txt' -['f',
'o', ....etc]??? Why?
The problem is that Python treats
non-ascii characters as multibyte characters - for
example, hex code for "Small Character A" in koi8-r is
0xc1, but Python interprets it as a sequence of
\xd0, \xb1 bytes.
Python is very unlikely to do that all by itself. Please show us the
script or whatever evidence you have. I strongly suggest that
immediately after "reading" a file name, you do
print repr(file_name)
NOT
print file_name
so that you can see *exactly* what you've got.

Are you sure about the \xb1??? Consider this:
>>'\xc1'.decode('koi8-r')
u'\u0430'
>>'\xc1'.decode('koi8-r').encode('utf8')
'\xd0\xb0'
>>>
Also:
>>import sys; sys.stdout.encoding
'cp850' # Win XP Pro, command prompt
>>>
What do you get when you do that?
>
What can I do so that Python interprets non-ascii
characters correctly?
Know how your non-ascii characters are encoded. Tell Python what to do
with them.

Read this:
http://www.amk.ca/python/howto/unicode

Hope this helps,
John
Jul 17 '07 #2
I want to create a script which reads files in a
current directory and renames them according to some
scheme. The file names are in Russian - sometimes
the names encoded as win-1251, sometimes as koi8-r etc.
I want to read in file name and convert it to list for
further processing. The problem is that Python treats
Apparently os.listdir returns a list of Unicode objects if the pathname
you give it is a Unicode object. So, Python should then convert the
Russian filenames to Unicode, using whatever encoding necessary. (I
don't know, however, how Python would know what to do if the filenames
are in a bunch of different encodings, as you say.)

If you can get the filenames into Unicode, then you can manipulate them
however you like.
--
For help at any time, press *H.
Jul 17 '07 #3
On Wed, 18 Jul 2007 08:29:58 +1000, John Machin <sj******@lexicon.netwrote:
...
I have a bunch of directories and files from different systems
(each directory contains files from the same system) which are
encoded differently (though all of them are in Russian), so the
following encodings are present: koi8-r, win-1251, utf-8 etc.,
and I want to transliterate them into a regular ASCII so that they
would be readable regardless of the system. Personally I use both
Linux and Windows. So what I do, is read file name using os.listdir,
convert to list ('foo.txt' =['f', 'o', ... , 't'], except that
file names are in Russian), transliterate (some letters in Russian
have to be transliterated into 2 or even 3 Latin letters),
and then rename file.

It seems though that after all I solved the problem - I thought
that my Windows (2000) used win-1251 and Linux used koi8-r and
because of that I couldn't understand what are those strange
codes I got while experimenting with locally created Cyrillic
file names, but in effect Linux uses utf-8, and Windows uses cp866,
so after getting it and reading the article you suggested I
solved the problem.

Thanks.

Jul 18 '07 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

3 posts views Thread by Mario | last post: by
25 posts views Thread by Yves Glodt | last post: by
3 posts views Thread by Chris Saunders | last post: by
11 posts views Thread by Dennis Allison | last post: by
4 posts views Thread by zolli | last post: by
3 posts views Thread by Chris Saunders | last post: by
32 posts views Thread by Adrian Herscu | last post: by
8 posts views Thread by Bern McCarty | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.