i18n: looking for expertise

[klappnase]

Hello all,

I am trying to internationaliz e my Tkinter program using gettext and
encountered various problems, so it looks like it's not a trivial
task.
After some "research" I made up a few rules for a concept that I hope
lets me avoid further encoding trouble, but I would feel more
confident if some of the experts here would have a look at the
thoughts I made so far and told me if I'm still going wrong somewhere
(BTW, the program is supposed to run on linux only). So here is what I
have so far:

1. use unicode instead of byte strings wherever possible. This can be
a little tricky, because in some situations I cannot know in advance
if a certain string is unicode or byte string; I wrote a helper module
for this which defines convenience methods for fail-safe
decoding/encoding of strings and a Tkinter.Unicode Var class which I
use to convert user input to unicode on the fly (see the code below).

2. so I will have to call gettext.install () with unicode=1

3. make sure to NEVER mix unicode and byte strings within one
expression

4. in order to maintain code readability it's better to risk excess
decode/encode cycles than having one too few.

5. file operations seem to be delicate; at least I got an error when I
passed a filename that contains special characters as unicode to
os.access(), so I guess that whenever I do file operations
(os.remove(), shutil.copy() ...) the filename should be encoded back
into system encoding before; The filename manipulations by the os.path
methods seem to be simply string manipulations so encoding the
filenames doesn't seem to be necessary.

6. messages that are printed to stdout should be encoded first, too;
the same with strings I use to call external shell commands.

############ file UnicodeHandler. py ############### ############### ####
# -*- coding: iso-8859-1 -*-
import Tkinter
import sys
import locale
import codecs

def _find_codec(enc oding):
# return True if the requested codec is available, else return
False
try:
codecs.lookup(e ncoding)
return 1
except LookupError:
print 'Warning: codec %s not found' % encoding
return 0

def _sysencoding():
# try to guess the system default encoding
try:
enc = locale.getprefe rredencoding(). lower()
if _find_codec(enc ):
print 'Setting locale to %s' % enc
return enc
except AttributeError:
# our python is too old, try something else
pass
enc = locale.getdefau ltlocale()[1].lower()
if _find_codec(enc ):
print 'Setting locale to %s' % enc
return enc
# the last try
enc = sys.stdin.encod ing.lower()
if _find_codec(enc ):
print 'Setting locale to %s' % enc
return enc
# aargh, nothing good found, fall back to latin1 and hope for the
best
print 'Warning: cannot find usable locale, using latin-1'
return 'iso-8859-1'

sysencoding = _sysencoding()

def fsdecode(input, errors='strict' ):
'''Fail-safe decodes a string into unicode.'''
if not isinstance(inpu t, unicode):
return unicode(input, sysencoding, errors)
return input

def fsencode(input, errors='strict' ):
'''Fail-safe encodes a unicode string into system default
encoding.'''
if isinstance(inpu t, unicode):
return input.encode(sy sencoding, errors)
return input
class UnicodeVar(Tkin ter.StringVar):
def __init__(self, master=None, errors='strict' ):
Tkinter.StringV ar.__init__(sel f, master)
self.errors = errors
self.trace('w', self._str2unico de)

def _str2unicode(se lf, *args):
old = self.get()
if not isinstance(old, unicode):
new = fsdecode(old, self.errors)
self.set(new)
############### ############### ############### ############### ###########

So before I start to mess up all of my code, maybe someone can give me
a hint if I still forgot something I should keep in mind or if I am
completely wrong somewhere.

Thanks in advance

Michael

[Neil Hodgson]

Michael:

5. file operations seem to be delicate; at least I got an error when I
passed a filename that contains special characters as unicode to
os.access(), so I guess that whenever I do file operations
(os.remove(), shutil.copy() ...) the filename should be encoded back
into system encoding before;

This can lead to failure on Windows when the true Unicode file name can
not be encoded in the current system encoding.

Neil

[klappnase]

"Neil Hodgson" <nh******@bigpo nd.net.au> wrote in message news:<6O******* *************@n ews-server.bigpond. net.au>...

Michael:

5. file operations seem to be delicate; at least I got an error when I
passed a filename that contains special characters as unicode to
os.access(), so I guess that whenever I do file operations
(os.remove(), shutil.copy() ...) the filename should be encoded back
into system encoding before;

This can lead to failure on Windows when the true Unicode file name can
not be encoded in the current system encoding.

Neil

Like I said, it's only supposed to run on linux; anyway, is it likely
that problems will arise when filenames I have to handle have
basically three sources:

1. already existing files

2. automatically generated filenames, which result from adding an
ascii-only suffix to an existing filename (like xy --> xy_bak2)

3. filenames created by user input

?
If yes, how to avoid these?

Any hints are appreciated

Michael

[Neil Hodgson]

Michael:

Like I said, it's only supposed to run on linux; anyway, is it likely
that problems will arise when filenames I have to handle have
basically three sources:
...
3. filenames created by user input

Have you worked out how you want to handle user input that is not
representable in the encoding? It is easy for users to input any characters
into a Unicode enabled UI either through invoking an input method or by
copying and pasting from another application or character chooser applet.

Neil

[klappnase]

"Neil Hodgson" <nh******@bigpo nd.net.au> wrote in message news:<Pq******* **********@news-server.bigpond. net.au>...

Michael:

Like I said, it's only supposed to run on linux; anyway, is it likely
that problems will arise when filenames I have to handle have
basically three sources:
...
3. filenames created by user input

Have you worked out how you want to handle user input that is not
representable in the encoding? It is easy for users to input any characters
into a Unicode enabled UI either through invoking an input method or by
copying and pasting from another application or character chooser applet.

Neil

As I must admit, no. I just couldn't figure out that someone will really do this.

I guess I could add a test like (pseudo code):

try:
test = fsdecode(input) # convert to unicode
test.encode(sys encoding)
except:
# show a message box with something like "Invalid file name"

Please tell me if you find any other possible gotchas.

Thanks so far

Michael

[Serge Orlov]

klappnase wrote:

Hello all,

I am trying to internationaliz e my Tkinter program using gettext and
encountered various problems, so it looks like it's not a trivial
task.
Considered that you decided to support old python versions, it's true.
Unicode support has gradually improved. If you choose to target old
python version, basically you're dealing with years old unicode
support.
After some "research" I made up a few rules for a concept that I hope
lets me avoid further encoding trouble, but I would feel more
confident if some of the experts here would have a look at the
thoughts I made so far and told me if I'm still going wrong somewhere
(BTW, the program is supposed to run on linux only). So here is what
I have so far:

1. use unicode instead of byte strings wherever possible. This can be
a little tricky, because in some situations I cannot know in advance
if a certain string is unicode or byte string; I wrote a helper
module for this which defines convenience methods for fail-safe
decoding/encoding of strings and a Tkinter.Unicode Var class which I
use to convert user input to unicode on the fly (see the code below).
I've never used tkinter, but I heard good things about it. Are you
sure it's not you who made it to return byte string sometimes?
Anyway, your idea is right, make IO libraries always return unicode.
3. make sure to NEVER mix unicode and byte strings within one
expression
As a rule of thumb you should convert byte strings into unicode
strings at input and back to byte strings at output. This way
the core of your program will have to deal only with unicode
strings.
4. in order to maintain code readability it's better to risk excess
decode/encode cycles than having one too few.
I don't think so. Either you need decode/encode or you don't.
5. file operations seem to be delicate;
You should be ready to handle unicode errors at file operations as
well as for example ENAMETOOLONG error. Any file system with path
argument can throw it, I don't think anything changed here with
introduction of unicode. For example access can return 11 (on
my linux system) error codes, consider unicode error to be twelveth.
at least I got an error when I
passed a filename that contains special characters as unicode to
os.access(), so I guess that whenever I do file operations
(os.remove(), shutil.copy() ...) the filename should be encoded back
into system encoding before;
I think python 2.3 handles that for you. (I'm not sure about the
version)
If you have to support older versions, you have to do it yourself.

6. messages that are printed to stdout should be encoded first, too;
the same with strings I use to call external shell commands.

If you use stdout as dump device just install the encoder in the
beginning of your program, something like

sys.stdout = codecs.getwrite r(...) ...
sys.stderr = codecs.getwrite r(...) ...
Serge.

[klappnase]

"Serge Orlov" <Se*********@gm ail.com> wrote in message news:<11******* **************@ o13g2000cwo.goo glegroups.com>. ..

I've never used tkinter, but I heard good things about it. Are you
sure it's not you who made it to return byte string sometimes?
Yes, I used a Tkinter.StringV ar to keep track of the contents of an
Entry widget; as long as I entered only ascii characters get() returns
a byte string, as soon as a special character is entered it returns
unicode.
Anyway, my UnicodeVar() class seems to be a handy way to avoid
problems here.

4. in order to maintain code readability it's better to risk excess
decode/encode cycles than having one too few.

I don't think so. Either you need decode/encode or you don't.

I use a bunch of modules that contain helper functions for frequently
repeated tasks. So it sometimes happens for example that I call one of
my module functions to convert user input into unicode and then call
the next module function to convert it back to byte string to start
some file operation; that's what I meant with "excess decode/encode
cycles". However, trying to avoid these ended in totally messing up
the code.

5. file operations seem to be delicate;

You should be ready to handle unicode errors at file operations as
well as for example ENAMETOOLONG error. Any file system with path
argument can throw it, I don't think anything changed here with
introduction of unicode. For example access can return 11 (on
my linux system) error codes, consider unicode error to be twelveth.

at least I got an error when I
passed a filename that contains special characters as unicode to
os.access(), so I guess that whenever I do file operations
(os.remove(), shutil.copy() ...) the filename should be encoded back
into system encoding before;

I think python 2.3 handles that for you. (I'm not sure about the
version)
If you have to support older versions, you have to do it yourself.

I am using python-2.3.4 and get unicode errors:

f = os.path.join(u'/home/pingu/phonoripper', u'\xc3\u20ac')
os.path.isfile( f) True os.access(f, os.R_OK) Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
24-25: ordinal not in range(128) f = f.encode('iso-8859-15')
os.access(f, os.R_OK) True

Thanks for the feedback

Michael

[Martin v. Löwis]

klappnase wrote:

I am using python-2.3.4 and get unicode errors:

f = os.path.join(u'/home/pingu/phonoripper', u'\xc3\u20ac')
os.path.isf ile(f)
True
os.access(f , os.R_OK)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position
24-25: ordinal not in range(128)

That's apparently a bug in os.access, which doesn't support Unicode file
names. As a work around, do

def access(name, mode, orig=os.access) :
try:
return orig(name, mode)
except UnicodeError:
return orig(name.encod e(sys.getfilesy stemencoding(), mode))
os.access=acces s

Apparently, access is used so rarely that nobody has noticed yet (or
didn't bother to report). os.path.isfile( ) builds on os.stat(), which
does support Unicode file names.

Regards,
Martin

[klappnase]

"Martin v. Löwis" <ma****@v.loewi s.de> wrote in message news:<42******* *************** @news.freenet.d e>...

That's apparently a bug in os.access, which doesn't support Unicode file
names. As a work around, do

def access(name, mode, orig=os.access) :
try:
return orig(name, mode)
except UnicodeError:
return orig(name.encod e(sys.getfilesy stemencoding(), mode))
os.access=acces s

Apparently, access is used so rarely that nobody has noticed yet (or
didn't bother to report). os.path.isfile( ) builds on os.stat(), which
does support Unicode file names.

Regards,
Martin

Ah, thanks!

Now another question arises: you use sys.getfilesyst emencoding() to
encode the
file name, which looks like it's the preferred method. However when I
tried to
find out how this works I got a little confused again (from the
library reference):

getfilesystemen coding()

Return the name of the encoding used to convert Unicode filenames into
system file names, or None if the system default encoding is used. The
result value depends on the operating system:
(...)
* On Unix, the encoding is the user's preference according to the
result of nl_langinfo(COD ESET), or None if the nl_langinfo(COD ESET)
failed.

On my box (mandrake-10.1) sys.getfilesyst emencoding() returns
'ISO-8859-15',
however :

locale.nl_langi nfo(locale.CODE SET) 'ANSI_X3.4-1968'

Anyway, my app currently runs with python-2.2 and I would like to keep
it that way if possible, so I wonder which is the preferred
replacement for sys.getfilesyst emencoding() on versions < 2.3 , or in
particular, will the method I use to determine "sysencodin g" I
described in my original post be safe or are there any traps I missed
(it's supposed to run on linux only)?

Thanks and best regards

Michael

[stewart.midwinter]

Michael:

on my box, (winXP SP2), sys.getfilesyst emencoding() returns 'mbcs'.

If you post your revised solution to this unicode problem, I'd be
delighted to test it on Windows. I'm working on a Tkinter front-end
for Vivian deSmedt's rsync.py and would like to address the issue of
accented characters in folder names.

thanks
Stewart
stewart dot midwinter at gmail dot com