I18n issue with optik

[Thorsten Kampe]

Hi,

I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?
Thorsten

[1]
Traceback (most recent call last):
File "script.py" , line 37, in <module>
options, args = cmdlineparser.p arse_args()
File "/usr/lib/python2.5/optparse.py", line 1378, in parse_args
stop = self._process_a rgs(largs, rargs, values)
File "/usr/lib/python2.5/optparse.py", line 1418, in _process_args
self._process_l ong_opt(rargs, values)
File "/usr/lib/python2.5/optparse.py", line 1493, in
_process_long_o pt
option.process( opt, value, values, self)
File "/usr/lib/python2.5/optparse.py", line 782, in process
self.action, self.dest, opt, value, values, parser)
File "/usr/lib/python2.5/optparse.py", line 804, in take_action
parser.print_he lp()
File "/usr/lib/python2.5/optparse.py", line 1648, in print_help
file.write(self .format_help(). encode(encoding , "replace"))
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc4 in position
264: ordinal not in range(128)

[Thorsten Kampe]

* paul (Mon, 02 Apr 2007 17:49:15 +0200)

Thorsten Kampe schrieb:
[snipp]

I got the tip to set a different encoding by
sys.stdout = codecs.EncodedF ile(sys.stdout, 'utf-8')

but unfortunately this does not change the encoding of any Terminal.
So my question is: how can I set a different encoding to sys.stdout
(or why can I set it without any error but nothing changes?)

AFAIK you can't. If the terminal is limited to ascii it won't be able to
display anything else; it might not even have the right font, so how are
you supposed to fix that?

Actually rxvt, Poderosa and console have the ability to display non-
ASCII characters. I use the dejavu fonts that support non-ASCII, too.

But the problem is even simpler: I can't even set the standard Windows
console ("cmd") to Windows 1252 in Python. Although directly executing
"chcp 1252" works.

Thorsten

[Damjan]

Actually rxvt, Poderosa and console have the ability to display non-
ASCII characters. I use the dejavu fonts that support non-ASCII, too.

But the problem is even simpler: I can't even set the standard Windows
console ("cmd") to Windows 1252 in Python. Although directly executing
"chcp 1252" works.

Maybe try to use http://sourceforge.net/projects/console it's claimed to be
muc better than the sucky CDM (I don't have windows to try it).
--
damjan

[Thorsten Kampe]

* Damjan (Mon, 02 Apr 2007 18:29:06 +0200)

Actually rxvt, Poderosa and console have the ability to display non-
ASCII characters. I use the dejavu fonts that support non-ASCII, too.

But the problem is even simpler: I can't even set the standard Windows
console ("cmd") to Windows 1252 in Python. Although directly executing
"chcp 1252" works.

Maybe try to use http://sourceforge.net/projects/console it's claimed to be
muc better than the sucky CDM (I don't have windows to try it).

It is definitely. But it just runs bash or cmd.exe so its capabilities
(encoding) are defined by Windows or Cygwin.

Thorsten

[Thorsten Kampe]

* Thorsten Kampe (Mon, 2 Apr 2007 16:05:25 +0100)

* Steven Bethard (Sun, 01 Apr 2007 10:21:40 -0600)

Thorsten Kampe wrote:
I'm not very experienced with internationaliz ation, but if you change::

gettext.install ('test')

to::

gettext.install ('test', unicode=True)

what happens?

Actually, this is the solution.

But there's one more problem: the solution only works when the
Terminal encoding is not US-ASCII. Unfortunately (almost) all
terminals I tried are set to US-ASCII (rxvt under Cygwin, Console[1]
running bash, Poderosa[2] running bash). Only the Windows Console is
CP852 and this works.

I got the tip to set a different encoding by
sys.stdout = codecs.EncodedF ile(sys.stdout, 'utf-8')

but unfortunately this does not change the encoding of any Terminal.
So my question is: how can I set a different encoding to sys.stdout
(or why can I set it without any error but nothing changes?)

I solved it (finally after two days with the help of a lot of people):

You have to set this...

sys.stdout = codecs.EncodedF ile(sys.stdout, 'iso-8859-15')
sys.stdout.enco ding = 'iso-8859-15'

....both of these and exactly in this order and not vice versa. It
doesn't have to be 'iso-8859-15', Windows-1252 is fine, too (but UTF-8
doesn't work). Now we have a new problem: the native Windows consoles
don't print the right characters. So you wrap this in a query:

if sys.platform in ['cygwin', 'linux2']:
sys.stdout = codecs.EncodedF ile(sys.stdout, 'iso-8859-15')
sys.stdout.enco ding = 'iso-8859-15'

This would be a problem if there's more than one translation (for
instance one with polish characters that aren't contained in iso-8859-
15). One could work around this with

if sys.platform in ['cygwin', 'linux2']:
sys.stdout = codecs.EncodedF ile(sys.stdout,
locale.getprefe rredencoding())
sys.stdout.enco ding = locale.getprefe rredencoding()

Funny (more or less): two days work to print "U" and "A" with double
points above.

Thanks to all who have helped to clear my confusion and to bring me
into the right direction for the solution.
Thorsten