I18n issue with optik

[Thorsten Kampe]

Hi,

I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?
Thorsten

[1]
Traceback (most recent call last):
File "script.py" , line 37, in <module>
options, args = cmdlineparser.p arse_args()
File "/usr/lib/python2.5/optparse.py", line 1378, in parse_args
stop = self._process_a rgs(largs, rargs, values)
File "/usr/lib/python2.5/optparse.py", line 1418, in _process_args
self._process_l ong_opt(rargs, values)
File "/usr/lib/python2.5/optparse.py", line 1493, in
_process_long_o pt
option.process( opt, value, values, self)
File "/usr/lib/python2.5/optparse.py", line 782, in process
self.action, self.dest, opt, value, values, parser)
File "/usr/lib/python2.5/optparse.py", line 804, in take_action
parser.print_he lp()
File "/usr/lib/python2.5/optparse.py", line 1648, in print_help
file.write(self .format_help(). encode(encoding , "replace"))
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc4 in position
264: ordinal not in range(128)

[Thorsten Kampe]

* Thorsten Kampe (Sun, 1 Apr 2007 20:08:39 +0100)

* Thorsten Kampe (Sun, 1 Apr 2007 19:45:59 +0100)

Yes, I could do that but I'd rather know first if my code is wrong or
the optparse code.

It might be the bug mentioned in
http://mail.python.org/pipermail/pyt...ay/065458.html

The patch although doesn't work. From my unicode-charset-codepage-
codeset-challenged point of view the encoding of sys.stdout doesn't
matter. The charset is defined in the .po/.mo file (but of course
optparse can't know if the message has been translated by gettext
("_").

If I "patch" line 1648 (the one mentioned in the traceback) of
optparse.py from

file.write(self .format_help(). encode(encoding , "replace"))
to
file.write(self .format_help())

....then everything works and is displayed fine (even without the
"unicode = True" parameter to gettext.install ).

But the "patch" might make other things fail, of course...

Thorsten

[Thorsten Kampe]

* Thorsten Kampe (Sun, 1 Apr 2007 20:22:51 +0100)

* Thorsten Kampe (Sun, 1 Apr 2007 20:08:39 +0100)

* Thorsten Kampe (Sun, 1 Apr 2007 19:45:59 +0100)

Yes, I could do that but I'd rather know first if my code is wrong or
the optparse code.

It might be the bug mentioned in
http://mail.python.org/pipermail/pyt...ay/065458.html

The patch although doesn't work. From my unicode-charset-codepage-
codeset-challenged point of view the encoding of sys.stdout doesn't
matter. The charset is defined in the .po/.mo file (but of course
optparse can't know if the message has been translated by gettext
("_").

If I "patch" line 1648 (the one mentioned in the traceback) of
optparse.py from

file.write(self .format_help(). encode(encoding , "replace"))
to
file.write(self .format_help())

...then everything works and is displayed fine [...]

....but only in Cygwin rxvt, the standard Windows console doesn't show
the right colors.

I give up and revert back to ASCII. This whole charset mess is not
meant to solved by mere mortals.

Thorsten

[Jarek Zgoda]

Thorsten Kampe napisa³(a):

>>Under Windows I get " File "G:\program files\python\li b\encodings
\cp1252.py" , line 12, in encode
return codecs.charmap_ encode(input,er rors,encoding_t able)"

I'm not very experienced with internationaliz ation, but if you change::

gettext.install ('test')

to::

gettext.install ('test', unicode=True)

what happens?

No traceback anymore from optparse but the non-ascii umlauts are
displayed as question marks ("?").

And this is expected behaviour of encode() with errors set to 'replace'.
I think this is the solution to your problem. I was a bit surprised I
never saw this error, but I always use the unicode=True setting to
gettext.install ()...

--
Jarek Zgoda
http://jpa.berlios.de/

[Leo Kislov]

On Apr 1, 8:47 am, Thorsten Kampe <thors...@thors tenkampe.dewrot e:

I guess the culprit is this snippet from optparse.py:

# used by test suite
def _get_encoding(s elf, file):
encoding = getattr(file, "encoding", None)
if not encoding:
encoding = sys.getdefaulte ncoding()
return encoding

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
encoding = self._get_encod ing(file)
file.write(self .format_help(). encode(encoding , "replace"))

So this means: when the encoding of sys.stdout is US-ASCII, Optparse
sets the encoding to of the help text to ASCII, too.

..encode() method doesn't set an encoding. It encodes unicode text into
bytes according to specified encoding. That means optparse needs ascii
or unicode (at least) for help text. In other words you'd better use
unicode throughout your program.

But that's
nonsense because the Encoding is declared in the Po (localisation)
file.

For backward compatibility gettext is working with bytes by default,
so the PO file encoding is not even involved. You need to use unicode
gettext.

How can I set the encoding of sys.stdout to another encoding?

What are you going to set it to? As I understand you're going to
distribute your program to some users. How are you going to find out
the encoding of the terminal of your users?

-- Leo

[Thorsten Kampe]

* Leo Kislov (1 Apr 2007 14:24:17 -0700)

On Apr 1, 8:47 am, Thorsten Kampe <thors...@thors tenkampe.dewrot e:

I guess the culprit is this snippet from optparse.py:

# used by test suite
def _get_encoding(s elf, file):
encoding = getattr(file, "encoding", None)
if not encoding:
encoding = sys.getdefaulte ncoding()
return encoding

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
encoding = self._get_encod ing(file)
file.write(self .format_help(). encode(encoding , "replace"))

So this means: when the encoding of sys.stdout is US-ASCII, Optparse
sets the encoding to of the help text to ASCII, too.

.encode() method doesn't set an encoding. It encodes unicode text into
bytes according to specified encoding. That means optparse needs ascii
or unicode (at least) for help text. In other words you'd better use
unicode throughout your program.

But that's
nonsense because the Encoding is declared in the Po (localisation)
file.

For backward compatibility gettext is working with bytes by default,
so the PO file encoding is not even involved. You need to use unicode
gettext.

You mean

gettext.install ('test', unicode = True)
and
description = _(u'THIS SOFTWARE COMES WITHOUT WARRANTY, LIABILITY OR
SUPPORT!') ?

If I modify my code like this, I don't get any traceback anymore, but
the non-ascii umlauts are still displayed as question marks.
Thorsten

[Thorsten Kampe]

* Jarek Zgoda (Sun, 01 Apr 2007 22:02:15 +0200)

Thorsten Kampe napisa?(a):

>Under Windows I get " File "G:\program files\python\li b\encodings
\cp1252.py", line 12, in encode
return codecs.charmap_ encode(input,er rors,encoding_t able)"
I'm not very experienced with internationaliz ation, but if you change::

gettext.install ('test')

to::

gettext.install ('test', unicode=True)

what happens?

No traceback anymore from optparse but the non-ascii umlauts are
displayed as question marks ("?").

And this is expected behaviour of encode() with errors set to 'replace'.
I think this is the solution to your problem. I was a bit surprised I
never saw this error, but I always use the unicode=True setting to
gettext.install ()...

I can't see the "solution" here. Is the optparse "print_help " function
wrong? Why should there even be errors if I use "unicode = True" with
gettext.install ?

I have ISO-8859-15 gettext translations and I want optparse to display
them correctly. What do I have to do?

Thorsten

[Jarek Zgoda]

Thorsten Kampe napisa³(a):

I can't see the "solution" here. Is the optparse "print_help " function
wrong? Why should there even be errors if I use "unicode = True" with
gettext.install ?

I have ISO-8859-15 gettext translations and I want optparse to display
them correctly. What do I have to do?

Please, see gettext module documentation on this topic.

The solution is: always install your translation with unicode=True
setting. This assures usage of ugettext() instead of gettext() and works
properly with character sets other than ASCII. Your messages are
internally decoded to unicode objects and passed to output. Then the
displayed output will be limited only by the encoding of your terminal,
but it will not crash your program on any inconsistency, you would see
question marks.

--
Jarek Zgoda

"We read Knuth so you don't have to."

[Thorsten Kampe]

* Steven Bethard (Sun, 01 Apr 2007 10:21:40 -0600)

Thorsten Kampe wrote:
I'm not very experienced with internationaliz ation, but if you change::

gettext.install ('test')

to::

gettext.install ('test', unicode=True)

what happens?

Actually, this is the solution.

But there's one more problem: the solution only works when the
Terminal encoding is not US-ASCII. Unfortunately (almost) all
terminals I tried are set to US-ASCII (rxvt under Cygwin, Console[1]
running bash, Poderosa[2] running bash). Only the Windows Console is
CP852 and this works.

I got the tip to set a different encoding by
sys.stdout = codecs.EncodedF ile(sys.stdout, 'utf-8')

but unfortunately this does not change the encoding of any Terminal.
So my question is: how can I set a different encoding to sys.stdout
(or why can I set it without any error but nothing changes?)
Thorsten

[1] http://sourceforge.net/project/scree...group_id=43764
[2] http://en.poderosa.org/present/about_poderosa.html

[Thorsten Kampe]

* Jarek Zgoda (Mon, 02 Apr 2007 17:52:34 +0200)

Thorsten Kampe napisa?(a):

I can't see the "solution" here. Is the optparse "print_help " function
wrong? Why should there even be errors if I use "unicode = True" with
gettext.install ?

I have ISO-8859-15 gettext translations and I want optparse to display
them correctly. What do I have to do?

Please, see gettext module documentation on this topic.

The solution is: always install your translation with unicode=True
setting. This assures usage of ugettext() instead of gettext() and works
properly with character sets other than ASCII. Your messages are
internally decoded to unicode objects and passed to output. Then the
displayed output will be limited only by the encoding of your terminal,

You are right. My problem is that all the terminals I use are set to
US-ASCII (rxvt under Cygwin, Console[1] running bash, Poderosa[2]
running bash). Even those who actually support non-ASCII characters.

I got the tip to set a different encoding by
sys.stdout = codecs.EncodedF ile(sys.stdout, 'utf-8')

but unfortunately this does not change the encoding.

So my question is: how can I set a different encoding to sys.stdout
(or why can I set it without any error but nothing changes?)
Thorsten

[1] http://sourceforge.net/project/scree...group_id=43764
[2] http://en.poderosa.org/present/about_poderosa.html

[paul]

Thorsten Kampe schrieb:
[snipp]

I got the tip to set a different encoding by
sys.stdout = codecs.EncodedF ile(sys.stdout, 'utf-8')

but unfortunately this does not change the encoding of any Terminal.
So my question is: how can I set a different encoding to sys.stdout
(or why can I set it without any error but nothing changes?)

AFAIK you can't. If the terminal is limited to ascii it won't be able to
display anything else; it might not even have the right font, so how are
you supposed to fix that? The .encode(encodin g, "replace") ensures safe
downgrades though.

cheers
Paul