I18n issue with optik

[Thorsten Kampe]

Hi,

I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?
Thorsten

[1]
Traceback (most recent call last):
File "script.py" , line 37, in <module>
options, args = cmdlineparser.p arse_args()
File "/usr/lib/python2.5/optparse.py", line 1378, in parse_args
stop = self._process_a rgs(largs, rargs, values)
File "/usr/lib/python2.5/optparse.py", line 1418, in _process_args
self._process_l ong_opt(rargs, values)
File "/usr/lib/python2.5/optparse.py", line 1493, in
_process_long_o pt
option.process( opt, value, values, self)
File "/usr/lib/python2.5/optparse.py", line 782, in process
self.action, self.dest, opt, value, values, parser)
File "/usr/lib/python2.5/optparse.py", line 804, in take_action
parser.print_he lp()
File "/usr/lib/python2.5/optparse.py", line 1648, in print_help
file.write(self .format_help(). encode(encoding , "replace"))
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc4 in position
264: ordinal not in range(128)

[Steven Bethard]

Thorsten Kampe wrote:

I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?

Yes, probably. Though if you can reduce it to the simplest test case
that produces the error, it'll increase your chances of having someone
look at it.

You could also try posting to the optik list:
http://lists.sourceforge.net/lists/listinfo/optik-users

STeVe

[Thorsten Kampe]

* Steven Bethard (Sat, 31 Mar 2007 20:08:45 -0600)

Thorsten Kampe wrote:

I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?

Yes, probably. Though if you can reduce it to the simplest test case
that produces the error, it'll increase your chances of having someone
look at it.

The most simple test.py is:

###
#! /usr/bin/env python

import gettext, \
os, \
sys

gettext.textdom ain('optparse')
gettext.install ('test')

from optparse import OptionParser, \
OptionGroup

cmdlineparser = OptionParser(de scription = _('THIS SOFTWARE COMES
WITHOUT WARRANTY, LIABILITY OR SUPPORT!'))

options, args = cmdlineparser.p arse_args()
###

When I run LANGUAGE=de ./test.py --help I get the error.

### This is the test.de.po file
# Copyright (C) 2006 Thorsten Kampe
# Thorsten Kampe <th******@thors tenkampe.de>, 2006

msgid ""
msgstr ""

"Project-Id-Version: Template 1.0\n"
"POT-Creation-Date: Tue Sep 7 22:20:34 2004\n"
"PO-Revision-Date: 2005-07-03 16:47+0200\n"
"Last-Translator: Thorsten Kampe <th******@thors tenkampe.de>\n"
"Language-Team: Thorsten Kampe <th******@thors tenkampe.de>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=ISO-8859-15\n"
"Content-Transfer-Encoding: 8-bit\n"
"Generated-By: pygettext.py 1.5\n"

msgid "THIS SOFTWARE COMES WITHOUT WARRANTY, LIABILITY OR SUPPORT!"
msgstr "DIESES PROGRAMM HAT WEDER GEWÄHRLEISTUNG, HAFTUNG NOCH
UNTERSTÜTZUNG!"
###

The localisation now produces an error in the localised optik files,
too.

Under Windows I get " File "G:\program files\python\li b\encodings
\cp1252.py", line 12, in encode
return codecs.charmap_ encode(input,er rors,encoding_t able)"

Is there something I have to do to put the terminal in "non-ascii
output mode"?

I tried

###
#! /usr/bin/env python
# -*- coding: ISO-8859-15 -*-

print "DIESES PROGRAMM HAT WEDER GEWÄHRLEISTUNG, HAFTUNG NOCH
UNTERSTÜTZUNG!"
###

....and this worked. That means that my terminal is willing to print,
right?!

You could also try posting to the optik list:
http://lists.sourceforge.net/lists/listinfo/optik-users

I already did this via Gmane (although the list seems pretty dead to
me). Sourceforge seems to have a bigger problem as [1] and [2] error.

Sorry for the confusion but this Unicode magic is far from being
rational. I guess most people just don't get it...
Thorsten
[1] http://sourceforge.net/mailarchive/f...um=optik-users
[2] https://lists.sourceforge.net/lists/listinfo

[Thorsten Kampe]

Just an addition : when I insert this statement...

print _('THIS SOFTWARE COMES WITHOUT WARRANTY, LIABILITY OR SUPPORT!')

into this skript, the line is printed out. So if my Skript can output
the localised text but Optparse can't it should be an optparse bug,
right?!

Thorsten

[Thorsten Kampe]

I guess the culprit is this snippet from optparse.py:

# used by test suite
def _get_encoding(s elf, file):
encoding = getattr(file, "encoding", None)
if not encoding:
encoding = sys.getdefaulte ncoding()
return encoding

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
encoding = self._get_encod ing(file)
file.write(self .format_help(). encode(encoding , "replace"))

So this means: when the encoding of sys.stdout is US-ASCII, Optparse
sets the encoding to of the help text to ASCII, too. But that's
nonsense because the Encoding is declared in the Po (localisation)
file.

How can I set the encoding of sys.stdout to another encoding? Of
course this would be a terrible hack if the encoding of the
localisation changes or different translators use different
encodings...

Thorsten

[Steven Bethard]

Thorsten Kampe wrote:

* Steven Bethard (Sat, 31 Mar 2007 20:08:45 -0600)

>Thorsten Kampe wrote:

>>I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?

Yes, probably. Though if you can reduce it to the simplest test case
that produces the error, it'll increase your chances of having someone
look at it.

The most simple test.py is:

###
#! /usr/bin/env python

import gettext, \
os, \
sys

gettext.textdom ain('optparse')
gettext.install ('test')

from optparse import OptionParser, \
OptionGroup

cmdlineparser = OptionParser(de scription = _('THIS SOFTWARE COMES
WITHOUT WARRANTY, LIABILITY OR SUPPORT!'))

options, args = cmdlineparser.p arse_args()
###

When I run LANGUAGE=de ./test.py --help I get the error.

### This is the test.de.po file
# Copyright (C) 2006 Thorsten Kampe
# Thorsten Kampe <th******@thors tenkampe.de>, 2006

msgid ""
msgstr ""

"Project-Id-Version: Template 1.0\n"
"POT-Creation-Date: Tue Sep 7 22:20:34 2004\n"
"PO-Revision-Date: 2005-07-03 16:47+0200\n"
"Last-Translator: Thorsten Kampe <th******@thors tenkampe.de>\n"
"Language-Team: Thorsten Kampe <th******@thors tenkampe.de>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=ISO-8859-15\n"
"Content-Transfer-Encoding: 8-bit\n"
"Generated-By: pygettext.py 1.5\n"

msgid "THIS SOFTWARE COMES WITHOUT WARRANTY, LIABILITY OR SUPPORT!"
msgstr "DIESES PROGRAMM HAT WEDER GEWÄHRLEISTUNG, HAFTUNG NOCH
UNTERSTÜTZUNG!"
###

The localisation now produces an error in the localised optik files,
too.

Under Windows I get " File "G:\program files\python\li b\encodings
\cp1252.py", line 12, in encode
return codecs.charmap_ encode(input,er rors,encoding_t able)"

I'm not very experienced with internationaliz ation, but if you change::

gettext.install ('test')

to::

gettext.install ('test', unicode=True)

what happens?

STeVe

[Steven Bethard]

Thorsten Kampe wrote:

I guess the culprit is this snippet from optparse.py:

# used by test suite
def _get_encoding(s elf, file):
encoding = getattr(file, "encoding", None)
if not encoding:
encoding = sys.getdefaulte ncoding()
return encoding

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
encoding = self._get_encod ing(file)
file.write(self .format_help(). encode(encoding , "replace"))

So this means: when the encoding of sys.stdout is US-ASCII, Optparse
sets the encoding to of the help text to ASCII, too. But that's
nonsense because the Encoding is declared in the Po (localisation)
file.

How can I set the encoding of sys.stdout to another encoding? Of
course this would be a terrible hack if the encoding of the
localisation changes or different translators use different
encodings...

If print_help() is what's wrong, you should probably hack print_help()
instead of sys.stdout. You could try something like::

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
file.write(self .format_help())

optparse.Option Parser.print_he lp = print_help

cmdlineparser = optparse.Option Parser(descript ion=...)
...

That is, you could monkey-patch print_help() before you create an
OptionParser.

STeVe

[Thorsten Kampe]

* Steven Bethard (Sun, 01 Apr 2007 10:21:40 -0600)

Thorsten Kampe wrote:

* Steven Bethard (Sat, 31 Mar 2007 20:08:45 -0600)

Thorsten Kampe wrote:
I've written a script which uses Optik/Optparse to display the
options (which works fine). The text for the help message is localised
(with german umlauts) and when I execute the script with the localised
environment variable set, I get this traceback[1]. The interesting
thing is that the localised optparse messages from displays fine -
it's only my localisation that errors.

From my understanding, my script doesn't put out anything, it's
optik/optparse who does that. My po file is directly copied from the
optik po file (who displays fine) and modified so the po file should
be fine, too.

What can I do to troubleshoot whether the culprit is my script, optik
or gettext?

Would it make sense to post the script and the mo or po files?
Yes, probably. Though if you can reduce it to the simplest test case
that produces the error, it'll increase your chances of having someone
look at it.

The most simple test.py is:

###
#! /usr/bin/env python

import gettext, \
os, \
sys

gettext.textdom ain('optparse')
gettext.install ('test')

from optparse import OptionParser, \
OptionGroup

cmdlineparser = OptionParser(de scription = _('THIS SOFTWARE COMES
WITHOUT WARRANTY, LIABILITY OR SUPPORT!'))

options, args = cmdlineparser.p arse_args()
###

When I run LANGUAGE=de ./test.py --help I get the error.

### This is the test.de.po file
# Copyright (C) 2006 Thorsten Kampe
# Thorsten Kampe <th******@thors tenkampe.de>, 2006

msgid ""
msgstr ""

"Project-Id-Version: Template 1.0\n"
"POT-Creation-Date: Tue Sep 7 22:20:34 2004\n"
"PO-Revision-Date: 2005-07-03 16:47+0200\n"
"Last-Translator: Thorsten Kampe <th******@thors tenkampe.de>\n"
"Language-Team: Thorsten Kampe <th******@thors tenkampe.de>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=ISO-8859-15\n"
"Content-Transfer-Encoding: 8-bit\n"
"Generated-By: pygettext.py 1.5\n"

msgid "THIS SOFTWARE COMES WITHOUT WARRANTY, LIABILITY OR SUPPORT!"
msgstr "DIESES PROGRAMM HAT WEDER GEWÄHRLEISTUNG, HAFTUNG NOCH
UNTERSTÜTZUNG!"
###

The localisation now produces an error in the localised optik files,
too.

Under Windows I get " File "G:\program files\python\li b\encodings
\cp1252.py", line 12, in encode
return codecs.charmap_ encode(input,er rors,encoding_t able)"

I'm not very experienced with internationaliz ation, but if you change::

gettext.install ('test')

to::

gettext.install ('test', unicode=True)

what happens?

No traceback anymore from optparse but the non-ascii umlauts are
displayed as question marks ("?").

Thorsten

[Thorsten Kampe]

* Steven Bethard (Sun, 01 Apr 2007 10:26:54 -0600)

Thorsten Kampe wrote:

I guess the culprit is this snippet from optparse.py:

# used by test suite
def _get_encoding(s elf, file):
encoding = getattr(file, "encoding", None)
if not encoding:
encoding = sys.getdefaulte ncoding()
return encoding

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
encoding = self._get_encod ing(file)
file.write(self .format_help(). encode(encoding , "replace"))

So this means: when the encoding of sys.stdout is US-ASCII, Optparse
sets the encoding to of the help text to ASCII, too. But that's
nonsense because the Encoding is declared in the Po (localisation)
file.

How can I set the encoding of sys.stdout to another encoding? Of
course this would be a terrible hack if the encoding of the
localisation changes or different translators use different
encodings...

If print_help() is what's wrong, you should probably hack print_help()
instead of sys.stdout. You could try something like::

def print_help(self , file=None):
"""print_help(f ile : file = stdout)

Print an extended help message, listing all options and any
help text provided with them, to 'file' (default stdout).
"""
if file is None:
file = sys.stdout
file.write(self .format_help())

optparse.Option Parser.print_he lp = print_help

cmdlineparser = optparse.Option Parser(descript ion=...)
...

That is, you could monkey-patch print_help() before you create an
OptionParser.

Yes, I could do that but I'd rather know first if my code is wrong or
the optparse code.

Thorsten

[Thorsten Kampe]

* Thorsten Kampe (Sun, 1 Apr 2007 19:45:59 +0100)

Yes, I could do that but I'd rather know first if my code is wrong or
the optparse code.

It might be the bug mentioned in
http://mail.python.org/pipermail/pyt...ay/065458.html

The patch although doesn't work. From my unicode-charset-codepage-
codeset-challenged point of view the encoding of sys.stdout doesn't
matter. The charset is defined in the .po/.mo file (but of course
optparse can't know if the message has been translated by gettext
("_").

Thorsten