Problem with mixing doctest with gettext _()

[Pierre Rouleau]

I have a problem writing self-testable modules using doctest when these
modules have internationalized strings using gettext _('...').

- The main module of an application (say app.py) calls gettext.install()
to install the special _ function inside Python builtin. Other modules,
taken from a general purpose collection of Python modules, also support
internationalisation and doctest testing.

For example:

utstring.py would contain some code like this:

def onOffStr(isOn) :
"""Return the "ON" string for True, "OFF" for False.

**Example**

onOffStr(True) u'ON' onOffStr(False) u'OFF'

"""
if isOn:
return _(u"ON") # notice the underscore
else:
return _(u"OFF")

The utstring module does not call any of the gettext calls, because some
other module does it when the application runs. So the doctest fails:

************************************************** ***************
Failure in example: onOffStr(True)
from line #4 of utstring.onOffStr
Exception raised:
Traceback (most recent call last):
File "C:\Python23\Lib\doctest.py", line 442, in _run_examples_inner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "C:\dev\python\utstring.py", line 513, in onOffStr
return _(u"ON")
TypeError: 'tuple' object is not callable
************************************************** ***************

I tried to define a _() function when testing with the code below but
the doctest still fails. The following code is at the end of my
utstring.py module

def _test():
"""_test() perform docstring test"""

import doctest, utstring
return doctest.testmod(utstring)

if __name__ == "__main__":
def _(aString):
# dummy _() attempting to get doctest to pass.
return aString

_test()
----------

Does anyone know why the doctest still fails when I define the dummy _()
function?
Thanks in advance.

Pierre

[Skip Montanaro]

Pierre> I tried to define a _() function when testing with the code
Pierre> below but the doctest still fails. The following code is at the
Pierre> end of my utstring.py module
...

I suspect it's because your dummy _ is not in builtins. Why not just call
gettext.install() where you are currently defining _?

Skip

[Pierre Rouleau]

Skip Montanaro wrote:

Pierre> I tried to define a _() function when testing with the code
Pierre> below but the doctest still fails. The following code is at the
Pierre> end of my utstring.py module
...

I suspect it's because your dummy _ is not in builtins. Why not just call
gettext.install() where you are currently defining _?

Skip, this looks like the way to go.

I was trying to avoid it because the translation files for these library
modules are constructed at the application level somewhere else. But I
was also considering creating a translation library for these module, so
I guess that is one more incentive to do so...

Thanks

Pierre

[Skip Montanaro]

I suspect it's because your dummy _ is not in builtins. Why not just
call gettext.install() where you are currently defining _?
Pierre> Skip, this looks like the way to go.

Pierre> I was trying to avoid it because the translation files for these
Pierre> library modules are constructed at the application level
Pierre> somewhere else. But I was also considering creating a
Pierre> translation library for these module, so I guess that is one
Pierre> more incentive to do so...

If you really want a dummy _() you can also stuff your version into
builtins:

import __builtin__
def foo(s): return s ... __builtin__._ = foo
_ <function foo at 0x1d6670> _("hi")

'hi'

Skip

[Pierre Rouleau]

Skip Montanaro wrote:

If you really want a dummy _() you can also stuff your version into
builtins:

>>> import __builtin__
>>> def foo(s): return s ... >>> __builtin__._ = foo
>>> _ <function foo at 0x1d6670> >>> _("hi") 'hi'

I tried that, but it only works for the first call...

[Shell buffer started: python.exe]
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

import __builtin__
def foo(s): return s .... __builtin__._ = foo
_ <function foo at 0x008F98B0> _ <function foo at 0x008F98B0> _('hi') 'hi' _ 'hi' _('Hello') Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'str' object is not callable

Pierre

[Peter Otten]

Pierre Rouleau wrote:

Skip Montanaro wrote:

If you really want a dummy _() you can also stuff your version into
builtins:

>>> import __builtin__
>>> def foo(s): return s

...

>>> __builtin__._ = foo
>>> _

<function foo at 0x1d6670>

>>> _("hi")

'hi'

I tried that, but it only works for the first call...

Setting __builtin__._ to the result of the last calculation is a side effect
of sys.displayhook. Therefore you need to change that too:

Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
[GCC 3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
def mydisplayhook(a): .... if a is not None: sys.stdout.write("%r\n" % a)
.... def foo(s): return s .... sys.displayhook = mydisplayhook
import __builtin__
__builtin__._ = foo
_("hi") 'hi' _("hello") 'hello'

Peter

[Pierre Rouleau]

Pierre Rouleau wrote:

Skip Montanaro wrote:

Pierre> I tried to define a _() function when testing with the code
Pierre> below but the doctest still fails. The following code is
at the
Pierre> end of my utstring.py module
...

I suspect it's because your dummy _ is not in builtins. Why not just
call
gettext.install() where you are currently defining _?

Skip, this looks like the way to go.

I was trying to avoid it because the translation files for these library
modules are constructed at the application level somewhere else. But I
was also considering creating a translation library for these module, so
I guess that is one more incentive to do so...

I implemented the dictionary and use it in the code just fine, but the
doctest still fails :(

My teststr.py module is:

#--[---------------------------------------------------------------
if __name__ == "__main__":
# install gettext for testing this module because of the string
translation
# performed in the code.
import gettext
gettext.install('impathpl', './locale', unicode=False)
presLan_en = gettext.translation('impathpl', "./locale",
languages=['en'])
presLan_en.install()

def onOffStr(isOn) :
"""Return the "ON" string for True, "OFF" for False.

**Example**

onOffStr(True) 'ON' onOffStr(False) 'OFF' """
if isOn:
return _("ON")
else:
return _("OFF")
def _test():
"""_test() perform docstring test"""

import doctest, teststr
return doctest.testmod(teststr)

if __name__ == "__main__":
_test()

#--]-------------------------------------------------
Running the following script shows that the module runs OK:

[Shell buffer started: python.exe]
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information. import teststr
import gettext
gettext.install('impathpl', './locale', unicode=False)
presLan_en = gettext.translation('impathpl', "./locale", languages=['en']) presLan_en.install()
teststr.onOffStr(True) 'ON'

BUT, the doctest fails:

D:\dev\python>teststr
************************************************** ***************
Failure in example: onOffStr(False)
from line #6 of teststr.onOffStr
Exception raised:
Traceback (most recent call last):
File "c:\Python23\lib\doctest.py", line 442, in _run_examples_inner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "teststr.py", line 23, in onOffStr
return _("OFF")
TypeError: 'str' object is not callable
************************************************** ***************
1 items had failures:
1 of 2 in teststr.onOffStr
***Test Failed*** 1 failures.
I'm still puzzled...

Pierre

[Pierre Rouleau]

Peter Otten wrote:

Pierre Rouleau wrote:

Skip Montanaro wrote:

If you really want a dummy _() you can also stuff your version into
builtins:

>>> import __builtin__
>>> def foo(s): return s
...
>>> __builtin__._ = foo
>>> _
<function foo at 0x1d6670>
>>> _("hi")
'hi'

I tried that, but it only works for the first call...

Setting __builtin__._ to the result of the last calculation is a side effect
of sys.displayhook. Therefore you need to change that too:

Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
[GCC 3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
def mydisplayhook(a):
... if a is not None: sys.stdout.write("%r\n" % a)
...
def foo(s): return s
...
sys.displayhook = mydisplayhook
import __builtin__
__builtin__._ = foo
_("hi")
'hi'
_("hello")

'hello'

Thanks Peter, it does work!

[Skip Montanaro]

Pierre> BUT, the doctest fails:
...

Looks like Peter Otten's sys.displayhook hack is required for doctest.

Skip

[Skip Montanaro]

Pierre> I tried that, but it only works for the first call...

:-)

Pierre> [Shell buffer started: python.exe]
Pierre> Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
Pierre> win32
Pierre> Type "help", "copyright", "credits" or "license" for more information.

import __builtin__
def foo(s): return s Pierre> ... __builtin__._ = foo
_ Pierre> <function foo at 0x008F98B0> _ Pierre> <function foo at 0x008F98B0> _('hi') Pierre> 'hi' _ Pierre> 'hi' _('Hello') Pierre> Traceback (most recent call last):
Pierre> File "<stdin>", line 1, in ?
Pierre> TypeError: 'str' object is not callable

That's true. In interactive mode _ is assigned the value of the last
expression evaluated. Try it from a script.

Skip

[Pierre Rouleau]

Pierre Rouleau wrote:

Peter Otten wrote:

Pierre Rouleau wrote:

Skip Montanaro wrote:
If you really want a dummy _() you can also stuff your version into
builtins:

>>> import __builtin__
>>> def foo(s): return s
...
>>> __builtin__._ = foo
>>> _
<function foo at 0x1d6670>
>>> _("hi")
'hi'
I tried that, but it only works for the first call...

Setting __builtin__._ to the result of the last calculation is a side
effect
of sys.displayhook. Therefore you need to change that too:

Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
[GCC 3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

> import sys
> def mydisplayhook(a):

... if a is not None: sys.stdout.write("%r\n" % a)
...

> def foo(s): return s

...

> sys.displayhook = mydisplayhook
> import __builtin__
> __builtin__._ = foo
> _("hi")

'hi'

> _("hello")

'hello'

Thanks Peter, it does work!

It worked, BUT only for a simple function, it fails if I add a another
simple function:

My updated teststr.py script:

#--[--------------------------------------
def onOffStr(isOn) :
"""Return the "ON" string for True, "OFF" for False.

**Example**

onOffStr(True) u'ON' onOffStr(False) u'OFF' """
if isOn:
return _(u"ON")
else:
return _(u"OFF")
def inList(longString, stringList) :
"""Return True if one of the string in `stringList` is inside
`longString`.

Also return the list index.
**Example**
L = ["**", "/*"]
inList("aksdkajshd",L) (False, 0) inList("aksdkajsh**d",L)

(True, 0)

"""

theIndex = 0
for theStr in stringList:
if longString.find(theStr) >= 0:
return (True,theIndex)
theIndex +=1
return (False,0)
def _test():
"""_test() perform docstring test"""

import doctest, teststr
return doctest.testmod(teststr)

if __name__ == "__main__":
import sys

def test_displayhook(a):
if a is not None: sys.stdout.write("%r\n" % a)

def test_translator(aString):
return aString

sys.displayhook = test_displayhook

import __builtin__
__builtin__._ = test_translator
_test()

#--]--------------------------------------

Running the test fails:

D:\dev\python>teststr
************************************************** ***************
Failure in example: inList("aksdkajshd",L)
from line #6 of teststr.inList
Exception raised:
Traceback (most recent call last):
File "c:\Python23\lib\doctest.py", line 442, in _run_examples_inner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "D:\dev\python\teststr.py", line 50, in test_displayhook
if a is not None: sys.stdout.write("%r\n" % a)
TypeError: not all arguments converted during string formatting
************************************************** ***************
Failure in example: inList("aksdkajsh**d",L)
from line #8 of teststr.inList
Exception raised:
Traceback (most recent call last):
File "c:\Python23\lib\doctest.py", line 442, in _run_examples_inner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "D:\dev\python\teststr.py", line 50, in test_displayhook
if a is not None: sys.stdout.write("%r\n" % a)
TypeError: not all arguments converted during string formatting
************************************************** ***************
1 items had failures:
2 of 3 in teststr.inList
***Test Failed*** 2 failures.

#------------------------------------------

So far, I don't have a solution for writing internationalized Python
that support doctest. Surely, I am not the first one trying to do that...
Pierre

[Peter Otten]

Pierre Rouleau wrote:

It worked, BUT only for a simple function, it fails if I add a another
simple function:

Haven't read your post completely, but judging from the error message, it's
just a minor glitch in mydisplayhook(). Try the following instead:

def mydisplayhook(a):

.... if a is not None:
.... sys.stdout.write("%r\n" % (a,))
....

That should be able to cope with tuples as commandline results.

Peter

[Pierre Rouleau]

Peter Otten wrote:

Pierre Rouleau wrote:

It worked, BUT only for a simple function, it fails if I add a another
simple function:

Haven't read your post completely, but judging from the error message, it's
just a minor glitch in mydisplayhook(). Try the following instead:

def mydisplayhook(a):

... if a is not None:
... sys.stdout.write("%r\n" % (a,))
...

That should be able to cope with tuples as commandline results.

Peter

You're right! It does work. I must admit, that I don't see why though.
(a,) makes a tuple out of the `a` argument. Does the %r conversion
require a tuple?

[Peter Otten]

Pierre Rouleau wrote:

Peter Otten wrote:

Pierre Rouleau wrote:

It worked, BUT only for a simple function, it fails if I add a another
simple function:

Haven't read your post completely, but judging from the error message,
it's just a minor glitch in mydisplayhook(). Try the following instead:

>def mydisplayhook(a):

... if a is not None:
... sys.stdout.write("%r\n" % (a,))
...

That should be able to cope with tuples as commandline results.

Peter

You're right! It does work. I must admit, that I don't see why though.
(a,) makes a tuple out of the `a` argument. Does the %r conversion
require a tuple?

The formatting operator behaves differently depending on whether the
righthand argument is a tuple or something else.

formatstr % tuple

ensures that there is a corresponding format expression - e.g. "%s" or "%d"
- for every item in the tuple, whereas

formatstr % nontuple

expects exactly one format expression. Therefore

"%r\n" % a

raises an exception when a is a tuple with more or less than one item and
wrongly prints the item's representation instead of the tuple's for
one-tuples.
Wrapping it like so (a,) fixes the problem because now we have always a
tuple with one item - where this item is sometimes a tuple.

An alternative approach would be to ensure that the righthand side is always
a nontuple:

"%s\n" % repr(a)

Peter

[Pierre Rouleau]

You're right! It does work. I must admit, that I don't see why though.
(a,) makes a tuple out of the `a` argument. Does the %r conversion
require a tuple?

The formatting operator behaves differently depending on whether the
righthand argument is a tuple or something else.

formatstr % tuple

ensures that there is a corresponding format expression - e.g. "%s" or "%d"
- for every item in the tuple, whereas

formatstr % nontuple

expects exactly one format expression. Therefore

"%r\n" % a

raises an exception when a is a tuple with more or less than one item and
wrongly prints the item's representation instead of the tuple's for
one-tuples.
Wrapping it like so (a,) fixes the problem because now we have always a
tuple with one item - where this item is sometimes a tuple.

An alternative approach would be to ensure that the righthand side is always
a nontuple:

"%s\n" % repr(a)

Thanks for this clear explanation!

Pierre