Problem with reimporting modules

[Christoph Zwerschke]

I'm currently investigating a problem that can hit you in TurboGears
when Kid template modules are reloaded in the background, because in
certain situations, global variables suddenly are set to None values.

I tracked it down to the following behavior of Python. Assume you have a
module hello.py like that:

---- hello. py ----
greeting = 'Hello!'
def print_hello():
print greeting
-------------------

Now run the following code:

from hello import print_hello
print_hello()
import sys
del sys.modules['hello'] # delete module
import hello # recreate module
print_hello()

The second print_hello() prints "None" instead of "Hello!". Why is that?
I had expected that it either prints an error or print "Hello!" as well.

Is this intended behavior of Python?

-- Christoph

[Dustan]

On Feb 11, 5:53 am, Christoph Zwerschke <[e-mail]:

I'm currently investigating a problem that can hit you in TurboGears
when Kid template modules are reloaded in the background, because in
certain situations, global variables suddenly are set to None values.

I tracked it down to the following behavior of Python. Assume you have a
module hello.py like that:

---- hello. py ----
greeting = 'Hello!'
def print_hello():
print greeting
-------------------

Now run the following code:

from hello import print_hello
print_hello()
import sys
del sys.modules['hello'] # delete module
import hello # recreate module
print_hello()

The second print_hello() prints "None" instead of "Hello!". Why is that?
I had expected that it either prints an error or print "Hello!" as well.

Is this intended behavior of Python?

-- Christoph

You're most likely looking for reload:
http://docs.python.org/lib/built-in-funcs.html#l2h-61
The documentation does imply that deleting a module from sys.modules
may not be what you want:
http://docs.python.org/lib/module-sys.html#l2h-5147

>>import sys
import warnings as previous_warnings # for example
# testing whether reload returns the same module object:

.... reload(previous_warnings) is previous_warnings
True

>># the following result is rather intuitive, but for

.... # the sake of demonstration, I'll do it anyway.
.... del sys.modules['warnings']

>>import warnings as after_warnings
after_warnings is previous_warnings

False

[Christoph Zwerschke]

Yes I know about reload(), but TurboGears (TurboKid) does not use it and
the docs say that removing modules from sys.module is possible to force
reloading of modules. I don't want to rewrite everything since it's a
pretty complex thing with modules which are compiled from templates
which can depend from other templates etc...

[Gabriel Genellina]

En Sun, 11 Feb 2007 15:56:16 -0300, Christoph Zwerschke <ci**@online.de>
escribió:

Yes I know about reload(), but TurboGears (TurboKid) does not use it and
the docs say that removing modules from sys.module is possible to force
reloading of modules. I don't want to rewrite everything since it's a
pretty complex thing with modules which are compiled from templates
which can depend from other templates etc...

If you remove the module from sys.modules, when you import it again you
end up with a *new*, fresh, module object, unrelated to the original one.

Quoting your original message again:

I tracked it down to the following behavior of Python. Assume you have a
module hello.py like that:
---- hello. py ----
greeting = 'Hello!'
def print_hello():
print greeting
-------------------
Now run the following code:
from hello import print_hello
print_hello()
import sys
del sys.modules['hello'] # delete module
import hello # recreate module
print_hello()
The second print_hello() prints "None" instead of "Hello!". Why is that?

Notice that you are mixing references here. You first import print_hello
from hello, and after deleting the module, you import hello (not
print_hello). And you expect that your old reference to print_hello now
refers to the new function. The whole point of reloading/reimporting a
module is to get the *new* contents, but that only works if you refer to
things using the module.function notation, not if you hold a reference to
the function (which will always be the original function).

In short, your code should be:

import hello
hello.print_hello()
import sys
del sys.modules['hello']
import hello
hello.print_hello()

or, using reload:

import hello
hello.print_hello()
reload(hello)
hello.print_hello()

If you think that always typing module.function is too much - well, don't
try to reload modules then :)
Somewhere I read that at Google the policy is to always import modules,
never functions, and this may be a good reason for it.

If you want to know the details: print_hello doesn't hold a reference to
the containing module, only to its namespace, in the func_globals
attribute. When you delete the last reference to the module, it gets
destroyed. At that time, all values in the module namespace are set to
None (for breaking possible cycles, I presume). print_hello now has a
func_globals with all names set to None. (Perhaps the names could have
been deleted instead, so print_hello() would raise a NameError, but I'm
just describing the current CPython implementation)

--
Gabriel Genellina

[Christoph Zwerschke]

Thanks for the detailed explanations, Gabriel.

At that time, all values in the module namespace are set to
None (for breaking possible cycles, I presume). print_hello now has a
func_globals with all names set to None. (Perhaps the names could have
been deleted instead, so print_hello() would raise a NameError, but I'm
just describing the current CPython implementation)

Yes, that was the thing that confused me a bit. I had expected that an
error is raised or that the namespace has a reference to the module and
is reestablished when the module has been reloaded.

Anyway, I have solved the problem in a different way now.

-- Christoph