By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,222 Members | 2,478 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,222 IT Pros & Developers. It's quick & easy.

re-importing modules

P: n/a
Hi,

When troubleshooting code that's saved in a text file, I often find that I
want to make a change to it, re-save it, then reimport it. However, just
typing

import myTestCode

doesn't always seem to import the newer version. Is it supposed to? I find
that right now I often have to close my iPython window, then reopen it and
import my recently modified code. I know this can't be the best way to do
this, but I don't know what is.

any suggestions/help welcome and appreciated,
trevis


Apr 30 '07 #1
Share this Question
Share on Google+
14 Replies


P: n/a
On Apr 30, 12:49 pm, "T. Crane" <tcr...@REMOVETHISuiuc.eduwrote:
Hi,

When troubleshooting code that's saved in a text file, I often find that I
want to make a change to it, re-save it, then reimport it. However, just
typing

import myTestCode

doesn't always seem to import the newer version. Is it supposed to? I find
that right now I often have to close my iPython window, then reopen it and
import my recently modified code. I know this can't be the best way to do
this, but I don't know what is.

any suggestions/help welcome and appreciated,
trevis
Hi,

Another person posted the same thing today. As with that person, you
probably need to use the reload() function. See this post for more
details:

http://www.python.org/search/hyperma...1993/0342.html

Mike

Apr 30 '07 #2

P: n/a
On Mon, 2007-04-30 at 12:44 -0700, ky******@gmail.com wrote:
On Apr 30, 12:49 pm, "T. Crane" <tcr...@REMOVETHISuiuc.eduwrote:
Hi,

When troubleshooting code that's saved in a text file, I often find that I
want to make a change to it, re-save it, then reimport it. However, just
typing

import myTestCode

doesn't always seem to import the newer version. Is it supposed to? I find
that right now I often have to close my iPython window, then reopen it and
import my recently modified code. I know this can't be the best way to do
this, but I don't know what is.

any suggestions/help welcome and appreciated,
trevis

Hi,

Another person posted the same thing today. As with that person, you
probably need to use the reload() function. See this post for more
details:

http://www.python.org/search/hyperma...1993/0342.html

Mike
In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to. For example:

Module code:
# dog.py
class dog(object):
def bark(self): print "Arf!"

Interactive session:
>>import dog
d = dog.dog()
d.bark()
Arf!
>># Now the module code is being changed in another window...
reload(dog)
<module 'dog' from 'dog.py'>
>># A new dog instance will now say Woof:
d2 = dog.dog()
d2.bark()
Woof!
>># But the dog instance from before still says Arf:
d.bark()
Arf!

This may or may not be a problem for you, but the bottom-line is that
reload must be used with caution.

-Carsten
Apr 30 '07 #3

P: n/a
On Apr 30, 3:00 pm, Carsten Haese <cars...@uniqsys.comwrote:
On Mon, 2007-04-30 at 12:44 -0700, kyoso...@gmail.com wrote:
On Apr 30, 12:49 pm, "T. Crane" <tcr...@REMOVETHISuiuc.eduwrote:
Hi,
When troubleshooting code that's saved in a text file, I often find that I
want to make a change to it, re-save it, then reimport it. However, just
typing
import myTestCode
doesn't always seem to import the newer version. Is it supposed to? I find
that right now I often have to close my iPython window, then reopen it and
import my recently modified code. I know this can't be the best way to do
this, but I don't know what is.
any suggestions/help welcome and appreciated,
trevis
Hi,
Another person posted the same thing today. As with that person, you
probably need to use the reload() function. See this post for more
details:
http://www.python.org/search/hyperma...1993/0342.html
Mike

In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to. For example:

Module code:
# dog.py
class dog(object):
def bark(self): print "Arf!"

Interactive session:>>import dog
>d = dog.dog()
d.bark()
Arf!
># Now the module code is being changed in another window...
reload(dog)

<module 'dog' from 'dog.py'>>># A new dog instance will now say Woof:
>d2 = dog.dog()
d2.bark()
Woof!
># But the dog instance from before still says Arf:
d.bark()

Arf!

This may or may not be a problem for you, but the bottom-line is that
reload must be used with caution.

-Carsten
Carsten,

Very good point. I don't recall ever seeing that explained before.
Thanks!

Mike

Apr 30 '07 #4

P: n/a
ky******@gmail.com wrote:
>>In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to.
Maybe reloading modules should be deprecated. The semantics
are awful, and it interferes with higher-performance implementations.

John Nagle
May 1 '07 #5

P: n/a
En Mon, 30 Apr 2007 21:32:20 -0300, John Nagle <na***@animats.com>
escribió:
ky******@gmail.com wrote:
>>In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to.

Maybe reloading modules should be deprecated. The semantics
are awful, and it interferes with higher-performance implementations.
Maybe all the drawbacks should be clearly stated instead, telling that
it's not a magic wand as someone could think.

--
Gabriel Genellina

May 1 '07 #6

P: n/a
Gabriel Genellina wrote:
En Mon, 30 Apr 2007 21:32:20 -0300, John Nagle <na***@animats.com>
escribió:
>ky******@gmail.com wrote:
>>>In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to.


Maybe reloading modules should be deprecated. The semantics
are awful, and it interferes with higher-performance implementations.


Maybe all the drawbacks should be clearly stated instead, telling that
it's not a magic wand as someone could think.
It's one of those things which is more of an artifact of the CPython
implementation than a designed-in feature. A more compile-oriented
implementation like ShedSkin has to go to considerable effort to support
a feature like this, and it's not all that useful anyway.

There's a discussion of this, as it affects Jython and PyPy, here:

http://mail.python.org/pipermail/edu...ch/007815.html

Some of the enthusiasm for reloading comes from attempts to emulate
the older LISP environments, where the development environment lived
in the same namespaces as the execution environment. That really
wasn't that great an idea even in the LISP era. The Symbolics LISP
machines were the greatest expression of that idea, and turned out
to be a dead end.

John Nagle
May 1 '07 #7

P: n/a
On Tue, 01 May 2007 00:32:20 +0000, John Nagle wrote:
ky******@gmail.com wrote:
>>>In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to.

Maybe reloading modules should be deprecated. The semantics
are awful, and it interferes with higher-performance implementations.

I'd hate for reload to disappear, it is great for interactive
development/debugging, at least under some circumstances. (If you have
complex and tangled class hierarchies, it might not be powerful enough.)

As for the semantics being awful, I disagree. reload() does exactly
what it claims to do, no more, no less. I wouldn't expect reload(module1)
to reload modules 2 through 5 just because they were imported by module1.
If I wanted them reloaded, I'd say so.

I'll admit to being puzzled for about five minutes the first time I found
my objects' behaviour wasn't being updated when I did a reload, but it
didn't take me long to slap myself in the head. Of course not -- if the
old objects are still around, they'll keep their old behaviour. That's
kind of an inconvenience, but unavoidable. I certainly DON'T want Python
to magically change objects (apart from module objects themselves) on a
reload.

Maybe reload() should be pulled out of the core Python language and turned
into an optional function. If (if!) Jython doesn't do interactive
development, then it doesn't need reload(); CPython, which does, does.

--
Steven D'Aprano

May 1 '07 #8

P: n/a
Steven D'Aprano wrote:
I'd hate for reload to disappear, it is great for interactive
development/debugging, at least under some circumstances. (If you have
complex and tangled class hierarchies, it might not be powerful enough.)

As for the semantics being awful, I disagree. reload() does exactly
what it claims to do, no more, no less.
It's more complicated than that. See

http://arcknowledge.com/lang.jython..../msg00017.html

Exactly what reloading should do is still an open question for some of
the hard cases.

"reload" as a debug facility is fine.
Trouble comes from production programs which use it as a
reinitialization facility.

Reloading a module with multiple threads running gets
complicated. It works in CPython because CPython doesn't have
real concurrency. Insisting that it work like CPython implies
an inefficient locking model.

John Nagle
May 1 '07 #9

P: n/a
On May 1, 2:17 pm, Steven D'Aprano <s...@REMOVEME.cybersource.com.au>
wrote:
On Tue, 01 May 2007 00:32:20 +0000, John Nagle wrote:
kyoso...@gmail.com wrote:
>>In addition to the warning that reload() does not recursively reload
modules that the reloaded module depends on, be warned that reloading a
module does not magically affect any functions or objects from the old
version that you may be holding on to.
Maybe reloading modules should be deprecated. The semantics
are awful, and it interferes with higher-performance implementations.

I'd hate for reload to disappear, it is great for interactive
development/debugging, at least under some circumstances. (If you have
complex and tangled class hierarchies, it might not be powerful enough.)

As for the semantics being awful, I disagree. reload() does exactly
what it claims to do, no more, no less. I wouldn't expect reload(module1)
to reload modules 2 through 5 just because they were imported by module1.
If I wanted them reloaded, I'd say so.
That it doesn't reload a parent when a child changes may be fine in an
interactive debugger, but can cause problems if not done where
automatic reloading is being done in a long running web application
where you want to avoid server restarts. For that very reason, the new
importer in mod_python 3.3 tracks parent/child relationships between
modules and is able to import a parent module when a child changes.
The new importer does a lot more besides that as well, including
distinguishing modules by full path rather than purely by name. For
some details on the mod_python importer see documentation for
import_module() in:

http://www.modpython.org/live/curren...pi-apmeth.html

To understand why the new importer does some of the things it does, it
may be helpful to read all the problems the previous importer was
causing:

http://www.dscpl.com.au/wiki/ModPyth...ortingIsBroken

Getting module importing with reloading working is frightfully tricky
in an application where multiple threads corresponding to concurrent
web requests are executing as one can't be reloading on top of
modules where code may be executing, one must also avoid triggering a
reload of a specific module half way through a request where the same
module is encountered twice and lots, plus lots of other gotchas.

Graham

May 1 '07 #10

P: n/a
Graham Dumpleton <Gr**************@gmail.comwrites:
That it doesn't reload a parent when a child changes may be fine in an
interactive debugger, but can cause problems if not done where
automatic reloading is being done in a long running web application
where you want to avoid server restarts.
Oh that sounds horrid. If you really have to do something like that,
it's time to consider adding serious hot-patching capability as I
believe Erlang has done. Better is to start a new server process and
transfer the live session data and active connections to it through
IPC mechanisms. I've had an open RFE for many years (#814689) about
ancillary messages on AF_UNIX sockets, which let you pass file
descriptors (such as those attached to open TCP connections) between
processes. It actually came up in conversation with someone about
something I'm currently working on, so maybe I'll have reason to try
coding it sometime soon (that's not at all definite though).
May 1 '07 #11

P: n/a
On May 1, 3:51 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
Graham Dumpleton <Graham.Dumple...@gmail.comwrites:
That it doesn't reload a parent when a child changes may be fine in an
interactive debugger, but can cause problems if not done where
automatic reloading is being done in a long running web application
where you want to avoid server restarts.

Oh that sounds horrid. If you really have to do something like that,
it's time to consider adding serious hot-patching capability as I
believe Erlang has done. Better is to start a new server process and
transfer the live session data and active connections to it through
IPC mechanisms. I've had an open RFE for many years (#814689) about
ancillary messages on AF_UNIX sockets, which let you pass file
descriptors (such as those attached to open TCP connections) between
processes. It actually came up in conversation with someone about
something I'm currently working on, so maybe I'll have reason to try
coding it sometime soon (that's not at all definite though).
It may sound horrible but mod_python's original module importer had
lots of problems because of things not being reloaded when they should
have in a consistent manner. People weren't going to want to give up
the reload feature altogether as restarting Apache all the time is
viewed as a worse solution, so the only practical solution was at
least make it reliable and this is one of the things that was required
to do it. The number of complaints against mod_python module importing
and reloading problems has basically vanished now with mod_python 3.3
and the whole thing is so much more stable now.

Do note that this reloading mechanism isn't applied across all Python
modules, only mod_python handler and associated modules which are used
within the context of URL/document space of your web server. Thus it
is quite specialised and has quite specific use case scenarios which
are well understood. In that context the whole mechanism works fine.
Thus, please try not to pass judgment on it without full understanding
the problem space it operates in and how it internally works. :-)

Graham

May 1 '07 #12

P: n/a
Graham Dumpleton wrote:
On May 1, 3:51 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
>>Graham Dumpleton <Graham.Dumple...@gmail.comwrites:
>>>That it doesn't reload a parent when a child changes may be fine in an
interactive debugger, but can cause problems if not done where
automatic reloading is being done in a long running web application
where you want to avoid server restarts.

Oh that sounds horrid.
Er, yes.
It may sound horrible but mod_python's original module importer had
lots of problems because of things not being reloaded when they should
have in a consistent manner. People weren't going to want to give up
the reload feature altogether as restarting Apache all the time is
viewed as a worse solution, so the only practical solution was at
least make it reliable and this is one of the things that was required
to do it. The number of complaints against mod_python module importing
and reloading problems has basically vanished now with mod_python 3.3
and the whole thing is so much more stable now.

Do note that this reloading mechanism isn't applied across all Python
modules, only mod_python handler and associated modules which are used
within the context of URL/document space of your web server. Thus it
is quite specialised and has quite specific use case scenarios which
are well understood. In that context the whole mechanism works fine.
Thus, please try not to pass judgment on it without full understanding
the problem space it operates in and how it internally works. :-)
Ouch. This is one of those things where you start out with what
looks like a simple idea, but scaling it up introduces excessive
complexity. I'm impressed that somebody worked through that mess.

Could be worse, though. Imagine reloading, mod_python, and Twisted
all interacting during a software upgrade.

There's something to be said for the FastCGI approach. Restarts
and upgrades are handled in a straightforward manner. You're just
running a program over and over, without reloading, and every N minutes,
or when it's asked to do so, the program exits and is replaced by a
freshly loaded version. There's no reloading, just loading.

John Nagle
May 1 '07 #13

P: n/a
John Nagle wrote:
Steven D'Aprano wrote:
I'd hate for reload to disappear, it is great for interactive
development/debugging, at least under some circumstances. (If you have
complex and tangled class hierarchies, it might not be powerful enough.)

As for the semantics being awful, I disagree. reload() does exactly
what it claims to do, no more, no less.

It's more complicated than that. See

http://arcknowledge.com/lang.jython..../msg00017.html

Exactly what reloading should do is still an open question for some of
the hard cases.

"reload" as a debug facility is fine.
Trouble comes from production programs which use it as a
reinitialization facility.
I tend to agree with this; our servers do graceful restarts to pick up
code changes (which, in addition to avoiding reload also means that
it's easier as an admin to control the view of the code than in
"automatically pick up new changes" models--in particular, you can
wait until all changes are there, send SIGUSR1, and pick up the whole
set of changes atomically).
Reloading a module with multiple threads running gets
complicated. It works in CPython because CPython doesn't have
real concurrency.
Point of clarity: the CPython interpreter is not concurrent.
Concurrency != multithreading, and multiprocess solutions run fine in
CPython.

May 1 '07 #14

P: n/a
On 5/1/07, John Nagle <na***@animats.comwrote:
Steven D'Aprano wrote:
I'd hate for reload to disappear, it is great for interactive
development/debugging, at least under some circumstances. (If you have
complex and tangled class hierarchies, it might not be powerful enough.)

As for the semantics being awful, I disagree. reload() does exactly
what it claims to do, no more, no less.

It's more complicated than that. See

http://arcknowledge.com/lang.jython..../msg00017.html

Exactly what reloading should do is still an open question for some of
the hard cases.

"reload" as a debug facility is fine.
Trouble comes from production programs which use it as a
reinitialization facility.

Reloading a module with multiple threads running gets
complicated. It works in CPython because CPython doesn't have
real concurrency. Insisting that it work like CPython implies
an inefficient locking model.

John Nagle
--
Not really. The problem is when people attempt to overload the
current, existing reload() semantics (which are sensible and simple
once you understand that they don't attempt magic, although people
tend to expect the magic and they're non-intuitive in that manner).
There shouldn't be any problem implementing reload() in anything that
implements import.

The problem is with the fancier, magical imports that people keep
expecting reload() to be, and thats what both the links you've
provided are attempting to write. There's potentially a place for that
sort of reload(), but it clearly can't have the semantics that the
current reload does, and it suffers from all the same problems that
hotpatching systems always have - that it's not always clear how they
should work or what they should do, and you need a very tight
specification of how they will work in all the corner cases, including
threading. But that's got nothing to do with the current reload(),
which is simple, straightforward, and easily implemented.

The GIL isn't important in this respect - there's nothing complicated
about reload() that isn't also complicated about import. You either
have thread-local modules (which is probably stupid, because it can
result in all sorts of crazy behavior if you pass objects defined in
different versions of the same module between threads) or you
serialize import and access to sys.modules (or whatever your
underlying implementation of the module cache is).
May 1 '07 #15

This discussion thread is closed

Replies have been disabled for this discussion.