Deprecating reload() ???

[Ellinghaus, Lance]

> > Other surprises: Deprecating reload()

Reload doesn't work the way most people think
it does: if you've got any references to the old module,
they stay around. They aren't replaced. It was a good idea, but the implementation simply
doesn't do what the idea promises.

I agree that it does not really work as most people think it does, but how
would you perform the same task as reload() without the reload()?

Would this be done by "del sys.modules['modulename']" and then perform an
'import'?

I use reload() for many purposes, knowing how it works/does not work.

Lance Ellinghaus

[David MacQuigg]

On Tue, 16 Mar 2004 14:52:54 -0600, Skip Montanaro <sk**@pobox.com >
wrote:

Dave> def h23(freq):
Dave> s = complex(2*pi*fr eq)
Dave> h0 = PZfuncs.h0
Dave> z1 = PZfuncs.z1; z2 = PZfuncs.z2
Dave> p1 = PZfuncs.p1; p2 = PZfuncs.p2; p3 = PZfuncs.p3
Dave> return h0*(s-z1)*(s-z2)/((s-p1)*(s-p2)*(s-p3))

Dave> Notice the clarity in that last formula.

Yeah, but h0, z1, z2, etc are not long-lived copies of attributes in
PZfuncs. If I execute:

>>> blah = h23(freq)
>>> reload(PZfuncs)
>>> blah = h23(freq)

things will work properly. It's only long-lived aliases that present a
problem:

def h23(freq, h0=PZfuncs.h0):
s = complex(2*pi*fr eq)
z1 = PZfuncs.z1; z2 = PZfuncs.z2
p1 = PZfuncs.p1; p2 = PZfuncs.p2; p3 = PZfuncs.p3
return h0*(s-z1)*(s-z2)/((s-p1)*(s-p2)*(s-p3))

I think we are saying the same thing. I moved the alias definitions
inside the loop, knowing they would be short-lived, and therefor not a
problem.
Dave> In this case we avoid the problem of local variables out-of-sync
Dave> with a reloaded module by refreshing those variables with every
Dave> call to the function. In other cases, this may add too much
Dave> overhead to the computation.

Unlikely. Creating local copies of frequently used globals (in this case
frequently used globals in another module) is almost always a win. In fact,
it's so much of a win that a fair amount of brain power has been devoted to
optimizing global access. See PEPs 266 and 267 and associated threads in
python-dev from about the time they were written. (Note that optimizing
global access is still an unsolved problem in Python.)

Interesting. I just ran a test comparing 10,000 calls to the original
h23 above with 10,000 calls to h23a below.

h0 = PZfuncs.h0
z1 = PZfuncs.z1; z2 = PZfuncs.z2
p1 = PZfuncs.p1; p2 = PZfuncs.p2; p3 = PZfuncs.p3
def h23a(freq):
s = complex(2*pi*fr eq)
return h0*(s-z1)*(s-z2)/((s-p1)*(s-p2)*(s-p3))

The results are very close:

time/loop (sec) %
Test 1: 6.48E-006 119.1 (class PZfuncs)
Test 2: 6.88E-006 126.4 (module PZfuncs)
Test 3: 5.44E-006 100.0 (z1, p1, etc outside loop)

There is not much difference in the original loop between accessing
the constants from a class vs accessing them from a module. There is
a significant difference ( but not as much as I expected ) if we move
the six assignments outside the loop. Then we are back to the problem
of ensuring that the aliases get updated each time we update the
module.

-- Dave

[Skip Montanaro]

Terry> To make the formula nearly as clear without the overhead, I would
Terry> consider

Terry> import PZfuncs as z
Terry> def h23(freq):
Terry> s = complex(2*pi*fr eq)
Terry> return z.h0 * (s-z.z1) * (s-z.z2) / ((s-z.p1) * (s-z.p2) * (s-z.p3))

Or even:

def h23(freq):
z = PZfuncs
s = complex(2*pi*fr eq)
return z.h0 * (s-z.z1) * (s-z.z2) / ((s-z.p1) * (s-z.p2) * (s-z.p3))

(slightly faster and avoids the module aliasing problem Dave is concerned
about).

Skip

[Skip Montanaro]

Dave> Interesting. I just ran a test comparing 10,000 calls to the
Dave> original h23 above with 10,000 calls to h23a below.

...

Dave> The results are very close:

Dave> time/loop (sec) %
Dave> Test 1: 6.48E-006 119.1 (class PZfuncs)
Dave> Test 2: 6.88E-006 126.4 (module PZfuncs)
Dave> Test 3: 5.44E-006 100.0 (z1, p1, etc outside loop)

I'm not sure what these particular tests are measuring. I can't tell which
are h23() calls and which are h23a() calls, but note that because h23() and
h23a() are actually quite simple, the time it takes to call them is going to
be a fair fraction of all calls.

For timing stuff like this I recommend you use timeit.py. Most people here
are getting used to looking at its output. Put something like:

import PZfuncs
h0 = PZfuncs.h0
z1 = PZfuncs.z1; z2 = PZfuncs.z2
p1 = PZfuncs.p1; p2 = PZfuncs.p2; p3 = PZfuncs.p3

def h23null(freq):
pass

def h23a(freq):
s = complex(2*pi*fr eq)
return h0*(s-z1)*(s-z2)/((s-p1)*(s-p2)*(s-p3))

def h23b(freq):
z = PZfuncs
s = complex(2*pi*fr eq)
return z.h0*(s-z.z1)*(s-z.z2)/((s-z.p1)*(s-z.p2)*(s-z.p3))

into h23.py then run timeit.py like:

timeit.py -s "import h23 ; freq = NNN" "h23null(fr eq)"
timeit.py -s "import h23 ; freq = NNN" "h23a(freq) "
timeit.py -s "import h23 ; freq = NNN" "h23b(freq) "

Its output is straightforward and pretty immediately comparable across the
runs. The h23null() run will give you some idea of the call overhead. You
can, of course, dream up h23[cdefg]() variants as well.

Post code and results and we'll be happy to throw darts... :-)

Skip

[David MacQuigg]

On Tue, 16 Mar 2004 19:48:06 -0600, Skip Montanaro <sk**@pobox.com >
wrote:

For timing stuff like this I recommend you use timeit.py. Most people here
are getting used to looking at its output.
Excellent utility. This ought to be highlighted in the docs on the
time module, at least listed under "See also". I just grabbed the
first thing that came up and wrote my own little routine around the
clock() function.

[...]Post code and results and we'll be happy to throw darts... :-)

# PZfuncs.py -- Timing test for reloaded constants.

# Local constants:
h0 = 1
z1 = 1; z2 = 1
p1 = -1 +1j; p2 = -1 -1j; p3 = -1
pi = 3.1415926535897 931

import Constants

def h23null(freq):
pass

def h23a(freq):
s = complex(2*pi*fr eq)
return h0*(s-z1)*(s-z2)/((s-p1)*(s-p2)*(s-p3))

def h23b(freq):
z = Constants
s = complex(2*pi*fr eq)
return z.h0*(s-z.z1)*(s-z.z2)/((s-z.p1)*(s-z.p2)*(s-z.p3))

def h23c(freq):
h0 = Constants.h0
z1 = Constants.z1; z2 = Constants.z2
p1 = Constants.p1; p2 = Constants.p2; p3 = Constants.p3
s = complex(2*pi*fr eq)
return h0*(s-z1)*(s-z2)/((s-p1)*(s-p2)*(s-p3))

% timeit.py -s "from PZfuncs import * ; freq = 2.0" "h23null(fr eq)"
1000000 loops, best of 3: 0.461 usec per loop
% timeit.py -s "from PZfuncs import * ; freq = 2.0" "h23a(freq) "
100000 loops, best of 3: 4.94 usec per loop
% timeit.py -s "from PZfuncs import * ; freq = 2.0" "h23b(freq) "
100000 loops, best of 3: 5.79 usec per loop
% timeit.py -s "from PZfuncs import * ; freq = 2.0" "h23c(freq) "
100000 loops, best of 3: 6.29 usec per loop

My conclusion is that I should stick with form c in my application.
The savings from moving these assignments outside the function (form
a) does not justify the cost in possible problems after a reload. The
savings in going to form b is negligible. Form a is the easiest to
read, but forms b and c are not much worse.

These functions will typically be used in the interactive part of the
program to set up plots with a few hundred points. The time-consuming
computations are all done in the simulator, which is written in C++.

-- Dave

[David MacQuigg]

On Thu, 11 Mar 2004 15:10:59 -0500, "Ellinghaus , Lance"
<la************ **@eds.com> wrote:

Reload doesn't work the way most people think
it does: if you've got any references to the old module,
they stay around. They aren't replaced.

It was a good idea, but the implementation simply
doesn't do what the idea promises.

I agree that it does not really work as most people think it does, but how
would you perform the same task as reload() without the reload()?

pzfuncs <open file 'PZfuncs.py', mode 'r' at 0x00A86160> exec pzfuncs
p3 -2

--- Edit PZfuncs.py here ---
pzfuncs.seek(0)
exec pzfuncs
p3 -3

The disadvantage compared to reload() is that you get direct
references to *all* the new objects in your current namespace. With
reload() you get only a reference to the reloaded module. With the
proposed super_reload (at least the version I would like) you get no
new references in your current namespace, just updates on the
references that are already there.

Hmm. Maybe we could reload(), then loop over the available names, and
replace any that exist in the current namespace.

-- Dave

[Hung Jung Lu]

Skip Montanaro <sk**@pobox.com > wrote in message news:<ma******* *************** *************@p ython.org>...

Sitting for long periods in an interactive session and expecting it to track
your changes is foreign to me.
...
Not sure whether this is related to what you are talking about. In
VC/VB, while debugging a program, it is very often to get into this
situation:

(a) you have loops somewhere, (say, from i=0 to i=2000000)
(b) your program fails at some particular points in the loop,
(c) your debugger tells you there is a problem (and maybe you have
some assertion points,) and the execution stops at that point.
(d) you want to add some more debugging code to narrow down the spot,
or narrow down or the condition of the error,
(e) if you do not have good IDE, you'll have to start your program all
over again. But in VC/VB, you just insert some more code, and resume
the execution, all in matter of seconds. And you have much better
insight into the source and nature of the bug. (Is the bug from the
code? Is the bug from the data? What to do? Is the bug from C++? Is
the bug coming from stored-procedure in the database?)

Is this pure theory talk? No, because I just need to use it, now.

Without interactive programming's edit-and-continue feature, very
often you have to stop the program, insert just a few lines of code,
and restart again. This turns really bad when the initial state setup
takes time. Of course, if your programs don't take much initial setup
time, then you won't be able to realize the need or benefit of
edit-and-continue.

Sure, you can unit test things all you want. But in real life,
interactive debugging is, and will always be, the king of bug killers,
especially in large and complex systems.
Maybe it's just the nature of your users and their background, but an
(edit/test/run)+ cycle seems much more common in the Python community than a
run/(edit/reload)+ cycle.

It all depends. For Zope's external methods (CGIs), you don't restart
the whole web/app server everytime you make changes to a CGI. The
run/(edit/reload) cycle is the typical behavior of long-running
applications. (Except for some earlier versions of Java and
Microsoft's web/app servers, where you DID have to restart. And that
was very annoying.)

An analogy is with Windows 95, where everytime you install/update an
application you need to reboot the OS. We know how annoying that is.
Edit-and-continue addresses a similar problem.

By the way, I am told that Common Lisp also has good edit-and-continue
feature.

regards,

Hung Jung

[David MacQuigg]

On Mon, 15 Mar 2004 13:50:47 -0600, Skip Montanaro <sk**@pobox.com >
wrote:

Please file a bug report on Sourceforge so your ideas don't get lost. Feel
free to assign it to me (sf id == "montanaro" ).

Done. SF bug ID is 919099. Here is the proposed addition to
reload(module), just after the first paragraph:

"""
When reload(module) is executed:

The objects defined in module are compiled and loaded into memory as
new objects.

The old objects remain in memory until all references to them are
gone, and they are removed by the normal garbage-collection process.

The names in the module namespace are updated to point to any new or
changed objects. Names of unchanged objects, or of objects no longer
present in the new module, remain pointing at the old objects.

Names in other modules that refer directly to the old objects (without
the module-name qualifier) remain unchanged and must be updated in
each namespace where they occur.
"""

Anyone with corrections or clarifications, speak now.

Also here is a bit that I'm not sure of from my write-up on reload()
at http://ece.arizona.edu/~edatools/Python/Reload.htm:

Footnotes
[1] Reload(M1) is equivalent to the following:

file = open(M1.__file_ _.rstrip('c'), 'r')
file.seek(0) # needed if this is not the first reload
exec file in M1.__dict__ # repeat from line 2
file.close()

-- Dave