module imports and memory usage

[Brad Tilley]

When memory usage is a concern, is it better to do:

from X import Y

or

import X

Also, is there a way to load and unload modules as they are needed. I
have some scripts that sleep for extended periods during a while loop
and I need to be as memory friendly as possible. I can post a detailed
script that currently uses ~ 10MB of memory if anyone is interested.

Thanks,
Brad

[Istvan Albert]

Brad Tilley wrote:

Also, is there a way to load and unload modules as they are needed. I
have some scripts that sleep for extended periods during a while loop
and I need to be as memory friendly as possible. I can post a detailed
script that currently uses ~ 10MB of memory if anyone is interested.

I believe that the rule of thumb for Python, C, Java
and probably most other portable languages is that
freed memory cannot be released back to the
operating system. The best you can expect (usually true)
is that allocating the same amount later will not
increase the memory footprint.

So the only way to write memory efficient programs
might be to:
1. either not load all the data
2. run the data intensive routines as a separate process
that terminates and exists

Istvan.

[David Bolen]

Brad Tilley <br********@gmail.com> writes:

When memory usage is a concern, is it better to do:

from X import Y

or

import X
Depending on "Y", the latter can technically use less memory, but it's
likely to be fairly small and depends on how many symbols from that
module you want to have local bindings. In general, it's not worth
worrying about.

For example, if module X has 1000 bound names in it, then doing an
"import X" loads the module and gives your current namespace a single
reference to the entire module. If however, you do a "from X import
*" you still load the X module, but also get local names bound for
each of the names in the X module. So you have double the references
to the objects created within X. Those references take up space (for
the textual name and the pointer) in your current namespace's
dictionary.

But if you just do a "from X import Y" where Y is a single symbol,
then it's a wash because you get Y bound as a local namespace name,
but you don't have a reference to the module itself. If you were to
import multiple symbols, then you'd have a few extra references locally.

Sometimes though it's not even the memory you can affect, but
performance. One place where I did find this to be noticeable, for
example, was with wxPython, where older releases typically often did a
"from wxPython.wx import *" since all the wxPython names already had a
"wx" prefix. But that yielded thousands of extra name bindings in the
local namespace. Switching to "from wxPython import wx" and then
using "wx.XXX" instead of just "XXX" actually made a fairly dramatic
decrease in load time. It did also drop memory because I had a bunch
of plugin modules, all of which were burning up a few thousand name
bindings for the same wxPython symbols. Switching them to just use
the module reference was a noticeable savings in that case.
Also, is there a way to load and unload modules as they are needed. I
have some scripts that sleep for extended periods during a while loop
and I need to be as memory friendly as possible. I can post a detailed
script that currently uses ~ 10MB of memory if anyone is interested.

You can play tricks by manually deleting a module out of sys.modules,
but there's no guarantee that the memory will actually be released
(either by Python or the OS). Unless you're working under very
specific external resources, I'd generally leave this to the OS. It
will figure out when some of your working set is unused for extended
periods and generally it should end up in swap space if you actually
need the memory for something else.

-- David

[Brad Tilley]

Brad Tilley wrote:

When memory usage is a concern, is it better to do:

from X import Y

or

import X

Also, is there a way to load and unload modules as they are needed. I
have some scripts that sleep for extended periods during a while loop
and I need to be as memory friendly as possible. I can post a detailed
script that currently uses ~ 10MB of memory if anyone is interested.

Thanks,
Brad

I discovered that when I wrap my code up in a function def and call it
that it uses around 4.6 MB of RAM all the time... even while sleeping.
However, when I don't put it in a function def it uses around 2.6MB of
data when it executes and only 100KB while sleeping. Why is this?

[Carlos Ribeiro]

On Tue, 30 Nov 2004 16:46:43 -0500, Brad Tilley <br********@gmail.com> wrote:

I discovered that when I wrap my code up in a function def and call it
that it uses around 4.6 MB of RAM all the time... even while sleeping.
However, when I don't put it in a function def it uses around 2.6MB of
data when it executes and only 100KB while sleeping. Why is this?

Totally unchecked, popped up from somewhere between my ears.

Is it possible that (in some situations) the function object keeps a
reference to its locals() somewhere, in such a way that it never gets
collected? Just wondering...
--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: ca********@gmail.com
mail: ca********@yahoo.com