John J. Lee wrote:
le***********@yahoo.com (lebo) writes:
I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.
So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?
[...]
On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.
This is not true any more these days!!! Consider the following program:
#include <stdio.h>
#include <stdlib.h>
int main()
{
char* blobbone;
printf("No blob yet, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");
blobbone = malloc(1024*1024);
printf("I have the blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");
free(blobbone);
printf("No more blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");
printf("Bye bye.\n");
return 0;
}
Using Ctrl-Z to suspend and Enter to continue, on Linux, we see:
[alex@lancelot sae]$ gcc mem.c
[alex@lancelot sae]$ ./a.out
No blob yet, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out
I have the blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out
No more blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out
Bye bye.
[alex@lancelot sae]$
See? the DRS grows from 1330 to 2362 K when the blob is allocated,
then drops back to 1334 when the blob is freed again. I'm pretty
sure on Windows, with a good modern VC++ version, you'll see the
same, apart from the slightly higher difficulty of the "measure
me now" task;-).
And in Python...? *SAME THING*!!! Watch:
print "No blob yet, measure me now, then press enter...",
x = raw_input()
print
blobbone = 1024*256*[None]
print "I have the blob, measure me now, then press enter...",
x = raw_input()
print
del blobbone
print "No more blob, measure me now, then press enter...",
x = raw_input()
print
print "Bye bye."
[alex@lancelot sae]$ python mem.py
No blob yet, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2504 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py
I have the blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 4892 3536 0.6 python mem.py
[alex@lancelot sae]$ fg
python mem.py
No more blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2508 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py
Bye bye.
[alex@lancelot sae]$
See? Just the same behavior -- DRS grows 3864->4892, then shrinks
back to 3864 when the blob is dropped.
The explanation of the behavior observed by the original poster is
quite different from the one you posit: rather, it is that
"del module" does NOT drop the module -- the module remains in
memory, fully available and loaded, and it is pointed to by
the appropriate entry sys.modules['module'] in case you want it
back. That's all there is to it -- modules, as opposed to, say,
lists, are NOT so easily disposed of:-).
Alex
