473,390 Members | 1,306 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,390 software developers and data experts.

Python embedded in C & memory releasing

Hi

I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

So then I tried embedded Python in C:
(1) c application base 1.6MBytes (a good number)
(2) Call Python function from c application - memory up to 2.6MByte
(OK number)
(3) Python function exits but memory usage stays at 2.6MByte

Why is memory not releasing? How can I tell if this Python behaviour
or OS behaviour?

C code snippet:
#include <Python.h>

int main(int argc, char *argv[])
{
int y = 0;
printf("Started...\n");
for (int count=0;count < 999999999;count++) {
y += 1;
}
printf("Initing Py\n");
Py_Initialize();
PyRun_SimpleString("from time import sleep\n"
"print 'I will sleep for a while'\n"
"sleep(10)\n"
"print 'Exiting Py'\n");
Py_Finalize();

for (int count=0;count < 999999999;count++) {
y += 1;
}
return 0;
}

Thanks, Leonard
Jul 18 '05 #1
4 3019
On 23 Sep 2003, lebo wrote:
I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

So then I tried embedded Python in C:
(1) c application base 1.6MBytes (a good number)
(2) Call Python function from c application - memory up to 2.6MByte
(OK number)
(3) Python function exits but memory usage stays at 2.6MByte

Why is memory not releasing? How can I tell if this Python behaviour
or OS behaviour?


this is how most runtime systems handle memory. they have a certain amount
of space allocated, and if they run out, they ask the OS for more, but
they never give it back. i think this is what most implementations of
malloc/free do; it's certainly what java does.

what you want is some way to tell python to give back some or all of its
spare memory. i'm not aware of such a thing myself.

tom

--
unconstrained by any considerations of humanity or decency

Jul 18 '05 #2
le***********@yahoo.com (lebo) writes:
I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

[...]

On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.
John
Jul 18 '05 #3
John J. Lee wrote:
le***********@yahoo.com (lebo) writes:
I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

[...]

On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.


This is not true any more these days!!! Consider the following program:

#include <stdio.h>
#include <stdlib.h>

int main()
{
char* blobbone;

printf("No blob yet, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

blobbone = malloc(1024*1024);

printf("I have the blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

free(blobbone);

printf("No more blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

printf("Bye bye.\n");
return 0;
}

Using Ctrl-Z to suspend and Enter to continue, on Linux, we see:

[alex@lancelot sae]$ gcc mem.c
[alex@lancelot sae]$ ./a.out
No blob yet, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out
I have the blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out
No more blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out
Bye bye.
[alex@lancelot sae]$

See? the DRS grows from 1330 to 2362 K when the blob is allocated,
then drops back to 1334 when the blob is freed again. I'm pretty
sure on Windows, with a good modern VC++ version, you'll see the
same, apart from the slightly higher difficulty of the "measure
me now" task;-).

And in Python...? *SAME THING*!!! Watch:

print "No blob yet, measure me now, then press enter...",
x = raw_input()
print

blobbone = 1024*256*[None]

print "I have the blob, measure me now, then press enter...",
x = raw_input()
print

del blobbone

print "No more blob, measure me now, then press enter...",
x = raw_input()
print

print "Bye bye."
[alex@lancelot sae]$ python mem.py
No blob yet, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2504 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py
I have the blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 4892 3536 0.6 python mem.py
[alex@lancelot sae]$ fg
python mem.py
No more blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2508 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py
Bye bye.
[alex@lancelot sae]$

See? Just the same behavior -- DRS grows 3864->4892, then shrinks
back to 3864 when the blob is dropped.
The explanation of the behavior observed by the original poster is
quite different from the one you posit: rather, it is that
"del module" does NOT drop the module -- the module remains in
memory, fully available and loaded, and it is pointed to by
the appropriate entry sys.modules['module'] in case you want it
back. That's all there is to it -- modules, as opposed to, say,
lists, are NOT so easily disposed of:-).
Alex

Jul 18 '05 #4
Alex Martelli <al***@aleax.it> writes:
John J. Lee wrote: [...]
On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.


This is not true any more these days!!! Consider the following program:

[...] PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out [...] 32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out [...] 32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out

[...]

Well, you live and learn. :-)
John
Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

53
by: john67 | last post by:
The company I work for is about to embark on developing a commercial application that will cost us tens-of-millions to develop. When all is said and done it will have thousands of business...
4
by: Dennis Clark | last post by:
Hi all, I've looked through the threads about embedded Python that are a year and a half old, and I thought that I'd ask this question now to see if anything has changed. Has anyone, or is...
8
by: Brennus | last post by:
I have a dll/so which embeds python. I can verify it works by compiling it as an executable and adding an appropriate main. I tried to write unit tests for this library with ctypes and a simple...
0
by: proteusguy | last post by:
I've got a large C++ application that we're adding web integration to, XMLRPC and HTTP GET/POST stuff. I'm jumping at the chance to use embedded python to handle this aspect of the application as...
13
by: placid | last post by:
Hi All, Just wondering when i run the following code; for i in range(1000000): print i the memory usage of Python spikes and when the range(..) block finishes execution the memory usage...
5
by: vishnu | last post by:
Hi there, I am embedding python 2.5 on embedded system running on RTOS where I had strict memory constraints. As python is a huge malloc intensive application, I observed huge memory...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.