473,548 Members | 2,744 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Python Memory Usage

I am using Python to process particle data from a physics simulation.
There are about 15 MB of data associated with each simulation, but
there are many simulations. I read the data from each simulation into
Numpy arrays and do a simple calculation on them that involves a few
eigenvalues of small matricies and quite a number of temporary
arrays. I had assumed that that generating lots of temporary arrays
would make my program run slowly, but I didn't think that it would
cause the program to consume all of the computer's memory, because I'm
only dealing with 10-20 MB at a time.

So, I have a function that reliably increases the virtual memory usage
by ~40 MB each time it's run. I'm measuring memory usage by looking
at the VmSize and VmRSS lines in the /proc/[pid]/status file on an
Ubuntu (edgy) system. This seems strange because I only have 15 MB of
data.

I started looking at the difference between what gc.get_objects( )
returns before and after my function. I expected to see zillions of
temporary Numpy arrays that I was somehow unintentionally maintaining
references to. However, I found that only 27 additional objects were
in the list that comes from get_objects(), and all of them look
small. A few strings, a few small tuples, a few small dicts, and a
Frame object.

I also found a tool called heapy (http://guppy-pe.sourceforge.net/)
which seems to be able to give useful information about memory usage
in Python. This seemed to confirm what I found from manual
inspection: only a few new objects are allocated by my function, and
they're small.

I found Evan Jones article about the Python 2.4 memory allocator never
freeing memory in certain circumstances: http://evanjones.ca/python-memory.html.
This sounds a lot like what's happening to me. However, his patch was
applied in Python 2.5 and I'm using Python 2.5. Nevertheless, it
looks an awful lot like Python doesn't think it's holding on to the
memory, but doesn't give it back to the operating system, either. Nor
does Python reuse the memory, since each successive call to my
function consumes an additional 40 MB. This continues until finally
the VM usage is gigabytes and I get a MemoryException .

I'm using Python 2.5 on an Ubuntu edgy box, and numpy 1.0.3. I'm also
using a few routines from scipy 0.5.2, but for this part of the code
it's just the eigenvalue routines.

It seems that the standard advice when someone has a bit of Python
code that progressively consumes all memory is to fork a process. I
guess that's not the worst thing in the world, but it certainly is
annoying. Given that others seem to have had this problem, is there a
slick package to do this? I envision:
value = call_in_separat e_process(my_fu nc, my_args)

Suggestions about how to proceed are welcome. Ideally I'd like to
know why this is going on and fix it. Short of that workarounds that
are more clever than the "separate process" one are also welcome.

Thanks,
Greg

Jun 20 '07 #1
0 3071

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
5007
by: ulysses | last post by:
hi, I'm working in python 5 months. I think it's very cool language. I do a p2p python program GUI. First I make a software by wxpython. But I find wxpython use many many memory. Second I use PYQT. But memory use still big. But but but when I minimize the windows to taskbar, a fantasy something happened. memory use very very low when...
8
2091
by: Sridhar R | last post by:
Hi, I am a little experienced python programmer (2 months). I am somewhat experienced in C/C++. I am planning (now in design stage) to write an IDE in python. The IDE will not be a simple one. I had an idea of writing the IDE in C/C++, as it is a big project, bcoz of the following 1. if python is used, then the memory required for...
0
1578
by: Robby Dermody | last post by:
Hey guys (thus begins a book of a post :), I'm in the process of writing a commercial VoIP call monitoring and recording application suite in python and pyrex. Basically, this software sits in a VoIP callcenter-type environment (complete with agent phones and VoIP servers), sniffs voice data off of the network, and allows users to listen...
25
5716
by: abhinav | last post by:
Hello guys, I am a novice in python.I have to implement a full fledged mail server ..But i am not able to choose the language.Should i go for C(socket API) or python for this project? What are the advantages of one over the other in implementing this server.which language will be easier? What are the performance issues?In what language are...
13
4465
by: placid | last post by:
Hi All, Just wondering when i run the following code; for i in range(1000000): print i the memory usage of Python spikes and when the range(..) block finishes execution the memory usage does not drop down. Is there a way of freeing this memory that range(..) allocated?
3
3425
by: test.07 | last post by:
I am wondering what happens to a thread in python in relation to win32com extensions. If I create a new thread, that uses the Dispatch method from win32com, what happens to the memory allocated in that thread when the thread is done. Will the Dispatch release the memory it created, or will the memory remain? The problem rises from the...
17
8454
by: frederic.pica | last post by:
Greets, I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. Here is a code sample, parsing an XML file under linux python 2.4 (same problem with windows 2.5, tried with the first example) : #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb...
1
1715
by: yzghan | last post by:
Hi all, I feel that my python script is leaking memory. And this is a test I have: log.write(" " + "test() ... memory usage: " + " ".join(repr(i/(1024*1024)) for i in getMemInfo()) + "\n") m = {} i = 1000*1000 while i 0:
1
2024
by: Jean-Paul Calderone | last post by:
On Tue, 22 Apr 2008 14:54:37 -0700 (PDT), yzghan@gmail.com wrote: The test doesn't demonstrate any leaks. It does demonstrate that memory usage can remain at or near peak memory usage even after the objects for which that memory was allocated are no longer live in the process. This is only a leak if peak memory goes up again each time you...
0
7444
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7954
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7467
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
1
5367
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5085
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3478
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1932
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1054
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
755
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.