473,706 Members | 6,324 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Python not freeing memory (?)

I have a Bayesian simulation package that keeps running into memory
allocation problems. I have a feeling this has something to do with
Python (2.5.1.1) not freeing memory. The program essentially
iterates n times, each time proposing new arrays (numpy) of values
that are evaluated using statistical likelihoods. All variables in
the loop are re-assigned at each iteration, so there should not be
a leak. Nevertheless, at approximately the same iteration every
time, I run into malloc errors:

Iteration 36200 at 5647.58165097

Iteration 36300 at 5664.8412981

Iteration 36400 at 5681.71009493
Python(28344,0x a000d000) malloc: *** vm_allocate(siz e=8421376)
failed (error code=3)
Python(28344,0x a000d000) malloc: *** error: can't allocate region
Python(28344,0x a000d000) malloc: *** set a breakpoint in szone_error
to debug
Traceback (most recent call last):
File "/Users/chris/EURING/detection/detection2.py", line 285,
in <module>
run()
File "/Users/chris/EURING/detection/detection2.py", line 268, in run
results = sampler.sample( iterations=iter ations, burn=burn, thin=thin)
File "/Library/Frameworks/Python.framewor k/Versions/2.5/lib/python2.5/site-
packages/PyMC/MCMC.py", line 3021, in sample
parameter.propo se(debug)
File "/Library/Frameworks/Python.framewor k/Versions/2.5/lib/python2.5/site-
packages/PyMC/MCMC.py", line 768, in propose
if not self._sampler.t est():
File "/Library/Frameworks/Python.framewor k/Versions/2.5/lib/python2.5/site-
packages/PyMC/MCMC.py", line 2899, in test
self()
File "/Library/Frameworks/Python.framewor k/Versions/2.5/lib/python2.5/site-
packages/PyMC/MCMC.py", line 2562, in __call__
return self.model()
File "/Users/chris/EURING/detection/detection2.py", line 238, in model
self.pd = invlogit(self.b eta0 + self.beta1 * self.wind + self.beta2 * (self.cloud>0) + beta3[self.series-
1])
MemoryError

Is there *any* way of getting around this?

Aug 12 '07 #1
2 2261
Is there *any* way of getting around this?

Sure: Determine the bug, and fix it. A prerequisite
is that you have the source code of all extension modules
which you are using, but that seems to be the case.

If you want others to help you in finding the bug, you
need to provide more detail, e.g. a specific piece of
code that reproducibly wastes memory. If you want to
study how Python objects are allocated and released,
you need to create a debug build of Python (and all
extension modules), and start, e.g., with looking at the
value of sys.gettotalref count() over time.

HTH,
Martin
Aug 12 '07 #2
Martin v. Löwis <martin <atv.loewis.dew rites:
If you want others to help you in finding the bug, you
need to provide more detail, e.g. a specific piece of
code that reproducibly wastes memory. If you want to
study how Python objects are allocated and released,
you need to create a debug build of Python (and all
extension modules), and start, e.g., with looking at the
value of sys.gettotalref count() over time.
I tried monitoring the refcount at every iteration, but it
does not change; at the same time, the memory use by the
python process increases. This is why I suspected that
python was not returning memory.

Below is the method that gets called iteratively; the *_like methods
are statistical likelihoods implemented in f2py. I dont see
anything that is obviously responsible:

def model(self):
# Specification of joint log-posterior

#alpha3 = concatenate(([0], self.alpha3))

# Linear model for surfacing wait time
#self.lamda = exp(self.alpha0 + self.alpha1 * self.wind + self.alpha2 *
self.air + alpha3[self.series-1])
gamma3 = concatenate(([0], self.gamma3))

# Linear model for at-surface probability
self.theta = invlogit(self.g amma0 + self.gamma1 * self.wind + self.gamma
2 * self.air + gamma3[self.series-1])

x, n, theta = transpose([[z[1], sum(z), t] for z, t in zip(self.downup ,
self.theta) if type(z)!=type(0 .0)])

# Binomial likelihood of available animals
self.binomial_l ike(x, n, theta, name='theta')

# Probability of availability (per survey)
self.pa = 1.0 - (1 - self.theta)**10

beta3 = concatenate(([0], self.beta3))

# Linearmodel for probability of detection
self.pd = invlogit(self.b eta0 + self.beta1 * self.wind + self.beta2 * (s
elf.cloud>0) + beta3[self.series-1])

# Binomial likelihood of detection
self.binomial_l ike(self.obs, self.present, self.pd * self.pa, name='pd')

zeta1 = concatenate(([0], self.zeta1))

# Probability of presence
self.pp = invlogit(self.z eta0 + zeta1[self.series-1] + self.zeta2 * self
..intake + self.zeta3 * (self.discharge - self.intake))

# Binomial likelihood of presence
self.binomial_l ike(self.presen t, self.absent + self.present, self.pp, na
me='pp')

# Correct flight counts for detection
self.N_flight = self.count / (self.pd * self.pa)

# Aggregate counts by series
N_series = [self.N_flight[self.series==i+ 1] for i in range(6)]

# Log-normal likelihood for N
#sum(self.logno rmal_like(X, log(N), T) for X, N, T in zip(N_series, self
..N, self.T))
for N, mu in zip(N_series, self.N):
self.poisson_li ke(N, mu)

Thanks in advance for anything that you might suggest.

cf

Aug 13 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
2254
by: Gary | last post by:
Hi- I've been searching the web for a while and I've been unable to find a way to access c data objects in python without using SWIG. I can do methods just fine but I can't access variables. It seems like PyModule_AddObject should work but it segfaults my program. There's a good chance I'm just using it wrong, or I should be using something else. Could someone give me a simple, complete example pretty please? Thanks!
0
1696
by: Johnathan Doe | last post by:
I've been thinking about what the issues would be in compiling Python into native machine code, and since type information is important in Python, it seems possible that Python code can be compiled into native machine code (albeit with a lot of extra effort). For instance, type information is discovered when something is assigned to a variable or an anonymous piece of data is used in a program. Compiling Python bytecode into native...
10
2235
by: Steven D'Aprano | last post by:
Can somebody help me please? I've spent a fruitless hour googling with no luck. I'm discussing memory allocation techniques with somebody, and I'm trying to find a quote from -- I think -- Tim Peters where he discusses the way Python allocates memory when you append to lists. In basic terms, he says that every time you try to append to a list that is already full, Python doubles the size of the list. This wastes no more than 50% of the...
0
1594
by: Robby Dermody | last post by:
Hey guys (thus begins a book of a post :), I'm in the process of writing a commercial VoIP call monitoring and recording application suite in python and pyrex. Basically, this software sits in a VoIP callcenter-type environment (complete with agent phones and VoIP servers), sniffs voice data off of the network, and allows users to listen into calls. It can record calls as well. The project is about a year and 3 months in the making and...
13
4484
by: placid | last post by:
Hi All, Just wondering when i run the following code; for i in range(1000000): print i the memory usage of Python spikes and when the range(..) block finishes execution the memory usage does not drop down. Is there a way of freeing this memory that range(..) allocated?
17
8476
by: frederic.pica | last post by:
Greets, I've some troubles getting my memory freed by python, how can I force it to release the memory ? I've tried del and gc.collect() with no success. Here is a code sample, parsing an XML file under linux python 2.4 (same problem with windows 2.5, tried with the first example) : #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared #Using http://www.pixelbeat.org/scripts/ps_mem.py to get memory information
0
3084
by: greg.novak | last post by:
I am using Python to process particle data from a physics simulation. There are about 15 MB of data associated with each simulation, but there are many simulations. I read the data from each simulation into Numpy arrays and do a simple calculation on them that involves a few eigenvalues of small matricies and quite a number of temporary arrays. I had assumed that that generating lots of temporary arrays would make my program run slowly,...
0
8695
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9285
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
9049
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8989
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6614
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5938
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
3145
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2499
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2092
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.