473,573 Members | 2,683 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

numarray memory leak

Calling the following function with a large enough 'n' causes memory leak.

import numarray as N

def loop(n, m=100):
for i in xrange(n):
a = N.zeros((m,m))
N.matrixmultipl y(a, a)

If the matrixmultiply line is commented out, there is no leak, the
program has a stable memory size.

I am using numarray 1.1, the result is the same with Python 2.3.4 and
2.4b2.

Peter
Jul 18 '05 #1
5 1584
Peter Dobcsanyi wrote:
Calling the following function with a large enough 'n' causes memory leak.

import numarray as N

def loop(n, m=100):
for i in xrange(n):
a = N.zeros((m,m))
N.matrixmultipl y(a, a)

If the matrixmultiply line is commented out, there is no leak, the
program has a stable memory size.

I am using numarray 1.1, the result is the same with Python 2.3.4 and
2.4b2.


I don't see such behavior on Mac OSX 10.3, Python 2.3, CVS numarray and
n=10000.

Could you try CVS numarray? What n did you use?

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
Jul 18 '05 #2
Robert Kern <rk***@ucsd.edu > wrote:
Peter Dobcsanyi wrote:
Calling the following function with a large enough 'n' causes memory leak.

import numarray as N

def loop(n, m=100):
for i in xrange(n):
a = N.zeros((m,m))
N.matrixmultipl y(a, a)

If the matrixmultiply line is commented out, there is no leak, the
program has a stable memory size.

I am using numarray 1.1, the result is the same with Python 2.3.4 and
2.4b2.


I don't see such behavior on Mac OSX 10.3, Python 2.3, CVS numarray and
n=10000.

Could you try CVS numarray? What n did you use?


With n a few hundreds (say n>=500), the continuous growth of the
program's memory footprint is quite noticeable. I am working on Linux
(Debian testing), I monitor the memory usage of the program by top.
The allocated memory is never released.

I have noticed this problem of memory leak in connection with numarray a
few months ago in a program of mine. The program processes the incidence
matrices of thousands of combinatorial structures read from a file one
by one. As the number of combinatorial objects went up I started running
out of memory even on a machine with 2 Gbyte memory. It took me a while
to pinpoint that it was caused by matrix multiplication.

I will try the CVS.

Peter

Jul 18 '05 #3
>>>>> "Peter" == Peter Dobcsanyi <pe***@designth eory.org> writes:
Peter> With n a few hundreds (say n>=500), the continuous growth
Peter> of the program's memory footprint is quite noticeable. I am
Peter> working on Linux (Debian testing), I monitor the memory
Peter> usage of the program by top. The allocated memory is never
Peter> released.

Peter> I have noticed this problem of memory leak in connection
Peter> with numarray a few months ago in a program of mine. The
Peter> program processes the incidence matrices of thousands of
Peter> combinatorial structures read from a file one by one. As
Peter> the number of combinatorial objects went up I started
Peter> running out of memory even on a machine with 2 Gbyte
Peter> memory. It took me a while to pinpoint that it was caused
Peter> by matrix multiplication.

Peter> I will try the CVS.

Yep, numarray 1.1 has a memory leak in dot, which Matrix multiply
uses. It also has a leak in cumulative boolean functions like
sometrue. Both are fixed in CVS.

JDH
Jul 18 '05 #4
John Hunter <jd******@ace.b sd.uchicago.edu > wrote:

Peter> I will try the CVS.

Yep, numarray 1.1 has a memory leak in dot, which Matrix multiply
uses. It also has a leak in cumulative boolean functions like
sometrue. Both are fixed in CVS.


Indeed, I tried the CVS version and that fixed the problem; for me, at
least. I am a bit worried about the users of my package depending on
numarray. Most of them would not be able (or willing to) pull down the
CVS tree.

Peter
Jul 18 '05 #5
>>>>> "Peter" == Peter Dobcsanyi <pe***@designth eory.org> writes:

Peter> Indeed, I tried the CVS version and that fixed the problem;
Peter> for me, at least. I am a bit worried about the users of my
Peter> package depending on numarray. Most of them would not be
Peter> able (or willing to) pull down the CVS tree.

You might ask on the numpy-discussion list when the release of
numarray 1.2 is expected. My guess is it won't be too long from now.
If it is going to be a while and this is a serious problem for your
package, you have the option of tarring up CVS yourself and/or
building a win32 installer and linking to this on your site for users
concerned about memory leaks.

There is a section of matplotlib code that depends on matrix multiply,
and I have this FAQ for memory leaks

http://matplotlib.sourceforge.net/faq.html#LEAKS

JDH
Jul 18 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2977
by: RJS | last post by:
Hi all, I can't get a py2exe compiled app to run with numarray (numarray-0.5.win32- py2.2). Also wxPythonWIN32-2.3.3.1-Py22 and ActivePython-2.2.1-222. In the sample below, commenting out "import numarray" allows the exe to run. Left in, I get "4.exe has generated errors" etc. I'm going around and around and there isn't much on Google....
4
1973
by: Marco Bubke | last post by:
Hi Ok, I get a reproduceable seqmentation fault. Ok, fist the code: cdef NumArray array_to_double(NumArray array): # maybe here is memoty leak! cdef NumArray flat_array_obj flat_array_obj = NA_InputArray(NA_updateDataPtr(array), tFloat64,
11
1976
by: grv | last post by:
So it is supposed to be very fast to have an array of say 5 million integers stored in a binary file and do a = numarray.fromfile('filename', (2, 2, 2)) numarray.add(a, 9, a) but how is that faster than reading the entire file into memory and then having a for loop in C: (loop over range) { *p++ += 9 }
3
1547
by: Mizrandir | last post by:
I'd like to subclass numarray's array. I'd like to add some methods and override others like __init__. I don't know how to do this and haven't found help searching the manual or the web, can someone help? For example imagine I just want to do something as simple as making a subclass "NewClass" with the __init__ method overridden so that the...
5
2025
by: Bill Mill | last post by:
Hello all, I've pickled a numarray array object using cPickle like so: pickle = cPickle.Pickler(fout, -1) pickle.dump((myarray, list1, list2)) and this seems to work fine, until I try to load the array back into memory:
8
3395
by: ranjeet.gupta | last post by:
Dear All Is the Root Cause of the Memory corruption is the Memory leak, ?? suppose If in the code there is Memory leak, Do this may lead to the Memory Corruption while executing the program ? In nut shell, what is/are the realtion/s between the Memory Leak and Memory Corruption. Juts Theoritical Assumtion below:
17
4786
by: José Joye | last post by:
Hi, I have implemented a Service that is responsible for getting messages from a MS MQ located on a remote machine. I'm getting memory leak from time to time (???). In some situation, it is easier to reproduce (e.g.: remote machine not available). After about 1 day, I get a usage of 300MB of memory. I have used .NET Memory Profiler tool to...
23
4524
by: James | last post by:
The following code will create memory leaks!!! using System; using System.Diagnostics; using System.Data; using System.Data.SqlClient; namespace MemoryLeak
22
9320
by: Peter | last post by:
I am using VS2008. I have a Windows Service application which creates Crystal Reports. This is a multi theaded application which can run several reports at one time. My problem - there is a memory leak someplace. I can not detect the memory leak by running several reports by hand, but when I run tha app as a servrice and process few...
0
7746
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7986
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8179
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
8036
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5556
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5258
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3701
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
2167
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1269
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.