469,649 Members | 1,729 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,649 developers. It's quick & easy.

Update on Memory problem with NumPy arrays

Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

I was able to reduce the code (see below) to its smallest size and
still
have the problem, albeit at a slower rate. The problem appears to come
from changing values in the array. Does this create another reference
to the
array, which can't be released?

Also, are there other python methods/extensions that can create
multi-deminsional
arrays?

thanks again to those who repsonded to the last post
Sonja

PS. to watch the memory usage I just used task manager

the code:
from numpy import *

y = ones((501,501))
z = zeros((501,501))
it = 50

for kk in xrange(it):
y[1,1] = 4
y[1,2] = 4
y[1,0] = 4
y[2,1] = 6

print "Iteration #:%s" %(kk)
for ee in xrange(0,501):
for ff in xrange(0,501):
if y[ee,ff] == 4 or y[ee,ff] == 6:
y[ee,ff] = 2
else:
pass

Jun 21 '06 #1
4 2512
sonjaa wrote:
Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

I was able to reduce the code (see below) to its smallest size and
still
have the problem, albeit at a slower rate.
Please post this to numpy-discussion instead of here. Also, please create a
ticket in our Trac:

http://projects.scipy.org/scipy/numpy
The problem appears to come
from changing values in the array. Does this create another reference
to the
array, which can't be released?
Since the array shouldn't be going away and no new arrays should be created that
wouldn't cause a problem.

It's possible that there is a bug with the scalar objects that are being created
when indexing into the arrays. I can reproduce abnormal memory consumption even
when I remove the line where you are setting value in the if: clause.
Also, are there other python methods/extensions that can create
multi-deminsional
arrays?


A few. numpy's predecessors, Numeric and numarray are still usable, but aren't
being actively developed. There's a pure-Python array package somewhere, but I
forget the name.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Jun 21 '06 #2
sonjaa wrote:
Also, are there other python methods/extensions that can create
multi-deminsional arrays?


if this example is typical for the code you're writing, you might as
well use nested Python lists:

def make_array(width, height, value):
out = []
for y in range(height):
out.append([value] * width)
return

y = make_array(501, 501, 1)
z = make_array(501, 501, 0)

y[ee][ff] = 4

etc

</F>

Jun 21 '06 #3
sonjaa wrote:
Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

[...]

Based on the numpy-discussion this seems to be fixed in the SVN now(?).

Anyway, you can use 'where' function to eliminate the loops:

from numpy import *

y = ones((501,501))
z = zeros((501,501))
it = 50

for kk in xrange(it):
y[1,1] = 4
y[1,2] = 4
y[1,0] = 4
y[2,1] = 6

print "Iteration #:%s" %(kk)
y = where((y == 4) | (y == 6), 2, y)
best,
fw

Jun 21 '06 #4
I've been in contact with Travis O, and he said it was fixed in the
SVN.
thanks for the suggestions, I'll try them out now.

best
Sonja
Filip Wasilewski wrote:
sonjaa wrote:
Hi

last week I posted a problem with running out of memory when changing
values in NumPy arrays. Since then I have tried many different
approaches and
work-arounds but to no avail.

[...]

Based on the numpy-discussion this seems to be fixed in the SVN now(?).

Anyway, you can use 'where' function to eliminate the loops:

from numpy import *

y = ones((501,501))
z = zeros((501,501))
it = 50

for kk in xrange(it):
y[1,1] = 4
y[1,2] = 4
y[1,0] = 4
y[2,1] = 6

print "Iteration #:%s" %(kk)
y = where((y == 4) | (y == 6), 2, y)
best,
fw


Jun 21 '06 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

7 posts views Thread by Ben Fairbank | last post: by
20 posts views Thread by mclaugb | last post: by
13 posts views Thread by sonjaa | last post: by
reply views Thread by greg.novak | last post: by
11 posts views Thread by Achim Domma | last post: by
2 posts views Thread by sapsi | last post: by
4 posts views Thread by Robert LaMarca | last post: by
reply views Thread by Robert Kern | last post: by
reply views Thread by gheharukoh7 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.