Python memory deallocate

[mariano.difelice]

Hi,
I've a big memory problem with my application.

First, an example:

If I write:

a = range(500*1024)

I see that python process allocate approximately 80Mb of memory.
What can i do for DEALLOCATE this memory, or good part of this?

My really problem is that my program work with more photos, which I
open with PIL package.
When I start it, my program allocate approximately 200Mb memory.

If I want abort actual work and restart without restarting python
process, the memory usage will go up approximately 380-400 Mb.

I would like find something that DEallocate the memory not used.

I've tried with gc python module, but don't work fine (it deallocate
approximately 20-30 Mb)
I've tried with Destroy, del command, but the memory don't show down.

Thx

I'm very desperate

[Tim N. van der Leeuw]

Hi,

'del a' should remove 'a', as a reference to the tuple created by the
'range' function.
If that is also the last reference, it can now be garbage-collected.

Of course, the question is if you really need to allocate such a big
amount of memory. If you just need to iterate over some numbers, it's
better to use 'xrange' instead of 'range': 'xrange' will not create the
whole list of numbers in advance, but will create an iterator,
producing the desired numbers one by one. With such large ranges, this
will reduce memory consumption significantly.

Cheers,

--Tim

[Eric.zhao.82]

O
I see

[Diez B. Roggisch]

ma************* *@gmail.com wrote:

Hi,
I've a big memory problem with my application.

First, an example:

If I write:

a = range(500*1024)

I see that python process allocate approximately 80Mb of memory.
What can i do for DEALLOCATE this memory, or good part of this?

My really problem is that my program work with more photos, which I
open with PIL package.
When I start it, my program allocate approximately 200Mb memory.

If I want abort actual work and restart without restarting python
process, the memory usage will go up approximately 380-400 Mb.

I would like find something that DEallocate the memory not used.

I've tried with gc python module, but don't work fine (it deallocate
approximately 20-30 Mb)
I've tried with Destroy, del command, but the memory don't show down.

Thx

I'm very desperate

No need to. Just because the process seems to be that large doesn't mean
that there is so much memory consumed by python. It's just that modern OSs
will not correctly display the amount of memory really consumed, as they
aren't too eager to remove once allocated virtual memory a process has been
granted. I guess it's an optimization scheme. For example if I do

a = range(500*1024 * 50)

the VmSize of the python process grows to 500MB. If I issue an
del a
it shrinks to 300MB. The reissuing
a = range(500*1024 * 50)
will make it grow - but only to about 400MB. Then I do
del a
a = [chr(i % 255) for i in xrange(500*1024 * 50)]

This time I used strings, to circumvene some internal caching python might
do on numbers. Yet still - the overall VmSize is 400MB

Bottomline: don't get confused by tsakmanagers memory size display. You only
have to be concerned when the consumption steadily grows - which indicates
a memory leak.

Diez

[mariano.difelice]

Well, it's right about range diff xrange, ok, but this was a simple
example...
I repeat that my program don't work with range or xrange...

I MUST find a system which deallocate memory...
Otherwise, my application crashes not hardly it's arrived to
break-point system

[Sion Arrowsmith]

<ma************ **@gmail.com> wrote:

If I write:

a = range(500*1024)

I see that python process allocate approximately 80Mb of memory.
What can i do for DEALLOCATE this memory, or good part of this?
[ ... ]
I've tried with Destroy, del command, but the memory don't show down.

It won't (much). When an object gets garbage collected Python
will keep hold of the memory and reuse it. Note how much memory
your process is using after assigning a above, then:

del a
Of course, you've seen this doesn't release back to the OS all
the memory being that was being used. But now do:
a = range(500*1024)

and you'll see that you're using no more memory than you were
after the first assignment. If your memory usage keeps on growing
then either (a) your program needs that much memory for the data,
and you'll just have to stick more in your box or deal with
swapping if this is causing you a problem or (b) you've got some
stray references left over to objects you think you've deleted.

--
\S -- si***@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

[mariano.difelice]

Ok, this is true.

Well, you consider that my app has a first windows, where I choose, for
example, the application 1.
The application 1 will be started, and it will allocate 200Mb total.
Now I want to abort this operation, and i will return to main initial
window. The memory usage remain to 200Mb, even if I've destroyed the
application 1 object with app1.Destroy() and del app1.
When from initial windows i re-choose application1, the memory usage
don't stay to 200Mb, but will increase to 360-380Mb memory, and I think
this is not good!!!

If I will repeat this procedure for 5-6 times, the application will
crash...

And it's not good!!!

[Heiko Wundram]

Am Donnerstag 11 Mai 2006 15:15 schrieb ma************* *@gmail.com:

I MUST find a system which deallocate memory...
Otherwise, my application crashes not hardly it's arrived to
break-point system

As was said before: as long as you keep a reference to an object, the object's
storage _will not be_ reused by Python for any other objects (which is
sensible, or would you like your object to be overwritten by other objects
before you're done with them?). Besides, even if Python did free the memory
that was used, the operating system wouldn't pick it up (in the general case)
anyway (because of fragmentation issues), so Python keeping the memory in an
internal free-list for new objects is a sensible choice the Python developers
took here.

Basically, what I think is happening is that you are loading all images that
you use for your program into memory at once, and these simply will eat up
your memory until your program crashes (because Python can't know you no
longer need them). As you keep a reference to the image (somewhere,
someplace), del(eting) the reference in one scope isn't going to free the
memory for Python (or PIL in this case) to reuse for the next image. An
object is kept alive because of a memory leak inherent to your application.

This is a programming problem, not a Python problem. And, if you don't post
any sources, we won't be able to help you much here, save to tell you to look
closely where you create objects, and where you store them to.

PIL isn't known to have any memory leaks, by the way (AFAICT), just to confirm
what I've written before, but the effbot should be of more help here...

--- Heiko.

[Serge Orlov]

Heiko Wundram wrote:

Am Donnerstag 11 Mai 2006 15:15 schrieb ma************* *@gmail.com:

I MUST find a system which deallocate memory...
Otherwise, my application crashes not hardly it's arrived to
break-point system

As was said before: as long as you keep a reference to an object, the object's
storage _will not be_ reused by Python for any other objects (which is
sensible, or would you like your object to be overwritten by other objects
before you're done with them?). Besides, even if Python did free the memory
that was used, the operating system wouldn't pick it up (in the general case)
anyway (because of fragmentation issues), so Python keeping the memory in an
internal free-list for new objects is a sensible choice the Python developers
took here.

BTW python 2.5 now returns free memory to OS, but if a program keeps
allocating more memory with each new iteration in python 2.4, it will
not help.

[bruno at modulix]

ma************* *@gmail.com wrote:

Ok, this is true.

Well, you consider that my app has a first windows, where I choose, for
example, the application 1.
The application 1 will be started, and it will allocate 200Mb total.
Now I want to abort this operation, and i will return to main initial
window. The memory usage remain to 200Mb, even if I've destroyed the
application 1 object with app1.Destroy() and del app1.
When from initial windows i re-choose application1, the memory usage
don't stay to 200Mb, but will increase to 360-380Mb memory, and I think
this is not good!!!

If I will repeat this procedure for 5-6 times, the application will
crash...

Either there are references somewhere keeping some objects alive, or
you're facing a known problem with pymalloc() [1]. In the second case,
the good news is that this problem seems to be solved in Python 2.5 [2].
The bad news is that Python 2.5 is still alpha...

Just a question: do you really need your app to be monolithic ? If
you're application is really composed of many applications (which is
what I understand from your example), you should probably have many
distinct applications, one of them being in charge of running the
others. This way, 'aborting' an app would kill the corresponding
process, and free memory.
[1] or it's yet another problem, of course - CS wouldn't be that fun if
it was so simple :(

[2] http://docs.python.org/dev/whatsnew/section-other.html

My 2 cents

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom. gro'.split('@')])"