memory allocation

[benevilent]

Hey,

I'm trying to debug the memory allocation in an embedded use of the
Python interpreter.

The longer I leave my program running the more memory it consumes. The
total number of objects in the system is not increasing (many are being
allocated and deallocated).

Using mtrace I have established that the only memory which is not being
freed is that which is allocated at Objects/obmalloc.c:429. It appears to
be allocating new arenas in proportion to it's running time.

I don't have an in-depth understanding of Python's object allocator. Does
anybody have any ideas as to what the problem may be?

Thanks,
Laurie

[benevilent]

My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the
program. If memory is freed, the program still occupies the amount of
memory occupied by these arenas. If allocations can be fulfilled without
creating new arenas, then no additional memory will be required. Hence as
time goes by the memory required by the program increases (assuming
increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?

On Sun, 19 Sep 2004 21:21:21 +1000, be********@optu snet.com.au wrote:

Hey,

I'm trying to debug the memory allocation in an embedded use of the
Python interpreter.

The longer I leave my program running the more memory it consumes. The
total number of objects in the system is not increasing (many are being
allocated and deallocated).

Using mtrace I have established that the only memory which is not being
freed is that which is allocated at Objects/obmalloc.c:429. It appears to
be allocating new arenas in proportion to it's running time.

I don't have an in-depth understanding of Python's object allocator. Does
anybody have any ideas as to what the problem may be?

Thanks,
Laurie

[David Pokorny]

<be********@opt usnet.com.au> wrote in message
news:pa******** *************** *****@optusnet. com.au...

My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the

This is all correct, but you're probably attacking the problem from the hard
end. Perhaps
some C code has forgotten to DECREF a dead object. See section 1.10 of

http://docs.python.org/ext/ext.html

David Pokorny

[benevilent]

On Sun, 19 Sep 2004 18:02:08 -0700, David Pokorny wrote:

<be********@opt usnet.com.au> wrote in message
news:pa******** *************** *****@optusnet. com.au...

My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the

This is all correct, but you're probably attacking the problem from the hard
end. Perhaps
some C code has forgotten to DECREF a dead object. See section 1.10 of

http://docs.python.org/ext/ext.html

Hey,

Thanks for the response.
As established earlier I have determined that this is not the case. Is it
possible to change the memory allocation scheme of Python so that I can
confirm this?

Thanks,
Laurie

[Tim Peters]

[be********@optu snet.com.au]

My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the
program.
I'm not sure what that means, but probably yes.
If memory is freed, the program still occupies the amount of
memory occupied by these arenas.
It's true that pymalloc never returns an arena to the system free().
It's also true, e.g., that space allocated for Python integers never
leaves a special internal free list for integers until Python shuts
down, and pymalloc plays no part in that.
If allocations can be fulfilled without creating new arenas, then no additional
memory will be required.
Not all parts of Python use pymalloc. If you're talking only about
the parts that do use pymalloc, then yes.
Hence as time goes by the memory required by the program increases
(assuming increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?
Possibly. I personally don't expect simplistic behavior from programs
that use only malloc and free, assuming that's what "an ordinary
program" intends to mean here. The relationship between the platform
malloc/free and "memory required by the program" is usually very
complex. Throwing pymalloc into the mix too makes it even more
complex. "memory required by the program" is ambiguous on its own on
machines with virtual memory, due to distinguishing between RAM in use
and paged-out virtual address space that's never referenced again.

[... later ...]
Is it possible to change the memory allocation scheme of Python so that I
can confirm this?

You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.

[benevilent]

Hey Tim,

Thanks for the response.

On Mon, 20 Sep 2004 01:35:31 -0400, Tim Peters wrote:

[..]

Hence as time goes by the memory required by the program increases
(assuming increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?
Possibly. I personally don't expect simplistic behavior from programs
that use only malloc and free, assuming that's what "an ordinary
program" intends to mean here. The relationship between the platform
malloc/free and "memory required by the program" is usually very
complex. Throwing pymalloc into the mix too makes it even more
complex. "memory required by the program" is ambiguous on its own on
machines with virtual memory, due to distinguishing between RAM in use
and paged-out virtual address space that's never referenced again.

For this discussion I will refer to the virtual memory occupied by the
program.
[... later ...]

Is it possible to change the memory allocation scheme of Python so that I
can confirm this?

You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.

I have built with this option, but to no avail. I can provide a sample
c program (which has it's virtual memory size expand and contract
dynamically), and a similar python program for which this does not
occur.

[Tim Peters]

[Tim Peters]

You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.

[be********@optu snet.com.au] I have built with this option, but to no avail.
Then the Python you built was not using pymalloc. I suppose you could
confirm that by setting breakpoints in pymalloc and noting that
they're never hit (or noting that the debugger won't let you set
breakpoints there anymore, because the pymalloc code no longer
exists!).
I can provide a sample c program (which has it's virtual memory size expand
and contract dynamically), and a similar python program for which this does
not occur.

If you're not using pymalloc anymore, then you're trying to out-guess
the behavior of your specific platform C's malloc/free system, and how
it interacts with your specific platform's operating system. There's
no guarantee that "returning" memory to the platform free() has any
effect on "the memory used by the program" as reported by the OS.
Indeed, if you have turned pymalloc off, Python itself is just another
C program using malloc and free. I'm not interested in staring at
your platform C+OS, but I'll note that it's common as mud, across many
platforms, to return memory to free() and see no effect on reported VM
size. As I said before, the behavior here is typically very complex,
depending on exact traces of when malloc() and free() are called, on
the exact arguments passed to them, and on myriad details of how
malloc/free are implemented on your platform. It's not actually
interesting to find a C program where "it works", the interesting bit
is finding C programs where "it doesn't work", because the latter show
the platform's weaknesses.

A common pattern, extended to exhaustion to make a point:

malloc(big_numb er)
malloc(little_n umber)
malloc(big_numb er)
malloc(little_n umber)
malloc(big_numb er)
malloc(little_n umber)
...
repeat until you run out of VM

This can fragment address space so badly that returning *all* the
memory allocated by the "big_number " calls nevertheless leaves
allocated little pieces scattered across the entire virtual address
space. So long as the little pieces remain allocated, there's nothing
that even a fiercely determined malloc/free could do to return any
part of the address space to the OS. In real life, malloc/free
typically don't try hard at all to return space to the OS, burning
just a few cycles after a free() to see whether it's obviously
possible to do. That's easily frustrated.

Depending on your platform C library, you may or may not be able to
ask its malloc to show you a map of allocated and available regions.
If you can, you'll probably find a fragmented address space. Figuring
out "why" it's fragmented is another big job, and then it may or may
not be possible to rearrange computations to avoid the cause(s)
discovered.

There's rarely an easy answer to one of these.