473,322 Members | 1,403 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

memory allocation

Hey,

I'm trying to debug the memory allocation in an embedded use of the
Python interpreter.

The longer I leave my program running the more memory it consumes. The
total number of objects in the system is not increasing (many are being
allocated and deallocated).

Using mtrace I have established that the only memory which is not being
freed is that which is allocated at Objects/obmalloc.c:429. It appears to
be allocating new arenas in proportion to it's running time.

I don't have an in-depth understanding of Python's object allocator. Does
anybody have any ideas as to what the problem may be?

Thanks,
Laurie

Jul 18 '05 #1
6 3818
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the
program. If memory is freed, the program still occupies the amount of
memory occupied by these arenas. If allocations can be fulfilled without
creating new arenas, then no additional memory will be required. Hence as
time goes by the memory required by the program increases (assuming
increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?

On Sun, 19 Sep 2004 21:21:21 +1000, be********@optusnet.com.au wrote:
Hey,

I'm trying to debug the memory allocation in an embedded use of the
Python interpreter.

The longer I leave my program running the more memory it consumes. The
total number of objects in the system is not increasing (many are being
allocated and deallocated).

Using mtrace I have established that the only memory which is not being
freed is that which is allocated at Objects/obmalloc.c:429. It appears to
be allocating new arenas in proportion to it's running time.

I don't have an in-depth understanding of Python's object allocator. Does
anybody have any ideas as to what the problem may be?

Thanks,
Laurie


Jul 18 '05 #2

<be********@optusnet.com.au> wrote in message
news:pa****************************@optusnet.com.a u...
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the


This is all correct, but you're probably attacking the problem from the hard
end. Perhaps
some C code has forgotten to DECREF a dead object. See section 1.10 of

http://docs.python.org/ext/ext.html

David Pokorny

Jul 18 '05 #3
On Sun, 19 Sep 2004 18:02:08 -0700, David Pokorny wrote:
<be********@optusnet.com.au> wrote in message
news:pa****************************@optusnet.com.a u...
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the


This is all correct, but you're probably attacking the problem from the hard
end. Perhaps
some C code has forgotten to DECREF a dead object. See section 1.10 of

http://docs.python.org/ext/ext.html


Hey,

Thanks for the response.
As established earlier I have determined that this is not the case. Is it
possible to change the memory allocation scheme of Python so that I can
confirm this?

Thanks,
Laurie
Jul 18 '05 #4
[be********@optusnet.com.au]
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the
program.
I'm not sure what that means, but probably yes.
If memory is freed, the program still occupies the amount of
memory occupied by these arenas.
It's true that pymalloc never returns an arena to the system free().
It's also true, e.g., that space allocated for Python integers never
leaves a special internal free list for integers until Python shuts
down, and pymalloc plays no part in that.
If allocations can be fulfilled without creating new arenas, then no additional
memory will be required.
Not all parts of Python use pymalloc. If you're talking only about
the parts that do use pymalloc, then yes.
Hence as time goes by the memory required by the program increases
(assuming increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?
Possibly. I personally don't expect simplistic behavior from programs
that use only malloc and free, assuming that's what "an ordinary
program" intends to mean here. The relationship between the platform
malloc/free and "memory required by the program" is usually very
complex. Throwing pymalloc into the mix too makes it even more
complex. "memory required by the program" is ambiguous on its own on
machines with virtual memory, due to distinguishing between RAM in use
and paged-out virtual address space that's never referenced again.

[... later ...]
Is it possible to change the memory allocation scheme of Python so that I
can confirm this?


You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.
Jul 18 '05 #5
Hey Tim,

Thanks for the response.

On Mon, 20 Sep 2004 01:35:31 -0400, Tim Peters wrote:

[..]
Hence as time goes by the memory required by the program increases
(assuming increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?
Possibly. I personally don't expect simplistic behavior from programs
that use only malloc and free, assuming that's what "an ordinary
program" intends to mean here. The relationship between the platform
malloc/free and "memory required by the program" is usually very
complex. Throwing pymalloc into the mix too makes it even more
complex. "memory required by the program" is ambiguous on its own on
machines with virtual memory, due to distinguishing between RAM in use
and paged-out virtual address space that's never referenced again.


For this discussion I will refer to the virtual memory occupied by the
program.
[... later ...]
Is it possible to change the memory allocation scheme of Python so that I
can confirm this?


You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.


I have built with this option, but to no avail. I can provide a sample
c program (which has it's virtual memory size expand and contract
dynamically), and a similar python program for which this does not
occur.

Jul 18 '05 #6
[Tim Peters]
You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.

[be********@optusnet.com.au] I have built with this option, but to no avail.
Then the Python you built was not using pymalloc. I suppose you could
confirm that by setting breakpoints in pymalloc and noting that
they're never hit (or noting that the debugger won't let you set
breakpoints there anymore, because the pymalloc code no longer
exists!).
I can provide a sample c program (which has it's virtual memory size expand
and contract dynamically), and a similar python program for which this does
not occur.


If you're not using pymalloc anymore, then you're trying to out-guess
the behavior of your specific platform C's malloc/free system, and how
it interacts with your specific platform's operating system. There's
no guarantee that "returning" memory to the platform free() has any
effect on "the memory used by the program" as reported by the OS.
Indeed, if you have turned pymalloc off, Python itself is just another
C program using malloc and free. I'm not interested in staring at
your platform C+OS, but I'll note that it's common as mud, across many
platforms, to return memory to free() and see no effect on reported VM
size. As I said before, the behavior here is typically very complex,
depending on exact traces of when malloc() and free() are called, on
the exact arguments passed to them, and on myriad details of how
malloc/free are implemented on your platform. It's not actually
interesting to find a C program where "it works", the interesting bit
is finding C programs where "it doesn't work", because the latter show
the platform's weaknesses.

A common pattern, extended to exhaustion to make a point:

malloc(big_number)
malloc(little_number)
malloc(big_number)
malloc(little_number)
malloc(big_number)
malloc(little_number)
...
repeat until you run out of VM

This can fragment address space so badly that returning *all* the
memory allocated by the "big_number" calls nevertheless leaves
allocated little pieces scattered across the entire virtual address
space. So long as the little pieces remain allocated, there's nothing
that even a fiercely determined malloc/free could do to return any
part of the address space to the OS. In real life, malloc/free
typically don't try hard at all to return space to the OS, burning
just a few cycles after a free() to see whether it's obviously
possible to do. That's easily frustrated.

Depending on your platform C library, you may or may not be able to
ask its malloc to show you a map of allocated and available regions.
If you can, you'll probably find a fragmented address space. Figuring
out "why" it's fragmented is another big job, and then it may or may
not be possible to rearrange computations to avoid the cause(s)
discovered.

There's rarely an easy answer to one of these.
Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: chris | last post by:
Hi all, I need to know, what is the difference between dynamic memory allocation, and stack allocation ? 1. If I have a class named DestinationAddress, when should I use dynamic memory...
4
by: PaulR | last post by:
Hi, We have a Server running SLES 8 and 3GB memory, with 1 DB2 instance and 2 active Databases. General info... DB2level = "DB2 v8.1.0.72", "s040914", "MI00086", and FixPak "7" uname -a =...
74
by: ballpointpenthief | last post by:
If I have malloc()'ed a pointer and want to read from it as if it were an array, I need to know that I won't be reading past the last index. If this is a pointer to a pointer, a common technique...
62
by: ivan.leben | last post by:
How can I really delete a preloaded image from memory/disk cache? Let's say I preload an image by creating an Image object and setting its src attribute to desired URL: var img = new Image();...
66
by: Johan Tibell | last post by:
I've written a piece of code that uses sockets a lot (I know that sockets aren't portable C, this is not a question about sockets per se). Much of my code ended up looking like this: if...
24
by: Ken | last post by:
In C programming, I want to know in what situations we should use static memory allocation instead of dynamic memory allocation. My understanding is that static memory allocation like using array...
1
by: Peterwkc | last post by:
Hello all expert, i have two program which make me desperate bu after i have noticed the forum, my future is become brightness back. By the way, my problem is like this i the first program was...
34
by: jacob navia | last post by:
Suppose that you have a module that always allocates memory without ever releasing it because the guy that wrote it was lazy, as lazy as me. Now, you want to reuse it in a loop. What do you do?...
14
by: vivek | last post by:
i have some doubts on dynamic memory allocation and stacks and heaps where is the dynamic memory allocation used? in function calls there are some counters like "i" in the below function. Is...
66
by: karthikbalaguru | last post by:
Hi, Will 'free' return the memory Immediately to the OS ? Thx in advans, Karthik Balaguru
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.