473,714 Members | 2,180 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

memory allocation

Hey,

I'm trying to debug the memory allocation in an embedded use of the
Python interpreter.

The longer I leave my program running the more memory it consumes. The
total number of objects in the system is not increasing (many are being
allocated and deallocated).

Using mtrace I have established that the only memory which is not being
freed is that which is allocated at Objects/obmalloc.c:429. It appears to
be allocating new arenas in proportion to it's running time.

I don't have an in-depth understanding of Python's object allocator. Does
anybody have any ideas as to what the problem may be?

Thanks,
Laurie

Jul 18 '05 #1
6 3844
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the
program. If memory is freed, the program still occupies the amount of
memory occupied by these arenas. If allocations can be fulfilled without
creating new arenas, then no additional memory will be required. Hence as
time goes by the memory required by the program increases (assuming
increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?

On Sun, 19 Sep 2004 21:21:21 +1000, be********@optu snet.com.au wrote:
Hey,

I'm trying to debug the memory allocation in an embedded use of the
Python interpreter.

The longer I leave my program running the more memory it consumes. The
total number of objects in the system is not increasing (many are being
allocated and deallocated).

Using mtrace I have established that the only memory which is not being
freed is that which is allocated at Objects/obmalloc.c:429. It appears to
be allocating new arenas in proportion to it's running time.

I don't have an in-depth understanding of Python's object allocator. Does
anybody have any ideas as to what the problem may be?

Thanks,
Laurie


Jul 18 '05 #2

<be********@opt usnet.com.au> wrote in message
news:pa******** *************** *****@optusnet. com.au...
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the


This is all correct, but you're probably attacking the problem from the hard
end. Perhaps
some C code has forgotten to DECREF a dead object. See section 1.10 of

http://docs.python.org/ext/ext.html

David Pokorny

Jul 18 '05 #3
On Sun, 19 Sep 2004 18:02:08 -0700, David Pokorny wrote:
<be********@opt usnet.com.au> wrote in message
news:pa******** *************** *****@optusnet. com.au...
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the


This is all correct, but you're probably attacking the problem from the hard
end. Perhaps
some C code has forgotten to DECREF a dead object. See section 1.10 of

http://docs.python.org/ext/ext.html


Hey,

Thanks for the response.
As established earlier I have determined that this is not the case. Is it
possible to change the memory allocation scheme of Python so that I can
confirm this?

Thanks,
Laurie
Jul 18 '05 #4
[be********@optu snet.com.au]
My latest understanding is that the default memory allocation will result
in arenas being created to fulfill the maximum memory requirements of the
program.
I'm not sure what that means, but probably yes.
If memory is freed, the program still occupies the amount of
memory occupied by these arenas.
It's true that pymalloc never returns an arena to the system free().
It's also true, e.g., that space allocated for Python integers never
leaves a special internal free list for integers until Python shuts
down, and pymalloc plays no part in that.
If allocations can be fulfilled without creating new arenas, then no additional
memory will be required.
Not all parts of Python use pymalloc. If you're talking only about
the parts that do use pymalloc, then yes.
Hence as time goes by the memory required by the program increases
(assuming increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?
Possibly. I personally don't expect simplistic behavior from programs
that use only malloc and free, assuming that's what "an ordinary
program" intends to mean here. The relationship between the platform
malloc/free and "memory required by the program" is usually very
complex. Throwing pymalloc into the mix too makes it even more
complex. "memory required by the program" is ambiguous on its own on
machines with virtual memory, due to distinguishing between RAM in use
and paged-out virtual address space that's never referenced again.

[... later ...]
Is it possible to change the memory allocation scheme of Python so that I
can confirm this?


You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.
Jul 18 '05 #5
Hey Tim,

Thanks for the response.

On Mon, 20 Sep 2004 01:35:31 -0400, Tim Peters wrote:

[..]
Hence as time goes by the memory required by the program increases
(assuming increasing memory requirements of the application), and it will not
dynamically expand and contract with the memory allocations/deallocations
as you would expect with an ordinary program. Is my understanding of this
correct?
Possibly. I personally don't expect simplistic behavior from programs
that use only malloc and free, assuming that's what "an ordinary
program" intends to mean here. The relationship between the platform
malloc/free and "memory required by the program" is usually very
complex. Throwing pymalloc into the mix too makes it even more
complex. "memory required by the program" is ambiguous on its own on
machines with virtual memory, due to distinguishing between RAM in use
and paged-out virtual address space that's never referenced again.


For this discussion I will refer to the virtual memory occupied by the
program.
[... later ...]
Is it possible to change the memory allocation scheme of Python so that I
can confirm this?


You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.


I have built with this option, but to no avail. I can provide a sample
c program (which has it's virtual memory size expand and contract
dynamically), and a similar python program for which this does not
occur.

Jul 18 '05 #6
[Tim Peters]
You can build Python without pymalloc. I've never done that myself,
so am not sure how to do it. Could be that passing --with-pymalloc=no
would suffice.

[be********@optu snet.com.au] I have built with this option, but to no avail.
Then the Python you built was not using pymalloc. I suppose you could
confirm that by setting breakpoints in pymalloc and noting that
they're never hit (or noting that the debugger won't let you set
breakpoints there anymore, because the pymalloc code no longer
exists!).
I can provide a sample c program (which has it's virtual memory size expand
and contract dynamically), and a similar python program for which this does
not occur.


If you're not using pymalloc anymore, then you're trying to out-guess
the behavior of your specific platform C's malloc/free system, and how
it interacts with your specific platform's operating system. There's
no guarantee that "returning" memory to the platform free() has any
effect on "the memory used by the program" as reported by the OS.
Indeed, if you have turned pymalloc off, Python itself is just another
C program using malloc and free. I'm not interested in staring at
your platform C+OS, but I'll note that it's common as mud, across many
platforms, to return memory to free() and see no effect on reported VM
size. As I said before, the behavior here is typically very complex,
depending on exact traces of when malloc() and free() are called, on
the exact arguments passed to them, and on myriad details of how
malloc/free are implemented on your platform. It's not actually
interesting to find a C program where "it works", the interesting bit
is finding C programs where "it doesn't work", because the latter show
the platform's weaknesses.

A common pattern, extended to exhaustion to make a point:

malloc(big_numb er)
malloc(little_n umber)
malloc(big_numb er)
malloc(little_n umber)
malloc(big_numb er)
malloc(little_n umber)
...
repeat until you run out of VM

This can fragment address space so badly that returning *all* the
memory allocated by the "big_number " calls nevertheless leaves
allocated little pieces scattered across the entire virtual address
space. So long as the little pieces remain allocated, there's nothing
that even a fiercely determined malloc/free could do to return any
part of the address space to the OS. In real life, malloc/free
typically don't try hard at all to return space to the OS, burning
just a few cycles after a free() to see whether it's obviously
possible to do. That's easily frustrated.

Depending on your platform C library, you may or may not be able to
ask its malloc to show you a map of allocated and available regions.
If you can, you'll probably find a fragmented address space. Figuring
out "why" it's fragmented is another big job, and then it may or may
not be possible to rearrange computations to avoid the cause(s)
discovered.

There's rarely an easy answer to one of these.
Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
8209
by: chris | last post by:
Hi all, I need to know, what is the difference between dynamic memory allocation, and stack allocation ? 1. If I have a class named DestinationAddress, when should I use dynamic memory allocation to create object of that class ? 2. If it says "dynamic memory allocation", is it mean the following code : DestinationAddress* dest = new DestinationAddress(); // code 1
4
6768
by: PaulR | last post by:
Hi, We have a Server running SLES 8 and 3GB memory, with 1 DB2 instance and 2 active Databases. General info... DB2level = "DB2 v8.1.0.72", "s040914", "MI00086", and FixPak "7" uname -a = Linux galahad 2.4.19-64GB-SMP #1 SMP /etc/sysctl.conf kernel.shmmax=268435456
74
4666
by: ballpointpenthief | last post by:
If I have malloc()'ed a pointer and want to read from it as if it were an array, I need to know that I won't be reading past the last index. If this is a pointer to a pointer, a common technique seems to be setting a NULL pointer to the end of the list, and here we know that the allocated memory has been exhausted. All good. When this is a pointer to another type, say int, I could have a variable that records how much memory is being...
62
17811
by: ivan.leben | last post by:
How can I really delete a preloaded image from memory/disk cache? Let's say I preload an image by creating an Image object and setting its src attribute to desired URL: var img = new Image(); img.src = ; Then I use the image a few more times by adding it into an Array object:
66
3619
by: Johan Tibell | last post by:
I've written a piece of code that uses sockets a lot (I know that sockets aren't portable C, this is not a question about sockets per se). Much of my code ended up looking like this: if (function(socket, args) == -1) { perror("function"); exit(EXIT_FAILURE); } I feel that the ifs destroy the readability of my code. Would it be
24
19083
by: Ken | last post by:
In C programming, I want to know in what situations we should use static memory allocation instead of dynamic memory allocation. My understanding is that static memory allocation like using array is faster than malloc, but dynamic memory allocation is more flexible. Please comment... thanks.
1
7967
by: Peterwkc | last post by:
Hello all expert, i have two program which make me desperate bu after i have noticed the forum, my future is become brightness back. By the way, my problem is like this i the first program was compiled and run without any erros but the second program has a run time error when the function return from allocate and the ptr become NULL. How to fixed this? Second Program: /* Best Method to allocate memory for 2D Array because it's ...
34
2572
by: jacob navia | last post by:
Suppose that you have a module that always allocates memory without ever releasing it because the guy that wrote it was lazy, as lazy as me. Now, you want to reuse it in a loop. What do you do? Contrary to some people that will start crying to that &@@""#~ programmer that wrote this sh!!!! you keep your cool and you do the following:
14
3825
by: vivek | last post by:
i have some doubts on dynamic memory allocation and stacks and heaps where is the dynamic memory allocation used? in function calls there are some counters like "i" in the below function. Is this stored in stack. If yes whether it will be deleted on exiting from the function. is dynamic memory allocation needed for this purpose
66
3688
by: karthikbalaguru | last post by:
Hi, Will 'free' return the memory Immediately to the OS ? Thx in advans, Karthik Balaguru
0
8713
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9187
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9080
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9033
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6638
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5961
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4467
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4730
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2529
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.