473,326 Members | 2,048 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Memory Allocation?

Is it possible to determine how much memory is allocated by an arbitrary
Python object? There doesn't seem to be anything in the docs about this,
but considering that Python manages memory allocation, why would such a
module be more difficult to design than say, the GC?
Jul 18 '05 #1
9 2457
Hello Chris,
I am sure there are many inaccuracies in this story but hey you asked
instead of seeking your owns answers so....
In general you need not worry about memory allocation.
Too be more specific objects have a size and most of them are known (at
least to a wizard named Tim) , but it doesn't really matter because it
doesn't work like that in Python.
CPython interpreter( I have never read a lick of the source this all
from late nite memory ) just grabs a chunk of memory and uses it as it
sees fit . Jython uses Java's GC . and etc..
Now tell me do you really want to take out the garbage or look at it?
Python does it for you so you don't have too.
But don't let me stop you.
You can probably still find some way to do it.
It is open source after all.
M.E.Farmer

Jul 18 '05 #2
Chris S. wrote:
Is it possible to determine how much memory is allocated by an arbitrary
Python object? There doesn't seem to be anything in the docs about this,
but considering that Python manages memory allocation, why would such a
module be more difficult to design than say, the GC?


Why do you want it?

regards,
Gerrit.

--
In the councils of government, we must guard against the acquisition of
unwarranted influence, whether sought or unsought, by the
military-industrial complex. The potential for the disastrous rise of
misplaced power exists and will persist.
-Dwight David Eisenhower, January 17, 1961
Jul 18 '05 #3
In article <8GCNd.9880$ya6.9761@trndny01>,
"Chris S." <ch*****@NOSPAM.udel.edu> wrote:
Is it possible to determine how much memory is allocated by an arbitrary
Python object? There doesn't seem to be anything in the docs about this,
but considering that Python manages memory allocation, why would such a
module be more difficult to design than say, the GC?


Sorry, I didn't follow that - such a module as what?

Along with the kind of complicated internal implementation
details, you may need to consider the possibility that the
platform malloc() may reserve more than the allocated amount,
for its own bookkeeping but also for alignment. It isn't
a reliable guide by any means, but something like this might
be at least entertaining -

class A: ... def __init__(self, a):
... self.a = a
... d = map(id, map(A, [0]*32))
d.sort()
k = 0
for i in d:

... print i - k
... k = i
...

This depends on the fact that id(a) returns a's storage
address.

I get very different results from one platform to another,
and I'm not sure what they mean, but at a guess, I think
you will see a fairly small number, like 40 or 48, that
represents the immediate allocation for the object, and
then a lot of intervals three or four times larger that
represent all the memory allocated in the course of creating
it. It isn't clear that this is all still allocated -
malloc() doesn't necessarily reuse a freed block right
away, and in fact the most interesting thing about this
experiment is how different this part looks on different
platforms. Of course we're still a bit in the dark as
to how much memory is really allocated for overhead.

Donn Cave, do**@u.washington.edu
Jul 18 '05 #4
Gerrit Holl wrote:
Chris S. wrote:
Is it possible to determine how much memory is allocated by an arbitrary
Python object? There doesn't seem to be anything in the docs about this,
but considering that Python manages memory allocation, why would such a
module be more difficult to design than say, the GC?

Why do you want it?


It would seem desirable to know how the components of one's program
occupies memory.
Jul 18 '05 #5
M.E.Farmer wrote:
Hello Chris,
I am sure there are many inaccuracies in this story but hey you asked
instead of seeking your owns answers so....
In general you need not worry about memory allocation.
Too be more specific objects have a size and most of them are known (at
least to a wizard named Tim) , but it doesn't really matter because it
doesn't work like that in Python.
CPython interpreter( I have never read a lick of the source this all
from late nite memory ) just grabs a chunk of memory and uses it as it
sees fit . Jython uses Java's GC . and etc..
Now tell me do you really want to take out the garbage or look at it?
Python does it for you so you don't have too.


Using similar logic, we shouldn't need access to the Garbage Collector
or Profiler. After all, why would anyone need to know how fast their
program is running or whether or not their garbage has been collected.
Python takes care of it.
Jul 18 '05 #6
Donn Cave wrote:
In article <8GCNd.9880$ya6.9761@trndny01>,
"Chris S." <ch*****@NOSPAM.udel.edu> wrote:

Is it possible to determine how much memory is allocated by an arbitrary
Python object? There doesn't seem to be anything in the docs about this,
but considering that Python manages memory allocation, why would such a
module be more difficult to design than say, the GC?

Sorry, I didn't follow that - such a module as what?


GC == Garbage Collector (http://docs.python.org/lib/module-gc.html)
Along with the kind of complicated internal implementation
details, you may need to consider the possibility that the
platform malloc() may reserve more than the allocated amount,
for its own bookkeeping but also for alignment. It isn't
a reliable guide by any means, but something like this might
be at least entertaining -
>>>
>>> class A: ... def __init__(self, a):
... self.a = a
... >>> d = map(id, map(A, [0]*32))
>>> d.sort()
>>> k = 0
>>> for i in d:

... print i - k
... k = i
...

This depends on the fact that id(a) returns a's storage
address.

I get very different results from one platform to another,
and I'm not sure what they mean, but at a guess, I think
you will see a fairly small number, like 40 or 48, that
represents the immediate allocation for the object, and
then a lot of intervals three or four times larger that
represent all the memory allocated in the course of creating
it. It isn't clear that this is all still allocated -
malloc() doesn't necessarily reuse a freed block right
away, and in fact the most interesting thing about this
experiment is how different this part looks on different
platforms. Of course we're still a bit in the dark as
to how much memory is really allocated for overhead.

Donn Cave, do**@u.washington.edu


Are you referring to Python's general method of memory management? I was
under the impression that the ISO C specification for malloc() dictates
allocation of a fixed amount of memory. free(), not malloc(), handles
deallocation. Am I wrong? Does Python use a custom non-standard
implementation of malloc()?
Jul 18 '05 #7
On 2005-02-07, Chris S. <ch*****@NOSPAM.udel.edu> wrote:
Is it possible to determine how much memory is allocated by an arbitrary
Python object?


The hardest part about answering this question is figuring out what
you want to count as being allocated for a particular object. That
varies widely depending on why you need this information, which is why
there isn't a "getobsize" or similar call.

For example, consider a dict. The expression

x = {'alpha': 1, 'beta': 2}

allocates a dict object, a hash table to hold the values [1], and the
strings and integers [2]. So how much memory is used by x? The dict
object structure and hash table obviously belong to x. How about the
contents? Those can be shared. If you count them, the answer is
misleading because deleting that object won't free up that much
memory; but if you don't count them, then your answer isn't very
useful because the large part of the object is probably the contents.
Another possibility is to count how much memory would be released if
the object were to go away, but this requires inspecting the reference
count of every object that can be reached from the subject, and it
might still be wrong if there are cycles.

I expect that if you want to know the size, you can decide on the
semantics you want for your application. The answer might even depend
on your application, since only you know which parts of the object
really counts toward its size (e.g., the names of attributes on your
object probably don't count). But the answer won't be the same for my
application, so a generic "getobsize" doesn't help.

Once you know what you want, it's pretty easy to get an estimate.
You'll have to make some assumptions about Python internals and it
wouldn't be portable across versions. Write a function that dispatches
on the type of its argument. For simple objects, return their basic
size; for containers, return the sum of the basic size, aux storage
size, and sizes of the contents (call your function on each of them).
For any object, the basic size is type(x).__basicsize__. The size of
the auxilary storage depends on the object and probably on the length
of its contents; e.g., a list allocates 4 bytes per element, and a
dict 12 bytes per slot [3]. Many do overallocation, so you have to
account for that too.

The only thing the interpreter can really help with is to be able to
ask an object about how much auxiliary memory it allocated. Only
builtin types can do that, and only that type knows the answer. Having
that would save you from having to know, for example, that a dict
allocates 12 bytes per slot. Everything else doesn't need the
interpreter's support, and pure Python works just as well in your
module as in the standard library (that said, if you do write it, I'm
sure others might find it useful--you aren't the first one with a
desire for this kind of information).

(The above is about CPython. JPython probably has its own set of issues.)

Dima.

[1] Small dicts don't need this extra allocation. We can ignore that
for now.

[2] Assuming, for the moment, that this isn't a constant expression.
When it's a constant, the integers are preloaded as constants in
the code object. Right now, that's an unnecessary complication.

[3] Numbers for CPython 2.4.
Jul 18 '05 #8
Hi Chris

Have a look at
http://www.python.org/doc/2.3.4/what...-pymalloc.html

for a description of what is going on.

Basically malloc is used to grab a big chunk of memory, then
python objects use bits of it.

Rgds

Tim

Chris S. wrote:
Donn Cave wrote:
In article <8GCNd.9880$ya6.9761@trndny01>,
"Chris S." <ch*****@NOSPAM.udel.edu> wrote:

Is it possible to determine how much memory is allocated by an
arbitrary Python object? There doesn't seem to be anything in the
docs about this, but considering that Python manages memory
allocation, why would such a module be more difficult to design than
say, the GC?


Sorry, I didn't follow that - such a module as what?

GC == Garbage Collector (http://docs.python.org/lib/module-gc.html)
Along with the kind of complicated internal implementation
details, you may need to consider the possibility that the
platform malloc() may reserve more than the allocated amount,
for its own bookkeeping but also for alignment. It isn't
a reliable guide by any means, but something like this might
be at least entertaining -
>>> >>> class A:

... def __init__(self, a):
... self.a = a
... >>> d = map(id, map(A, [0]*32))
>>> d.sort()
>>> k = 0
>>> for i in d:

... print i - k
... k = i
...
This depends on the fact that id(a) returns a's storage
address.

I get very different results from one platform to another,
and I'm not sure what they mean, but at a guess, I think
you will see a fairly small number, like 40 or 48, that
represents the immediate allocation for the object, and
then a lot of intervals three or four times larger that
represent all the memory allocated in the course of creating
it. It isn't clear that this is all still allocated -
malloc() doesn't necessarily reuse a freed block right
away, and in fact the most interesting thing about this
experiment is how different this part looks on different
platforms. Of course we're still a bit in the dark as
to how much memory is really allocated for overhead.

Donn Cave, do**@u.washington.edu

Are you referring to Python's general method of memory management? I was
under the impression that the ISO C specification for malloc() dictates
allocation of a fixed amount of memory. free(), not malloc(), handles
deallocation. Am I wrong? Does Python use a custom non-standard
implementation of malloc()?

Jul 18 '05 #9
Chris S. wrote:
Using similar logic, we shouldn't need access to the Garbage Collector
or Profiler. After all, why would anyone need to know how fast their
program is running or whether or not their garbage has been collected.
Python takes care of it.

Exactly! Even though there are a few who just won't listen ;)
I answered earlier that the *way* CPython allocates and uses memory it
was kinda useless. Similar objects in CPython and Jython and .etc..
have different implementions and consequently have different memory
footprints. Do you just want to keep track of average memory? Or do you
want know exactly when an object is collected and the memory is freed?
If so, that is also one of those grey areas of Python, objects are
collected and memory is freed at differing times , depending on several
factors plus the phase of the moon.
Lots of devils and plenty of details. If I am wrong someone correct me
;)
If you search around you can find many pointers to memory effecient
coding style in Python( never add a string together, use a list and
append then join, etc.)
Stick to profiling it will serve you better. But do not let me stop you
from writing a memory profiler ;)
M.E.Farmer

Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: chris | last post by:
Hi all, I need to know, what is the difference between dynamic memory allocation, and stack allocation ? 1. If I have a class named DestinationAddress, when should I use dynamic memory...
4
by: PaulR | last post by:
Hi, We have a Server running SLES 8 and 3GB memory, with 1 DB2 instance and 2 active Databases. General info... DB2level = "DB2 v8.1.0.72", "s040914", "MI00086", and FixPak "7" uname -a =...
74
by: ballpointpenthief | last post by:
If I have malloc()'ed a pointer and want to read from it as if it were an array, I need to know that I won't be reading past the last index. If this is a pointer to a pointer, a common technique...
62
by: ivan.leben | last post by:
How can I really delete a preloaded image from memory/disk cache? Let's say I preload an image by creating an Image object and setting its src attribute to desired URL: var img = new Image();...
66
by: Johan Tibell | last post by:
I've written a piece of code that uses sockets a lot (I know that sockets aren't portable C, this is not a question about sockets per se). Much of my code ended up looking like this: if...
24
by: Ken | last post by:
In C programming, I want to know in what situations we should use static memory allocation instead of dynamic memory allocation. My understanding is that static memory allocation like using array...
1
by: Peterwkc | last post by:
Hello all expert, i have two program which make me desperate bu after i have noticed the forum, my future is become brightness back. By the way, my problem is like this i the first program was...
34
by: jacob navia | last post by:
Suppose that you have a module that always allocates memory without ever releasing it because the guy that wrote it was lazy, as lazy as me. Now, you want to reuse it in a loop. What do you do?...
14
by: vivek | last post by:
i have some doubts on dynamic memory allocation and stacks and heaps where is the dynamic memory allocation used? in function calls there are some counters like "i" in the below function. Is...
66
by: karthikbalaguru | last post by:
Hi, Will 'free' return the memory Immediately to the OS ? Thx in advans, Karthik Balaguru
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.