memory usage of a specific function

[Hermann Maier]

hi,

i need to find out the memory usage of a specific function that i use in
my program. this function does some recursive calculations and i want my
program to display the amount of memory the function used to calculate a
specific value.

thx

[Sverker Nilsson]

Hermann Maier wrote:

hi,

i need to find out the memory usage of a specific function that i use in
my program. this function does some recursive calculations and i want my
program to display the amount of memory the function used to calculate a
specific value.

thx

I was thinking that Heapy in Guppy-PE (http://guppy-pe.sourceforge.net)
could be of help, but I can not give any specific advice based on only
the information given.

Maybe you could be more specific about your requirements, and post
the actual function code or some other illustrating example.

Regards,

Sverker

[Stephen Kellett]

In message <11**********************@g44g2000cwa.googlegroups .com>,
Sverker Nilsson <sv********@home.se> writes

i need to find out the memory usage of a specific function that i use in
my program. this function does some recursive calculations and i want my
program to display the amount of memory the function used to calculate a
specific value.

Python Memory Validator.

Run your program to completion.
Switch to the hotspots tab.
Search for your function.
All memory used in that function will be shown in the tree (with the
effective callstack) underneath that function node in the tree.

http://www.softwareverify.com

Stephen
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting

[Sverker Nilsson]

Stephen Kellett wrote:

In message <[e-mail] .com>,

Sverker Nilsson <[e-mail]> writes

[Note that actually it was Hermann Maier that wrote the following
but as quoted, it may look like it was I that wrote it.]

i need to find out the memory usage of a specific function that i use in
my program. this function does some recursive calculations and i want my
program to display the amount of memory the function used to calculate a
specific value.
Python Memory Validator.

Run your program to completion.
Switch to the hotspots tab.
Search for your function.
All memory used in that function will be shown in the tree (with the
effective callstack) underneath that function node in the tree.

http://www.softwareverify.com
I can't try it out easily since it doesn't seem to support Linux. Also
looking at the home page I couldn't find any description of what it
was actually doing. The info links didn't work, they showed blank
pages with my browser (Netscape 3.01.)

So if you would like to explain here, it would be helpful.
Especially, it isn't clear to me what this means (requoting):
All memory used in that function

Since Python has a reference-counting allocator, and various
kinds of objects use specialized allocation pools, it seems
unclear and implementation defined what is meant with
'all memory used'.

I could think of some different things this could mean.

1. The total memory allocated, regardless of whether it is released
again. If it is to mean really all the memory, it should include
memory allocated by special allocators such as the allocator for
ints. In this case, the following example function

def f():
x = 0
while x < 1000000:
x += 1

would have allocated (at least) 1000000 objects,
one for each iteration since int's are immutable
and one is allocated each time x is assigned to
(although each one is also deallocated.)

So according to the meaning (1.), the
'total memory used' would be 1000000 * the size of ints.
[ Though with some alternative interpreter implementation
it might be an entirely different value such as 0 bytes since
it may not need to allocate every integer in the heap. Would
'allocating' on the stack count? In a CPU register?]

Is something like this what would be reported by your system?

2. The second alternative meaning would be the memory allocated in
the function, MINUS the memory deallocated in the function.

Is this what is reported by your system?

If the case is according to the 2nd alternative, 'memory used in' f
would
be 0 bytes since all memory allocated is deallocated, and for a
function like the following

def g():
return range(1000)

it would be the size allocated for the list of length 1000 + any int
objects allocated to fill the list. Given the above function g, its
memory usage could be tested using heapy and the following function.
def test(g):
from guppy import hpy
hp=hpy()
hp.setrelheap()
hp.setrelheap() # Has to be done twice after the first import (XXX)
x=g()
print hp.heap()

test(g) Partition of a set of 902 objects. Total size = 15364 bytes.
Index Count % Size % Cumulative % Kind (class / dict of
class)
0 900 100 10800 70 10800 70 int
1 1 0 4128 27 14928 97 list
2 1 0 436 3 15364 100 types.FrameType

[ Why it shows an object of type FrameType is something of a mystery,
I'm not sure of the best way to get rid of it.]

I guess I could add a function somewhat like test(g) to Heapy,
depending on what common use cases seem to require.

On the other hand, test for memory usage according to alternative (1)
would be harder to add, and it is somewhat undefined what it
means. And it is perhaps not so useful as the 2nd alternative?

Sverker

Project home page: http://guppy-pe.sourceforge.net

[Stephen Kellett]

In message <11**********************@g44g2000cwa.googlegroups .com>,
Sverker Nilsson <sv********@home.se> writes

Python Memory Validator.

Run your program to completion.
Switch to the hotspots tab.
Search for your function.
All memory used in that function will be shown in the tree (with the
effective callstack) underneath that function node in the tree.

http://www.softwareverify.com

I can't try it out easily since it doesn't seem to support Linux.

Correct. Windows NT and above only at this time.
Also
looking at the home page I couldn't find any description of what it
was actually doing. The info links didn't work, they showed blank
pages with my browser (Netscape 3.01.)
IE 6, Opera and Firefox. No idea with NS 3.0

Maybe this more direct link helps.

http://www.softwareverify.com/python...tor/index.html
So if you would like to explain here, it would be helpful.
Especially, it isn't clear to me what this means (requoting):

All memory used in that function

Exactly that. In a GC language (Python, Ruby, Java, Javascript) each
object needs to memory to exist. PMV tracks the creation/destruction of
objects (and in the case of Python, the other special non-object
datatypes such as integers that have their own pool).

Consider the following psuedo code

Func 1
allocates A
calls Func 2
calls Func 3
calls Func 4

Func 2
allocates B
allocates C

Func 3
allocates B
calls Func 5
allocates C

Func 4
allocates D
allocates E

Func 5
loop 5 times
allocates F
allocates G

This gives you the following tree showing all memory allocations

Func1
A
Func 2
B
C
Func 3
B
Func 5
5 x F
5 x G
C
Func 4
D
E

The tree also includes stats so that you know which node has more (in %
terms) data in it (in bytes) that another node. From that tree we can
see that the following objects are created.
A 1
B 2
C 2
D 1
E 1
F 5
G 5
memory usage could be tested using heapy and the following function.
You cannot reliably use native GC language code to monitor the same
application's memory usage. You need to do this externally (from C or
what takes your fancy - something that won't create/destroy objects on
the native language).
On the other hand, test for memory usage according to alternative (1)
would be harder to add, and it is somewhat undefined what it
means. And it is perhaps not so useful as the 2nd alternative?

PMV provides both. If you want stats on the number of objects created,
no problem. If you want callstacks for all objects created and stats for
the memory allocated at these locations, no problem. If you want stats
for each generation of objects (a generation being the group allocated
between GC invocations), no problem. Plus a whole load of analysis and
query functions to allow you to find information on particular object
types etc, plus data export functions (HTML and XML).

Stephen
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting