473,572 Members | 3,269 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

best way to discover this process's current memory usage, cross-platform?

Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn't properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed. Being in a hurry, I originally used a q&d hack...:
if sys.platform in ('linux2', 'darwin'):
def _memsize():
""" this function tries to return a measurement of how much memory
this process is consuming, in some arbitrary unit (if it doesn't
manage to, it returns 0).
"""
gc.collect()
try:
x = int(os.popen('p s -p %d -o vsz|tail -1' % os.getpid()).re ad())
except:
x = 0
return x
else:
def _memsize():
return 0

Having a _memsize() function available, the test then does:
before = _memsize()
# a lot of repeated executions of code that should not consume
# any net memory, but used to when the leak was there
after = _memsize()
and checks that after==before.

However, that _memsize is just too much of a hack, and I really want to
clean it up. It's also not cross-platform enough. Besides, I got a bug
report from a user on a Linux platform different from those I had tested
myself, and it boils down to the fact that once in a while on his
machine it turns our that after is before+4 (for any large number of
repetitions of the code in the above comment) -- I'm not sure what the
unit of measure is supposed to be (maybe blocks of 512 byte, with a page
size of 2048? whatever...), but clearly an extra page is getting used
somewhere.

So, I thought I'd turn to the "wisdom of crowds"... how would YOU guys
go about adding to your automated regression tests one that checks that
a certain memory leak has not recurred, as cross-platform as feasible?
In particular, how would you code _memsize() "cross-platformly"? (I can
easily use C rather than Python if needed, adding it as an auxiliary
function for testing purposes to my existing extension).
TIA,

Alex
Nov 22 '05 #1
35 5480
Not sure if I should start a new thread or not, but
since this is closely related, I'll just leave it as is.

Alex Martelli wrote:
Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn't properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed.


I would like to investigate how much memory is used by
Python objects. My motive is 98% pure intellectual
curiosity and 2% optimization.

I wonder whether I can do something like this:

obj = something()
bytes_used = sizeof(obj)

(obviously there is no built-in function sizeof...
wait, let me check... nope, not a built-in)

I've read the docs for gc and pdb and nothing stands
out to me as doing anything like this.
--
Steven.

Nov 22 '05 #2
Steven D'Aprano <st***@REMOVEME cyber.com.au> wrote:
Not sure if I should start a new thread or not, but
since this is closely related, I'll just leave it as is.

Alex Martelli wrote:
Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn't properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed.


I would like to investigate how much memory is used by
Python objects. My motive is 98% pure intellectual
curiosity and 2% optimization.


I believe that's the purpose of the PySizer project (one of the "Google
Summer of Code" projects), which was recently announced on this group
(I'm sure any search engine will be able to direct you to it, anyway).

I have not checked it out, because my purpose is different -- mine is
not a Python-related leak at all, just a leak within C code (which
happens coincidentally to be a Python extension module).
Alex
Nov 22 '05 #3
Alex Martelli wrote:

So, I thought I'd turn to the "wisdom of crowds"... how would YOU guys
go about adding to your automated regression tests one that checks that
a certain memory leak has not recurred, as cross-platform as feasible?
In particular, how would you code _memsize() "cross-platformly"? (I can
easily use C rather than Python if needed, adding it as an auxiliary
function for testing purposes to my existing extension).


If you are doing Unix, can you use getrusage(2)?
import resource
r = resource.getrus age(resource.RU SAGE_SELF)
print r[2:5]


I get zeroes on my gentoo amd64 box. Not sure why. I thought maybe it
was Python, but C gives the same results.

Another possibiity is to call sbrk(0) which should return the top of
the heap. You could then return this value and check it. It requires
a tiny C module, but should be easy and work on most unixes. You can
determine direction heap grows by comparing it with id(0) which should
have been allocated early in the interpreters life.

I realize this isn't perfect as memory becomes fragmented, but might
work. Since 2.3 and beyond use pymalloc, fragmentation may not be much
of an issue. As memory is allocated in a big hunk, then doled out as
necessary.

These techniques could apply to Windows with some caveats. If you are
interested in Windows, see:
http://msdn.microsoft.com/library/de...l/UCMGch09.asp

Can't think of anything fool-proof though.

HTH,
n

Nov 22 '05 #4
My suggestion would also be to use sbrk() as it provides a high-water
mark for the memory usage of the process.

Below is the function hiwm() I used on Linux (RedHat). MacOS X and
Unix versions are straigthforward . Not sure about Windows.

/Jean Brouwers

#if _LINUX
#include <malloc.h>

size_t hiwm (void) {
/* info.arena - number of bytes allocated
* info.hblkhd - size of the mmap'ed space
* info.uordblks - number of bytes used (?)
*/
struct mallinfo info = mallinfo();
size_t s = (size_t) info.arena + (size_t) info.hblkhd;
return (s);
}

#elif _MAXOSX || _UNIX
#include <unistd.h>

size_t hiwm (void) {
size_t s = (size_t) sbrk(0);
return (s);
}

#elif _WINDOWS
size_t hiwm (void) {
size_t s = (size_t) 0; /* ??? */
return (s);
}

#endif

Nov 22 '05 #5
Neal Norwitz <nn******@gmail .com> wrote:
Alex Martelli wrote:

So, I thought I'd turn to the "wisdom of crowds"... how would YOU guys
go about adding to your automated regression tests one that checks that
a certain memory leak has not recurred, as cross-platform as feasible?
In particular, how would you code _memsize() "cross-platformly"? (I can
easily use C rather than Python if needed, adding it as an auxiliary
function for testing purposes to my existing extension).
If you are doing Unix, can you use getrusage(2)?


On Unix, I could; on Linux, nope. According to man getrusage on Linux,

"""
The above struct was taken from BSD 4.3 Reno. Not all fields are
meaningful under Linux. Right now (Linux 2.4, 2.6) only the
fields ru_utime, ru_stime, ru_minflt, ru_majflt, and ru_nswap are
maintained.
"""
and indeed the memory-usage parts are zero.

import resource
r = resource.getrus age(resource.RU SAGE_SELF)
print r[2:5]
I get zeroes on my gentoo amd64 box. Not sure why. I thought maybe it
was Python, but C gives the same results.


Yep -- at least, on Linux, this misbehavior is clearly documented in the
manpage; on Darwin, aka MacOSX, you _also_ get zeros but there is no
indication in the manpage leading you to expect that.

Unfortunately I don't have any "real Unix" box around -- only Linux and
Darwin... I could try booting up OpenBSD again to check that it works
there, but given that I know it doesn't work under the most widespread
unixoid systems, it wouldn't be much use anyway, sigh.

Another possibiity is to call sbrk(0) which should return the top of
the heap. You could then return this value and check it. It requires
a tiny C module, but should be easy and work on most unixes. You can
As I said, I'm looking for leaks in a C-coded module, so it's no problem
to add some auxiliary C code to that module to help test it --
unfortunately, this approach doesn't work, see below...
determine direction heap grows by comparing it with id(0) which should
have been allocated early in the interpreters life.

I realize this isn't perfect as memory becomes fragmented, but might
work. Since 2.3 and beyond use pymalloc, fragmentation may not be much
of an issue. As memory is allocated in a big hunk, then doled out as
necessary.
But exactly because of that, sbrk(0) doesn't mean much. Consider the
tiny extension which I've just uploaded to
http://www.aleax.it/Python/memtry.c -- it essentially exposes a type
that does malloc when constructed and free when freed, and a function
sbrk0 which returns sbrk(0). What I see on my MacOSX 10.4, Python
2.4.1, gcc 4.1, is (with a little auxiliary memi.py module that does
from memtry import *
import os
def memsiz():
return int(os.popen('p s -p %d -o vsz|tail -1' % os.getpid()).re ad())
)...:

Helen:~/memtry alex$ python -ic 'import memi'
memi.memsiz() 35824 memi.sbrk0() 16809984 a=memi.mem(9999 99)
memi.sbrk0() 16809984 memi.memsiz()

40900

See? While the process's memory size grows as expected (by 500+ "units"
when allocating one meg, confirming the hypothesis that a unit is
2Kbyte), sbrk(0) just doesn't budge.

As the MacOSX "man sbrk" says,
"""
The brk and sbrk functions are historical curiosities left over from
earlier days before the advent of virtual memory management.
"""
and apparently it's now quite hard to make any USE of those quaint
oddities, in presence of any attempt, anywhere in any library linked
with the process, to do some "smart" memory allocation &c.

These techniques could apply to Windows with some caveats. If you are
interested in Windows, see:
http://msdn.microsoft.com/library/de...n-us/dnucmg/ht
ml/UCMGch09.asp

Can't think of anything fool-proof though.


Fool-proof is way beyond what I'm looking for now -- I'd settle for
"reasonably clean, works in Linux, Mac and Windows over 90% of the time,
and I can detect somehow when it isn't working";-)
Since people DO need to keep an eye on their code's memory consumption,
I'm getting convinced that the major functional lack in today's Python
standard library is some minimal set of tools to help with that task.
PySizer appears to be a start in the right direction (although it may be
at too early a stage to make sense for the standard library of Python
2.5), but (unless I'm missing something about it) it won't help with
memory leaks not directly related to Python. Maybe we SHOULD have some
function in sys to return the best guess at current memory consumption
of the whole process, implemented by appropriate techniques on each
platform -- right now, though, I'm trying to find out which these
appropriate techniques are on today's most widespread unixoid systems,
Linux and MacOSX. (As I used to be a Win32 API guru in a previous life,
I'm confident that I can find out about _that_ platform by sweating
enough blood on MSDN -- problem here is I don't have any Windows machine
with the appropriate development system to build Python, so testing
would be pretty hard, but maybe I can interest somebody who DOES have
such a setup...;-)
Alex
Nov 22 '05 #6
MrJean1 <Mr*****@gmail. com> wrote:
My suggestion would also be to use sbrk() as it provides a high-water
mark for the memory usage of the process.
That's definitely what I would have used in the '70s -- nowadays, alas,
it ain't that easy.
Below is the function hiwm() I used on Linux (RedHat). MacOS X and
Unix versions are straigthforward . Not sure about Windows.
The MacOSX version using sbrk is indeed straightforward , it just doesn't
work. See my response to Neal's post and my little Python extension
module at http://www.aleax.it/Python/memtry.c -- on a Mac (OSX 10.4,
Python 2.4.1, gcc 4.1) sbrk(0) returns the same value as the process's
virtual memory consumption goes up and down (as revealed by ps). As the
MacOSX's manpage says, "The brk and sbrk functions are historical
curiosities left over from earlier days before the advent of virtual
memory management."

Guess I'll now try the linux version you suggest, with mallinfo:
#if _LINUX
#include <malloc.h>

size_t hiwm (void) {
/* info.arena - number of bytes allocated
* info.hblkhd - size of the mmap'ed space
* info.uordblks - number of bytes used (?)
*/
struct mallinfo info = mallinfo();
size_t s = (size_t) info.arena + (size_t) info.hblkhd;
return (s);
}


and see if and how it works.

I do wonder why both Linux and MacOSX "implemente d" getrusage, which
would be the obviously right way to do it, as such a useless empty husk
(as far as memory consumption is concerned). Ah well!-(
Alex
Nov 22 '05 #7
For some more details on Linux' mallinfo, see
<ftp://gee.cs.oswego.e du/pub/misc/malloc.h> and maybe function mSTATs()
in glibc/malloc/malloc.c (RedHat).

/Jean Brouwers

Nov 22 '05 #8
For some more details on Linux' mallinfo, see
<ftp://gee.cs.oswego.e du/pub/misc/malloc.h> and maybe function mSTATs()
in glibc/malloc/malloc.c (RedHat).

/Jean Brouwers

Nov 22 '05 #9
Perhaps you could extend Valgrind (http://www.valgrind.org) so it works
with python C extensions? (x86 only)

matt

Nov 22 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
546
by: Ronen | last post by:
Hi All, I am running a client against a web application installed on a remote server written in C# which involve infragistics for GUI . The .NET framework I use is from version 1.1.4322 and we use the following Infragistics components: * WebSchedule Component Version: 1.0.20033.6 * UltraWebTree and UltraWebMenu components Version:...
2
8650
by: w2jin | last post by:
Hi All Some my SQL Server are experience high memory usage. 1. How can I detect which process which process cause the big memory usage and not released? 2. Which sql server components in this memory, and what are their usage distribution?
0
2613
by: Dirk Reske | last post by:
Hello, how can I get the memory and cpu usage of a Process? I get the process infos with Process proc = Process.GetProcessByID(...) which value is the Memory usage, I get in the windows Task manager? and how do I get the cpu usage of this process like in the win task manager? thx
3
3350
by: MattC | last post by:
I have an ASP.NET app that is running at around 60MB for the w3wp.exe. This often jumps very quickly to 100/120 MB before shrinking again. It is also using around 250MB of virtual memory. Is it normal for the memory usage to jump around so much? Also the server is a PIII 730 with 256MB RAM, running Windows 2003 Server. I would...
2
2182
by: Sidharth | last post by:
Hello, We are experiencing some issues with the aspnet_wp process on our live servers. The problems are intermittent and we cannot reproduce it on our dev and test servers. Currently around twice a day the aspnet_wp process takes 100% of the cpu and causes the whole website to come to a stand still. We then have to end the process for...
22
2998
by: Zen | last post by:
Hi, My production machine has 2G of memory, when aspnet_wp.exe goes up to about ~1.2G of memory usage, I start get out-of-memory exception. Other processes don't use as much memory and I added all the peak memory usage of all the processes (including aspnet_wp.exe), it goes up to no more than 1.5. How is that possible? Would anyone know...
4
2038
by: =?Utf-8?B?VzFsZDBuZTc0?= | last post by:
When one architects a new project one of the first steps in the decision is to decide on the layers. (In my opinion anyway) One architecture that I have used before is to go solid OO and create objects, which normally are very small and only deals with the stuff pertaining to that object, then break it down into Business Process, Process...
6
7440
by: Terry Carroll | last post by:
I am trying to do something with a very large tarfile from within Python, and am running into memory constraints. The tarfile in question is a 4-gigabyte datafile from freedb.org, http://ftp.freedb.org/pub/freedb/ , and has about 2.5 million members in it. Here's a simple toy program that just goes through and counts the number of members...
1
9518
by: bizt | last post by:
Hi, I have a process where every so often I refresh a table's data using a setTimeout() .. and when its no longer needed, a clearTimeout(). The following is a simple example of how this is done: var goMenuTimeout = null; function LoadMenu() { // cancel any repeat actions
3
3163
by: Salim Fadhley | last post by:
I'm trying to create a python unit-test which will test a complex program which includes a number of functions which have been implemented in C or C++. The unit-test needs to check that after the functions have been run a few thousand times all of the memory used by those functions has been un-allocated, i.e. that there are no memory leaks....
0
7733
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7640
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8155
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
8000
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5524
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3685
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3673
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1240
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
975
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.