Detecting memory leaks on apache, mod_python

[Ilias Lazaridis]

How to detect memory leaks of python programms, which run in an
environment like this:

* Suse Linux 9.3
* Apache
* mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

context:

http://dev.lazaridis.com/base/ticket/148

[Ilias Lazaridis]

On Dec 17, 8:41 am, Ilias Lazaridis <[e-mail]:

How to detect memory leaks of python programms, which run in an
environment like this:

* Suse Linux 9.3
* Apache
* mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

No tool available to detect memory leaks for python applications?

context:

http://dev.lazaridis.com/base/ticket/148

[Ilias Lazaridis]

On Dec 19, 5:40 am, Ilias Lazaridis <[e-mail]:

On Dec 17, 8:41 am, Ilias Lazaridis <[e-mail]:

How to detect memory leaks of python programms, which run in an
environment like this:

* Suse Linux 9.3
* Apache
* mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

No tool available to detect memory leaks for python applications?

So, anyone who hit's on this thread via a search will think

a) that there's really no memory leak detection for python
b) that this community is not very helpful

>

context:

http://dev.lazaridis.com/base/ticket/148

[Graham Dumpleton]

On Dec 21, 7:42*pm, Ilias Lazaridis <[e-mail]:

On Dec 19, 5:40 am, Ilias Lazaridis <[e-mail]:

On Dec 17, 8:41 am, Ilias Lazaridis <[e-mail]:

How to detect memory leaks of python programms, which run in an
environment like this:

** Suse Linux 9.3
** Apache
**mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

No tool available to detect memory leaks for python applications?

So, anyone who hit's on this thread via a search will think

a) that there's really no memory leak detection for python
b) that this community is not very helpful

Comments like (b) will not help your chances one bit in respect of
getting an answer from anyone.

Maybe you should read:

http://www.catb.org/~esr/faqs/smart-questions.html

Two things to note in here. First, choose your forum appropriately and
secondly show some courtesy rather than making accusations against the
community if no one answers.

If you want to see perhaps how you might be viewed by the open source
community when you make such a comment, perhaps also watch:

http://video.google.com/videoplay?do...11961522818645

Now, since you think this is a Trac problem, why don't you go ask on
the Trac mailing list.

http://groups.google.com/group/trac-users?lnk=li

Even a search of that forum will most likely yield previous
discussions about growing memory use of Trac as a result of things
like Python wrappers for Subversion or certain database adapters. It
may be a case of using a different version, or in some cases
configuration of your hosting environment, if using Apache, to recycle
Apache child processes after a set number of requests so as to restore
process sizes back to a low level.

So, do some research first in the correct places and then perhaps ask
any additional questions in the correct place also.

Graham

[Diez B. Roggisch]

>So, anyone who hit's on this thread via a search will think

>>
a) that there's really no memory leak detection for python
b) that this community is not very helpful

c) That finally people in this forum are smart enough to detect your
flamebait & refuse to comment on it, Ilias...

Comments like (b) will not help your chances one bit in respect of
getting an answer from anyone.

Maybe you should read:

http://www.catb.org/~esr/faqs/smart-questions.html

Two things to note in here. First, choose your forum appropriately and
secondly show some courtesy rather than making accusations against the
community if no one answers.

If you want to see perhaps how you might be viewed by the open source
community when you make such a comment, perhaps also watch:

http://video.google.com/videoplay?do...11961522818645

Now, since you think this is a Trac problem, why don't you go ask on
the Trac mailing list.

http://groups.google.com/group/trac-users?lnk=li

Even a search of that forum will most likely yield previous
discussions about growing memory use of Trac as a result of things
like Python wrappers for Subversion or certain database adapters. It
may be a case of using a different version, or in some cases
configuration of your hosting environment, if using Apache, to recycle
Apache child processes after a set number of requests so as to restore
process sizes back to a low level.

So, do some research first in the correct places and then perhaps ask
any additional questions in the correct place also.

Graham, I'm not sure how long you have been around here - but googling
for Ilias in general and specificly in this group will reveal that he
often posts spiteful & inflammatory messages and has a tendency to get
cross with open source projects. It's debatable if he can be called a
troll - but he surely is a nuisance best let alone, so he doesn't gain
traction.

Diez

[Ilias Lazaridis]

On Dec 21, 12:25 pm, Graham Dumpleton <[e-mail]>
wrote:

On Dec 21, 7:42 pm, Ilias Lazaridis <[e-mail]:

On Dec 19, 5:40 am, Ilias Lazaridis <[e-mail]:

On Dec 17, 8:41 am, Ilias Lazaridis <[e-mail]:
How to detect memory leaks of python programms, which run in an
environment like this:

* Suse Linux 9.3
* Apache
*mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

No tool available to detect memory leaks for python applications?

So, anyone who hit's on this thread via a search will think

a) that there's really no memory leak detection for python
b) that this community is not very helpful

Comments like (b) will not help your chances one bit in respect of
getting an answer from anyone.

[...]

I'm really just looking for a low-level python (memory) profiling
tool, and thus i'm asking in c.l.p. for experiences, mainly in context
of an apache mod_python environment.

As to Open Source etc.:

http://case.lazaridis.com/wiki/PythonAudit
http://case.lazaridis.com/wiki/CoreLiveEval

Than you for your reply.

..

[Ilias Lazaridis]

On Dec 21, 3:21 pm, "Diez B. Roggisch" <[e-mail]:
[...]

Please get serious, Mr.!

(and avoid further off-topics)

..

[Steven D'Aprano]

On Fri, 21 Dec 2007 14:21:27 +0100, Diez B. Roggisch wrote:

>>So, anyone who hit's on this thread via a search will think

a) that there's really no memory leak detection for python b) that
this community is not very helpful

c) That finally people in this forum are smart enough to detect your
flamebait & refuse to comment on it, Ilias...

Not me. I didn't (still don't actually) recognise Ilias' name. I was
actually thinking:

(d) Python is so great that there can never be any memory leaks!

but I guess that's probably wishful thinking.

So... how do you measure memory usage in Python? Every programming
language I've used before (not a huge range, I'll admit) had *some* sort
of facility to measure memory usage, typically things like:

* how much memory is free in the stack?

* how much memory is free in the heap?

* how big a block does this pointer point to?

* how much memory does this record/struct/object/string use?

Unless I've missed something, I don't believe Python does *any* of that.
It could be quite useful. The timeit module is good for measuring the
time taken for code to run, but as near as I can tell, there's no way to
optimize memory use except prematurely.

--
Steven.

[Fredrik Lundh]

Steven D'Aprano wrote:

Not me.

You're quite knew to this internet thing, aren't you? ;-)

So... how do you measure memory usage in Python? Every programming
language I've used before (not a huge range, I'll admit) had *some* sort
of facility to measure memory usage, typically things like:

* how much memory is free in the stack?

* how much memory is free in the heap?

* how big a block does this pointer point to?

* how much memory does this record/struct/object/string use?

And what languages would that be? I cannot think of a single modern
language that does any of that. Including low-level stuff like C/C++.

And things like "how much memory is free in the heap" isn't even a
meaningful concept on a modern machine, thanks to the wonders of virtual
memory (especially the overcommitting kind).

For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns. Just run the program under a few different
scenarios, and see what it does. If the memory use looks suspicious,
use standard debugging techniques to locate the problematic area, and
standard benchmarking techniques to look for unexpected blowups and leaks.

For really hard problems, use the "gc" module, instrumenting memory
allocation libraries (e.g. dmalloc), or C/C++-level debuggers.

</F>

[Steven D'Aprano]

On Sat, 22 Dec 2007 08:09:50 +0100, Fredrik Lundh wrote:

Steven D'Aprano wrote:

Not me.

You're quite knew to this internet thing, aren't you? ;-)

:-D

>So... how do you measure memory usage in Python? Every programming
language I've used before (not a huge range, I'll admit) had *some*
sort of facility to measure memory usage, typically things like:

* how much memory is free in the stack?

* how much memory is free in the heap?

* how big a block does this pointer point to?

* how much memory does this record/struct/object/string use?

And what languages would that be? I cannot think of a single modern
language that does any of that. Including low-level stuff like C/C++.

I didn't actually say they were *modern* languages. E.g. THINK Pascal for
Apple Mac, circa 1990.

And things like "how much memory is free in the heap" isn't even a
meaningful concept on a modern machine, thanks to the wonders of virtual
memory (especially the overcommitting kind).

Maybe the memory model used in modern multi-tasking virtual-memory PCs
makes the concept of "free memory" obsolete. Although if so, somebody
should have a quiet word with the author of the Linux free command:

$ free
total used free shared buffers cached
Mem: 1002524 988736 13788 0 7044 98916
-/+ buffers/cache: 882776 119748
Swap: 4241080 3939736 301344

(Admittedly that's system-wide memory usage, rather than for a single
process.)

For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns. Just run the program under a few different
scenarios, and see what it does.

Are you saying that Python programs can't monitor their own memory use?

I'm happy to accept that "free memory" is not a meaningful concept for a
process in a modern system. That makes sense. But surely it is reasonable
for a process to have an idea of how much memory it has actually used.
Yes? No?

If the memory use looks suspicious,
use standard debugging techniques to locate the problematic area, and
standard benchmarking techniques to look for unexpected blowups and
leaks.

What sort of "standard debugging techniques" work in the absence of any
way (that I know of) to measure memory usage? In Python, standard
debugging techniques usually start with the print statement, but one
can't do anything like this:

# problematic area of code
last = memory()
for i in xrange(100):
x = foo()
if memory() >= last:
print "memory use increased", memory()
So what are you suggesting is standard?

For really hard problems, use the "gc" module, instrumenting memory
allocation libraries (e.g. dmalloc), or C/C++-level debuggers.

I'm not so much concerned about the really hard problems as I am about
the really easy ones.

--
Steven

[Fredrik Lundh]

Steven D'Aprano wrote:

> Not me.

You're quite knew to this internet thing, aren't you? ;-)

:-D

(hmm. why is that whenever you make some silly last-second addition to a
post, you end up making a stupid typo?)

>And things like "how much memory is free in the heap" isn't even a
meaningful concept on a modern machine, thanks to the wonders of virtual
memory (especially the overcommitting kind).

Maybe the memory model used in modern multi-tasking virtual-memory PCs
makes the concept of "free memory" obsolete. Although if so, somebody
should have a quiet word with the author of the Linux free command:

$ free
total used free shared buffers cached
Mem: 1002524 988736 13788 0 7044 98916
-/+ buffers/cache: 882776 119748
Swap: 4241080 3939736 301344

(Admittedly that's system-wide memory usage, rather than for a single
process.)

the problem isn't that you can get a number, it's that the number you
get has no *direct* correlation to the amount of memory your process can
actually allocate -- or at least allocate without causing excessive
swapping or triggering the OOM killer or otherwise annoying all the
other processes on the machine.

>For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns. Just run the program under a few different
scenarios, and see what it does.

Are you saying that Python programs can't monitor their own memory use?

I'm happy to accept that "free memory" is not a meaningful concept for a
process in a modern system. That makes sense. But surely it is reasonable
for a process to have an idea of how much memory it has actually used.
Yes? No?

unless you do utterly trivial things, a process allocates memory from a
lot of different resource pools (i/o buffers, virtual memory buffers,
network buffers, graphics resources, etc). there's simply no way for
the process itself to know how much memory it uses. if you want to
know, ask the operating system.

>If the memory use looks suspicious,
use standard debugging techniques to locate the problematic area, and
standard benchmarking techniques to look for unexpected blowups and
leaks.

What sort of "standard debugging techniques" work in the absence of any
way (that I know of) to measure memory usage?

as I said, use standard process monitoring tools (i.e. "ps" and "top"
etc on Unix, the task manager on Windows) takes you a long way.

In Python, standard
debugging techniques usually start with the print statement
but one can't do anything like this:

# problematic area of code
last = memory()
for i in xrange(100):
x = foo()
if memory() >= last:
print "memory use increased", memory()

it doesn't have to be harder than this:

# problematic area of code
raw_input("before: check memory here")
for i in xrange(100):
x = foo()
raw_input("after: check memory here")

on unix, you can also do e.g. os.system("ps ux") or os.system("ps -F -p
%d" % os.getpid()) or some variation thereof.

So what are you suggesting is standard?

running individual modules/classes/functions under test harnesses, so
you can analyze their behaviour under well-known conditions.

</F>

[Ilias Lazaridis]

On 22 Äåê, 09:09, Fredrik Lundh <[e-mail]:
[...]

For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns.

[...]

which "standard process monitoring tools" are those?

Aren't there any python specific tools (e.g. which list the python
modules which use the memory)?

.

http://case.lazaridis.com/wiki/PythonAudit

[Steven D'Aprano]

On Sat, 22 Dec 2007 13:05:23 -0800, Dennis Lee Bieber wrote:

I've never encountered such items
supported by the language.

OS specific extensions MIGHT supply it...

Picky picky... but of course you are right. When I said that programming
languages I have used before had facilities to measure memory usage, I
meant that the *implementation* had those facilities rather than the
language itself.
--
Steven

[Ilias Lazaridis]

On Dec 23, 2:47 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.auwrote:

On Sat, 22 Dec 2007 13:05:23 -0800, Dennis Lee Bieber wrote:

I've never encountered such items
supported by the language.

OS specific extensions MIGHT supply it...

Picky picky... but of course you are right. When I said that programming
languages I have used before had facilities to measure memory usage, I
meant that the *implementation* had those facilities rather than the
language itself.

yes, that's what this thread is about.

"
I'm really just looking for a low-level python (memory) profiling
tool, and thus i'm asking in c.l.p. for experiences, mainly in context
of an apache mod_python environment.
"

But seeing the responses within this thread, it looks like there's no
such tool available.

..

http://case.lazaridis.com/wiki/PythonAudit