473,320 Members | 2,020 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Re: How do you debug memory usage?

On May 6, 6:27 am, David <wizza...@gmail.comwrote:
Hi list.
What is the best way to debug memory usage in a Python script?
...
Are there any tools/modules/etc I can use like this?
David.
You need to use the debug build of Python to get exact numbers,
but there are a few tricks you can use with the standard build
to detect memory leaks. The simplest thing is to simply watch the
RSS column output of `ps aux` (I'm assuming you are using UNIX).

The other trick I got from Chris Siebenmann
http://utcc.utoronto.ca/~cks/space/b.../GetAllObjects
I modified his example a little bit. This does not tell you how
many bytes of memory your running code is using, but it will
show you the number of objects. When looking for memory leaks,
counting the number of objects is sufficient to detect leaks.
For example, say you suspect a function is leaking memory.
You could call it in a loop like this and watch the count of
objects before and after each call.

while True:
print "Number objects before:", len(get_all_objects())
suspect_function()
print "Number objects after:", len(get_all_objects())

Here is my modified version of Chris' get_all_objects() function.
All I did was force garbage collection using gc.collect().
This makes sure that you are not counting objects that Python has
left in memory, but plans on deleting at some point.

import gc
# Recursively expand slist's objects
# into olist, using seen to track
# already processed objects.
def _getr(slist, olist, seen):
for e in slist:
if id(e) in seen:
continue
seen[id(e)] = None
olist.append(e)
tl = gc.get_referents(e)
if tl:
_getr(tl, olist, seen)
# The public function.
def get_all_objects():
"""Return a list of all live Python objects, not including the
list itself."""
gc.collect()
gcl = gc.get_objects()
olist = []
seen = {}
# Just in case:
seen[id(gcl)] = None
seen[id(olist)] = None
seen[id(seen)] = None
# _getr does the real work.
_getr(gcl, olist, seen)
return olist

--
Noah
Jun 27 '08 #1
3 2376
>
Here is my modified version of Chris' get_all_objects() function.
All I did was force garbage collection using gc.collect().
This makes sure that you are not counting objects that Python has
left in memory, but plans on deleting at some point.
Thanks for the logic.

I want to debug rdiff-backup (Python backup tool for Linux) - it's
using 2 GB of memory (1GB ram, 1GB swap) on a backup server at work.

I'll use your method to find out why this is happening. Even if it
doesn't give me exact details, it should be enough info to go by.

David
Jun 27 '08 #2
On May 6, 2:19 pm, David <wizza...@gmail.comwrote:
I want to debug rdiff-backup (Python backup tool for Linux) - it's
using 2 GB of memory (1GB ram, 1GB swap) on a backup server at work.
...
David
Rsync uses a lot of memory:
http://www.samba.org/rsync/FAQ.html#4
rdiff-backup uses librsync, not rsync.
I'm not sure if rsync uses librsync, but one could speculate that
they share some code and perhaps some of the same problems.
But 2GB seems excessive unless you are dealing with millions of files.
A memory leak seems more likely.
--
Noah
Jun 27 '08 #3
Rsync uses a lot of memory:
http://www.samba.org/rsync/FAQ.html#4
rdiff-backup uses librsync, not rsync.
I'm not sure if rsync uses librsync, but one could speculate that
they share some code and perhaps some of the same problems.
But 2GB seems excessive unless you are dealing with millions of files.
A memory leak seems more likely.
Thanks for your input. In case you're interested here's a link to my
post to a local Linux user group:

http://lists.clug.org.za/pipermail/c...ay/040532.html

David.
Jun 27 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Justice | last post by:
Currently I'm doing some experimenting with the XMLHTTP object in Javascript. Now, the XMLHttp object is asynchronous (at least in this case), and the following code causes a significant memory...
2
by: tomvr | last post by:
Hello I have noticed some 'weird' memory usage in a vb.net windows app The situation is as follows I have an app (heavy on images) with 2 forms (actually there are more forms and on starting...
6
by: Tom | last post by:
We have a VERY simple .NET C# Form Application, that has about a 23MB Memory Footprint. It starts a window runs a process and does a regular expression. I have done a GC.Collect to make sure that,...
3
by: Ian Taite | last post by:
Hello, I'm exploring why one of my C# .NET apps has "high" memory usage, and whether I can reduce the memory usage. I have an app that wakes up and processes text files into a database...
5
by: greg.merideth | last post by:
I'm working on a service for a project that is producing some bizarre behavior. In debug mode while the service is running, the memory usage of the service (watching with process explorer) goes...
2
by: kevinding | last post by:
Hi All, We meet an evil condition for our project. Our project has 3 layers. A C# layer to do some business logic, and Managed C++ layer translate managed values to native ones or vice...
1
by: Daniel | last post by:
i seem to be having memory issues, will deplying a release build instead of a debug build help with memory usage of my .net process?
1
by: Jean-Paul Calderone | last post by:
On Tue, 22 Apr 2008 14:54:37 -0700 (PDT), yzghan@gmail.com wrote: The test doesn't demonstrate any leaks. It does demonstrate that memory usage can remain at or near peak memory usage even after...
0
by: David | last post by:
On Tue, May 6, 2008 at 4:21 PM, Banibrata Dutta <banibrata.dutta@gmail.comwrote: Thanks for your post. I didn't think of Googling for "python memory profiling". That search returns a lot of...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.