473,231 Members | 1,503 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,231 software developers and data experts.

python gc performance in large apps


Hey guys (thus begins a book of a post :),

I'm in the process of writing a commercial VoIP call monitoring and
recording application suite in python and pyrex. Basically, this
software sits in a VoIP callcenter-type environment (complete with agent
phones and VoIP servers), sniffs voice data off of the network, and
allows users to listen into calls. It can record calls as well. The
project is about a year and 3 months in the making and lately the
codebase has stabilized enough to where it can be used by some of our
clients. The entire project has about 37,000 lines of python and pyrex
code (along with 1-2K lines of unrelated java code).

Now, some disjointed rambling about the architecture of this software.
This software has two long-running server-type components. One
component, the "director" application, is written in pure python and
makes use of the twisted, nevow, and kinterbasdb libraries (which I
realize link to some C extensions). The other component, the
"harvester", is a mixture of python and pyrex, and makes use of the
twisted library, along with using the C libs libpcap and glib on the
pyrex end. Basically, the director is the "master" component. A single
director process interacts with users of the system through a web and/or
pygtk client application interface and can coordinate 1 to n harvesters
spread about the world. The harvester is the "heavy lifter" component
that sniffs the network traffic and sifts out the voice and signalling
data. It then updates the director of call status changes, and can
provide users of the system access to the data. It records the data to
disk as well. The scalibility of this thing is really cool: given a
single director sitting somewhere coordinating the list of agents,
multiple harvester can be placed anywhere there is voice traffic. A user
that logs into the director can end up seeing the activity of all of
these seperate voice networks presented like a single giant mesh.

Overall, I have been very pleased with python and the 3rd party
libraries that I use (twisted, nevow, kinterbasdb and pygtk). It is a
joy to program with, and I think the python community has done a fine
job. However, as I have been running the software lately and profiling
its memory usage, the one and only Big Problem I have seen is that of
the memory usage. Ideally, the server application(s) should be able to
run indefinitely, but from the results I'm seeing I will end up
exhausting the memory on a 2 GB machine in 2 to 3 days of heavy load.

Now normally I would not raise up an issue like this on this list, but
based on the conversations held on this list lately, and the work done
by Evan Jones (http://evanjones.ca/python-memory.html), I am led to
believe that this memory usage -- while partially due to some probably
leaks in my program -- is largely due to the current python gc. I have
some graphs I made to show the extent of this memory usage growth:

http://public.robbyd.fastmail.fm/iq-graph1.gif

http://public.robbyd.fastmail.fm/iq-...rector-rss.gif

http://public.robbyd.fastmail.fm/iq-graph-harv-rss.gif

The preceding three diagrams are the result of running the 1 director
process and 1 harvester process on the same machine for about 48 hours.
This is the most basic configuration of this software. I was running
this application through /usr/bin/python (CPython) on a Debian 'testing'
box running Linux 2.4 with 2GB of memory and Python version 2.3.5.
During that time, I gathered the resident and virtual memory size of
each component at 120 second intervals. I then imported this data into
MINITAB and did some plots. The first one is a graph of the resident
(RSS) and virtual memory usage of the two applications. The second one
is a zoomed in graph of the director's resident memory usage (complete
with a best fit quadratic), and the 3rd one is a zoomed in graph of the
harvester's resident memory usage.

To give you an idea of the network load these apps were undergoing
during this sampling time, by the time 48 hours had passed, the
harvester had gathered and parsed about 900 million packets. During the
day there will be 50-70 agents talking. This number goes to 10-30 at night.

In the diagrams above, one can see the night-day separation clearly. At
night, the memory usage growth seemed to all but stop, but with the
increased call volume of the day, it started shooting off again. When I
first started gathering this data, I was hoping for a logarithmic curve,
but at least after 48 hours, it looks like the usage increase is almost
linear. (Although logarithmic may still be the case after it exceeds a
gig or two of used memory. :) I'm not sure if this is something that I
should expect from the current gc, and when it would stop.

Now, as I stated above, I am certain that at least some of this
increased memory usage is due to either un-collectable objects in the
python code, or memory leaks in the pyrex code (where I make some use of
malloc/free). I am working on finding and removing these issues, but
from what I've seen with the help of gc UNCOLLECTABLE traces, there are
not many un-collectable reference issues at least. Yes, there are some
but definitely not enough to justify growth like I am seeing. The pyrex
side should not be leaking too much, I'm very good about freeing what I
allocate in pyrex/C land. I will be running that linked to a memory leak
finding library in the next few days. Past the code reviews I've done,
what makes me think that I don't have any *wild* leaks going on at least
with the pyrex code is that I am seeing the same type of growth patterns
in both apps, and I don't use any pyrex with the director. Yes, the
harvester is consuming much more memory, but it also does the majority
of the heavy lifting.

I am alright with the app not freeing all the memory it can between high
and low activity times, but what puzzles me is how the memory usage just
keeps on growing and growing. Will it ever stop?

What I would like to know if others on this list have had similar
problems with python's gc in long running, larger python applications.
Am I crazy or is this a real problem with python's gc itself? If it's a
python gc issue, then it's my opinion that we will need to enhance the
gc before python can really gain leverage as a language suitable for
"enterprise-class" applications. I have surprised many other programmers
that I'm writing an application like this in python/pyrex that works
just as well and even more efficiently than the C/C++/Java competitors.
The only thing I have left to show is that the app lasts as long between
restarts. ;)
Robby
Oct 21 '05 #1
0 1550

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Michael L. Labbe | last post by:
Hello. I'm an experienced programmer who is evaluating learning Python if it is applicable to a few projects. The programs I am going to list are production software - they are not throwaway toy...
8
by: Sridhar R | last post by:
Hi, I am a little experienced python programmer (2 months). I am somewhat experienced in C/C++. I am planning (now in design stage) to write an IDE in python. The IDE will not be a simple...
9
by: limor | last post by:
Hi, I am considering using Python in a new testing tool application we intend to build for out product. I must get references before starting develope in this language , since although lots of...
7
by: Ixokai | last post by:
Hello all. :) I've been a long time Python fan, and have fairly recently (with the support of a coworker who surprised me by mentioning my pet language one day) convinced my company to begin the...
81
by: julio | last post by:
Sorry but there is no another way, c# .net and mono are going to rip python, not because python is a bad lenguage, but because is to darn old and it refuses to innovate things, to fix wrong things,...
9
by: F. GEIGER | last post by:
I've dev'ed a Python prototype of an app, that besides the internals making it up has a gui. While test-driven dev'ing the app's internals in Python is fun as usual, dev'ing the GUI is not so...
114
by: Maurice LING | last post by:
This may be a dumb thing to ask, but besides the penalty for dynamic typing, is there any other real reasons that Python is slower than Java? maurice
113
by: John Nagle | last post by:
The major complaint I have about Python is that the packages which connect it to other software components all seem to have serious problems. As long as you don't need to talk to anything outside...
34
by: Victor Kryukov | last post by:
Hello list, our team is going to rewrite our existing web-site, which has a lot of dynamic content and was quickly prototyped some time ago. Today, as we get better idea of what we need, we're...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.