468,736 Members | 1,780 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,736 developers. It's quick & easy.

Program slowing down with greater memory use


I have two different python programs that are slowing down quite a bit as
their memory use goes up.

I'm not really sure if this is some sort of CPU cache effect, or if it's
something about python's garbage collector taking more time, or what.

One of the programs is one of those "how fast is data moving through the
pipe" measurement tools, called reblock. I'd initially expected that the
program would run faster with large block sizes, because then you're
wasting less time with in the C library, system call interface and context
switches.

But that turned out to be false. If I use a blocksize of about 2**18, it
runs much faster than if I use a blocksize of 2**22.

Then another program, which is a search engine, will slow way down if I
try to run it against too many search keywords. The program inhales a
bunch of filenames at the beginning, and if I run it against n files, n>m,
and later run it against m files, then the n file run will be quite a bit
slower even after it has handled only m files. The program implements a
memory cache to speed database operations, but even when the size of my
cache is nowhere near the size of the system's physical memory, it still
slows way down in this enigmatic way.

This makes me wonder.

What's the deal here? Is the garbage collector working overtime on these
programs that have a lot of objects? One of the programs above (reblock)
doesn't precisely have a large number of objects - just one huge string.

Thanks!

Jun 22 '06 #1
3 1612
In <pa****************************@dcs.nac.uci.edu> , Dan Stromberg wrote:
What's the deal here?


Very hard to tell without more details.

Ciao,
Marc 'BlackJack' Rintsch
Jun 23 '06 #2
Dan Stromberg:
What's the deal here?


The sketchy information in your post doesn't rule out any possibility.

--
René Pijlman
Jun 23 '06 #3
Dan Stromberg <st******@dcs.nac.uci.edu> writes on Thu, 22 Jun 2006 23:36:00 GMT:
I have two different python programs that are slowing down quite a bit as
their memory use goes up.


I have seen this too with Zope.

I do not know where it comes from -- maybe from degraded locality.
Dieter
Jun 25 '06 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

16 posts views Thread by TheKeith | last post: by
3 posts views Thread by Daragoth | last post: by
5 posts views Thread by Droopy | last post: by
5 posts views Thread by tony | last post: by
334 posts views Thread by Antoninus Twink | last post: by
Banfa
2 posts views Thread by Banfa | last post: by
reply views Thread by =?Utf-8?B?UnVkeSBNb29yZQ==?= | last post: by
xarzu
2 posts views Thread by xarzu | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.