473,839 Members | 1,398 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Is this a bug? Python intermittently stops dead for seconds

Below is a simple program that will cause python to intermittently
stop executing for a few seconds. it's 100% reproducible on my machine.

I'd be tempted to say this is a nasty garbage collection performance
issue except that there is no major memory to be garbage collected in
this script. I'd be tempted to say it was a unix virtual memory
issue except this is occuring at around 1/5th of my physical memory
size. So something else unexplained is going on

Class Foo instances create and hold a list of size nfoo of integers.
(nfoo =10)

Class Bar instances create and hold a list of size nbar of Foo
objects. (nbar =100)

When the code runs it starts creating and storing Bar objects in a
list while watching for real-time glitches in how long it takes to
create the next Foo object. If this is longer than 1/2 of a second
then it reports it.

On my computer after creating 1500 Bar objects, the rate of
creation of new Foo suddenly has a periodic glitch. This glitch re-
occurs about every 300 Bar Objects, and the duration of the glitch
keeps getting longer--growing to many seconds!!!!

Platform: 800Mhz powermac g 4 1Gb of memory
python: python 2.4.2

Note: since I an using absolute time threshold for reporting the
glitches, the first one may take more iterations before it occurs on
a fast computer. You may need to increase nbar or nfoo.

import sys
from time import time
class Foo(object):
def __init__(me,nfo o):
me.memory = [1]*nfoo

class Bar(object):
def __init__(me,nba r,nfoo):
tgx.set_tag("Ba r") # timer

b = [None]*nbar
for i in xrange(nbar):
b[i]=Foo(nfoo) # make a foo, add it to list
tgx.jump("Bar"+ `i`) #timer

me.b = b #store the list in my instance memory


# just a utility class to time things.
class gtime:
def __init__(me,f=s ys.stderr):
me.last = time()
me.f=f
me.tag = ""
me.ticks = 0

def set_tag(me,tag) :
me.tag = tag
me.ticks = 0

def reset(me):
me.ticks = 0

def jump(me,tag="NO NE",allowed_gap =0.5):
me.t = time()
me.ticks +=1
if me.t-me.last>allowed _gap:
print >>me.f,"Big Gap:",me.t-me.last,"second s
",me.tag,tag,me .ticks
me.last = time()

tgx = gtime() # instance of the timer
# main loop
nbar = 100
nfoo = 10

ad_nauseum = 20000
final = [None]*ad_nauseum

for i in xrange(ad_nause um ):
if i%100 == 0 :print >>sys.stderr,"B ars Made: ",i
final[i] = Bar(nbar,nfoo)

sample Output:

Bars Made: 0
Bars Made: 100
Bars Made: 200
Bars Made: 300
Bars Made: 400
Bars Made: 500
Bars Made: 600
Bars Made: 700
Bars Made: 800
Bars Made: 900
Bars Made: 1000
Bars Made: 1100
Bars Made: 1200
Bars Made: 1300
Bars Made: 1400
Bars Made: 1500
Big Gap: 0.618862867355 seconds Bar Bar5 6
Bars Made: 1600
Bars Made: 1700
Bars Made: 1800
Big Gap: 0.748915195465 seconds Bar Bar76 77
Bars Made: 1900
Bars Made: 2000
Bars Made: 2100
Big Gap: 0.809149980545 seconds Bar Bar45 46
Bars Made: 2200
Bars Made: 2300
Bars Made: 2400
Big Gap: 0.898494958878 seconds Bar Bar15 16
Bars Made: 2500
Bars Made: 2600
Bars Made: 2700
Big Gap: 1.01110696793 seconds Bar Bar86 87
Bars Made: 2800
Bars Made: 2900
Bars Made: 3000
Big Gap: 1.12396192551 seconds Bar Bar55 56
Bars Made: 3100
Bars Made: 3200
Bars Made: 3300
Big Gap: 1.39006495476 seconds Bar Bar25 26
Bars Made: 3400
Bars Made: 3500
Bars Made: 3600
Big Gap: 1.55699706078 seconds Bar Bar96 97
Bars Made: 3700
Bars Made: 3800
Bars Made: 3900
Big Gap: 1.49929594994 seconds Bar Bar65 66
Bars Made: 4000
Bars Made: 4100
Bars Made: 4200
Big Gap: 1.64840602875 seconds Bar Bar35 36
Bars Made: 4300
Bars Made: 4400
Bars Made: 4500
Big Gap: 1.728484869 seconds Bar Bar5 6
Bars Made: 4600
Bars Made: 4700
Bars Made: 4800


Oct 1 '06 #1
5 1591
charlie strauss wrote:
Below is a simple program that will cause python to intermittently
stop executing for a few seconds. it's 100% reproducible on my
machine.
Confirmed with Python 2.4.2 on Windows.

gc.disable() fixes it, so it looks like you found an inefficiency in the
Python's GC. I have no idea whether this would be considered a bug by Python's
developer, but you can try opening a bugreport...
--
Giovanni Bajo
Oct 1 '06 #2

Giovanni Bajo wrote:
charlie strauss wrote:
Below is a simple program that will cause python to intermittently
stop executing for a few seconds. it's 100% reproducible on my
machine.

Confirmed with Python 2.4.2 on Windows.

gc.disable() fixes it, so it looks like you found an inefficiency in the
Python's GC. I have no idea whether this would be considered a bug by Python's
developer, but you can try opening a bugreport...
Reproduced on 2.4.3 and 2.5 on Windows.
Disabling GC fixes the speed problem as Giovanni said, but doesn't
reduce the amount of memory taken (measured by increase in "page file
used" display in Task Manager). At about 520 Mb, this seems rather too
high to me.

Definitely worth reporting, even if the outcome is only(!) a timbottian
dissertation of why it's not a problem -- at least we'd learn something
:-)

Cheers,
John

Oct 1 '06 #3
Below is a simple program that will cause python to intermittently
stop executing for a few seconds. it's 100% reproducible on my
machine.
Confirmed with Python 2.4.2 on Windows.

gc.disable() fixes it, so it looks like you found an inefficiency in the
Python's GC. I have no idea whether this would be considered a bug by
Python's
developer, but you can try opening a bugreport...

Reproduced on 2.4.3 and 2.5 on Windows.
Disabling GC fixes the speed problem as Giovanni said, but doesn't
reduce the amount of memory taken (measured by increase in "page file
used" display in Task Manager). At about 520 Mb, this seems rather too
high to me.

Definitely worth reporting, even if the outcome is only(!) a timbottian
dissertation of why it's not a problem -- at least we'd learn something
:-)
This is because the OP violated the Style Guide (PEP 8) and used 'me'
instead of 'self' as the first argument of instance methods, the
couple of seconds delay in runtime is the revenge of the vicious
interpreter :)
Oct 1 '06 #4
On Sun, 01 Oct 2006 07:13:25 GMT, Giovanni Bajo <no***@sorry.co mwrote:
charlie strauss wrote:
>Below is a simple program that will cause python to intermittently
stop executing for a few seconds. it's 100% reproducible on my
machine.

Confirmed with Python 2.4.2 on Windows.
And Python 2.3.5 on Linux, amd64. In fact, it causes heavy swapping so it
will disrupt other processes, too.

I didn't read the code, stupid as I am, but I trust that the behavior
doesn't match what the code actually tries to do.

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.dyn dns.org R'lyeh wgah'nagl fhtagn!
Oct 1 '06 #5
[charlie strauss]
>>Below is a simple program that will cause python to intermittently
stop executing for a few seconds. it's 100% reproducible on my
machine.
[Giovanni Bajo]
>Confirmed with Python 2.4.2 on Windows.
[Jorgen Grahn]
And Python 2.3.5 on Linux, amd64. In fact, it causes heavy swapping so it
will disrupt other processes, too.

I didn't read the code, stupid as I am, but I trust that the behavior
doesn't match what the code actually tries to do.
No, that's what it does: as it goes on, it creates an ever-increasing
and unbounded number of immortal objects and lists. The "time burps"
merely reflect that the more live containers and objects you have, the
longer it takes for cyclic gc to determine that they're not trash.
Oct 1 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

147
7809
by: Sateesh | last post by:
Hi, I am a beginner in Python, and am wondering what is it about the indentation in Python, without which python scripts do not work properly. Why can't the indentation not so strict so as to give better freedom to the user? Is there any plausible reason behind this? Cheers! Sateesh
2
3021
by: Karl Ehr | last post by:
I have written the following simple program to monitor a single URL, and notify me via email whenever this URL changes. Intermittently, I raise the following exception: Traceback (most recent call last): File "./urlwatch.py", line 34, in ? server.sendmail(sourceAddress, emailAddress, message) File "/usr/lib/python2.2/smtplib.py", line 621, in sendmail if not (200 <= self.ehlo() <= 299): File "/usr/lib/python2.2/smtplib.py", line...
86
4119
by: Matthias Kaeppler | last post by:
Hi, sorry for my ignorance, but after reading the Python tutorial on python.org, I'm sort of, well surprised about the lack of OOP capabilities in python. Honestly, I don't even see the point at all of how OO actually works in Python. For one, is there any good reason why I should ever inherit from a class? ^^ There is no functionality to check if a subclass correctly implements an inherited interface and polymorphism seems to be...
3
2613
by: zxo102 | last post by:
Hi there, I have a python application (many python scripts) and I start the application like this python myServer.py start in window. It is running in dos window. Now I would like to put it in background as NT service. I got a example code: SmallestService.py from chapter 18 of the book "Python Programming On Win32" by Mark Hammond etc. The code is as follows and runs well as an NT service.
0
361
by: Kurt B. Kaiser | last post by:
Patch / Bug Summary ___________________ Patches : 378 open ( +3) / 3298 closed (+34) / 3676 total (+37) Bugs : 886 open (-24) / 5926 closed (+75) / 6812 total (+51) RFE : 224 open ( +7) / 227 closed ( +7) / 451 total (+14) New / Reopened Patches ______________________
10
2492
by: Mythmon | last post by:
I am trying to make a program that will basically simulate a chess clock in python. To do this I have two threads running, one that updates the currently running clock, and one that watches for a keypress. I am using the effbot Console module, and that is where I get the events for the keypresses. But when I press space it crashes shortly after. The program stops, the Console closes, and windows says that the program pythonw.exe has had an...
852
29005
by: Mark Tarver | last post by:
How do you compare Python to Lisp? What specific advantages do you think that one has over the other? Note I'm not a Python person and I have no axes to grind here. This is just a question for my general education. Mark
2
3177
by: falloutphil | last post by:
Hi, I'm running a CGI script written in python that tars up the photos in user selected directories (via a form). It's running on Apache 1.3.31.1 on Solaris 5.8. It works well for a small amount of files, but when I try to use it for large archives (typically over 10MB, but this seems to vary) my download stops short. Occasionally I get an error message, this is usually a Broken Pipe (IO Error 32), but sometimes I don't see anything at...
3
8518
by: falloutphil | last post by:
Hi, First of all sorry for the double post - this is on the Python page too, but as far as I can see this is an Apache issue now. Mods - feel free to delete the similar titled posts from me on Python, if this is the case (can't seem to do it myself!). Anyway, I'm running a CGI script written in python that tars up the photos in user selected directories (via a form). It's running on Apache 1.3.31.1 on Solaris 5.8. It works well for a...
0
9856
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9698
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
10654
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10298
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7833
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5867
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4494
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4066
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3136
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.