473,699 Members | 2,479 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Problems with profiling results (hotshot or normal) : almost allare incorrect

Hello
I haven't received any responses on my original posting,
and it could very well be that it never made it to the newsgroup
(because I can't find it back on my news server here).
So I'm posting this again. Sorry if it *did* make it to
the group.. Please consider trying my test-program that is given
here and let me know how it works on your python installation.

Thanks for any insight on this.

--Irmen

[original message follows]

Irmen de Jong wrote:
Okay I tried some profiling, but am uncertain about the
results I'm getting. They confuse the hell out of me.

I have a test program (see below) that essentially has
two loops that get called repeatedly. One that is an idle
wait loop, and one that is a busy-wait CPU hogger.
I wanted to see what profiling results that would give.
The total runtime of the program is 10 seconds, where 5
seconds are spent in the CPU-loop and 5 seconds in the idle
wait loop.

But what I'm getting is not what I expected.

For instance, with python2.3.4 on windows, it says:
(hotshot:) 21 function calls in 35.772 CPU seconds
(normal: ) 23 function calls in 10.008 CPU seconds
Python 2.4b1 on windows gives:
(hotshot:) 21 function calls in 0.001 CPU seconds
(normal: ) 260362 function calls in 6.724 CPU seconds

I know that timing information on Windows is sometimes
difficult to get, but the above is really weird. Anyway
so I also tried it on my Linux (mandrake 10) computer:

Python 2.3.4 on linux:
(hotshot:) 21 function calls in 17.542 CPU seconds
(normal: ) 23 function calls in 5.000 CPU seconds
Python 2.4b1 on linux:
(hotshot:) 21 function calls in 0.000 CPU seconds
(normal: ) 52474 function calls in 1.650 CPU seconds

Conclusion: only *one* of the above test runs gives the
right summary (in my opinion): the normal profiler of
python 2.3.4 on linux (5 CPU seconds).
The normal profiler of python 2.3.4 on windows also counts
the idle loop time apparently, and gives 10 CPU seconds.
But all the others are waaaay off!

(and I haven't event talked about the detailed function
call profiling statistics).

Can anybody please help me out? Am I doing something wrong,
or is the profiler just not reliable yet?

Confused-ly y'rs,
--Irmen de Jong
PS using the 'time' command on linux, when running the test
program, gives 0:20.19elapsed 50%CPU which is what I expected.
-------------------- test program ---------------
import time

def foo():
# idle wait loop
time.sleep(0.5)
def bar():
# busy (CPU) wait loop
s=time.time()
while time.time()<(s+ 0.5):
pass

def test():
for i in range(10):
foo()
bar()

if __name__=="__ma in__":
import hotshot, hotshot.stats, wait
import profile, pstats

print "HOTSHOT profiling..."
prof = hotshot.Profile ("wait.prof" )
prof.runcall(wa it.test)
prof.close()
print "PROFILE DONE"
stats = hotshot.stats.l oad("wait.prof" )
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(40)

print "Normal profiler..."
profile.run('wa it.test()', 'wait.prof')
print "profile done"
stats = pstats.Stats('w ait.prof')
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(40)

Jul 18 '05 #1
2 2157
On Wed, 03 Nov 2004 00:43:31 +0100, Irmen de Jong <irmen@-nospam-remove-this-xs4all.nl> wrote:
Hello
I haven't received any responses on my original posting,
and it could very well be that it never made it to the newsgroup
(because I can't find it back on my news server here).
So I'm posting this again. Sorry if it *did* make it to
the group.. Please consider trying my test-program that is given
here and let me know how it works on your python installation.

Thanks for any insight on this.

Traceback (most recent call last):
File "C:\pywk\clp\ir men2.py", line 86, in ?
import hotshot, hotshot.stats, wait
ImportError: No module named wait

Where does wait come from?

Regards,
Bengt Richter
Jul 18 '05 #2
Bengt Richter wrote:
On Wed, 03 Nov 2004 00:43:31 +0100, Irmen de Jong <irmen@-nospam-remove-this-xs4all.nl> wrote:

Hello
I haven't received any responses on my original posting,
and it could very well be that it never made it to the newsgroup
(because I can't find it back on my news server here).
So I'm posting this again. Sorry if it *did* make it to
the group.. Please consider trying my test-program that is given
here and let me know how it works on your python installation.

Thanks for any insight on this.


Traceback (most recent call last):
File "C:\pywk\clp\ir men2.py", line 86, in ?
import hotshot, hotshot.stats, wait
ImportError: No module named wait

Where does wait come from?


Aww, terribly sorry. Should test any code that I post.
(I called the test script "wait.py" on my system, that's
why it worked here, locally)

Here's a fixed version:

import time

def foo():
# idle wait loop
time.sleep(0.5)
def bar():
# busy (CPU) wait loop
s=time.time()
while time.time()<(s+ 0.5):
pass

def test():
for i in range(10):
foo()
bar()

if __name__=="__ma in__":
import hotshot, hotshot.stats
import profile, pstats

print "HOTSHOT profiling..."
prof = hotshot.Profile ("wait.prof" )
prof.runcall(te st)
prof.close()
print "PROFILE DONE"
stats = hotshot.stats.l oad("wait.prof" )
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(40)

print "Normal profiler..."
profile.run('te st()', 'wait.prof')
print "profile done"
stats = pstats.Stats('w ait.prof')
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(40)
Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
1702
by: KefX | last post by:
Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed Python into a plugin for a freeware (closed-source) Windows music program called Jeskola Buzz. (Man, I can't believe how many people keep telling me "Extend Python instead!" when clearly I can't do that in this case. If somebody here tells me this, I will explode in an appropriately gory fashion.) It's a great program, and it works mainly on plugins other people...
4
2146
by: KefX | last post by:
Hey...as for what I'm doing with Python, look at my post "Strange Hotshot problem". To make a long story short, I'm embedding Python in order to write a plugin to a freeware music program; the plugin interfaces to the program with C++, but the plugin calls Python routines to do much (but not all) of the work. Which brings me to the problem. It's SLOW! Well, that's hardly surprising, but anyway...I'm using Psyco, and it's definitely at...
0
1591
by: Irmen de Jong | last post by:
Hi, when using the hotshot profiler, I can see no difference in the measurement of a function that is busy eating CPU cycles, and a function that is blocking/waiting on IO (or sleeping)... For instance I have this output: 3 function calls in 26.931 CPU seconds Ordered by: internal time, call count
0
1230
by: Rodrigo Daunaravicius | last post by:
I'm having trouble understanding hotshot's results. It's the first time I use a profiler. I'm trying to profile a loop that takes almost six hours (~21000 seconds) to execute in production. The profiler stats, however, give me a total of 16.903 CPU seconds (that's ~17 seconds, right?). The results *are* relevant, as they helped me to pull execution time down from the original 80 hours, but I wonder if I'm missing something that might help...
0
464
by: Irmen de Jong | last post by:
Okay I tried some profiling, but am uncertain about the results I'm getting. They confuse the hell out of me. I have a test program (see below) that essentially has two loops that get called repeatedly. One that is an idle wait loop, and one that is a busy-wait CPU hogger. I wanted to see what profiling results that would give. The total runtime of the program is 10 seconds, where 5 seconds are spent in the CPU-loop and 5 seconds in the...
0
1401
by: Charles Hartman | last post by:
I've used profile before, but wanted to get more information so I thought I'd try hotshot instead. It works fine when used like profile. But when I run it using this line, prof = hotshot.Profile('ScanHot.prof', lineevents=1) though it runs all right, when I try to load the resulting file I get this: >>> hs = hotshot.stats.load("ScanHot.prof") Traceback (most recent call last): File "<stdin>", line 1, in ?
0
1203
by: Charles Hartman | last post by:
(I asked this a day or two ago; if there was an answer, I missed it. Anybody using hotshot?) I've used profile before, but wanted to get more information so I thought I'd try hotshot instead. It works fine when used like profile. But when I run it using this line, prof = hotshot.Profile('ScanHot.prof', lineevents=1) though it runs all right, when I try to load the resulting file I get this:
6
3746
by: cournape | last post by:
Hi there, I have some scientific application written in python. There is a good deal of list processing, but also some "simple" computation such as basic linear algebra involved. I would like to speed things up implementing some of the functions in C. So I need profiling. I first tried to use the default python profiler, but profiling my application multiplies the execution time by a factor between 10 and 100 ! So I decided to give a...
1
1762
by: Geert Jansen | last post by:
Hi! I'm trying to profile an application that I believe is blocking on I/O for a significant amount of time. In trying to dig down where this happens, I profiled the application with hotshot. The results are not really usable however as it seems to display the amount of CPU time which for my application is much lower than the total run time. Is possible to use hotshot with wall clock time, i.e. is it possible to have the code fragment...
13
1644
by: guitarromantic | last post by:
Hey everyone. I'm editing some stuff I did last summer, trying to bugfix and improve stuff. One improvement (or an oversight of the original design) is adding dynamic <title> tags to my pages. Here's how I originally structured my php pages: include header.php (contents are basically my HTML/CSS up to the <!--begin content--> point) <php code>
0
8700
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8623
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9048
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8892
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7767
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6539
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5878
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4635
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2359
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.