473,574 Members | 3,026 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

python profiling, hotspot and strange execution time

Hi there,

I have some scientific application written in python. There is a
good deal of list processing, but also some "simple" computation such
as basic linear algebra involved. I would like to speed things up
implementing some of the functions in C. So I need profiling.

I first tried to use the default python profiler, but profiling my
application multiplies the execution time by a factor between 10 and
100 ! So I decided to give a try to hotspot. I just followed the
example of the python library reference, but I have some strange
results concerning cpu time. My profiling script is something like the
following:

def run_foo():
print time.clock()

function_to_pro file()

print time.clock()

prof = hotshot.Profile ("essai.prof ")
benchtime= prof.runcall(ru n_foo)
prof.close()
stats = hotshot.stats.l oad("essai.prof ")
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(20)

The goal is to profile the function function_to_pro file(). Running this
script gives me a CPU executime time of around 2 seconds, whereas the
difference between the two clock calls is around 10 seconds ! And I
don't run any other cpu consuming tasks at the same time, so this
cannot come from other running processes. Is there something perticular
about hotspot timing I should know ? I am not sure how I can get more
accurate results with hotspot.

I would appreciate any help,

Thanks

Sep 6 '05 #1
6 3739
co******@gmail. com writes:
I have some scientific application written in python. There is a
good deal of list processing, but also some "simple" computation such
as basic linear algebra involved. I would like to speed things up
implementing some of the functions in C. So I need profiling.


Why don't you use numarray for the linear algebra?
Sep 6 '05 #2
In message <11************ **********@g47g 2000cwa.googleg roups.com>,
co******@gmail. com writes
Hi there,

I have some scientific application written in python. There is a
good deal of list processing, but also some "simple" computation such
as basic linear algebra involved. I would like to speed things up
implementing some of the functions in C. So I need profiling.


You haven't said which platform you are on. If you are on Windows you
may want to try Python Performance Validator.

http://www.softwareverify.com

Stephen
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
Sep 6 '05 #3
co******@gmail. com wrote:
Hi there,

I have some scientific application written in python. There is a
good deal of list processing, but also some "simple" computation such
as basic linear algebra involved. I would like to speed things up
implementing some of the functions in C. So I need profiling.

I first tried to use the default python profiler, but profiling my
application multiplies the execution time by a factor between 10 and
100 ! So I decided to give a try to hotspot.
OK - first of all, as someone else has asked, what platform are you
running? I'm assuming it's windows since you're referring to
time.clock() and then later saying "wall clock".

Next, what are you hoping that the profiler will give you? If you're
expecting it to give you the big picture of your application's
performance and give you "real runtime numbers", you may be
disappointed. It is a deterministic profiler and will give you CPU time
spent in different areas of code rather than and overall "how long did
this thing take to run?".
I just followed the
example of the python library reference, but I have some strange
results concerning cpu time. My profiling script is something like the
following:

def run_foo():
print time.clock()

function_to_pro file()

print time.clock()

prof = hotshot.Profile ("essai.prof ")
benchtime= prof.runcall(ru n_foo)
prof.close()
stats = hotshot.stats.l oad("essai.prof ")
stats.strip_di rs()
stats.sort_sta ts('time', 'calls')
stats.print_st ats(20)


Well, let's just add more confusion to the pot, shall we? Look at this
example (a slight hack from yours)::

import time
import hotshot
import hotshot.stats
def run_foo():
print time.clock()
print time.time()

time.sleep(5)

print time.clock()
print time.time()

prof = hotshot.Profile ("essai.prof ")
benchtime= prof.runcall(ru n_foo)
prof.close()
stats = hotshot.stats.l oad("essai.prof ")
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(20)

and the output::

0.24
1126011669.55
0.24
1126011674.55
1 function calls in 0.000 CPU seconds

Ordered by: internal time, call count

ncalls tottime percall cumtime percall filename:lineno (function)
1 0.000 0.000 0.000 0.000 tmphQNKbq.py:6( run_foo)
0 0.000 0.000 profile:0(profi ler)

I inserted a time.time() call since I'm on Linux and time.clock()
returns a process's CPU time and wanted to show the "wall clock time" as
it were. So, the stats show 0 time taken, whereas time.time() shows 5
seconds. It's because the time.sleep() took a negligable amount of CPU
time which is what the profiler looks at.
The goal is to profile the function function_to_pro file(). Running this
script gives me a CPU executime time of around 2 seconds, whereas the
difference between the two clock calls is around 10 seconds !
I would attribute the wall clock and profile time difference to the
overhead of hotshot. While hotshot is miles better than the "regular"
profiler, it can still take a little time to profile code.
And I
don't run any other cpu consuming tasks at the same time, so this
cannot come from other running processes. Is there something perticular
about hotspot timing I should know ? I am not sure how I can get more
accurate results with hotspot.

I would appreciate any help,

Thanks

HTH,
JMJ
Sep 6 '05 #4

OK - first of all, as someone else has asked, what platform are you
running? I'm assuming it's windows since you're referring to
time.clock() and then later saying "wall clock".
Actually, no. I am working on a x86 linux (HT disabled for this
testing, as I thought it may introduce some subtilities). I am not sure
aht you mean by wall clock ?
Next, what are you hoping that the profiler will give you? If you're
expecting it to give you the big picture of your application's
performance and give you "real runtime numbers", you may be
disappointed. It is a deterministic profiler and will give you CPU time
spent in different areas of code rather than and overall "how long did
this thing take to run?".
I am not sure to understand the big difference between "time spent in
different areas of code" and "how long did this thing take to run?".
Looking at python doc for deterministic profiling, I understand the
implementation difference, and the performance implications, but I
don't see why deterministic profiling would not give me an overall
picture ?
Well, let's just add more confusion to the pot, shall we? Look at this
example (a slight hack from yours)::

import time
import hotshot
import hotshot.stats
def run_foo():
print time.clock()
print time.time()

time.sleep(5)

print time.clock()
print time.time()

prof = hotshot.Profile ("essai.prof ")
benchtime= prof.runcall(ru n_foo)
prof.close()
stats = hotshot.stats.l oad("essai.prof ")
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(20)

and the output::

0.24
1126011669.55
0.24
1126011674.55
1 function calls in 0.000 CPU seconds

Ordered by: internal time, call count

ncalls tottime percall cumtime percall filename:lineno (function)
1 0.000 0.000 0.000 0.000 tmphQNKbq.py:6( run_foo)
0 0.000 0.000 profile:0(profi ler)

I inserted a time.time() call since I'm on Linux and time.clock()
returns a process's CPU time and wanted to show the "wall clock time" as
it were. So, the stats show 0 time taken, whereas time.time() shows 5
seconds. It's because the time.sleep() took a negligable amount of CPU
time which is what the profiler looks at.
Well, your example make actually more sense to me :) I understand the
difference between CPU time and time spent in the python code (even if
I was not clear in my previous post about it...). But this case does
not apply to my code, as my code is never "idled", takes 100 % of the
cpu, with no other CPU consuming task
I would attribute the wall clock and profile time difference to the
overhead of hotshot. While hotshot is miles better than the "regular"
profiler, it can still take a little time to profile code.


Well, if hotshot reported a timing which is longer than the execution
time without it, I would have considered that to be normal. Even in C,
using gprof has a non negligeable overhead, most of the time.

What I don't understand is why hotshot reports that do_foo is executed
in 2 seconds whereas it effectively takes more than 10 seconds ? Is it
because I don't understand what deterministic profiling is about ?

David

Sep 8 '05 #5
co******@gmail. com wrote:
<snip>
I am not sure to understand the big difference between "time spent in
different areas of code" and "how long did this thing take to run?".
Looking at python doc for deterministic profiling, I understand the
implementati on difference, and the performance implications, but I
don't see why deterministic profiling would not give me an overall
picture ?

I think from below you said you were more clear on this. Cool.

<snip>
Well, your example make actually more sense to me :) I understand the
difference between CPU time and time spent in the python code (even if
I was not clear in my previous post about it...). But this case does
not apply to my code, as my code is never "idled", takes 100 % of the
cpu, with no other CPU consuming task
I would attribute the wall clock and profile time difference to the
overhead of hotshot. While hotshot is miles better than the "regular"
profiler, it can still take a little time to profile code.
Well, if hotshot reported a timing which is longer than the execution
time without it, I would have considered that to be normal. Even in C,
using gprof has a non negligeable overhead, most of the time.

Actually, I'd expect the opposite, but not as extreme for your case. I
would expect it to *report* that a piece of code took less time to
execute than I *observed* it taking. Reasons in the snipped area
above. Unless you're calling a C extension, in which case, IIRC, it's
supposed to report the actual execution time of the C call (and I guess
plus any overhead that hotshot may cause it to incur) in which case you
would be IMO 100% correct. I hope you're not calling a C extension, or
my head's gonna explode.
What I don't understand is why hotshot reports that do_foo is executed
in 2 seconds whereas it effectively takes more than 10 seconds ? Is it
because I don't understand what deterministic profiling is about ?

The profiler is supposed to be smart about how it tracks time spent in
execution so it doesn't get readings that are tainted by other processes
running or other stuff. I could easily see a 2->10 second disparity if
your process were idling somehow. Now, if you're doing a lot of IO, I
wonder if the profiler isn't taking into consideration any blocking
calls that may max out the CPU in IOWAIT... Are you doing a lot of IO?
David

JMJ
Sep 8 '05 #6
I was unhappy with both hotshot and the standard python profiler, so
I wrote my own, which may be what you are looking for. I've submitted
it as a patch at:

http://sourceforge.net/tracker/index...70&atid=305470

It should add a minimum of overhead, give real numbers and also
gives stats on child calls. However, it is not compatible with
the stats module.

You can compile it as a standalone module.
co******@gmail. com wrote:
Hi there,

I have some scientific application written in python. There is a
good deal of list processing, but also some "simple" computation such
as basic linear algebra involved. I would like to speed things up
implementing some of the functions in C. So I need profiling.

I first tried to use the default python profiler, but profiling my
application multiplies the execution time by a factor between 10 and
100 ! So I decided to give a try to hotspot. I just followed the
example of the python library reference, but I have some strange
results concerning cpu time. My profiling script is something like the
following:

def run_foo():
print time.clock()

function_to_pro file()

print time.clock()

prof = hotshot.Profile ("essai.prof ")
benchtime= prof.runcall(ru n_foo)
prof.close()
stats = hotshot.stats.l oad("essai.prof ")
stats.strip_dir s()
stats.sort_stat s('time', 'calls')
stats.print_sta ts(20)

The goal is to profile the function function_to_pro file(). Running this
script gives me a CPU executime time of around 2 seconds, whereas the
difference between the two clock calls is around 10 seconds ! And I
don't run any other cpu consuming tasks at the same time, so this
cannot come from other running processes. Is there something perticular
about hotspot timing I should know ? I am not sure how I can get more
accurate results with hotspot.

I would appreciate any help,

Thanks


Sep 8 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

467
21317
by: mike420 | last post by:
THE GOOD: 1. pickle 2. simplicity and uniformity 3. big library (bigger would be even better) THE BAD:
46
6232
by: Jon Perez | last post by:
Can one run a 1.5 .pyc file with the 2.x version interpreters and vice versa? How about running a 2.x .pyc using a 2.y interpreter?
114
9741
by: Maurice LING | last post by:
This may be a dumb thing to ask, but besides the penalty for dynamic typing, is there any other real reasons that Python is slower than Java? maurice
3
5372
by: dot | last post by:
Hi all, I have written a Python huffman Encoding Module, for my own amusement. I thought it might be educational/entertaining for other people, so I've put it on my website and wrote about it a little. http://gumuz.looze.net/wordpress/index.php/archives/2004/11/25/huffman-encoding/ Your comments are highly appreciated! cheers,
0
1426
by: Ian Bicking | last post by:
The Chicago Python User Group, ChiPy, will have its next meeting on Thursday, December 9th, starting at 7pm. For more information on ChiPy see http://chipy.org Topic ----- This month's topic is testing and profiling strategies in Python. Chris McAvoy will talk about doctest. Brian Ray will talk about the hotspot profiler. Ian...
118
6658
by: 63q2o4i02 | last post by:
Hi, I've been thinking about Python vs. Lisp. I've been learning Python the past few months and like it very much. A few years ago I had an AI class where we had to use Lisp, and I absolutely hated it, having learned C++ a few years prior. They didn't teach Lisp at all and instead expected us to learn on our own. I wasn't aware I had to...
25
3769
by: jwrweatherley | last post by:
I'm pretty new to python, but am very happy with it. As well as using it at work I've been using it to solve various puzzles on the Project Euler site - http://projecteuler.net. So far it has not let me down, but it has proved surprisingly slow on one puzzle. The puzzle is: p is the perimeter of a right angle triangle with integral length...
0
8234
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7815
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8098
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6452
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5620
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5300
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3740
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
1
1339
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
1060
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.