473,395 Members | 2,079 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,395 software developers and data experts.

Extracting results from a large hotshot profile

I'm profiling some code that screens a large database. The algorithm
efficiency is heavily variable, based upon the different records in
the database. In order to get a sense of the best place to start
optimizing the code I did a hotshot profile. To make this profile took
about a day. However, I've been trying to extract results from the
profile for the past two days. It's eating up all the memory (2G
memory) on the machine and is slowing sucking up the swap space
(another 2G).

I'm hesitant to kill it for I might get results out of it some day
(even if I may have to bequeath them to my grand-children), but is
there a better way to do this?

I'm using the following script to print my results:
import sys
import hotshot.stats
stats = hotshot.stats.load(sys.argv[1])
stats.strip_dirs()
stats.sort_stats('time', 'calls')
stats.print_stats(20)

The profile is 355 MB large. Is there anything I can do to extract that data?

Thanks in advance,
Brian Cole
Jan 16 '06 #1
0 995

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: KefX | last post by:
Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed Python into a plugin for a freeware (closed-source) Windows music program called Jeskola Buzz. (Man, I can't believe...
0
by: Irmen de Jong | last post by:
Hi, when using the hotshot profiler, I can see no difference in the measurement of a function that is busy eating CPU cycles, and a function that is blocking/waiting on IO (or sleeping)... For...
0
by: Rodrigo Daunaravicius | last post by:
I'm having trouble understanding hotshot's results. It's the first time I use a profiler. I'm trying to profile a loop that takes almost six hours (~21000 seconds) to execute in production. The...
0
by: Irmen de Jong | last post by:
Okay I tried some profiling, but am uncertain about the results I'm getting. They confuse the hell out of me. I have a test program (see below) that essentially has two loops that get called...
0
by: Charles Hartman | last post by:
I've used profile before, but wanted to get more information so I thought I'd try hotshot instead. It works fine when used like profile. But when I run it using this line, prof =...
0
by: Charles Hartman | last post by:
(I asked this a day or two ago; if there was an answer, I missed it. Anybody using hotshot?) I've used profile before, but wanted to get more information so I thought I'd try hotshot instead....
1
by: Andreas Lobinger | last post by:
Aloha, hotshot.Profile has flags for recording timing per line and line events. Even if i had both set to 1 i still get only the standard data (time per call). Is there any document available...
0
by: Lars Woetmann | last post by:
what is the definition of sub-functions in the hotshot profiler? and just as important what is not sub-functions from the output I can see that functions I made and functions in...
1
by: Yang | last post by:
Note: I realize hotshot is obsoleted by cProfile, but 2.5 breaks several packages I depend on. I'm using Python 2.4.3. I'm getting an AssertionError on "assert not self._stack" when calling...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.