Profiling evaluation tools?

[Jens Theisen]

Hello,

I want to apologise in advance for this being off topic. It's not neither
A C nor a C++ question, but to profiling in general, though I my chances
are best to find the answer in the C/C++ community.

I have a C++ program to profile and went about it by producing large
history files of calling dependencies with associated times. It is
presumably similar to gprof's data format and could be converted.

I'm now looking for utilities that can evaluate such a tree. I only
looked at the gprof's manual page and that seemed too basic to me.

I already wrote some Python scripts that can extract stuff like:

- Break the time down into that spent in each (libraries, files,
functions, ...), exclusive to that spent on their behalf in other
such entities. Print it in order.

- Give a tree of (libraries, files, functions) according to their
calling dependencies, but don't recurse into those subentities below
a certain threshhold of consumed time. Give the number of calls and
time (including the time spent in subentities).

All this is already quite useful, but I also dearly want to have:

- My script is already "merging together" subtrees on the same
level. For instance, if f calls g a number of times, and g h a
number of times, you only get f->g->h once in the tree view of the
data, the number at the line of h being the total calls to h through
this callchain. However, if a call to f occurs at a number of
different places in the code, those occurences will not be merged
together: I can't say, view any call to f as a the "whole program"
and forget other calls, merging all these subtrees into one.

- I have a general structure of attributes to each node (ie library, file
and function names), but I also want to have additional attributes like
the value of a parameter the function was called with, which may only
apply to some functions. I need to filter out certain values of these
attributes or break down to their values.

- Reverse tree view: Given a (library, file, function), how often was
it called and how much time did it spend on the behalf of all it's
calling parent, grandparents, etc., again breaking recursion on a
certain time threshold. A special case of this is the ability to
extract a backtrace from the profiling data, which is useful on it's
own.

I have a number of vague ideas how to go about this.

My scripts are in Python, but they are messy and slow. An exotic
approach would be xslt, since it's designed to operate on tree-data. I
have zero experience in this regard - is this possible? how does it
perform? Another one is trying to use databases as a storage for
trees, but I'm not sure if all of these operations will be efficient.

The most realistic option I would go about in the absence of further
advice and existing utilities is to rewrite my scripts in C++.

All of this needs properly thinking through, so I'd really much rather
use an exisiting solution. Is there one?

Cheers,

Jens

[Flash Gordon]

Jens Theisen wrote:

Hello,

I want to apologise in advance for this being off topic. It's not neither
A C nor a C++ question,
If you know it is not a C or C++ question and know it is therefor off
topic here, don't ask it here.
but to profiling in general, though I my chances
are best to find the answer in the C/C++ community.
No, you are most likely to get complained at for posting in what you
know is the wrong place.
I have a C++ program to profile and went about it by producing large
history files of calling dependencies with associated times. It is
presumably similar to gprof's data format and could be converted.

<snip>

Since you are using GNU tools, try asking in GNU mailing lists.
Alternatively try asking in a more general group or an OS specific
group. Don't ask in what you *know* is the wrong group.
--
Flash Gordon
Living in interesting times.
Web site - http://home.flash-gordon.me.uk/
comp.lang.c posting guidlines and intro -
http://clc-wiki.net/wiki/Intro_to_clc

[Ico]

In comp.lang.c Jens Theisen <jt***@arcor.de> wrote:

I have a C++ program to profile and went about it by producing large
history files of calling dependencies with associated times. It is
presumably similar to gprof's data format and could be converted.

[... snipped a lot of requested features ...]

<offtopic answer following>

If you happen to work on linux, take a look at the excellent callgrind
and kcachegrind tools. Callgrind is a tool that runs your code in a
virtual machine. It can do profiling, call tracing, cache profiling and
a lot more. Kcachegrind is a gui for showing the profiling results in
various ways.

http://kcachegrind.sourceforge.net/cgi-bin/show.cgi
--
:wq
^X^Cy^K^X^C^C^C^C

[Jens Theisen]

Flash wrote:

Since you are using GNU tools, try asking in GNU mailing lists.
Alternatively try asking in a more general group or an OS specific
group. Don't ask in what you *know* is the wrong group.

An OS specific group would be just as wrong, since it's not an OS specific
problem.

I searched for newsgroups or mailinglists for profiling related talks but
failed in finding them.

I don't know what the right group is to ask, and obviously you don't
either.

Jens

[Jens Theisen]

Ico wrote:

http://kcachegrind.sourceforge.net/cgi-bin/show.cgi

That's indeed an interesting start, thank you very much!

Jens

[Mark McIntyre]

On 15 Feb 2006 13:27:00 +0000, in comp.lang.c , jt***@arcor.de (Jens
Theisen) wrote:

Flash wrote:

Since you are using GNU tools, try asking in GNU mailing lists.
Alternatively try asking in a more general group or an OS specific
group. Don't ask in what you *know* is the wrong group.

An OS specific group would be just as wrong, since it's not an OS specific
problem.

The thing is, there is NO generic C answer to this as its a toolset
thing, not a C thing. You need to ask in a group specialising in each
OS or toolset you're interested in. From your original post, you are
interested in only one OS and toolset, so ask over there.
Mark McIntyre
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan

----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----

[Jens Theisen]

Mark wrote:

The thing is, there is NO generic C answer to this as its a toolset
thing, not a C thing. You need to ask in a group specialising in each
OS or toolset you're interested in. From your original post, you are
interested in only one OS and toolset, so ask over there.
Mark McIntyre

I think my original posting was very clear in that it's not at all
specific to a certain toolset.

Jens

[CBFalconer]

Jens Theisen wrote:

Mark wrote:

The thing is, there is NO generic C answer to this as its a toolset
thing, not a C thing. You need to ask in a group specialising in each
OS or toolset you're interested in. From your original post, you are
interested in only one OS and toolset, so ask over there.
Mark McIntyre

I think my original posting was very clear in that it's not at all
specific to a certain toolset.

Maybe not, but the solution necessarily must be. Either the
compiler/runtime system has special provisions, or you must distort
the original source code to inject something or other. You don't
have a namespace to play in, since things are either reserved for
the implementation or for the user.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943

[Keith Thompson]

jt***@arcor.de (Jens Theisen) writes:

Flash wrote:

Since you are using GNU tools, try asking in GNU mailing lists.
Alternatively try asking in a more general group or an OS specific
group. Don't ask in what you *know* is the wrong group.

An OS specific group would be just as wrong, since it's not an OS specific
problem.

I searched for newsgroups or mailinglists for profiling related talks but
failed in finding them.

I don't know what the right group is to ask, and obviously you don't
either.

That doesn't imply that *this* is the right group.

You might try comp.programming.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Flash Gordon]

Jens Theisen wrote:

Flash wrote:

Since you are using GNU tools, try asking in GNU mailing lists.
Alternatively try asking in a more general group or an OS specific
group. Don't ask in what you *know* is the wrong group.
An OS specific group would be just as wrong, since it's not an OS specific
problem.

Many OS specific groups deal with tool chains.
I searched for newsgroups or mailinglists for profiling related talks but
failed in finding them.

I don't know what the right group is to ask, and obviously you don't
either.

There are a number of possible general groups including alt.comp.misc
which from the name is a very general group to comp.programmer which is
another very general group amongst others. Is it really so hard to try a
generalist group when you can't find a specific group?

Would you go in to a Hi-fi shop and ask for advice on how to repair a
printer just because you can't find a printer specialist, or would you
go in to a general computer repair centre? They both have a lot of
electronics in, after all.
--
Flash Gordon
Living in interesting times.
Web site - http://home.flash-gordon.me.uk/
comp.lang.c posting guidlines and intro -
http://clc-wiki.net/wiki/Intro_to_clc

[usenet]

In comp.lang.c Flash Gordon <sp**@flash-gordon.me.uk> wrote:

Jens Theisen wrote:

Flash wrote:

Since you are using GNU tools, try asking in GNU mailing lists.
Alternatively try asking in a more general group or an OS specific
group. Don't ask in what you *know* is the wrong group.

An OS specific group would be just as wrong, since it's not an OS specific
problem.

Would you go in to a Hi-fi shop and ask for advice on how to repair a
printer just because you can't find a printer specialist, or would you
go in to a general computer repair centre? They both have a lot of
electronics in, after all.

If I can't find a computer repair centre in my town, I'd go into the
Hi-fi shop and friendly ask something like:

"Good morning, I want to apologise in advance for this being in the
wrong shop. It's not neither A receiver nor CD-player question, but
electronics in general, though I my chances are best to find the answer
in the local hi-fi shop"

They might just be able to answer my question (did you plug the power
cord in?), or kindly tell me that there *is* a general computer repair
center down the road, that I didn't know about.

Whatever the outcome, everybody would be kind, friendly and happy.

[Neil Cerutti]

["Followup-To:" header set to comp.lang.c++.]
On 2006-02-16, us****@zevv.nl <us****@zevv.nl> wrote:

Would you go in to a Hi-fi shop and ask for advice on how to repair
a printer just because you can't find a printer specialist, or
would you go in to a general computer repair centre? They both have
a lot of electronics in, after all.
If I can't find a computer repair centre in my town, I'd go into the
Hi-fi shop and friendly ask something like:

"Good morning, I want to apologise in advance for this being in the
wrong shop. It's not neither A receiver nor CD-player question, but
electronics in general, though I my chances are best to find the
answer in the local hi-fi shop"

Hi-fi shops are used to incredibly strange questions, so while they
probably would be no help, they might get a laugh. Like the time time
my brother was asked for "a new needle for my 8-track player". The guy
was extremely disappointed in the answer. ;-)
They might just be able to answer my question (did you plug the power
cord in?), or kindly tell me that there *is* a general computer repair
center down the road, that I didn't know about.

Whatever the outcome, everybody would be kind, friendly and happy.

That depends on several unknown factors. Rudeness in electronics shops
is quite possible.

--
Neil Cerutti

[Michael Wojcik]

In article <43***************@yahoo.com>, CBFalconer <cb********@yahoo.com> writes:

Jens Theisen wrote:

Mark wrote:

The thing is, there is NO generic C answer to this as its a toolset
thing, not a C thing.

I think my original posting was very clear in that it's not at all
specific to a certain toolset.

Maybe not, but the solution necessarily must be. Either the
compiler/runtime system has special provisions, or you must distort
the original source code to inject something or other.

While I agree that the query was OT for c.l.c (and presumably for
c.l.c++), both Mark's response and yours are incorrect. There are
profiling tools which are specific to language implementations (which
it appears is what "toolset" refers to here), but there are also
those that are not.

Of the three profiling techniques I'm familiar with, only one -
instrumentation - is specific to a language implementation. Your
post describes an instrumenting profiler. That is not the only
mechanism for profiling.

Sampling and virtualizing profilers need not be tied to a particular
implementation, and indeed often are not. They are generally created
for a particular OS, since they require fairly intimate interaction
with the execution environment. (Obviously hardware-based sampling
profilers can be at least somewhat OS-independent.)

Given all of that, OS-specific groups do indeed appear to be the
correct place to ask.

--
Michael Wojcik mi************@microfocus.com

Even 300 years later, you should plan it in detail, when it comes to your
summer vacation. -- Pizzicato Five

[Jens Theisen]

Michael wrote:

While I agree that the query was OT for c.l.c (and presumably for
c.l.c++), both Mark's response and yours are incorrect. There are
profiling tools which are specific to language implementations (which
it appears is what "toolset" refers to here), but there are also
those that are not.

Why I also agree that my OP was off-topic, I do so for opposite reasons.

I don't need a Profiler, I need something to convert the data of an
already profiled application into information.

Such a solution could be based on xslt as well as sql or something
completely different.

That said, you're all right that this is not the place for that
discussion.

Jens