Hi Ira,
This post was rejected by comp.lang.c++.moderated since it is getting
slightly off-topic. We should consider continuing the discussion
somewhere else or by directly contacting each other via email, if there
is still interest.
See my comments inline.
BR,
Roman
[color=blue][color=green][color=darkred]
> > >> <romixnews@googlemail.com> wrote in message
> > >> news:1144182134.961761.287840@g10g2000cwb.googlegr oups.com...
> > >
> > >>> > Hi,
> > >>> >
> > >>> > I'm facing the problem of analyzing a memory allocation dynamic and
> > >>> > object creation dynamics of a very big C++ application ...
> > >
> > >>> > [many types of statistics... some methods for collecting them]
> > >
> > >>> > b) Doing the instrumentation automatically
> > >>> >
> > >>> > Automating the instrumentation of C++ code makes the task much[/color]
> > easier.[color=darkred]
> > >>> > This can be done either at the source code level or at the machine
> > >>> > code level.
> > >>> >
> > >>> > When we speak about automated source-code instrumentation, some[/color][/color]
> tools[color=green][color=darkred]
> > >>> > like Aspect++ or OpenC++, as well as some other source-to-source
> > >>> > transformation tools come up to my mind. As it can be easily seen,
> > >>> > they are coming from such areas like aspect-oriented programming,
> > >>> > meta-object protocols, etc.
> > >
> > >> Well, some come from there. Our perspective is that massive change
> > >> is a capability that should be in the hands of programmers, and has
> > >> nothing specific to do with aspects (expect they are special case).[/color]
> >
> > If the performance statistics collection as well as some logging of the
> > activities in the application is considered, the usage of AOP is quite
> > natural. This performance collection activity can be viewed as an
> > OPTIONAL orthogonal aspect (i.e. a cross-cutting concern) of an
> > application. It has nothing to do with the business logic of this app.
> > It should be possible to use this aspect only if required and
> > completely forget it, if it is not needed.[/color]
>[color=green]
> > BTW, when you say "in hands of programmers", it sounds as if AOP does
> > not let programmers to hold it in their hands. But why? AOP gives you
> > as a programmer the possibility to tell, where and what should be
> > applied. You have the choice.[/color]
>
> AOP requires you write aspects ("advice"), and run a weaver.
> If the aspects are simply separate
> from the main code, and you don't run the weaver, the aspects
> simply aren't present. You can think of such advice
> as just being specialized ways to state a limited class
> of transformations (e.g., usually before-and-after function-call
> insertions) and a source-to-source transformation system can easily
> implement them. Just like AOP and *not* running a weaver,
> if you use a source-to-source transformation system, application
> of the instrumenter is completely optional
> So the source-to-source systems can do AOP, and a lot
> more.[/color]
Yes. But I never argued against it. I even stated that source-to-source
systems are more powerful and can do AOP (at least source code level
AOP ), and a lot more.
[color=blue]
> And you aren't limited to "before and after"
> function advice. Your examples included requirements
> to count accesses to specific classes; this implies the need
> the instrument every class access. With source to source
> transforms, you can instrument any combination
> of events (e.g., "count the number of times procedures
> are call with the same value in the first and 3rd parameters").[/color]
In principle, AOP is not limited only to "before and after". You can
instrument every class. You can even instrument on the object instance
level. Have a look at AspectWerkz, really. You won't regret it.
Actually, some ideas (e.g. pointcut specification languages and
semantics) could be probably even interesting and useful also for
source-to-source tools. But I generally agree that some very complex
static(!!!) conditions can be probably more naturally expressed by
means of source-to-source transoformers. At the same time, some of the
comlex dynamic conditions (e.g. where the applicability of the
interceptor is dependent on the dynamic state and previous behavior of
the system) are likely to be easier to express by means of run-time
based AOP engines.
[color=blue][color=green]
> > When AOP is applied at the source level, then it is probably less
> > powerful than some general purpose source-to-source transformation
> > tools. But the nice thing about Java AOP tools and some others is that
> > you can apply the same transformations at the run-time or deployment
> > time.[/color]
>
> At two costs. 1) You have to apply it at runtime; this doesn't help
> performance.
> Many applications have (even soft) realtime requirements, and
> instrumentation
> overhead (especially runtime, unoptimized) can sometimes prevent
> the instrumentation from being useful. Think "HeisenProbes".[/color]
Actually, Java AOP engines apply changes and than it is JITted. So, I;d
not call it unoptimized. But I agree that a static call is probably
still cheaper.
[color=blue]
> With source-to-source, you can often instrument special cases
> specially, reducing overhead. SD's test coverage and profiling
> instrumenters, based on DMS, generate probe code that typically compiles
> to a single machine instruction.[/color]
No doubts, that if you have more control, you can place you
instrumentation more precisely, resulting in less overhead.
[color=blue]
> 2) You have generally have a "neat" object code (such as the Java JVM).
> (This is a weak substitute for actually having the source).
> That doesn't apply to C++, the topic of this newsgroup, since C++ codes
> are generally machine object codes.[/color]
You are absolutely right. We are drifting a bit from C++ into the AOP,
Java and compiler-related waters. May be we should move this part of
the thread there, so that moderators and other readers do not get
annoyed? ;)
[color=blue]
> (In another thread, I see
> you are exploring the Sun tool, DTrace. Sounds like a great tool.
> If you are running Solaris. Most people aren't.[/color]
There are some plans to support it on BSD and Linux systems.
The approach itself is not very Solaris based. But it is probaly a big
code base, so porting will take some time...
[color=blue]
> How does it capture trace data? I was under the impression that the OS was
> involved).[/color]
As far as I understand, they patch the binray image of a process in
memory. They are able to patch entries and exits from functions in
shared libs and system calls.
Additionally, they have some special providers in the OS kernel and
some important subsystems (IO, process mgmt, memory management, etc).
These providers have some explicit Dtrace-tracepoints in their code.
They are switched off by default.
So, when you use dtrace tool it activates (and eventually inserts) the
tracepoints according to the patterns you give. A pattern specify
certain event and can say something like "every time when a given
function in a given process is called and the first parameter is like
this, do the following actions". Have a look at the dtrace docs for
more information.
[color=blue][color=green]
> >No special builds are required. No access to the sources is required.[/color]
>
> Eh? How can you possibly code a AOP before-piece of advice without
> knowing something about the function name which is the target
> of the before-advice?
> You're clearly looking at the source code.
> So you must have access to it.[/color]
Well, first of all there are not only before-advices. Secondly, of
course you have to know the name of the function. But this is not the
same as having the source code, or? In Java, you can obtain enough
information from the compiled class. But even in the case of C or C++,
if you have a binray, you can still see function names of external
functions (just use nm objectfile.o) and if the binary is not stripped
then you can also see the names of all the internal functions. All that
without access to the real source code. It is enough to have _some
artifacts_ that are preserved from source-code after compilation (e.g.
function names, etc). Of course, if you want to to something that is
very deeply based on the logic of the original function being
intercepted than you probably need access to the sources. But in this
case, AOP is not a right tool for a task. AOP is for orthogonal
features, which is not the case in such scenarios.
[color=blue][color=green]
> > This gives a lot of freedom. And this flexibility compared to
> > the source-to-source transformation comes partly due to the fact that
> > AOP is _less powerful_ and more restricted with reagrd to source code.
> > Basically, it restricts itself not to the syntactic constructs, but to
> > the semantic constucts and artifacts that are still present even in the
> > compiled form, e.g. to the classes, methods, fields, function
> > invocations, etc. Exactly this fact gives the opportunity to apply
> > aspects even at the run-time.[/color]
>
> The "semantic constructs" require you have *full access* to
> the "source code" and the implied langauge semantics.
> With DMS, we simulate access to the full source with
> by providing a source-code isomorphic abstract syntax
> tree. Implied langauge semantics are just that; this
> information often has to be encoded in auxiliary
> analyses and custom transforms that take such semantics
> into account.[/color]
Depends on how much semantics you wish to use. With Java you have
access to the classes, objects and their fields and functions. This is
the granularity you can use for instrumentation. All that does not
require any source code and can be done at the startup-time/run-time,
since it is preserved in the compiled classes.
If you wish to instrument some smaller constructs, e.g. conditional
statements or something like that, then you cannot do it with AOP. Then
you really need a source code and some source-to-source tools.
[color=blue]
> AFAIK, AOP's "less powerful" facilities restrict the
> places where advice can be inserted, and often has
> the "single-point-of-insertion" phenomenon, which makes
> it extremely difficult to collect data whose triggering events
> are at two widely separated places in the source code.[/color]
I cannot follow you here. The whole idea of AOP is that you use
pointcuts to describe a set of places where an advice should be
inserted. And then the AOP engines applies the advice at those places.
Such pointcuts can be defined with a granularity I explained before.
Additionally, annotations in the source code can be (optionally!) used
to make it even more precise. This was introduced in Java in the latest
JDK 1.5, where Sun has introduced the annotation concept. Annotations
are preserved in the classes. So, still no need for source code at the
time, where you apply the advice.
[color=blue][color=green]
> > Now, if AOP means are not good enough for your use-case and cannot
> > capture the abstractions and statistics you'd like to have, than
> > source-to-source transformation tools come into play and can be very
> > useful.[/color]
>
> I'll agree that to the extent that AOP will allow you instrument
> your code, you can use it. So far, AOP only seems to be available
> for Java.[/color]
And .Net, which is conceptually very close to Java.
[color=blue]
> (Yes, I'm familiar with AspectC++; that's a research
> implementation, and I don't think it handles the full C++ langauge).[/color]
I'm also not quite sure about the maturity of this project.
[color=blue]
> So in practice, there are no AOP tools for C++.[/color]
And that is a pity. I hope the situation will improve.
How about releasing a (preferably free) AOP product for C++ ? :-} You
have already everything you might need for it in place. You just need
to make your system less powerful and looking more like AOP. And
reducing the system functionality is always easy :)
[color=blue]
> DMS exists, is production with 6 dialects of C++,
> and has been used for astonishingly hard transformational tasks.[/color]
There are no doubts that you product can do it. But it is a commercial
product and probably not very cheap, or? Now, almost all AOP tools and
also many source-to-source tools are free, though they are probably not
very powerful compared to your tools. So, many developers would still
use not-so-advanced AspectC++ or something else.
You argued that not everybody uses Dtrace and Solaris,even though they
are free now. True. But I'd argue that evem less people use DMS
products. And for open-source projects, e.g. Mozilla, it is a good
style to use mostly open-source tools, so that everybody has the
ability to work on the project and has access to all the required
tools. Therefore commericial products are not very popular for these
kinds of projects.
(Don't get me wrong. I'm not a purist of the open-source approach. I
work for a very big corporation and use commercial development tools on
every day basis with a great success. But for open-source projects,
usage of open-source development tools is almost a must.)
And again, you speak about some "astonishingly hard transformational
tasks". What I need as well as many other performance testers just does
not falll into this category, in my opinion. It is just overkill for
many simpler and less astonishing tasks.
[color=blue][color=green][color=darkred]
> > >>> > I guess, it is possible to instrument any C++ application
> > >>> > for a statistics collection using these tools. But again, this would
> > >>> > introduce some changes at the source level, which can be[/color]
> > considered as a drawback in some situations.[color=darkred]
> > >
> > >> I don't see it that way. If you can easily instrument your code,
> > >> you can instrument as you desire, collect your data, and throw
> > >> the instrumentated code away. A really good instrumentation tool
> > >> knows the
> > >> complete language,
> > >> can insert arbitrary probes, and can optimize probe insertion in
> > >> appropriate circumstances to minimize the Heisenprobe effect.
> > >> Where's the disadvantage?[/color]
> >
> > Main disadvantage is that you can not do it at run-time or deployment
> > time. Sometimes you do not have access to the source code, e.g. if you
> > have a 3rd party library in a binary form.[/color]
>
> Yep. That's a problem with source-to-source. However,
> how can you do you do intrusive instrumentation on such code without having
> its source? (Yes, I can see how you can trap calls to and
> from library routines in binary code. How can you trap accesses
> to data, when you don't know its name or structure?)[/color]
OK. First of all, I was speaking mainly about Java AOP tools. There you
can trap all reads or writes even at the single field level!
When we speak about C/C++, trapping access to the data selectively
(e.g. based on the type of the variables) is certainly not possible
without some support from the compiler (that should have a
corresponding support) or instrumentation at the source level (this is
where DMS could come into play). Of course, it is possible to trap
every memory read or write access, though it is not as precise as a
selective trapping. This is exactly what tolls like Pruify or Valgrind
do.
BTW, GCC already has option "-finstrument-functions" (Generate
instrumentation calls for entry and exit to functions.) Taking this
idea forward, one can imagine that it could also instrument access to
the data structures. The difference from source-to-source approach is
that it does not affect sources in any form.
[color=blue][color=green]
> >Doing it at run-time is less
> > intrusive, since it exists only during the execution of the process.
> > Binary image of a program is not affected at all.[/color]
>
> In some simple cases, yes, if the JVM has hooks in function
> call logic. But in general if you are doing arbitrary
> runtime instrumentation, how does
> the "binary" not get affected?
> Either you have byte-code editors
> for probe insertion, or somebody painfully writes reflection
> code *before* the code was compiled. That reflection
> code wasn't part of the application, but does end up in the
> binary.[/color]
I meant binary executable file code on the filesystem. An AOP engine
changes and patches the running binary image in memory. In case of
Java, it indeed uses byte-code editing libraries and modifies the
loaded classes on the fly. With Java, you can do it pretty easily and
efficiently. Of course, this exists only during the run-time. May be it
is also somehow possible to save a modified version in a persistent
store, so that next time you don't need to patch it again.
[color=blue]
> For C++ object code, in general I think you
> have to build binary code patchers, which does affect
> the binary. And it gets into the problem of distinguishing
> code from data. It isn't good when an object code
> editor patches data bytes by inserting machine code.[/color]
Yes. Dtrace does patch the running image of the program in memory. But
what is a problem in distinguishing code from data? Code sections (i.e.
,text) are usually marked as those either in the ELF/PE executable or
even in memory using special memory protection mechanisms. So, this is
not such a big problem. If you mean that code can be mixed with data, I
agree. It could be an issue. But usually standard prologue and epilogue
code sequences are well known for a given platform, so they are not a
problem. Now, if you wish to patch in the middle of the function then
you have a problem! Doing so at the binary level at run-time could
require a very complex analysis (actually disassembly) and probably not
possible in general case, unless you as a developer explicitly mark
such places in you source code. So, in this case you really go for
source-to-source transformations or try extend AOP to support join
point in the middle of the functions.
[color=blue][color=green]
> >And as a developer
> > using a debugger, you do not see any code that you have not written.[/color]
>
> So, are you debugging while you are collecting instrumentation?
> Yes, that could be a problem. But I don't see why this is different
> with AspectJ, can you explain why?[/color]
Good point. This is not different from AspectJ at all, at least from
its previous versions that existed before they merged it with
AspectWerkz. AspectJ is source-to-source tool with limitations of AOP
approach. But most of the best Java AOP engines do not work at the
source level. They work at the run-time. Again, have a look at
AspectWerks or JBoss AOP. Of course with "invisible" patches introduced
by these tools you have another kind of problem. Something goes wrong
in the advice code, but you as a developer don't see it ;) But this is
another issue. Depends on what you want.
[color=blue]
> And finally, we're back
> the fact that this discussion is about C++.[/color]
That's true.
[color=blue][color=green]
> > And since it is the same production build, even customers can do the
> > peroformance monitoring on their own under real conditions.[/color]
>
> Sampling profiling is nice for this. More intrusive instrumentation...
> I don't see how you intend to accomplish it.[/color]
Well, Dtarce does it by producing some sampling every N milliseconds.
You can configure it. No instrumentation is needed at all, as far as I
understand (or on the opposite, you could instrument every entry and
exit from any function of the process or even every machine
instruction). It just looks at the PC register and looks up which
function in the current proces it is. But of course, it is not as
precise, as it probably could be with a cleverer instrumenation.
[color=blue][color=green]
> > Doing it at run-time, especially for some languages with a virtual
> > machine concept, e.g. Java platofrm or .Net CLR, is also more
> > convenient, because than you are not programming language dependent.
> > Applications written in any language compiled into a bytecode for these
> > VMs can profit from the performance statistics collection tools. In the
> > source-to-source case, you'd need to have special front-ends for each
> > of the source languages.[/color][/color]
[color=blue]
> So, "if you can get the infrastructure, VMs are a good thing".
> The object to source-to-source here is, "you need special front ends
> for each language". First, the front ends don't need to be special
> except that they have to be able to deal with the languages.[/color]
Under special I meant a separate front-end per language. Not that they
are very special in any other sense.
[color=blue]
> Second, they have to be available.
> DMS has front ends for most popular langauages and dialects.[/color]
Yes. If you customer has it, this is very good for him. But VMs are
installed on almost any computer and are usually free and standardized.
One could argue that there will be more tools for this kind of
infrastructure. And of course the argumentation about free/commercial
is still applicable here.
[color=blue][color=green]
> > I personally also think that applying changes at the source level to
> > huge code bases like Mozilla is not the best idea, specially if you do
> > not own the code and just want to check some properties of it. Doing it
> > by source-to-source transformation may be too "heavy weight".[/color]
>
> Well, if you can get the instrumentation by some other means,
> I suppose. So, what specific instrumentation method do you
> propose for Mozilla? DMS and its source-to-source transformation
> facility can do this now. (See below paper :)[/color]
Looking at what the Mozilla developers use now
(
http://www.mozilla.org/performance/tools.html), almost anything would
be a win :-) :-)
Now seriously, at the moment the following methods are used:
- different malloc debug libraries (dmalloc, Boehm GC)
- some parts of the code are instrumented by hand to collect some
statistics about memory allocation, etc
- It is theoretically possible to use Pruify, if you have it
Since I'm not a Mozilla contributor (I'm just starting to look into
Mozilla code), and not a great expert in the Mozilla code base,
consider the following as my personal opinion based on the reading of
the documentation and analysis of the code:
The lowest layer of the project (NSPR - portable runtime) is
implemented in C++ and they use reference counting for grabage
collection. Such concepts like memory areans are used for memory
allocation at this layer. Higher layers are implemented in JavaScript
and XUL. JavaScript uses mark&sweep grabage collector for its own
objects management. Higher levels can access C++ objects from the lower
layers using special proxy objects exposed by the lower layers.
A problem is that there are some circular dependencies in the objects
graph between lower and higher layers. As a result, objects in such
cycles are not collected by a reference counting GC and memory is
leaked.
Another problem is the way how memory allocation is done at C++ layer.
It can and should be probably greatly improved. But this requires a
thorough analysis of its behavior today. That is one of the reasons, I
started this thread.
[color=blue][color=green][color=darkred]
> > >> The DMS Software Reengineering Toolkit can (and is) be used
> > >> for this effect. The page,
> > >>
http://www.semanticdesigns.com/Produ...ers/index.html
> > >> has a white paper on how to insert a particular type of code coverage
> > >> probe; we do this for many langauges including C++.
> > >> Changing probe styles isn't particularly hard, and many
> > >> of the this of data collection activities you described could
> > >> easily be implemented this way.[/color]
> >
> > I've read the paper. Looks nice. And it can do some things that AOP
> > cannot do, i.e. instrumenting branches. But due to the reasons
> > described before, I still think source-to-source is an overkill for
> > just a performance statistics collection.[/color]
>
> Depends on what you mean by "performance statistics". For simple
> "how many times did this function get executed" statistics,
> I might agree with you. You started this thread with
> requirements for all kinds of data collection, which I don't
> think fit this simple model.[/color]
Still, for the most requirements I had, AOP (especially Java-like) can
do the job. For something like code coverage - probably not, since
separate statements are not handled by typical AOP tools. You product
could do it easily, I guess.
[color=blue][color=green]
> > But for very specialized and
> > more complicated kinds of instrumentations it can be a way to go.[/color]
> We agree :-}[/color]
Good that we agree at least on something common :-}
BR,
Roman
