Java performance

[JavaPhreak]

So I work for a firm where Java performance is highly debated (e.g. threading, garbage collection, overuse of abstraction, interprocess latency, write once, debug everywhere, debuggers for real-time apps, etc). The C and C++ applications developed in-house are used for our real-time systems (high speed prop trading systems) based on high volume/low latency mulit-cast messaging. We do not have memory, latency, debugging issues, etc with these applications, honestly they have been solid. The in-house Java applications perform like crap in semi-real time on the same platforms using Sun Java 1.5.x and 1.6.x.

With the C and C++ applications, our developers addressed security, memory management and instrumention in the design from the start. The Java applications are a different story. My question is without saying hey put Solaris 10 and Dtrace on every server, how are other people here monitoring latency, handling garbage collection, instrumenting their code, tuning, etc without using O/S specific tools and code?

All of our applications run on Red Hat Linux 2.6.x kernel, on Solaris 9 and Solaris 10 x86 the Java applications are very problematic. Dtrace is beyond the comprehension of most of the Java developers and it makes absolutely no sense to replace Linux just to use Dtrace and get poor performance in return. The hardware base is a mix of Intel Xeon and AMD Opteron dual core processors, GbE, 16 GB min, 15K RPM HDDs, EMC fiber attached storage, high end network fabric, etc.

Before I start an O/S and H/W war, the question above is related to Java not the O/S, etc :)

[JosAH]

Maybe the discussion should be about C/C++ programmers versus the Java
programmers in your company. While garbage collection can hinder true real
time applications it has relatively low impact on the performance of applications
nowadays. Maybe the real functionality of your Java applications are burried
in too many frameworks, technologies and what else.

kind regards,

Jos

[JavaPhreak]

You are correct about the many frameworks being used by various Java developers, the core messaging systems for the Java and C/C++ applications are the same products (Tibco and IBM MQ Series). In the world of C++ doing hello world using STL can use around 1,500 lines of code at compile time and I agree 'our' C/C++ developers are more aware of what design and coding shortcuts can do to the environments their applications are deployed.

Some of our lead Java developers like to find 'stuff' on the net for their projects and blindly plug it in to their code, without looking beyond simple JUnit test cases. It allows them to make short deadlines, which for some simple one-off applet may be fine. However on the server side these inefficiencies are leading to a purchasing more and bigger H/W to address the issue syndrome. This money is coming from our training and software budgets, which is hurting our teams longterm.

We have seen Java stall on mulitcore systems during full GCs using Sun Java 1.5 and 1.6. Which incurs large latency spikes, which then become slow consumer problems. My question is what methods, instrumentation and/or tools are real-time Java shops using to monitor, predict and triage applications? Either in-house techniques, Open Source or commercial options.

Before anyone thinks I'm fueling a C++ vs. Java war, I code in C/C++/Java/Perl and work in both groups here.

Thanks

[JosAH]

Maybe you should have your C/C++ developers talk with/to your Java developers.
Capable C/C++ developers are also aware of the rammifications of a garbage
collector keeping other code on a tight (speed) leash. Garbage collection isn't
a 'one size fits all' kind of thing that frees you from the responsibilities and the
awareness that memory is still a finite thing. Object pooling and all that can
keep the garbage collector from waking up and doing its dirty deeds; but it all
depends on your applications.

Before you resort to all sorts of testing, profiling, whatever tools; the wetware
between the developers' ears should know what this ugly world is all about: the
garbage collector is no free lunch ;-)

kind regards,

Jos

[gagegage]

You guys should try out JRockit. The JRockit JVM incorporates a technology called the Deterministic GC which can guarantee an upper bound worst case pause time for the garbage collector.
Performance Analysis of the WebLogic Real Time 1.0 "Trader" Application

It's also possible to profile your application for latencies so you can find out the exact code line that caused the latency, and if it was I/O or lock related.The Mission Control 3.0 Latency Analyser