benchmarks? java vs .net

[King Raz]

The shootout site has benchmarks comparing different languages. It
includes C# Mono vs Java but not C# .NET vs Java. So I went through
all the benchmark on the site ...

http://kingrazi.blogspot.com/2008/05...enchmarks.html

Just to keep the post on topic for my friends at comp.lang.c++, how do
I play default windows sounds with C++?

[Rudy Velthuis]

Arne Vajhøj wrote:

IOW, there are arguments for both approaches. The GC one has the big
advantage that one big cause of errors, all errors regarding memory
use, are more or less completely eliminated. But I doubt I would
call speed one of the main factors to choose a GC.

Actually GC speed is very good.

The problem people complain over is the non deterministic
aspect of it.

People also complained about messaging and the non-linear aspect of it
when they moved from DOS to Windows. I guess, to many, this is a
similar issue, i.e. they sense a loss of control. <g>
--
Rudy Velthuis http://rvelthuis.de

"We don't make mistakes, we just have happy little accidents."
-- Bob Ross, "The Joy of Painting"

[Jon Harrop]

Arne Vajhøj wrote:

Jon Harrop wrote:

>Razii wrote:

>>On Wed, 04 Jun 2008 20:34:59 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:
You are still timing Cygwin's implementation of Unix pipes which has
nothing to do with anything.
I am still waiting for you to verify and demonstrate it has any
effect.

Your inexplicably anomalous results already proved that.

For anyone with just a minimum of understanding of logic: not.

My Cygwin-free results are up to 4x faster than Razii's Cygwin-burdened
results. How else do you explain that?

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Arne Vajhøj wrote:

Or why do you think the Cygwin stuff can explain the difference in the two
tests?

According to Razii, Cygwin is the only significant difference between our
setups. From my results, we know Razii's results for .NET are up to 4x
slower than they should be. Therefore, Cygwin is the only logical
explanation.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Arne Vajhøj wrote:

Razii wrote:

>On Sat, 07 Jun 2008 19:35:08 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>>The shootout doesn't use 3 gig max memory.
How do you know that?

They list the options they use. Besides, the computer they are using,
Pentium 4, has only 512 MB ram.

Xmx is a reliable max on memory usage.

The Java program with the 512Mb limit actually uses 800Mb here.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Arne Vajhøj wrote:

Jon Harrop wrote:

>You are ignoring all of the overheads of a GC, like thread
synchronization, stack walking and limitations placed upon the code
generator required to keep the GC happy.

I would expect non-GC solutions to need more thread synchronization
than GC because it will need it many more times.

That is just more unfounded speculation.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Mon, 09 Jun 2008 14:17:13 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Firstly, your tweak did not break the program for other (previously valid)
inputs as Razii's does.

Have some decency and don't lie. My tweak doesn't break the program
for any previously valid input. You are whining since you can't tweak
..NET which is pathetically twice slower.

So here what we have now:

regexdna (.NET slower)
binarytrees (.NET twice slower).
recursive (.NET slower).

Only in partialsums .NET is twice fast but that speed comes at the
cost that results for sin and cos are inconsistent on different
hardware. In other words, .NET is not portable. Almost everything
else, .NET is slower.

[Razii]

On Mon, 9 Jun 2008 06:44:26 -0700 (PDT), "Jon Skeet [C# MVP]"
<sk***@pobox.comwrote:

>But the point is that it didn't fail for n=20, which is the test in
question.

Even with n=16, GC runs with my tweak of -Xms512m etc. Harpo is lying.
He now has zero credibility.

[Jon Skeet [C# MVP]]

On Jun 9, 4:49 pm, Razii <[e-mail]:

<snip>

So here what we have now:

regexdna (.NET slower)
binarytrees (.NET twice slower).
recursive (.NET slower).

I haven't looked at regexdna or recursive yet, but for binarytrees I
didn't see .NET being twice as slow. With both sets of tweaks, .NET
was about 50% slower than Java on my box, and by your own stats (15s
vs 25s) it's only ~66% slower. Exaggerating the figures undermines
your point rather than improving it. (The 15s vs 25s was in message
<gd********************************@4ax.com>.)

Now, on how *applicable* these benchmarks are to real world
performance, I'm entirely with Jon Harrop: they may be somewhat
interesting in and of themselves, but they're not good predictors of
what real applications will do. For real world applications, you can
rarely get a massive improvement by tweaking the garbage collector,
for instance - partly because real world applications tend to execute
a far wider range of code. Benchmarks are naturally myopic.

As I've said before in this thread (I think) performance isn't
generally a good reason to choose between Java and .NET. There are far
more important criteria.

Jon

[Razii]

On Mon, 9 Jun 2008 09:09:12 -0700 (PDT), "Jon Skeet [C# MVP]"
<sk***@pobox.comwrote:

>I haven't looked at regexdna or recursive yet, but for binarytrees I
didn't see .NET being twice as slow. With both sets of tweaks, .NET
was about 50% slower than Java on my box, and by your own stats (15s
vs 25s) it's only ~66% slower. Exaggerating the figures undermines
your point rather than improving it. (The 15s vs 25s was in message
<gd********************************@4ax.com>.)

In your message, you said with -server it was ~15s vs ~28s. That's 2
second short of twice slower.

As for recursive, .NET is 1.5 times slower:

http://shootout.alioth.debian.org/gp...ng=javaxx&id=0
http://shootout.alioth.debian.org/gp...ng=csharp&id=0
Regexdna. .NET is about 25% slower:
http://shootout.alioth.debian.org/gp...ng=javaxx&id=4

http://shootout.alioth.debian.org/gp...ng=csharp&id=0
The input file for regexdna is generated with FASTA benchmark

http://shootout.alioth.debian.org/gp...ng=javaxx&id=2

use n = 500,000 with Fasta benchmark, and the output would be the
input-file for regexdna.

[Razii]

On Mon, 9 Jun 2008 09:09:12 -0700 (PDT), "Jon Skeet [C# MVP]"
<sk***@pobox.comwrote:

>Now, on how *applicable* these benchmarks are to real world
performance

The benchmark shows that Java's GC is easier to tune for given
situation. Many options are available to tune GC

http://java.sun.com/javase/technolog.../vmoptions.jsp

[Jon Skeet [C# MVP]]

Razii <py******@gmail.comwrote:

On Mon, 9 Jun 2008 09:09:12 -0700 (PDT), "Jon Skeet [C# MVP]"
<sk***@pobox.comwrote:

Now, on how *applicable* these benchmarks are to real world
performance

The benchmark shows that Java's GC is easier to tune for given
situation. Many options are available to tune GC

http://java.sun.com/javase/technolog.../vmoptions.jsp

.... and it's rarely the case that it's worth tuning them, in my
experience. There are exceptions, of course, but my initial instinctive
reaction is to leave the defaults as they are.

--
Jon Skeet - <sk***@pobox.com>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com

[Jon Skeet [C# MVP]]

Razii <py******@gmail.comwrote:

I haven't looked at regexdna or recursive yet, but for binarytrees I
didn't see .NET being twice as slow. With both sets of tweaks, .NET
was about 50% slower than Java on my box, and by your own stats (15s
vs 25s) it's only ~66% slower. Exaggerating the figures undermines
your point rather than improving it. (The 15s vs 25s was in message
<gd********************************@4ax.com>.)

In your message, you said with -server it was ~15s vs ~28s. That's 2
second short of twice slower.

Yes, which still doesn't mean "twice as slow" - and notice your choice
to take the worst possible set of figures (mine) instead of yours.
(Despite using your own in every other case.)

However, you snipped the most important bit of my previous post:

<quote>
Now, on how *applicable* these benchmarks are to real world
performance, I'm entirely with Jon Harrop: they may be somewhat
interesting in and of themselves, but they're not good predictors of
what real applications will do. For real world applications, you can
rarely get a massive improvement by tweaking the garbage collector,
for instance - partly because real world applications tend to execute
a far wider range of code. Benchmarks are naturally myopic.

As I've said before in this thread (I think) performance isn't
generally a good reason to choose between Java and .NET. There are far
more important criteria.
</quote>

The benchmarks test small, very precise pieces of a program. Optimising
for those pieces will often perturb the performance of the rest of the
program.

As an interesting exercise, could you find the "best" *consistent* set
of JVM options to run *all* your tests under? It still won't be
representative of a real program (which doesn't tend to have
bottlenecks of this sort) but it will be more reasonable. In real life
you don't get to change the garbage collection options just for a
single piece of code - you get one set for the whole program.

I will agree that it *is* a bit of a pity that the .NET GC isn't more
tweakable, as there are certainly occasions where it could help. I
suspect it would be used in plenty of *other* situations, mind you...

--
Jon Skeet - <sk***@pobox.com>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com

[Mark Thornton]

Jon Skeet [C# MVP] wrote:

Razii <py******@gmail.comwrote:

>On Mon, 9 Jun 2008 09:09:12 -0700 (PDT), "Jon Skeet [C# MVP]"
<sk***@pobox.comwrote:

>>Now, on how *applicable* these benchmarks are to real world
performance

The benchmark shows that Java's GC is easier to tune for given
situation. Many options are available to tune GC

http://java.sun.com/javase/technolog.../vmoptions.jsp

... and it's rarely the case that it's worth tuning them, in my
experience. There are exceptions, of course, but my initial instinctive
reaction is to leave the defaults as they are.

It is worthwhile selecting one of the low pause collectors if that is
what you need. If you know you will need a 500MB heap then you can save
a bit of time by setting the minimum size to 500MB (this doesn't save as
much time as it used to).

Mark Thornton

[Lew]

Razii <py******@gmail.comwrote:

>>situation. Many options are available to tune GC

http://java.sun.com/javase/technolog.../vmoptions.jsp

Jon Skeet [C# MVP] wrote:

>... and it's rarely the case that it's worth tuning them, in my
experience. There are exceptions, of course, but my initial
instinctive reaction is to leave the defaults as they are.

Mark Thornton wrote:

It is worthwhile selecting one of the low pause collectors if that is
what you need. If you know you will need a 500MB heap then you can save
a bit of time by setting the minimum size to 500MB (this doesn't save as
much time as it used to).

Likewise it is worthwhile switching to a parallel collector in a
multiprocessor environment.

Jon's experience must be limited. Where I work they tune the GC quite
carefully for their various application suites. They definitely find it
worthwhile, given the high volumes they process.

--
Lew

[Jon Harrop]

Razii wrote:

On Mon, 09 Jun 2008 14:17:13 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>Firstly, your tweak did not break the program for other (previously valid)
inputs as Razii's does.

Have some decency and don't lie. My tweak doesn't break the program
for any previously valid input.

The program used to work for n=22 on my machine. With your tweak it no
longer works for n=22 because it runs out of heap space.

So you broke the program for previously valid input.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Mon, 09 Jun 2008 16:56:48 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>I cranked up the heap until the GC never ran. Surely we can agree that the
GC was effectively off when it never even ran?!

GC never runs if you enter n=7, even without any command-line
arguments. You are a fool. Just because GC doesn't need to run
doesn't mean it's off. When the memory is full, GC will run...

[Razii]

On Mon, 09 Jun 2008 16:56:48 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

On a different machine with a different VM the exact same
test might very well fail.

It won't fail. You have posted ZERO evidence that the test fails ony
any machine.

[Razii]

On Mon, 09 Jun 2008 16:56:48 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>In contrast, I have posted numerous reproducible counter examples
demonstrating the GC never being invoked.

You have posted no such thing other than making a fool out of yourself
in public.

[kwikius]

On Jun 10, 1:22*am, King Ripper<[e-mail]:

Yawn.

Hey Ratboy. I know youre tired mate, but now what you want to do is
get some beans. Get plenty of them and eat em. Then wash em down with
some mashed potato. Then get a hunk of salami. eat the lot. Then what
you want to do is jump up and down. Then you gotta hold it, you got
hold it as long as you can. Then when you can't hold it in any more...
Let it go bud...

You'll get a massive ripper, but dont forget to keep some tissues
handy in case you follow through and flood out your troll cave.

regards
Andy Little

[Razii]

On Tue, 10 Jun 2008 01:18:40 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>The program used to work for n=22 on my machine. With your tweak it no
longer works for n=22 because it runs out of heap space.

The program works fine with n=22 with my options. Nothing is "broken."

java -server -Xms512m -Xmx512m -XX:NewRatio=1 binarytrees 22
stretch tree of depth 23 check: -1
8388608 trees of depth 4 check: -8388608
2097152 trees of depth 6 check: -2097152
524288 trees of depth 8 check: -524288
131072 trees of depth 10 check: -131072
32768 trees of depth 12 check: -32768
8192 trees of depth 14 check: -8192
2048 trees of depth 16 check: -2048
512 trees of depth 18 check: -512
128 trees of depth 20 check: -128
32 trees of depth 22 check: -32
long lived tree of depth 22 check: -1

real 7m0.076s

In fact, the with n=22, the better argumenst are:

[Razii]

On Mon, 9 Jun 2008 17:56:43 -0700 (PDT), kwikius
<an**@servocomm.freeserve.co.ukwrote:

>
regards
Andy Little

PLONK

[Razii]

On Mon, 09 Jun 2008 20:07:46 -0500, Razii <py******@gmail.comwrote:

>On Mon, 9 Jun 2008 17:56:43 -0700 (PDT), kwikius
<an**@servocomm.freeserve.co.ukwrote:

>>
regards
Andy Little

*PLONK*

The nut has gotten more attention than he deserves (given he has
nothing interesting to say anyway).

[Razii]

On Mon, 09 Jun 2008 20:07:04 -0500, Razii <py******@gmail.comwrote:

>In fact, the with n=22, the better argumenst are:

$ time java -server -Xms1600m -Xmx1600m -XX:NewRatio=1 binarytrees 22
stretch tree of depth 23 check: -1
8388608 trees of depth 4 check: -8388608
2097152 trees of depth 6 check: -2097152
524288 trees of depth 8 check: -524288
131072 trees of depth 10 check: -131072
32768 trees of depth 12 check: -32768
8192 trees of depth 14 check: -8192
2048 trees of depth 16 check: -2048
512 trees of depth 18 check: -512
128 trees of depth 20 check: -128
32 trees of depth 22 check: -32
long lived tree of depth 22 check: -1

real 1m17.265s

in any case, nothing is "broken."

[kwikius]

On Jun 10, 2:07*am, Razii <[e-mail]:

On Mon, 9 Jun 2008 17:56:43 -0700 (PDT), kwikius

<[e-mail]:

regards
Andy Little

PLONK

Forgot to stick your follow ups back on that one didnt ya?

You stupid little twat.

regards
Andy Little

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 01:18:40 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>The program used to work for n=22 on my machine. With your tweak it no
longer works for n=22 because it runs out of heap space.

The program works fine with n=22 with my options. Nothing is "broken."

Your program is broken:

$ time java -server -Xms512m -Xmx512m -XX:NewRatio=1 binarytrees 22
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :58)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.access$000(binarytrees.java:3 9)
at binarytrees.main(binarytrees.java:19)

real 0m34.124s
user 0m24.458s
sys 0m3.112s

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Razii wrote:

On Mon, 09 Jun 2008 16:56:48 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>On a different machine with a different VM the exact same
test might very well fail.

It won't fail. You have posted ZERO evidence that the test fails ony
any machine.

Here is your program failing on my machine (again) where others succeeded:

$ time java -server -Xms512m -Xmx512m -XX:NewRatio=1 binarytrees 22
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :58)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.bottomUpTree(binarytrees.java :51)
at binarytrees$TreeNode.access$000(binarytrees.java:3 9)
at binarytrees.main(binarytrees.java:19)

real 0m34.124s
user 0m24.458s
sys 0m3.112s

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Razii wrote:

On Mon, 9 Jun 2008 09:09:12 -0700 (PDT), "Jon Skeet [C# MVP]"
<sk***@pobox.comwrote:

>>I haven't looked at regexdna or recursive yet, but for binarytrees I
didn't see .NET being twice as slow. With both sets of tweaks, .NET
was about 50% slower than Java on my box, and by your own stats (15s
vs 25s) it's only ~66% slower. Exaggerating the figures undermines
your point rather than improving it. (The 15s vs 25s was in message
<gd********************************@4ax.com>.)

In your message, you said with -server it was ~15s vs ~28s. That's 2
second short of twice slower.

I get:

n=16
Java: 2.17s
F#: 2.21s

n=20
Java: 41s
F#: 45s

n=22
Java: 737s
F#: 311s

The only significant difference is the last result where .NET is >2x faster
than Java and I had to use the original Java because your tweaked version
is broken and cannot even handle that computation.

As for recursive, .NET is 1.5 times slower:
...

I get:
F#: 3.0s
Java: 3.6s

So F#/.NET is 20% faster than Java on recursive here.

Regexdna. .NET is about 25% slower:
...

I get:

Java: 5.17s
F#: 4.08s

So .NET is 25% faster than Java here as well.

Java is still slower on revcomp and is also slower on four SciMark2
benchmarks:

F#:
FFT 337 MFLOPS .NET is 4% faster than Java
SOR 612 MFLOPS .NET is 19% faster than Java
Monte Carlo 102 MFLOPS .NET is 38% faster than Java
Sparse 451 MFLOPS
LU 757 MFLOPS .NET is 11% faster than Java
Average 452

Java:
FFT 324 MFLOPS
SOR 514 MFLOPS
Monte Carlo 74 MFLOPS
Sparse 461 MFLOPS
LU 683 MFLOPS
Average 411

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Tue, 10 Jun 2008 02:51:01 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Your program is broken:

It isn't "broken" on mine as I showed. If on your computer it doesn't
run, increase the heap-size. Nothing is "broken." C# has access to the
entire ram (and it uses over 700 MB with n=22).

[Razii]

On Tue, 10 Jun 2008 02:53:16 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Here is your program failing on my machine (again) where others succeeded:

Since you have posted other claims that were not reproducible, I will
take this claim with a grain of slat unless someone else verifies it.
The program works fine on my computer.

Second, C# has access to entire RAM. Increase Xmx and Xms to 1600m
and java version again is faster

$ time binarytrees 22 (C#)
stretch tree of depth 23 check: -1
8388608 trees of depth 4 check: -8388608
2097152 trees of depth 6 check: -2097152
524288 trees of depth 8 check: -524288
131072 trees of depth 10 check: -131072
32768 trees of depth 12 check: -32768
8192 trees of depth 14 check: -8192
2048 trees of depth 16 check: -2048
512 trees of depth 18 check: -512
128 trees of depth 20 check: -128
32 trees of depth 22 check: -32
long lived tree of depth 22 check: -1

real 2m6.072s
$ time java -server -Xms1600m -Xmx1600m -XX:NewRatio=1 binarytrees 22
stretch tree of depth 23 check: -1
8388608 trees of depth 4 check: -8388608
2097152 trees of depth 6 check: -2097152
524288 trees of depth 8 check: -524288
131072 trees of depth 10 check: -131072
32768 trees of depth 12 check: -32768
8192 trees of depth 14 check: -8192
2048 trees of depth 16 check: -2048
512 trees of depth 18 check: -512
128 trees of depth 20 check: -128
32 trees of depth 22 check: -32
long lived tree of depth 22 check: -1

real 1m22.414s

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 02:51:01 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>Your program is broken:

It isn't "broken" on mine as I showed.

Working only some of the time isn't good enough.

If on your computer it doesn't run, increase the heap-size. Nothing
is "broken." C# has access to the entire ram (and it uses over 700 MB with
n=22).

In other words you want to use different Java entrants for different inputs.
So your benchmarking methodology is broken as well as your program...

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Tue, 10 Jun 2008 02:59:42 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>The only significant difference is the last result where .NET is >2x faster
than Java and I had to use the original Java because your tweaked version
is broken and cannot even handle that computation.

Your numbers are not reproducible on my computer. They are not
reproducible on Skeet computer. .NET is slower, much slower, on my
machine. F# is slower than C#.

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 02:53:16 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>Here is your program failing on my machine (again) where others succeeded:

Since you have posted other claims that were not reproducible, I will
take this claim with a grain of slat unless someone else verifies it.
The program works fine on my computer.

Second, C# has access to entire RAM. Increase Xmx and Xms to 1600m
and java version again is faster...

So the F# solution is not only as fast as Java but it doesn't need to be
changed every time you alter the input. I think we have a clear winner...

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Tue, 10 Jun 2008 02:59:42 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>So .NET is 25% faster than Java here as well.

..NET is 25% slower on regexdna and Harpo admitted it last time in the
other thread. Note that now Harpo has changed his tune and is claiming
that .NET is faster in regexdna.

He did the same thing with binarytrees. Earlier he admitted that Java
is 2x times faster in binarytrees.

http://groups.google.sh/group/comp.l...9c011eb5833dec

"Java is only 2x faster in one benchmark (binarytrees) here and that
is not a well formed benchmark."

Later he changed his claim and started claiming .NET is faster. I
don't get these numbers, nor did John Skeet. F# is slower than C# on
..NET.

He has yet to post C# version of Mersenne Twister too, where he
claimed it's twice faster.

[Razii]

On Tue, 10 Jun 2008 03:22:02 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>So the F# solution is not only as fast as Java but it doesn't need to be
changed every time you alter the input. I think we have a clear winner...

F# not as "fast" as java. It's slower than C#, forget Java.

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 02:59:42 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>The only significant difference is the last result where .NET is >2x
faster than Java and I had to use the original Java because your tweaked
version is broken and cannot even handle that computation.

Your numbers are not reproducible on my computer. They are not
reproducible on Skeet computer. .NET is slower, much slower, on my
machine. F# is slower than C#.

Your F# results are much slower than mine even though your machine is
faster. That proves you are doing something wrong when running the F#.

I suspect you are either still using Mono instead of .NET, still have not
turned compiler optimizations on or you are still using the original Mono
code without the optimizations that I posted. Indeed, you recently cited
only the unoptimized Mono code on the shootout when much faster solutions
were already available elsewhere.

Whichever it is we know your benchmark results for F# are flawed, your
conclusions are therefore flawed, your benchmarking methodology is flawed
and your submissions randomly fail where others do not.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Tue, 10 Jun 2008 03:17:19 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>In other words you want to use different Java entrants for different inputs.
So your benchmarking methodology is broken as well as your program...

In other words, I can tune Java GC to fit the situation and make it
faster. You don't have that choice. No wonder .NET is slow.

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 02:59:42 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>So .NET is 25% faster than Java here as well.

.NET is 25% slower on regexdna and Harpo admitted it last time in the
other thread. Note that now Harpo has changed his tune and is claiming
that .NET is faster in regexdna.

Here is my optimized F# solution that is 25% faster than the fastest Java
solution:

#light

#I @"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
#r @"C:\Program Files\Microsoft Parallel Extensions Jun08
CTP\System.Threading.dll"

open System.Text.RegularExpressions

let variants = ["agggtaaa|tttaccct"; "[cgt]gggtaaa|tttaccc[acg]";
"a[act]ggtaaa|tttacc[agt]t"; "ag[act]gtaaa|tttac[agt]ct";
"agg[act]taaa|ttta[agt]cct"; "aggg[acg]aaa|ttt[cgt]ccct";
"agggt[cgt]aa|tt[acg]accct"; "agggta[cgt]a|t[acg]taccct";
"agggtaa[cgt]|[acg]ttaccct"]

let subst = ["B", "(c|g|t)"; "D", "(a|g|t)"; "H", "(a|c|t)"; "K", "(g|t)";
"M", "(a|c)"; "N", "(a|c|g|t)"; "R", "(a|g)"; "S", "(c|g)";
"V", "(a|c|g)"; "W", "(a|t)"; "Y", "(c|t)"]

let file_data =
use stream = System.Console.OpenStandardInput()
use stream = new System.IO.BufferedStream(stream)
use stream = new System.IO.StreamReader(stream)
stream.ReadToEnd()

let file_length = String.length file_data

let dna = Regex(">.*\n|\n", RegexOptions.Compiled).Replace(file_data, "")
let code_length = String.length dna

let () =
let futures =
[ for re in variants ->
System.Threading.Tasks.Future.Create(fun () ->
let mutable count = 0
let r = new Regex(re, RegexOptions.Compiled);
let mutable m = r.Match(dna)
while m.Success do
count <- count + 1
m <- m.NextMatch()
re, count) ]
for future in futures do
let re, count = future.Value
printf "%s %i\n" re count
let mutable b = dna
for re, s in subst do
b <- Regex(re, RegexOptions.Compiled).Replace(b, s)
printf "\n%i\n%i\n%i\n" file_length code_length (String.length b)

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Razii wrote:

Later he changed his claim and started claiming .NET is faster.

You tweaked Java's GC parameters so Jon Skeet tweaked .NET's GC parameters
and we all got different results.

F# is slower than C# on .NET.

You are doing something wrong to make the F# run so slowly (given that it
runs much faster on my much slower machine).

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 03:22:02 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>So the F# solution is not only as fast as Java but it doesn't need to be
changed every time you alter the input. I think we have a clear winner...

F# not as "fast" as java. It's slower than C#, forget Java.

Then you are doing something wrong. What versions of the benchmark source
code, F#, .NET and the TPL are you using and what is your compile line?

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Razii]

On Tue, 10 Jun 2008 03:30:09 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Your F# results are much slower than mine even though your machine is
faster. That proves you are doing something wrong when running the F#.

I am doing nothing wrong. My C# results are same as John Skeet and F#
(that is also running on .NET like C#) was only 2 seconds slower than
C#.

[Razii]

On Tue, 10 Jun 2008 03:41:19 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Then you are doing something wrong. What versions of the benchmark source
code, F#, .NET and the TPL are you using and what is your compile line?

F# 1.9.4.17 http://research.microsoft.com/fsharp/release.aspx

fsc -O3 binarytreesf.fs

binarytreesf.fs(23,30): warning FS0062: This construct is for
compatibility with
OCaml. The expression form 'expr.(expr)' is for use when OCaml
compatibility is
enabled. In F# code you may use 'expr.[expr]'. A type annotation may
be require
d to indicate the first expression is an array. This warning can be
disabled usi
ng '--ml-compatibility', '--no-warn 62' or '#nowarn "62"'

binarytreesf.fs(23,16): warning FS0062: This construct is for
compatibility with
OCaml. Consider using the overloaded operator 'int' instead. This
warning can b
e disabled using '--ml-compatibility', '--no-warn 62' or '#nowarn
"62"'

[Razii]

On Tue, 10 Jun 2008 03:35:18 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Here is my optimized F# solution that is 25% faster than the fastest Java
solution:

This doesn't compile

fsc -O3 regexdnaf.fs

regexdnaf.fs(4,0): error FS0191: Unable to find the file C:\Program
Files\Micros
oft Parallel Extensions Jun08
CTP\System.Threading.dll in any of
C:\Windows\Microsoft.NET\Framework\v2.0.50727\
C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5
C:\MyApps\shootout\Agressive
C:\Program Files\FSharp-1.9.4.17\bin.

Unlike yours, C# compiled fine...

csc regexdna.cs
Microsoft (R) Visual C# 2008 Compiler version 3.5.21022.8
for Microsoft (R) .NET Framework version 3.5
Copyright (C) Microsoft Corporation. All rights reserved.
$ time regexdna < reg3.txt
agggtaaa|tttaccct 178
[cgt]gggtaaa|tttaccc[acg] 624
a[act]ggtaaa|tttacc[agt]t 2120
ag[act]gtaaa|tttac[agt]ct 1444
agg[act]taaa|ttta[agt]cct 2723
aggg[acg]aaa|ttt[cgt]ccct 767
agggt[cgt]aa|tt[acg]accct 713
agggta[cgt]a|t[acg]taccct 805
agggtaa[cgt]|[acg]ttaccct 1086

25416745
25000000
33399408

real 0m25.642s
user 0m0.030s
sys 0m0.015s

$ time java -server regexdna < reg3.txt
agggtaaa|tttaccct 178
[cgt]gggtaaa|tttaccc[acg] 624
a[act]ggtaaa|tttacc[agt]t 2120
ag[act]gtaaa|tttac[agt]ct 1444
agg[act]taaa|ttta[agt]cct 2723
aggg[acg]aaa|ttt[cgt]ccct 767
agggt[cgt]aa|tt[acg]accct 713
agggta[cgt]a|t[acg]taccct 805
agggtaa[cgt]|[acg]ttaccct 1086

25416745
25000000
33399408

real 0m18.103s
user 0m0.015s
sys 0m0.000s

[Razii]

On Tue, 10 Jun 2008 03:37:38 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>You tweaked Java's GC parameters so Jon Skeet tweaked .NET's GC parameters
and we all got different results.

No, you shifted your claim before Jon Skeet's tweak. The google
archive shows that. You are just inconsistent.

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 03:30:09 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>Your F# results are much slower than mine even though your machine is
faster. That proves you are doing something wrong when running the F#.

I am doing nothing wrong.

Then why are your F# results four times slower than they should be
(extrapolating from my results)?

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 03:35:18 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>Here is my optimized F# solution that is 25% faster than the fastest Java
solution:

This doesn't compile

fsc -O3 regexdnaf.fs

regexdnaf.fs(4,0): error FS0191: Unable to find the file C:\Program
Files\Micros
oft Parallel Extensions Jun08
CTP\System.Threading.dll in any of
C:\Windows\Microsoft.NET\Framework\v2.0.50727\
C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5
C:\MyApps\shootout\Agressive
C:\Program Files\FSharp-1.9.4.17\bin.

Unlike yours, C# compiled fine...

You don't even have the necessary software installed on your machine so you
cannot possibly have been posting benchmark results for my optimized
implementations as you claimed.

Install the TPL and the use my optimized implementations.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Skeet [C# MVP]]

Lew <co****@lewscanon.com.invalidwrote:

Mark Thornton wrote:

It is worthwhile selecting one of the low pause collectors if that is
what you need. If you know you will need a 500MB heap then you can save
a bit of time by setting the minimum size to 500MB (this doesn't save as
much time as it used to).

Likewise it is worthwhile switching to a parallel collector in a
multiprocessor environment.

Jon's experience must be limited. Where I work they tune the GC quite
carefully for their various application suites. They definitely find it
worthwhile, given the high volumes they process.

Oh we most definitely tune the VM where I work too. (Heck, we tune just
about everything.) It can certainly be useful. I just think they're
more the exception than the rule. I suspect that tweaking the GC
*isn't* the exception, but getting a significant and predictable
benefit from it is. (I suspect that most companies don't do enough
measurement with accurate loads to properly judge what's an improvement
in the first place.)

Maybe my recollection of the Java defaults is inaccurate, however - I
thought they were pretty good. (In particular, much better than they
used to be in terms of the max heap size, which used to *always* have
to be turned up for any significant server app.)
On a different note, having just opened up my newsreader and found
dozens of new posts on this thread, I've discovered in myself a
complete lack of interest in pursuing all of this any further. Thanks
for the interesting bits (which were almost entirely aside from the
main argument). No doubt I'll be posting in cljp again before long.

--
Jon Skeet - <sk***@pobox.com>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com

[Razii]

On Tue, 10 Jun 2008 06:03:58 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>Then why are your F# results four times slower than they should be
(extrapolating from my results)?

If you are claiming that F# is 4 times faster than C# even though both
are running on .NET. then your results are not credible. Jon Skeet
numbers for C# were similar to mine.

[Razii]

On Tue, 10 Jun 2008 06:08:22 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>You don't even have the necessary software installed on your machine so you
cannot possibly have been posting benchmark results for my optimized
implementations as you claimed.

No, I was using your version of binarytrees and it compiled fine. It
was just slower -- slower than C# and definitely slower than Java.

fsc -O3 binarytreesf.fs

binarytreesf.fs(23,30): warning FS0062: This construct is for
compatibility with
OCaml. The expression form 'expr.(expr)' is for use when OCaml
compatibility is
enabled. In F# code you may use 'expr.[expr]'. A type annotation may
be require
d to indicate the first expression is an array. This warning can be
disabled usi
ng '--ml-compatibility', '--no-warn 62' or '#nowarn "62"'

binarytreesf.fs(23,16): warning FS0062: This construct is for
compatibility with
OCaml. Consider using the overloaded operator 'int' instead. This
warning can b
e disabled using '--ml-compatibility', '--no-warn 62' or '#nowarn
"62"'

..NET is slow here. No twist will change that.

[Lew]

Lew wrote:

>Where I work they tune the GC quite
carefully for their various application suites. They definitely find it
worthwhile, given the high volumes they process.

Jon Skeet [C# MVP] wrote:

Oh we most definitely tune the VM where I work too. (Heck, we tune just
about everything.) It can certainly be useful. I just think they're
more the exception than the rule. I suspect that tweaking the GC
*isn't* the exception, but getting a significant and predictable
benefit from it is. (I suspect that most companies don't do enough
measurement with accurate loads to properly judge what's an improvement
in the first place.)

I work on a fairly large-scale system involving several load-balanced,
multi-processor servers and associated stuff. Those ops guys do measure
outcomes, and have various tweaks that help overall. They do have
requirements amenable to tuning, such as that throughput is more important
than pause time.

Maybe my recollection of the Java defaults is inaccurate, however - I
thought they were pretty good. (In particular, much better than they
used to be in terms of the max heap size, which used to *always* have
to be turned up for any significant server app.)

Like many shops, where I am they are still on Java 1.3 in production.

I suspect that when they finally transition to 5 (just in time for its
funeral!) that the tuning will change.

On a different note, having just opened up my newsreader and found
dozens of new posts on this thread, I've discovered in myself a
complete lack of interest in pursuing all of this any further. Thanks
for the interesting bits (which were almost entirely aside from the
main argument). No doubt I'll be posting in cljp again before long.

Unusually for this recurring thread, much of the discussion has been useful,
relevant and interesting, particularly your posts, Mr. Skeet. I don't care
much about the "shootout" results, but the concomitant discussion of how to
tune Java and C# yields some fruit.

--
Lew

[Jon Harrop]

Razii wrote:

On Tue, 10 Jun 2008 06:03:58 +0100, Jon Harrop <jo*@ffconsultancy.com>
wrote:

>>Then why are your F# results four times slower than they should be
(extrapolating from my results)?

If you are claiming that F# is 4 times faster than C# even though both
are running on .NET. then your results are not credible.

For completely different programs?

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u