Division Error?

[rhl]

I just came across this weird division result with VB.NET.

2/3 = 0.66666666666666663

With VB6, I get

2/3 =0.666666666666667

Using the Windows Calculator results in
0.66666666666666666666666666666667

I'm sure I'm not the first one to notice this...what gives?

rhl

[Scott M.]

The differences are due to the math being done on different data types.

VB 6's, VB .NET's and Windows don't all have comprable data types.
"rhl" <ra******@gmail.comwrote in message
news:11*********************@z24g2000prh.googlegro ups.com...

>I just came across this weird division result with VB.NET.

2/3 = 0.66666666666666663

With VB6, I get

2/3 =0.666666666666667

Using the Windows Calculator results in
0.66666666666666666666666666666667

I'm sure I'm not the first one to notice this...what gives?

rhl

[Patrice]

Exact code ? I've got the same result than in VB6 with Debug.WriteLine(2 /
3)

Anyway, all those results are wrong and obviously you'll never be able to
display or even compute the correct result. More generally this is a well
know problem with "real" numbers.

You have several ways to workaround this depending on the problem you have
with this. You could use a better type (such as decimal) for increasing (and
possibly "guaranteed" precision), you could store this a a fraction etc...

--
Patrice
"rhl" <ra******@gmail.coma écrit dans le message de news:
11*********************@z24g2000prh.googlegroups.c om...

>I just came across this weird division result with VB.NET.

2/3 = 0.66666666666666663

With VB6, I get

2/3 =0.666666666666667

Using the Windows Calculator results in
0.66666666666666666666666666666667

I'm sure I'm not the first one to notice this...what gives?

rhl