VB.Net Maths Bug?? Please Help.

[Scott Gunn]

Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.0099999904632 5684

Its not just this one either try this:

1.001 - 0.001 = 0.9999999999999 89

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS

[Marina]

Welcome to floating point arithmetic.

Use the Decimal datatype to avoid this.

"Scott Gunn" <ne**@thebigspi der.co.uk> wrote in message
news:O9******** ********@TK2MSF TNGP09.phx.gbl. ..

Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.0099999904632 5684

Its not just this one either try this:

1.001 - 0.001 = 0.9999999999999 89

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS

[Charles D. Quarles]

Hi Scott,

"Scott Gunn" <ne**@thebigspi der.co.uk> wrote in message
news:O9******** ********@TK2MSF TNGP09.phx.gbl. ..

Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.0099999904632 5684

Its not just this one either try this:

1.001 - 0.001 = 0.9999999999999 89

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS You are seeing the expected result of an operation on two numbers that
cannot be exactly represented in the IEEE format for floating point numbers,
whether single precision or double precision. It is not a bug. Go to this
site http://docs.sun.com/source/806-3568/ncg_goldberg.html and other similar
explanations of the limitations of limited precision math.

Charles

[Scott Gunn]

Crikey...

I didn't realise maths was so complicated. This page is for Einstine!
http://docs.sun.com/source/806-3568/ncg_goldberg.html

From now on I'll use:
Dim Calc1 As Decimal = 3.53D
Dim Calc2 As Decimal = 3.52D

and cdec instead of cdbl

Many Thanks
TBS

"Scott Gunn" <ne**@thebigspi der.co.uk> wrote in message
news:O9******** ********@TK2MSF TNGP09.phx.gbl. ..

Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.0099999904632 5684

Its not just this one either try this:

1.001 - 0.001 = 0.9999999999999 89

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS

[Michael A. Covington]

Welcome to the world of computers!

Computers do arithmetic in binary. The numbers 0.1, 0.01, etc., have the
same problem in binary that 1/3 does in decimal: no matter how many digits
you have, you can never get them exact. (1/3 is close to 0.3333, closer to
0.33333, and so forth.)

..NET Framework provides the Decimal data type for exact calculations with
multiples of 1/10 and 1/100.

[Jon Skeet [C# MVP]]

Scott Gunn <ne**@thebigspi der.co.uk> wrote:

I didn't realise maths was so complicated. This page is for Einstine!
http://docs.sun.com/source/806-3568/ncg_goldberg.html

From now on I'll use:
Dim Calc1 As Decimal = 3.53D
Dim Calc2 As Decimal = 3.52D

and cdec instead of cdbl

Using decimal only shifts the problem around a bit.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html for another
description. It's not really that hard.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too