By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
457,887 Members | 1,003 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 457,887 IT Pros & Developers. It's quick & easy.

VB.Net Maths Bug?? Please Help.

P: n/a
Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.00999999046325684

Its not just this one either try this:

1.001 - 0.001 = 0.999999999999989

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS
Nov 22 '05 #1
Share this Question
Share on Google+
5 Replies


P: n/a
Welcome to floating point arithmetic.

Use the Decimal datatype to avoid this.

"Scott Gunn" <ne**@thebigspider.co.uk> wrote in message
news:O9****************@TK2MSFTNGP09.phx.gbl...
Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.00999999046325684

Its not just this one either try this:

1.001 - 0.001 = 0.999999999999989

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS

Nov 22 '05 #2

P: n/a
Hi Scott,

"Scott Gunn" <ne**@thebigspider.co.uk> wrote in message
news:O9****************@TK2MSFTNGP09.phx.gbl...
Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.00999999046325684

Its not just this one either try this:

1.001 - 0.001 = 0.999999999999989

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS You are seeing the expected result of an operation on two numbers that
cannot be exactly represented in the IEEE format for floating point numbers,
whether single precision or double precision. It is not a bug. Go to this
site http://docs.sun.com/source/806-3568/ncg_goldberg.html and other similar
explanations of the limitations of limited precision math.

Charles

Nov 22 '05 #3

P: n/a
Crikey...

I didn't realise maths was so complicated. This page is for Einstine!
http://docs.sun.com/source/806-3568/ncg_goldberg.html

From now on I'll use:
Dim Calc1 As Decimal = 3.53D
Dim Calc2 As Decimal = 3.52D

and cdec instead of cdbl

Many Thanks
TBS

"Scott Gunn" <ne**@thebigspider.co.uk> wrote in message
news:O9****************@TK2MSFTNGP09.phx.gbl...
Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.00999999046325684

Its not just this one either try this:

1.001 - 0.001 = 0.999999999999989

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS

Nov 22 '05 #4

P: n/a
Welcome to the world of computers!

Computers do arithmetic in binary. The numbers 0.1, 0.01, etc., have the
same problem in binary that 1/3 does in decimal: no matter how many digits
you have, you can never get them exact. (1/3 is close to 0.3333, closer to
0.33333, and so forth.)

..NET Framework provides the Decimal data type for exact calculations with
multiples of 1/10 and 1/100.
Nov 22 '05 #5

P: n/a
Scott Gunn <ne**@thebigspider.co.uk> wrote:
I didn't realise maths was so complicated. This page is for Einstine!
http://docs.sun.com/source/806-3568/ncg_goldberg.html

From now on I'll use:
Dim Calc1 As Decimal = 3.53D
Dim Calc2 As Decimal = 3.52D

and cdec instead of cdbl


Using decimal only shifts the problem around a bit.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html for another
description. It's not really that hard.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 22 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.