457,887 Members | 1,003 Online Need help? Post your question and get tips & solutions from a community of 457,887 IT Pros & Developers. It's quick & easy.

 P: n/a Hello All, In VB.Net 2003 why is this the result to a simple maths calculation? 3.53 - 3.52 = 0.00999999046325684 Its not just this one either try this: 1.001 - 0.001 = 0.999999999999989 Coding example: Dim Calc1 As Double = 3.53F Dim Calc2 As Double = 3.52F Msgbox(Calc1 - Calc2) Why? Please help TBS Nov 22 '05 #1
5 Replies

 P: n/a Welcome to floating point arithmetic. Use the Decimal datatype to avoid this. "Scott Gunn" wrote in message news:O9****************@TK2MSFTNGP09.phx.gbl... Hello All, In VB.Net 2003 why is this the result to a simple maths calculation? 3.53 - 3.52 = 0.00999999046325684 Its not just this one either try this: 1.001 - 0.001 = 0.999999999999989 Coding example: Dim Calc1 As Double = 3.53F Dim Calc2 As Double = 3.52F Msgbox(Calc1 - Calc2) Why? Please help TBS Nov 22 '05 #2

 P: n/a Hi Scott, "Scott Gunn" wrote in message news:O9****************@TK2MSFTNGP09.phx.gbl... Hello All, In VB.Net 2003 why is this the result to a simple maths calculation? 3.53 - 3.52 = 0.00999999046325684 Its not just this one either try this: 1.001 - 0.001 = 0.999999999999989 Coding example: Dim Calc1 As Double = 3.53F Dim Calc2 As Double = 3.52F Msgbox(Calc1 - Calc2) Why? Please help TBS You are seeing the expected result of an operation on two numbers that cannot be exactly represented in the IEEE format for floating point numbers, whether single precision or double precision. It is not a bug. Go to this site http://docs.sun.com/source/806-3568/ncg_goldberg.html and other similar explanations of the limitations of limited precision math. Charles Nov 22 '05 #3

 P: n/a Crikey... I didn't realise maths was so complicated. This page is for Einstine! http://docs.sun.com/source/806-3568/ncg_goldberg.html From now on I'll use: Dim Calc1 As Decimal = 3.53D Dim Calc2 As Decimal = 3.52D and cdec instead of cdbl Many Thanks TBS "Scott Gunn" wrote in message news:O9****************@TK2MSFTNGP09.phx.gbl... Hello All, In VB.Net 2003 why is this the result to a simple maths calculation? 3.53 - 3.52 = 0.00999999046325684 Its not just this one either try this: 1.001 - 0.001 = 0.999999999999989 Coding example: Dim Calc1 As Double = 3.53F Dim Calc2 As Double = 3.52F Msgbox(Calc1 - Calc2) Why? Please help TBS Nov 22 '05 #4

 P: n/a Welcome to the world of computers! Computers do arithmetic in binary. The numbers 0.1, 0.01, etc., have the same problem in binary that 1/3 does in decimal: no matter how many digits you have, you can never get them exact. (1/3 is close to 0.3333, closer to 0.33333, and so forth.) ..NET Framework provides the Decimal data type for exact calculations with multiples of 1/10 and 1/100. Nov 22 '05 #5

 P: n/a Scott Gunn wrote: I didn't realise maths was so complicated. This page is for Einstine! http://docs.sun.com/source/806-3568/ncg_goldberg.html From now on I'll use: Dim Calc1 As Decimal = 3.53D Dim Calc2 As Decimal = 3.52D and cdec instead of cdbl Using decimal only shifts the problem around a bit. See http://www.pobox.com/~skeet/csharp/floatingpoint.html for another description. It's not really that hard. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Nov 22 '05 #6

### This discussion thread is closed

Replies have been disabled for this discussion. 