How do I make my computer unstupid?

[JAM]

I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
(0.01999999999999996) what is wrong with it?

[Herfried K. Wagner [MVP]]

"JAM" <JA*@discussions.microsoft.com> schrieb:

I tell my computer a math problem (19.98-20.00) and it gives me a wako
answer
(0.01999999999999996) what is wrong with it?

Nothing is wrong. This behavior is by design.

Binary floating point and .NET
<URL:http://www.yoda.arachsys.com/csharp/floatingpoint.html>

--
M S Herfried K. Wagner
M V P <URL:http://dotnet.mvps.org/>
V B <URL:http://classicvb.org/petition/>

[bob]

Jam,

I don' know much about arithmetic, but I can think of a lot of "stupid"
answers to your question, such as: To make your computer unstupid,
don't unfix it!

Good luck.

JAM wrote:

I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
(0.01999999999999996) what is wrong with it?

[M. Posseth]

a computer is not a human :-)

from a computers point of view the answer is completely valid

computers work with the binary system humans work with the decimal system

so to get a human expected answer you should convert the values to the
decimal system and then perform your math

(CType(19.98, Decimal) - CType(20.0, Decimal)
regards

Michel Posseth


"JAM" wrote:

I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
(0.01999999999999996) what is wrong with it?

[M. Posseth]

this is a nice article that explains the subject clearly

http://support.microsoft.com/default...b;en-us;214118

note that in VB.net we have the decimal datatype now that solves this issue
however it comes with a prize and that is resources ( speed, memory )

regards

Michel Posseth [MCP]
"JAM" wrote:

I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
(0.01999999999999996) what is wrong with it?

[Jay B. Harlow [MVP - Outlook]]

M.Posseth,
I would recommend:

Dim d As Decimal = 19.98D - 20.0D

Which causes the 19.98 & 20.0 to be Decimal literals & not Double literals.

Being Decimal literals saves you from converting the values to Decimal at
run time...

At the very least it makes the expression easier to read ;-)

--
Hope this helps
Jay B. Harlow [MVP - Outlook]
..NET Application Architect, Enthusiast, & Evangelist
T.S. Bradley - http://www.tsbradley.net
"M. Posseth" <MP******@discussions.microsoft.com> wrote in message
news:1D**********************************@microsof t.com...
|
| a computer is not a human :-)
|
| from a computers point of view the answer is completely valid
|
| computers work with the binary system humans work with the decimal system
|
| so to get a human expected answer you should convert the values to the
| decimal system and then perform your math
|
| (CType(19.98, Decimal) - CType(20.0, Decimal)
|
|
| regards
|
| Michel Posseth
|
|
|
|
| "JAM" wrote:
|
| > I tell my computer a math problem (19.98-20.00) and it gives me a wako
answer
| > (0.01999999999999996) what is wrong with it?