Fuzzy math!

[schiefaw]

Hi,

I am having a slight problem with some code. I perform some additions
and subtractions on some 2 digit decimals, multiply by -1.0 and wind up
with a number with 12 digits of precision.

? (83538.18 + 0.0 + 0.0 - 40000.0 - 0.0) * -1.0
-43538.179999999993
I have replaced all the variables with their values for this example.
Does anyone know of a problem with VB's handling of decimals? And, if
so, what can I do about it (other than just rounding it off)?

Thanks,
Tony

[Larry Lard]

sc******@computer.org wrote:

Hi,

I am having a slight problem with some code. I perform some additions
and subtractions on some 2 digit decimals
No, you perform some additions and subtractions on some *floating-point
numbers*. Floating point arithmetic is not exact.

, multiply by -1.0 and wind up with a number with 12 digits of precision.

? (83538.18 + 0.0 + 0.0 - 40000.0 - 0.0) * -1.0
-43538.179999999993
? (83538.18D + 0.0D + 0.0D - 40000.0D - 0.0D) * -1.0D
-43538.18D

(I use the deprecated type declaraction character 'D' here; CDec(...)
has the same effect)


I have replaced all the variables with their values for this example.
Does anyone know of a problem with VB's handling of decimals?
Here's one you might like:

? 0.1 + 0.1 + 0.1 - 0.3
And, if
so, what can I do about it (other than just rounding it off)?

Use fixed-point types to hold exact values.

--
Larry Lard
Replies to group please

[schiefaw]

Thanks,

I switched from doubles to decimals and that seems to work.

[Herfried K. Wagner [MVP]]

<sc******@computer.org> schrieb:

I am having a slight problem with some code. I perform some additions
and subtractions on some 2 digit decimals, multiply by -1.0 and wind up
with a number with 12 digits of precision.

? (83538.18 + 0.0 + 0.0 - 40000.0 - 0.0) * -1.0
-43538.179999999993
I have replaced all the variables with their values for this example.
Does anyone know of a problem with VB's handling of decimals? And, if
so, what can I do about it (other than just rounding it off)?

In addition to the other replies:

That's a characteristic of floating point numbers.

<URL:http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>

--
M S Herfried K. Wagner
M V P <URL:http://dotnet.mvps.org/>
V B <URL:http://classicvb.org/petition/>