473,473 Members | 1,975 Online
Bytes | Software Development & Data Engineering Community
Create Post

Home Posts Topics Members FAQ

Precision of Double->Wrong calculations

Hello all,

we just got a little problem with the correctness of VB.Net.

What would u expect as the result of
4.35 * 100.0

435.0 u think ?

well, VB.Net tells us, that the result of this
calculation will be:

4.35*100.0 434.99999999999994 Double

Who is right here, and how can we avoid these kind of miscalculations.
Using Decimal instead of Double?
Thanks for reading and answering this post.
Thomas

Nov 21 '05 #1
2 4322
"Th.Krebs" <kl****@hotmail.com> schrieb:
What would u expect as the result of
4.35 * 100.0

435.0 u think ?

well, VB.Net tells us, that the result of this
calculation will be:

4.35*100.0 434.99999999999994 Double

Who is right here, and how can we avoid these kind of miscalculations.
Using Decimal instead of Double?


For the reasons, see:

<URL:http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<URL:http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html>
<URL:http://support.microsoft.com/?scid=kb;[LN];42980>

MSDN on 'Decimal':

---
The 'Decimal' value type represents 'Decimal' numbers ranging from positive
79,228,162,514,264,337,593,543,950,335 to negative
79,228,162,514,264,337,593,543,950,335. The 'Decimal' value type is
appropriate for financial calculations requiring large numbers of
significant integral and fractional digits and no round-off errors.
---

--
M S Herfried K. Wagner
M V P <URL:http://dotnet.mvps.org/>
V B <URL:http://dotnet.mvps.org/dotnet/faqs/>

Nov 21 '05 #2
Thomas,
I would expect something close to 434.99999999999994...

In addition to the links that Herfried gave see:

http://www.yoda.arachsys.com/csharp/floatingpoint.html
http://www.yoda.arachsys.com/csharp/decimal.html

Hope this helps
Jay
"Th.Krebs" <kl****@hotmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
Hello all,

we just got a little problem with the correctness of VB.Net.

What would u expect as the result of
4.35 * 100.0

435.0 u think ?

well, VB.Net tells us, that the result of this
calculation will be:

4.35*100.0 434.99999999999994 Double

Who is right here, and how can we avoid these kind of miscalculations.
Using Decimal instead of Double?
Thanks for reading and answering this post.
Thomas

Nov 21 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
by: Philipp | last post by:
Hello (not sure this is the right forum for that question so please redirect me if necessary) How can I know how many double values are available between 0 and 1? On my machine (pentium 3) I get...
5
by: buddyletts | last post by:
Below is some test code: #include <iostream> #include <cmath> using namespace std; int main() { double P = -1943.6650940829325; double R = 3.6683591763449312;
19
by: Allagappan | last post by:
Hello everyone, Just check this code: #include<stdio.h> int main() { double a,b; scanf("%lf",&a); scanf("%lf",&b); printf("\nNumber a: %0.16lf",a);
15
by: michael.mcgarry | last post by:
Hi, I have a question about floating point precision in C. What is the minimum distinguishable difference between 2 floating point numbers? Does this differ for various computers? Is this...
15
by: giff | last post by:
Hi all, I have a doubt, I'll try to expose it to you as clearly as I can, maybe it is not completely in topic, sorry about that. I have a few vectors of doubles (output of some calculations)...
116
by: Dilip | last post by:
Recently in our code, I ran into a situation where were stuffing a float inside a double. The precision was extended automatically because of that. To make a long story short, this caused...
10
by: Bo Yang | last post by:
Hi, I am confused by the double type in C++. I don't know whether below is legal or possible: double PI = 3.141592675932; Now, can I get another double variable from PI with lower precision,...
8
by: Grant Edwards | last post by:
I'm pretty sure the answer is "no", but before I give up on the idea, I thought I'd ask... Is there any way to do single-precision floating point calculations in Python? I know the various...
6
by: Matthew | last post by:
Hi, I want to change the precision level of floating point variables and calculations which is done in php.ini. However the server I rent for my domain does not give me access to php.ini,...
10
by: Guillermo_Lopez | last post by:
Hello All, I am using VBA in access to perform some calculations. There is a particular sumation that is wrong (barely). this code is withing a loop. TDist = TDist + TempDist Both TDist...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
1
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
1
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...
0
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The...
1
muto222
php
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.