473,408 Members | 1,729 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,408 software developers and data experts.

SUM() increases precision (decimal point) ?

dlite922
1,584 Expert 1GB
I have a query with number fields that contain floats with zero to three decimal places. When I SUM even just two of them together the sum() field fills up the precision up to the float max value.

example

1933.92 + 2335.78 summed -> 4269.70007324219

Calculator says: 4269.7 so Where did the 0.00007324219 come from?

Why does SUM() fill up the entire field. (ie if it was double, it would be even higher precision). I have to use ROUND() to put some sense back into it.

I'm no math genius but when you add two non-integer real numbers you get a result that has as many decimal points as the number that had the most decmial places.

e.g. 2.0013 + 5.2 = 7.2013


Any insight or explanation will be greatly appreciated!




Dan
Jan 31 '09 #1
4 11114
dlite922
1,584 Expert 1GB
By the way, this is because my field is declared as a float.

If your field is a DECIMAL or FLOAT(12,2) this will not happen in MySQL 5 and up.

I've head this is because some numbers cannot be represented exactly in binary (computer base) vs base 10 (decimal).

I don't know exactly how and why not but Windows Calculator just says zero when I try to convert 0.5 to binary.

I thought base was just that, a base, any number can be represented in any base.




Dan
Jan 31 '09 #2
Atli
5,058 Expert 4TB
Floating point numbers are note considered 100% accurate.
I am not exactly sure of the reason for this (although, I suspect Google may help with that), but I have seen minor calculation errors like these when using them.

If you need to store your fractions with 100% accuracy (like to store monetary values), use DECIMAL.

@dlite922
As far as I know (and based on all that I know) this is not true.
All data can (and is) stored as binary on a computer. Computers run on binary, so any number that is used by a computer must therefore be convertible from and to binary.

Edit
After a little research... it turns out, some numbers (0.1 and 0.01) can not be represented *accurately* in binary. Their actual representation would be an infinite loop (similar to the decimal fractional value of 1/3), so they are actually represented as 4 counts of the loop and a 1 (in some cases).

As a result, when converting these numbers back from binary, these deliberate inaccuracies have to be taken into account, but it appears this is not always possible when using floats, which can sometimes lead to slight loss of accuracy.

Note that I didn't spend a lot of time researching this, so there may be more to it
Feb 1 '09 #3
r035198x
13,262 8TB
What every computer scientist should know about floating-point arithmetic

PS:What makes you sure your numbers are only zero to three decimal places?
Feb 3 '09 #4
dlite922
1,584 Expert 1GB
@r035198x
tl;dr; , i'll read it later.

Numbers are dollar amounts imported.





Dan
Feb 4 '09 #5

Sign in to post your reply or Sign up for a free account.

Similar topics

15
by: Ladvánszky Károly | last post by:
Entering 3.4 in Python yields 3.3999999999999999. I know it is due to the fact that 3.4 can not be precisely expressed by the powers of 2. Can the float handling rules of the underlying layers be...
4
by: Roger Leigh | last post by:
Hello, I'm writing a fixed-precision floating point class, based on the ideas in the example fixed_pt class in the "Practical C++ Programming" book by Steve Oualline (O' Reilly). This uses a...
2
by: Brian van den Broek | last post by:
Hi all, I guess it is more of a maths question than a programming one, but it involves use of the decimal module, so here goes: As a self-directed learning exercise I've been working on a...
1
by: Shreyas Kulkarni | last post by:
hi there, recently i have got a problem regarding calculation of sum of digits in a floating point or precision number. the weird behaviour of compiler/language is preventing me from calculating...
3
by: Madan | last post by:
Hi all, I had problem regarding float/double arithmetic only with + and - operations, which gives inaccurate precisions. I would like to know how the arithmetic operations are internally handled...
15
by: Donkey | last post by:
Hi, The precision of built-in date type of C is very low. Even using long double float type or double float type, we can only use 12 or 16 digits after the decimal point. What can we do if we want...
3
by: Vamsi | last post by:
Hi, I have a column defined as decimal(4,3). I get the following error when I try to insert "12.50000000000000" CLI0111E Numeric value out of range. SQLSTATE=22003 sqlstate = 22003 I even...
9
by: asaguiar | last post by:
Hi, Given the pseudocode explanation of the Kahan algorithm at http://en.wikipedia.org/wiki/Kahan_summation_algorithm, I tried to implement it in C. It is supposed to minimize the effect of base...
2
by: Tim | last post by:
Folks, Can anyone thow some clarifying light on the following? I have come across a column with the same name and same data contents defined on different tables, on some the column is defined...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.