473,320 Members | 1,991 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

FAQ Topic - Why does simple decimal arithmetic give strange results? (2008-10-11)

-----------------------------------------------------------------------
FAQ Topic - Why does simple decimal arithmetic give strange
results?
-----------------------------------------------------------------------

For example, ` 5 * 1.015 ` does not give exactly
` 5.075 ` and ` 0.06+0.01 ` does
not give exactly ` 0.07 ` in javascript.

ECMAScript numbers are represented in binary as IEEE-754 (IEC 559)
Doubles, with a resolution of 53 bits, giving an accuracy of
15-16 decimal digits; integers up to about ` 9e15 ` are precise, but
few decimal fractions are. Given this, arithmetic is as exact
as possible, but no more. Operations on integers are exact if
the true result and all intermediates are integers within that
range.

In particular, non-integer results should not normally be
compared for equality; and non-integer computed results
commonly need rounding; see 4.6.

http://msdn2.microsoft.com/en-us/library/7wkd9z69.aspx

http://www.merlyn.demon.co.uk/js-misc0.htm#DW4

Otherwise, use ` Math.round ` on the results of expressions which
should be of integer value.
--
Postings such as this are automatically sent once a day. Their
goal is to answer repeated questions, and to offer the content to
the community for continuous evaluation/improvement. The complete
comp.lang.javascript FAQ is at http://jibbering.com/faq/index.html.
The FAQ workers are a group of volunteers. The sendings of these
daily posts are proficiently hosted by http://www.pair.com.

Oct 10 '08 #1
2 1881
In comp.lang.javascript message <48***********************@news.sunsite.
dk>, Fri, 10 Oct 2008 23:00:02, FAQ server <ja********@dotinternet.be>
posted:
>ECMAScript numbers are represented in binary as IEEE-754 (IEC 559)
Doubles, with a resolution of 53 bits, giving an accuracy of
15-16 decimal digits; integers up to about ` 9e15 ` are precise, but
few decimal fractions are. Given this, arithmetic is as exact
as possible, but no more. Operations on integers are exact if
the true result and all intermediates are integers within that
range.
The last sentence is correct. However, the semi-literate may not
recognise the difference between the concepts expressed by "if" and
"only if".

One can say that "Operations on Numbers are exact if the true result
and all intermediates can be expressed exactly as binary fractions with
a 53-bit mantissa and an exponent in the range -X to +Y" which is also
(I think) exact and covers more, but not all, cases. One can say that
"Operations on Numbers are exact if the true result and all
intermediates can be expressed exactly as IEEE Doubles", but that won't
mean anything to the intended audience.

Can the wording be briefly improved so that naive readers will realise
that operations on numbers which are multiples of 0.5, 0,25, 0,125,
.... about 1.1102230246251565404236316680908e-16 are safe? Or is "few
decimal fractions" good enough?

--
(c) John Stockton, Surrey, UK. ?@merlyn.demon.co.uk Turnpike v6.05 MIME.
Web <URL:http://www.merlyn.demon.co.uk/- FAQish topics, acronyms, & links.
Proper <= 4-line sig. separator as above, a line exactly "-- " (SonOfRFC1036)
Do not Mail News to me. Before a reply, quote with ">" or "" (SonOfRFC1036)
Oct 11 '08 #2
Dr J R Stockton wrote:
In comp.lang.javascript message <48***********************@news.sunsite.
dk>, Fri, 10 Oct 2008 23:00:02, FAQ server <ja********@dotinternet.be>
posted:
>ECMAScript numbers are represented in binary as IEEE-754 (IEC 559)
Doubles, with a resolution of 53 bits, giving an accuracy of
15-16 decimal digits; integers up to about ` 9e15 ` are precise, but
few decimal fractions are. Given this, arithmetic is as exact
as possible, but no more. Operations on integers are exact if
the true result and all intermediates are integers within that
range.

The last sentence is correct. However, the semi-literate may not
recognise the difference between the concepts expressed by "if" and
"only if".

One can say that "Operations on Numbers are exact if the true result
and all intermediates can be expressed exactly as binary fractions with
a 53-bit mantissa and an exponent in the range -X to +Y" which is also
(I think) exact and covers more, but not all, cases. One can say that
"Operations on Numbers are exact if the true result and all
intermediates can be expressed exactly as IEEE Doubles", but that won't
mean anything to the intended audience.

Can the wording be briefly improved so that naive readers will realise
that operations on numbers which are multiples of 0.5, 0,25, 0,125,
... about 1.1102230246251565404236316680908e-16 are safe? Or is "few
decimal fractions" good enough?
There could be an example, but then that might warrant more explanation
that cannot be quickly explained.

Garrett

--
comp.lang.javascript FAQ <URL: http://jibbering.com/faq/ >
Oct 12 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Neculai Macarie | last post by:
Hi! Using Union and Order By gives strange behaviour in the following test-case: drop table if exists gallery; drop table if exists gallery_categ; # create test tables create table gallery...
2
by: FAQ server | last post by:
----------------------------------------------------------------------- FAQ Topic - Why does simple decimal arithmetic give strange results?...
1
by: FAQ server | last post by:
----------------------------------------------------------------------- FAQ Topic - Why does simple decimal arithmetic give strange results?...
1
by: FAQ server | last post by:
----------------------------------------------------------------------- FAQ Topic - Why does simple decimal arithmetic give strange results?...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.