473,692 Members | 1,963 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Javascript Floating Point Precision Problem

2 New Member
Hello everyone! I'm new here but have been programming for the web in various languages for 15 years or so. I'm certainly no "expert" but can keep myself out of trouble (or in it?) most of the time.

This particular problem has plagued me for years; it is making me very, very, old. :-( It deals with the way Javascript's method of floating point precision takes the simplest math calculations and steals a penny - in the example below, a simple 636.35 is turned into 636.34 (636.3499999999 9999 !)

I know the truncate function below is not only lame, it turns the number into a string - but that is not the issue, as you can see the problem exists before truncate gets to it, and persists even if it is taken out of the script. I won't bore you will all the various methods I've tried, the most obvious being multiply everything by 100 then divide by 100 at the end - all failed

PLEASE can someone paste and test this code, offer a solution? Just load it, hit calculate, watch the alerts. I'm turning gray by the minute. :-( THANK YOU! -- Bill

Expand|Select|Wrap|Line Numbers
  1. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
  2. <html>
  3. <head>
  4.     <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
  5.     <title>Javascript Floating Point Precision Problem</title>
  6. </head>
  7.  
  8. <body>
  9. <h2>Javascript Floating Point Precision Problem</h2>
  10. <form name="po_form" id="po_form" onSubmit="return false;">
  11. <input type="hidden" name="number_of_items" id="number_of_items" value="3">
  12. <table>
  13.  <tr><td><input type="text" name="pn_1" id="pn_1" size="15" maxlength="20" value="3H00NT"  tabindex="18"></td>
  14.      <td> <input type="text" name="description_1" id="description_1" size="18" maxlength="255" value="Fuser Roller for HP4200DT" tabindex="19"></td>
  15.      <td><input type="text" onChange="calcForm(this.form);" name="quantity_1" id="quantity_1" size="2" maxlength="12" value="1" tabindex="21"></td>
  16.      <td> $ <input type="text" onChange="calcForm(this.form);" name="price_1" id="price_1" size="6" maxlength="12" value="540.80" tabindex="22"></td>
  17.      <td> $ <input type="text" onChange="calcForm(this.form);" name="shipping_1" id="shipping_1" size="6" maxlength="12" value="0.00" tabindex="23"></td>
  18.      <td> <strong>$</strong> <input type="text" name="subtotal_1" id="subtotal_1" onFocus="blur();" size="6" maxlength="12" value="540.80" tabindex="5018"></td>
  19.  </tr>
  20.  <tr><td><input type="text" name="pn_2" id="pn_2" size="15" maxlength="20" value="3H0006"  tabindex="24"></td>
  21.   <td> <input type="text" name="description_2" id="description_2" size="18" maxlength="255" value="Feed Roller" tabindex="25"></td>
  22.   <td><input type="text" onChange="calcForm(this.form);" name="quantity_2" id="quantity_2" size="2" maxlength="12" value="1" tabindex="27"></td>
  23.   <td> $ <input type="text" onChange="calcForm(this.form);" name="price_2" id="price_2" size="6" maxlength="12" value="48.75" tabindex="28"></td>
  24.   <td> $ <input type="text" onChange="calcForm(this.form);" name="shipping_2" id="shipping_2" size="6" maxlength="12" value="0.00" tabindex="29"></td>
  25.   <td> <strong>$</strong> <input type="text" name="subtotal_2" id="subtotal_2" onFocus="blur();" size="6" maxlength="12" value="48.75" tabindex="5024"></td>
  26.  </tr>
  27.  <tr><td><input type="text" name="pn_3" id="pn_3" size="15" maxlength="20" value="3H007B"  tabindex="30"></td>
  28.    <td> <input type="text" name="description_3" id="description_3" size="18" maxlength="255" value="Roller" tabindex="31"></td>
  29.    <td><input type="text" onChange="calcForm(this.form);" name="quantity_3" id="quantity_3" size="2" maxlength="12" value="1" tabindex="33"></td>
  30.    <td> $ <input type="text" onChange="calcForm(this.form);" name="price_3" id="price_3" size="6" maxlength="12" value="46.80" tabindex="34"></td>
  31.    <td> $ <input type="text" onChange="calcForm(this.form);" name="shipping_3" id="shipping_3" size="6" maxlength="12" value="0.00" tabindex="35"></td>
  32.    <td> <strong>$</strong> <input type="text" name="subtotal_3" id="subtotal_3" onFocus="blur();" size="6" maxlength="12" value="46.80" tabindex="5030"></td>
  33.  </tr>
  34.  <tr>
  35.   <td colspan="4">&nbsp;</td>
  36.   <td> TOTAL:</td>
  37.   <td> <strong>$</strong> <input type="text" name="po_total" id="po_total" onFocus="blur();" size="6" maxlength="12" value="636.34" tabindex="5500"> </td>
  38.  </tr>
  39.  <tr><td colspan="6"> <hr width="100%" size="1"></td></tr>
  40.  <tr>
  41.   <td><input type="submit" name="addLineItem" id="addLineItem" value="Add New Item" tabindex="2001" onClick="alert('Different function, ignore.'); return false;"></td>
  42.   <td> &nbsp; </td>
  43.   <td colspan="4">
  44.     <input type="button" id="recalcButton" onClick="calcForm(this.form);" value="Recalculate Total" tabindex="2002">
  45.     <input type="submit" name="submitButton" id="submitButton" onClick="checkForm(this.form); return false;" value="Submit PO" tabindex="2003">
  46.    </td>
  47. </tr>
  48. </table>
  49. </form>
  50.  
  51.        <script type="text/javascript">
  52.  
  53.             function calcForm (form) {
  54.  
  55.                    var subtotal=quan=price=ship=total=0;
  56.                    var sub,cancelled;
  57.                    var tot       = document.getElementById('po_total');
  58.                    var num_items = document.getElementById('number_of_items').value;
  59.  
  60.                    for (i=1;i<=num_items;i++) {
  61.                       subtotal = 0;
  62.  
  63.                       sub   = document.getElementById('subtotal_' + i);
  64.                       ship  = document.getElementById('shipping_' + i);
  65.                       q     = document.getElementById('quantity_' + i);
  66.                       p     = document.getElementById('price_' + i);
  67.                       subtotal = (q.value * p.value) + (ship.value * 1);
  68.                       total += subtotal;
  69.                       alert(subtotal + ' ' + total);
  70.                       sub.value = truncate(subtotal);
  71.  
  72.                    }
  73.                    tot.value = truncate(total);
  74.             }
  75.  
  76.             function checkForm (form) {
  77.                  calcForm(form);
  78.                  alert('various checks, then submit form.');
  79.             }
  80.  
  81.             function truncate(num) {
  82.                  string = "" + num;
  83.                  if (string.indexOf('.') == -1)
  84.                  return string + '.00';
  85.                  seperation = string.length - string.indexOf('.');
  86.                  if (seperation > 3)
  87.                  return string.substring(0,string.length-seperation+3);
  88.                  else if (seperation == 2)
  89.                  return string + '0';
  90.                  return string;
  91.             }
  92.           </script>
  93.  
  94. </body>
  95. </html>
  96.  
Aug 1 '06 #1
5 13205
iam_clint
1,208 Recognized Expert Top Contributor
This might make you mad my very simple solution if this isn't what your looking for let me know and i will give you another solution!

num.toFixed(2)
:)


Expand|Select|Wrap|Line Numbers
  1. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
  2. <html>
  3. <head>
  4.     <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
  5.     <title>Javascript Floating Point Precision Problem</title>
  6. </head>
  7.  
  8. <body>
  9. <h2>Javascript Floating Point Precision Problem</h2>
  10. <form name="po_form" id="po_form" onSubmit="return false;">
  11. <input type="hidden" name="number_of_items" id="number_of_items" value="3">
  12. <table>
  13.  <tr><td><input type="text" name="pn_1" id="pn_1" size="15" maxlength="20" value="3H00NT"  tabindex="18"></td>
  14.      <td> <input type="text" name="description_1" id="description_1" size="18" maxlength="255" value="Fuser Roller for HP4200DT" tabindex="19"></td>
  15.      <td><input type="text" onChange="calcForm(this.form);" name="quantity_1" id="quantity_1" size="2" maxlength="12" value="1" tabindex="21"></td>
  16.      <td> $ <input type="text" onChange="calcForm(this.form);" name="price_1" id="price_1" size="6" maxlength="12" value="540.80" tabindex="22"></td>
  17.      <td> $ <input type="text" onChange="calcForm(this.form);" name="shipping_1" id="shipping_1" size="6" maxlength="12" value="0.00" tabindex="23"></td>
  18.      <td> <strong>$</strong> <input type="text" name="subtotal_1" id="subtotal_1" onFocus="blur();" size="6" maxlength="12" value="540.80" tabindex="5018"></td>
  19.  </tr>
  20.  <tr><td><input type="text" name="pn_2" id="pn_2" size="15" maxlength="20" value="3H0006"  tabindex="24"></td>
  21.   <td> <input type="text" name="description_2" id="description_2" size="18" maxlength="255" value="Feed Roller" tabindex="25"></td>
  22.   <td><input type="text" onChange="calcForm(this.form);" name="quantity_2" id="quantity_2" size="2" maxlength="12" value="1" tabindex="27"></td>
  23.   <td> $ <input type="text" onChange="calcForm(this.form);" name="price_2" id="price_2" size="6" maxlength="12" value="48.75" tabindex="28"></td>
  24.   <td> $ <input type="text" onChange="calcForm(this.form);" name="shipping_2" id="shipping_2" size="6" maxlength="12" value="0.00" tabindex="29"></td>
  25.   <td> <strong>$</strong> <input type="text" name="subtotal_2" id="subtotal_2" onFocus="blur();" size="6" maxlength="12" value="48.75" tabindex="5024"></td>
  26.  </tr>
  27.  <tr><td><input type="text" name="pn_3" id="pn_3" size="15" maxlength="20" value="3H007B"  tabindex="30"></td>
  28.    <td> <input type="text" name="description_3" id="description_3" size="18" maxlength="255" value="Roller" tabindex="31"></td>
  29.    <td><input type="text" onChange="calcForm(this.form);" name="quantity_3" id="quantity_3" size="2" maxlength="12" value="1" tabindex="33"></td>
  30.    <td> $ <input type="text" onChange="calcForm(this.form);" name="price_3" id="price_3" size="6" maxlength="12" value="46.80" tabindex="34"></td>
  31.    <td> $ <input type="text" onChange="calcForm(this.form);" name="shipping_3" id="shipping_3" size="6" maxlength="12" value="0.00" tabindex="35"></td>
  32.    <td> <strong>$</strong> <input type="text" name="subtotal_3" id="subtotal_3" onFocus="blur();" size="6" maxlength="12" value="46.80" tabindex="5030"></td>
  33.  </tr>
  34.  <tr>
  35.   <td colspan="4">&nbsp;</td>
  36.   <td> TOTAL:</td>
  37.   <td> <strong>$</strong> <input type="text" name="po_total" id="po_total" onFocus="blur();" size="6" maxlength="12" value="636.34" tabindex="5500"> </td>
  38.  </tr>
  39.  <tr><td colspan="6"> <hr width="100%" size="1"></td></tr>
  40.  <tr>
  41.   <td><input type="submit" name="addLineItem" id="addLineItem" value="Add New Item" tabindex="2001" onClick="alert('Different function, ignore.'); return false;"></td>
  42.   <td> &nbsp; </td>
  43.   <td colspan="4">
  44.     <input type="button" id="recalcButton" onClick="calcForm(this.form);" value="Recalculate Total" tabindex="2002">
  45.     <input type="submit" name="submitButton" id="submitButton" onClick="checkForm(this.form); return false;" value="Submit PO" tabindex="2003">
  46.    </td>
  47. </tr>
  48. </table>
  49. </form>
  50.  
  51.        <script type="text/javascript">
  52.  
  53.             function calcForm (form) {
  54.  
  55.                    var subtotal=quan=price=ship=total=0;
  56.                    var sub,cancelled;
  57.                    var tot       = document.getElementById('po_total');
  58.                    var num_items = document.getElementById('number_of_items').value;
  59.  
  60.                    for (i=1;i<=num_items;i++) {
  61.                       subtotal = 0;
  62.  
  63.                       sub   = document.getElementById('subtotal_' + i);
  64.                       ship  = document.getElementById('shipping_' + i);
  65.                       q     = document.getElementById('quantity_' + i);
  66.                       p     = document.getElementById('price_' + i);
  67.                       subtotal = (q.value * p.value) + (ship.value * 1);
  68.                       total += subtotal;
  69.                       alert(subtotal + ' ' + total);
  70.                       sub.value = truncate(subtotal);
  71.  
  72.                    }
  73.                    tot.value = truncate(total);
  74.             }
  75.  
  76.             function checkForm (form) {
  77.                  calcForm(form);
  78.                  alert('various checks, then submit form.');
  79.             }
  80.  
  81.             function truncate(num) {
  82.                  string = num.toFixed(2);
  83.                  return string;
  84.             }
  85.           </script>
  86.  
  87. </body>
  88. </html>
  89.  
Aug 1 '06 #2
rocknbil
2 New Member
Clint, if anything on the Internet makes one mad, it's time to back away from the computer! :-) Thanks for your reply!

Funny you should mention toFixed() - I had discovered it through other means (today.) But the problem with the numbers exists long before it gets to truncate() - in fact, by using toFixed() I was able to eliminate that stupid function altogether. Hereis my revised Javascript in its entirety, notice that in the alerts it still displays 636.34999999999 9 but that toFixed() properly rounds it up to .35. WOO HOO!

Many thanks for looking, i have been fighting this Javascript "feature" off and on for years . . . . .


Expand|Select|Wrap|Line Numbers
  1.        <script type="text/javascript">
  2.  
  3.             function calcForm (form) {
  4.  
  5.                    var subtotal,q,p,ship;
  6.                    var total = 0;
  7.  
  8.                    var sub,cancelled;
  9.                    var tot       = document.getElementById('po_total');
  10.                    var num_items = document.getElementById('number_of_items').value;
  11.  
  12.                    for (i=1;i<=num_items;i++) {
  13.                       subtotal = 0;
  14.                       sub = document.getElementById('subtotal_' + i);
  15.                       ship = document.getElementById('shipping_' + i).value;
  16.                       q = document.getElementById('quantity_' + i).value;
  17.                       p = document.getElementById('price_' + i).value;
  18.  
  19.                       subtotal = (q * p) + (ship * 1);
  20.                       total += (subtotal);
  21.                       alert(subtotal + ' ' + total);
  22.                       sub.value = subtotal.toFixed(2); 
  23.  
  24.                    }
  25.                    total = total.toFixed(2);
  26.                    tot.value = total;
  27.             }
  28.  
  29.             function checkForm (form) {
  30.                  calcForm(form);
  31.                  alert('various checks, then submit form.');
  32.             }
  33.           </script>
  34.  
Aug 2 '06 #3
iam_clint
1,208 Recognized Expert Top Contributor
Yeah i didn't think about what you really said i just got the proper number to display but yes for your answer .toFixed(2) is probably the best way to fix your problem -- i didn't really worry about the alert... all you gotta do is look what i did and change the variable that the alert is coming from and do the toFixed(2) but glad you got it working if you have any more question give us a post.
Aug 2 '06 #4
Banfa
9,065 Recognized Expert Moderator Expert
This isn't actually a Javascript feature but a feature of the way the IEEE define how floating point numbers are stored in memory. Any language using this definaition suffers from this "feature", for instance C does as well and since javascript is often implemented in C so does Javascript.

The problem is that Javascript is using 32 bits of data which have 4294967296 different combinations to hold any value in the range

1.7976931348623 158e+308 to 2.2250738585072 014e–308

It does this by using a smaller presision value and using some of the bits as a exponent (this can also be done in 16 bits with a smaller range and less presision) which results in it being able to approximate any value in the range but not exactly represent them all (because in real number terms the are an infinaite number of values between any 2 given values).

As part of your calculcation clearly the internal representation of the value is going outside the available presision (15 digits for a 32 bit number but only 6 digits for a 16 digit number so perhaps Javascript uses 16 bit floating point values) and you are ending up with an approximation to the value instead of an exact value.

We all have to code round this.

Sometimes it is easier (particularly with money) when you don't really need the huge range of a floating point value to write your code so that internally it deals with cents (or pence) and only converts to dollars (pounds) for display to the user.

This results in all your calculations being performed in integer arithmatic which does not have the imprecisions of floating point arrithmatic. Of course this wont work if you are anticipating many orders greater than $42,949,672.9 :D
Aug 2 '06 #5
iam_clint
1,208 Recognized Expert Top Contributor
LOL thanks for the explanation Banfa, i wasn't completely clear on toFixed but now I am. I just knew of several times i have used it to fix the same problem so thats what i gave him.
Aug 2 '06 #6

Sign in to post your reply or Sign up for a free account.

Similar topics

4
3308
by: Dave | last post by:
Hi folks, I am trying to develop a routine that will handle sphere-sphere and sphere-triangle collisions and interactions. My aim is to develop a quake style collision engine where a player can interact with a rich 3D environment. Seem to be 90% of the way there! My problems are related to calculations where the result tends to zero (or another defined limit.) Have loads of cases where this kind of interaction occurs but this one
31
3661
by: JS | last post by:
We have the same floating point intensive C++ program that runs on Windows on Intel chip and on Sun Solaris on SPARC chips. The program reads the exactly the same input files on the two platforms. However, they generate slightly different results for floating point numbers. Are they really supposed to generate exactly the same results? I guess so because both platforms are supposed to be IEEE floating point standard (754?) compliant. ...
5
3740
by: Anton Noll | last post by:
We are using Visual Studio 2003.NET (C++) for the development of our software in the fields digital signal processing and numerical acoustics. One of our programs was working correctly if we are using the Debug-Version of the program, but it fails (or leads to false results) if we are using the Release-Version. After a long debugging session we found out, that our program was working correctly, but the floating point processing...
687
23429
by: cody | last post by:
no this is no trollposting and please don't get it wrong but iam very curious why people still use C instead of other languages especially C++. i heard people say C++ is slower than C but i can't believe that. in pieces of the application where speed really matters you can still use "normal" functions or even static methods which is basically the same. in C there arent the simplest things present like constants, each struct and enum...
5
2410
by: Steffen | last post by:
Hi, is it possible to have two fractions, which (mathematically) have the order a/b < c/d (a,b,c,d integers), but when (correctly) converted into floating point representation just have the opposite order? The idea is that the two fractions are almost identical and that the error introduced by going to floating point representation is bigger than the exact difference, but different for the two fractions such that it somehow turns...
10
2404
by: chanma | last post by:
code1:var x=0xf0000000; alert(x); output:4026531840 code2:var b=0xf<<28; alert(b); output:-268435456
1
2762
by: Shhnwz.a | last post by:
Hi, I have a problem regarding handling floating point error, specially Denormalisation error. I want to know is there any trick or technique to find bit length of the resultant before using and floating point operation. e.g before using pow(0.5,100000);
137
6657
by: mathieu.dutour | last post by:
Dear all, I want to do multiprecision floating point, i.e. I want to go beyond single precision, double precision and have quadruple precision, octuple precision and the like, and possibly with high speed. What would be the possible alternatives? Thanks for any help
39
3559
by: rembremading | last post by:
Hi all! The following piece of code has (for me) completely unexpected behaviour. (I compile it with gcc-Version 4.0.3) Something goes wrong with the integer to float conversion. Maybe somebody out there understands what happens. Essentially, when I subtract the (double) function value GRID_POINT(2) from a variable which has been assigned the same value before this gives a non-zero result and I really do not understand why.
0
8603
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9081
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8955
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8796
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8800
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
6459
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5818
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
1
2970
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2236
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.