Linux rounding error???

[Kelly Mah]

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}
------------------------------------------------

Anybody know why?

[Martin Ambuhl]

Kelly Mah wrote:

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h> There is no point in including headers you don't use.
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}
------------------------------------------------

Anybody know why?

This is yet another variation on the never-ending question: "I don't
understand floating-point arithmetic and am too lazy to check the FAQ.
Why don't floating-point numbers have infinit precision?" Try the
following on your computer:

#include <stdio.h>
#include <float.h>

int main(void)
{
double x = 0.0010;
double y = 0.420;
long double xl = 0.0010L;
long double yl = 0.420L;
float xf = 0.0010f;
float yf = 0.420f;

printf("In all cases, bogus precision is printed;\n"
"xxx_DIG + 4 is used rather than xxx_DIG\n\n");
printf("y/x = %.*g/%.*g = %.*g, (int)(y/x)= %d\n", DBL_DIG + 4, y,
DBL_DIG + 4, x, DBL_DIG + 4, y / x, (int) (y / x));
printf("yl/xl = %.*Lg/%.*Lg = %.*Lg, (int)(yl/xl)= %d\n",
LDBL_DIG + 4, yl, LDBL_DIG + 4, xl, LDBL_DIG + 4, yl / xl,
(int) (yl / xl));
printf("yf/xf = %.*g/%.*g = %.*g, (int)(yf/xf)= %d\n", FLT_DIG + 4,
yf, FLT_DIG + 4, xf, FLT_DIG + 4, yf / xf, (int) (yf / xf));
return 0;
}

In all cases, bogus precision is printed;
xxx_DIG + 4 is used rather than xxx_DIG

y/x = 0.4199999999999 999845/0.0010000000000 00000021 = 420, (int)(y/x)= 420
yl/xl = 0.4199999999999 999999827/0.0009999999999 999999999133 = 420,
(int)(yl/xl)= 420
yf/xf = 0.4199999869/0.001000000047 = 419.9999695, (int)(yf/xf)= 419

[infobahn]

Kelly Mah wrote:

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}
------------------------------------------------

Anybody know why?

Let's assume (for the purposes of illustration) that Platform X has
16 bits of precision below the point. On Platform X, the closest we
can get to 0.001 is 0.0010070800781 25 or so. In binary, this is
0.0000000001000 010. The closest we can get to 0.42 on Platform X is
0.4199981689453 125 (in binary, that is 0.0110101110000 101).

Dividing one by the other, we get 417.04545... which is considerably
less accurate than your own findings on your platforms. Now, we can
get better precision by using more bits below the point. How much
better can we get? Obviously, it depends on how many bits we use.

And there's your explanation. :-)

[CBFalconer]

Kelly Mah wrote:

I seem to be running into a rounding issue on a Linux AS2.1 box
using gcc 2.95. This program always returns 419 on Linux, but on
a Solaris machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}
------------------------------------------------

Anybody know why?

Yes. The line "result = y/x" computes 419.99999... (the cast is
pointless, and like most casts should not be present) and the
conversion to int on assignment properly truncates. Or the line
computes 420.00000000000 0000001, etc. The point is floating point
arithmetic is not exact.

You can get more consistent results by controlling rounding:

result = y/x + 0.5;

--
Chuck F (cb********@yah oo.com) (cb********@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Gregory Toomey]

Kelly Mah wrote:

Sorry if I got the wrong group.

I seem to be running into a rounding issue on a Linux AS2.1 box using
gcc 2.95. This program always returns 419 on Linux, but on a Solaris
machine, I get the expected 420 output.

------------------------------------------------
#include <math.h>
#include <stdio.h>

int main()
{
double x = 0.0010;
double y = 0.420;
int result = 0;

result = (int)(y/x);
printf("y/x is %d\n",result);
return 0;
}
------------------------------------------------

Anybody know why?

I see division, not rounding.
Learn the difference.

gtoomey