Double+Int16

[Doug Bell]

Hi,
I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and
1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?

Thanks

Doug

[Jon Skeet [C# MVP]]

Doug Bell <du*@bigpond.com> wrote:

I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and
1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?

Your misunderstanding of floating point numbers is faulty.

dblMins isn't actually 1.2, as that isn't representable in binary
floating point. That's just how the debugger (or whatever) is showing
it.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Doug Bell]

Thanks Jon

"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message
news:MP************************@msnews.microsoft.c om...

Doug Bell <du*@bigpond.com> wrote:

I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and 1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?

Your misunderstanding of floating point numbers is faulty.

dblMins isn't actually 1.2, as that isn't representable in binary
floating point. That's just how the debugger (or whatever) is showing
it.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too