Strange BUG in teh Framework?

[David]

hello.
when doing the simple following computation, the value put into the variable
numMinusOne is NOT the same as what the computation is showed to be in the
Watch window!!
here is the code:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim num As Integer = CInt(Math.Floor(xSng * yInt))
Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1)

After running these 4 lines, numMinusOne will be:52383776
while the Watch window will claim that the value of :
CInt(Math.Floor(xSng * yInt) - 1) = 52383775

BTW, the Watch window is correct.

how come the numMinusOne does not get the correct value?

[Jay B. Harlow [MVP - Outlook]]

David,
I don't believe its a bug per se, its due to the inexact nature of how
floating point numbers are stored.

If you change xSng to a Double or Decimal, you get the correct result.
For a complete discussion (and explaination) of how floating point numbers
work in .NET see:

http://www.yoda.arachsys.com/csharp/floatingpoint.html

http://www.yoda.arachsys.com/csharp/decimal.html

Hope this helps
Jay
"David" <Da*********@nowhere.com> wrote in message
news:ep**************@TK2MSFTNGP10.phx.gbl...

hello.
when doing the simple following computation, the value put into the
variable
numMinusOne is NOT the same as what the computation is showed to be in the
Watch window!!
here is the code:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim num As Integer = CInt(Math.Floor(xSng * yInt))
Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1)

After running these 4 lines, numMinusOne will be:52383776
while the Watch window will claim that the value of :
CInt(Math.Floor(xSng * yInt) - 1) = 52383775

BTW, the Watch window is correct.

how come the numMinusOne does not get the correct value?

[Jon Skeet [C# MVP]]

David <Da*********@nowhere.com> wrote:

when doing the simple following computation, the value put into the variable
numMinusOne is NOT the same as what the computation is showed to be in the
Watch window!!
here is the code:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim num As Integer = CInt(Math.Floor(xSng * yInt))
Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1)

After running these 4 lines, numMinusOne will be:52383776
while the Watch window will claim that the value of :
CInt(Math.Floor(xSng * yInt) - 1) = 52383775

BTW, the Watch window is correct.

how come the numMinusOne does not get the correct value?

See http://www.pobox.com/~skeet/csharp/floatingpoint.html

You're actually calculating 6547.97216796875 * 8000 and then taking the
floor of that.

The watch window is probably doing computation in a slightly different
way - e.g. using a Double instead of a Single.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[csmba]

> You're actually calculating 6547.97216796875 * 8000 and then taking the

floor of that.

hemm... why is that? Dim xSng As Single = 6547.972
means I want to use... 6547.972, not 6547.97216796875 . are you telling me
that the best representation for 6547.972 is 6547.97216796875 ??

[csmba]

I also think that it has to do with .NET, and not a general floating point
issue. because of the following:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim d As Double = xSng * yInt
Dim s As Single = xSng * yInt

xSng * yInt should be : 52383776.0 . I still don't see how this can be any
different, even after reading the problems with floating point. in any case,
it either can or can not store 6547.972 as 6547.972.
how can it be that
d = 52383777.34375
s = 52383776.0
the runtime multiplies the numbers differently according to the left side of
the equation? this is against anything I know about order of operations.
if not, if the runtime does the same calculation, then why would the storing
of the same result give such a different number in d and in s ? such a
casting can make a small difference, not a 1.3x difference....

"csmba" <cs***@nowhere.com> wrote in message
news:uk**************@TK2MSFTNGP12.phx.gbl...

You're actually calculating 6547.97216796875 * 8000 and then taking the
floor of that.

hemm... why is that? Dim xSng As Single = 6547.972
means I want to use... 6547.972, not 6547.97216796875 . are you telling me
that the best representation for 6547.972 is 6547.97216796875 ??

[David Browne]

"csmba" <cs***@nowhere.com> wrote in message
news:%2****************@TK2MSFTNGP11.phx.gbl...

I also think that it has to do with .NET, and not a general floating point
issue. because of the following:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim d As Double = xSng * yInt
Dim s As Single = xSng * yInt

xSng * yInt should be : 52383776.0 . I still don't see how this can be
any
different, even after reading the problems with floating point. in any
case,
it either can or can not store 6547.972 as 6547.972.
how can it be that
d = 52383777.34375
s = 52383776.0
the runtime multiplies the numbers differently according to the left side
of
the equation? this is against anything I know about order of operations.

No it's not. You are forgetting about the data type conversion operations.
tRemember that you usually only have arithmetic operations defined as binary
operations taking two inputs of the same type and returning that same type.
There is no multiplication operator coded to take an Integer and a Single
and return a Double. If there were, it would probably give the right
answer.

Look at it this way. How would you compute:

Dim d as Double = xSng * yInt

There are two possible ways, and both have problems.

First you could do it like this

A: Dim d as Double = CDbl(CSng(xSng * CSng(yInt)))

Or like this

B: Dim d as Double = CDbl(xSng) * CDbl(yInt)

For some values of xSng and yInt A will be more accurate, but for some
values the result of xSng * yInt won't even fit in a Single. If you use A
there, then you won't even get an answer!

EG

Dim xSng As Single = Single.MaxValue
Dim yInt As Integer = 8000

So the compiler chooses B as the default order of operations. If you don't
like that order, force your own.

David

[Alan LeHun]

In article <#d**************@TK2MSFTNGP11.phx.gbl>, cs***@nowhere.com
says...

I also think that it has to do with .NET, and not a general floating point
issue. because of the following:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim d As Double = xSng * yInt
What happens when you do
Dim d As Double = xSng * Cdbl(yInt)

Dim s As Single = xSng * yInt

and
Dim s As Single= xSng * CSng(yInt) ?
--
Alan LeHun

[csmba]

thanks David. what you say is true... so the compiler decided to cast it to
double for the valid reason you mentioned...

so the only technical thing I guess I was missing was:
how come casting 6547.972 to double gives me a LESS accurate number? I
guessed that double is MORE accurate then single. however, a simple program
shows that all my problems are caused by the simple fact that:
Dim xSng As Single = 6547.972
dim d as double = CDbl(xSng)
--> gives that d's value is : 6547.97216796875

so that little cast from single to double causes a very significant
difference in value. I didn't expect a double to not be able to keep a value
when it comes from a single (float)...
did you know that this can happen?

Hernan.

[csmba]

as David.B pointed out, the issue is how the compiler decides to cast the
variables... it turns out that as long as it does it in Single, it works. if
it does it in Doubles, it will not.
looking into it, it all boils down to the fact that :
Dim xSng As Single = 6547.972
dim d as double = CDbl(xSng)
--> gives that d's value is : 6547.97216796875
in short, for some reason Double can not represent 6547.972, but adds that
167... to it which kills me in the calculation.
so to your question,:

Dim d As Double = xSng * Cdbl(yInt) d= 52383777.34375 Dim s As Single= xSng * CSng(yInt) ?

d=52383776.0

However, this still makes little sense, since a DOUBLE can easily save
6547.972 as in :
Dim xDbl As Double = 6547.972
check out xDbl value, it is 6547.972

so I still have no idea why this happens.

[csmba]

BTW, I just checked, and a double CAN save the value 6547.972 just fine. as
in
Dim xDbl As Double= 6547.972
check out xDbl value, it is 6547.972 and not 6547.972167...
so I still don't understand why the problem happens just when casting...

[Jon Skeet [C# MVP]]

csmba <cs***@nowhere.com> wrote:

You're actually calculating 6547.97216796875 * 8000 and then taking the
floor of that.

hemm... why is that? Dim xSng As Single = 6547.972
means I want to use... 6547.972, not 6547.97216796875 . are you telling me
that the best representation for 6547.972 is 6547.97216796875 ??

Yes. If you can find a closer floating point representation in 32 bit
IEEE 754, please provide it.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

csmba <cs***@nowhere.com> wrote:

as David.B pointed out, the issue is how the compiler decides to cast the
variables... it turns out that as long as it does it in Single, it works. if
it does it in Doubles, it will not.
looking into it, it all boils down to the fact that :
Dim xSng As Single = 6547.972
dim d as double = CDbl(xSng)
--> gives that d's value is : 6547.97216796875
in short, for some reason Double can not represent 6547.972, but adds that
167... to it which kills me in the calculation.
so to your question,:

Dim d As Double = xSng * Cdbl(yInt)

d= 52383777.34375

Dim s As Single= xSng * CSng(yInt) ?

d=52383776.0

However, this still makes little sense, since a DOUBLE can easily save
6547.972 as in :
Dim xDbl As Double = 6547.972
check out xDbl value, it is 6547.972

so I still have no idea why this happens.

Rather than take each of your posts in turn, here's what's going on:

1) 6547.972 is *not* exactly representable as either a double or a
single precision floating point value. Just because it shows it as
6547.972 in the debugger doesn't mean that's the exact value - it just
means that the debugger is doing a bit of rounding on the display.
Trust my DoubleConverter here (which is downloadable - see the page I
referred you to before). Don't rely on what the debugger or framework
displays - there's no simple way of getting the framework to display
the exact value without something like DoubleConverter.

2) The framework is free to perform calculations at a higher precision
than the operands, and that's why you're seeing the difference you are.
(Chances are that the operation is actually being carried out in 80
bits, not 64.) It's not that the single version is being more accurate
- it's being *less* accurate, but in a way that happens to be in the
opposite direction from the way that the original numbers are
inaccurately stored, so the two inaccuracies happen to cancel out.

3) If this is mucking your application up, you need to fix your
application so it doesn't rely on floating point values being
infinitely precise. They're not, and they're not going to be. There's
no bug here - what you're seeing *is* a perfectly normal set of
floating point results. If anything, you're actually being lucky - the
double result is the exact value of the multiplication of the two exact
values involved (8000 and 6547.97216796875). That certainly isn't
always the case.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[csmba]

thanks Jon, I appreciate your time. your answers are long and informative.

I understand everything you said and I am not claiming that anything should
have been different. the behavior which I could not understand was not the
fact that single and double are not a perfect representation, but the one
summarized in the next simple example:
Dim xSng As Single = 6547.972
Dim xDbl As Double = 6547.972

Dim d As Double = xDbl * CDbl(yInt)
Dim s As Single = xSng * CSng(yInt)

both d and s are calculated "correctly".
so I don't see how to explain this with your previous post: you say double
can not represent 6547.972 and neither can single (and that I was lucky that
a single gave me a right result). by why then would I get the correct answer
if I start from a double, and only face the problem of 6547.97216796875 when
it is a cast from single to double?

as for the business use in my case: I do not as you say need infinitely
precise numbers, all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is a
8000 samples/sec, this is all the precision I need. I know I can probably
treat it all as a integer counting in some small unit such as milliseconds,
but the data I get is in float so transforming it to INT might have the same
problems has I have here...

hernan.

[guy]

Further to the other replies if this is 'Really' a problem consider using a
Decimal instead of Double or Single

hth

guy

"David" wrote:

hello.
when doing the simple following computation, the value put into the variable
numMinusOne is NOT the same as what the computation is showed to be in the
Watch window!!
here is the code:
Dim xSng As Single = 6547.972
Dim yInt As Integer = 8000
Dim num As Integer = CInt(Math.Floor(xSng * yInt))
Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1)

After running these 4 lines, numMinusOne will be:52383776
while the Watch window will claim that the value of :
CInt(Math.Floor(xSng * yInt) - 1) = 52383775

BTW, the Watch window is correct.

how come the numMinusOne does not get the correct value?

[Jay B. Harlow [MVP - Outlook]]

csmba,

all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is a
8000 samples/sec, this is all the precision I need.
Consider using Decimal.

Dim x As Decimal = 6547.972D
Dim y As Integer = 8000

Dim d As Decimal = x * y

Hope this helps
Jay
"csmba" <cs***@nowhere.com> wrote in message
news:e7**************@TK2MSFTNGP11.phx.gbl... thanks Jon, I appreciate your time. your answers are long and informative.

I understand everything you said and I am not claiming that anything
should
have been different. the behavior which I could not understand was not the
fact that single and double are not a perfect representation, but the one
summarized in the next simple example:
Dim xSng As Single = 6547.972
Dim xDbl As Double = 6547.972

Dim d As Double = xDbl * CDbl(yInt)
Dim s As Single = xSng * CSng(yInt)

both d and s are calculated "correctly".
so I don't see how to explain this with your previous post: you say double
can not represent 6547.972 and neither can single (and that I was lucky
that
a single gave me a right result). by why then would I get the correct
answer
if I start from a double, and only face the problem of 6547.97216796875
when
it is a cast from single to double?

as for the business use in my case: I do not as you say need infinitely
precise numbers, all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is a
8000 samples/sec, this is all the precision I need. I know I can probably
treat it all as a integer counting in some small unit such as
milliseconds,
but the data I get is in float so transforming it to INT might have the
same
problems has I have here...

hernan.

[Jon Skeet [C# MVP]]

csmba <cs***@nowhere.com> wrote:

thanks Jon, I appreciate your time. your answers are long and informative.
So long as they're actually helping - there's nothing worse than
writing a long answer which only makes things *less* clear!
I understand everything you said and I am not claiming that anything should
have been different. the behavior which I could not understand was not the
fact that single and double are not a perfect representation, but the one
summarized in the next simple example:
Dim xSng As Single = 6547.972
Dim xDbl As Double = 6547.972

Dim d As Double = xDbl * CDbl(yInt)
Dim s As Single = xSng * CSng(yInt)

both d and s are calculated "correctly".
so I don't see how to explain this with your previous post: you say double
can not represent 6547.972 and neither can single (and that I was lucky that
a single gave me a right result). by why then would I get the correct answer
if I start from a double, and only face the problem of 6547.97216796875 when
it is a cast from single to double?
Just chance. The error in truncating the result of the multiplication
happened to go the other way from the error in the original
representation - two "wrongs" happened to make a "right".
as for the business use in my case: I do not as you say need infinitely
precise numbers, all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is a
8000 samples/sec, this is all the precision I need. I know I can probably
treat it all as a integer counting in some small unit such as milliseconds,
but the data I get is in float so transforming it to INT might have the same
problems has I have here...

How are you getting it as a float though? From a database, or parsing
it from a file, or what?

So long as you don't do very many operations, you'll certainly keep 3
significant digits correctly, but you might want to consider using
decimal instead, as others have suggested.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[csmba]

I know about Decimal, it is not the first choice becuase of performance.
can you comment on my other quesion? which was why this works:

Dim xSng As Single = 6547.972
Dim xDbl As Double = 6547.972

Dim d As Double = xDbl * CDbl(yInt)
Dim s As Single = xSng * CSng(yInt)
both d and s are calculated "correctly". acording to what you said, the double version should not have worked... but
it does.
h.

"Jay B. Harlow [MVP - Outlook]" <Ja************@msn.com> wrote in message
news:On****************@TK2MSFTNGP12.phx.gbl...

csmba,

all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is

a 8000 samples/sec, this is all the precision I need.

Consider using Decimal.

Dim x As Decimal = 6547.972D
Dim y As Integer = 8000

Dim d As Decimal = x * y

Hope this helps
Jay
"csmba" <cs***@nowhere.com> wrote in message
news:e7**************@TK2MSFTNGP11.phx.gbl...

thanks Jon, I appreciate your time. your answers are long and informative.
I understand everything you said and I am not claiming that anything
should
have been different. the behavior which I could not understand was not the fact that single and double are not a perfect representation, but the one summarized in the next simple example:
Dim xSng As Single = 6547.972
Dim xDbl As Double = 6547.972

Dim d As Double = xDbl * CDbl(yInt)
Dim s As Single = xSng * CSng(yInt)

both d and s are calculated "correctly".
so I don't see how to explain this with your previous post: you say double can not represent 6547.972 and neither can single (and that I was lucky
that
a single gave me a right result). by why then would I get the correct
answer
if I start from a double, and only face the problem of 6547.97216796875
when
it is a cast from single to double?

as for the business use in my case: I do not as you say need infinitely
precise numbers, all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. I know I can probably treat it all as a integer counting in some small unit such as
milliseconds,
but the data I get is in float so transforming it to INT might have the
same
problems has I have here...

hernan.

[csmba]

well... I get it from a COM object (BLHaaA)
it is a c++ object, that loads a wav file to memory and I need to ask it
what is the length of the file. it returns a float.

Decimal just seems soooo slow. is there a trick to use a single but tell it
to keep only 3 points precision? in that case, I will not have a problem
when I multiply by 8000.

h.

"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message
news:MP************************@msnews.microsoft.c om...

csmba <cs***@nowhere.com> wrote:

thanks Jon, I appreciate your time. your answers are long and informative.

So long as they're actually helping - there's nothing worse than
writing a long answer which only makes things *less* clear!

I understand everything you said and I am not claiming that anything

should have been different. the behavior which I could not understand was not the fact that single and double are not a perfect representation, but the one summarized in the next simple example:
Dim xSng As Single = 6547.972
Dim xDbl As Double = 6547.972

Dim d As Double = xDbl * CDbl(yInt)
Dim s As Single = xSng * CSng(yInt)

both d and s are calculated "correctly".
so I don't see how to explain this with your previous post: you say double can not represent 6547.972 and neither can single (and that I was lucky that a single gave me a right result). by why then would I get the correct answer if I start from a double, and only face the problem of 6547.97216796875 when it is a cast from single to double?

Just chance. The error in truncating the result of the multiplication
happened to go the other way from the error in the original
representation - two "wrongs" happened to make a "right".

as for the business use in my case: I do not as you say need infinitely
precise numbers, all I need actually is 3 digits to the right precision.
this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. I know I can probably treat it all as a integer counting in some small unit such as milliseconds, but the data I get is in float so transforming it to INT might have the same problems has I have here...

How are you getting it as a float though? From a database, or parsing
it from a file, or what?

So long as you don't do very many operations, you'll certainly keep 3
significant digits correctly, but you might want to consider using
decimal instead, as others have suggested.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

csmba <cs***@nowhere.com> wrote:

well... I get it from a COM object (BLHaaA)
Oh dear :( On the other hand, it means you've probably done the
floating point conversion already, so any attempts to make it
"accurate" again (by using Decimal) are likely to be fruitless. Maybe
I'm wrong though... it depends on how exactly the COM object is doing
things, I guess.
it is a c++ object, that loads a wav file to memory and I need to ask it
what is the length of the file. it returns a float.
That's the length of the file in seconds?
Decimal just seems soooo slow.
Well, it's significantly slower than float - but is this actually the
bottleneck in your app?
is there a trick to use a single but tell it to keep only 3 points
precision? in that case, I will not have a problem when I multiply by
8000.

No, but you could always round the float afterwards. I'm not sure how I
see why that's a good thing though, to be honest.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[csmba]

yes, this 6547.972 is the length in seconds of a wav file.
and because of the calculation I am using (8000 is the sampling rate), all
that is actually true is the .xxx digits. anything that the double adds to
it like the .xxx167... is just wrong. and was never there to begin with (the
number is a perfect factor of 8000).
so since you said that "actually" the number is stored as 6547.972164... if
I can ignore the digits behind the first 3 I get exactly what I want.... (I
can do it of course by multiplying by 1000, then flooring the number to a
int, then dividing by 1000 again)--> but if I end up storing it ina single,
and he again adds those random 167.. it will do no good.

Decimal is out of the question, I am doing hundred of thousands of such
calculation per file.

while you said it is luck that
dim d as double = 6547.972
works, I find it more then strange. it can not be luck since I tested many
different numbers, all work the same: they all work when set like above. but
some return "strange" values when cast from single to double.
it is as if the problem is in the casting.

as for rounding the float: what I wanted is a way to guarantee that
6547.972 * 8000 gives the right answer. it seems like a fair thing to do. I
am surprised that except for the suggestion of using decimal, I did not find
any real answer to this.
again, it actually does work, as long as the 6547.972 value is stored in a
double or in a single. it FAILS only when it is stored in a single and CAST
to a double as in CDbl(sing)*8000 where sing is a single parameter with
6547.972 as a value.

H.

well... I get it from a COM object (BLHaaA)

Oh dear :( On the other hand, it means you've probably done the
floating point conversion already, so any attempts to make it
"accurate" again (by using Decimal) are likely to be fruitless. Maybe
I'm wrong though... it depends on how exactly the COM object is doing
things, I guess.

it is a c++ object, that loads a wav file to memory and I need to ask it
what is the length of the file. it returns a float.

That's the length of the file in seconds?

Decimal just seems soooo slow.

Well, it's significantly slower than float - but is this actually the
bottleneck in your app?

is there a trick to use a single but tell it to keep only 3 points
precision? in that case, I will not have a problem when I multiply by
8000.

No, but you could always round the float afterwards. I'm not sure how I
see why that's a good thing though, to be honest.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Daniel O'Connell [C# MVP]]

"csmba" <cs***@nowhere.com> wrote in message
news:up**************@TK2MSFTNGP11.phx.gbl...

yes, this 6547.972 is the length in seconds of a wav file.
and because of the calculation I am using (8000 is the sampling rate), all
that is actually true is the .xxx digits. anything that the double adds to
it like the .xxx167... is just wrong. and was never there to begin with
(the
number is a perfect factor of 8000).

You can't ask for number of samples or parse it yourself? It seems to me you
are basically doing a sample to length to sample conversion.

I do not believe that there is ever a circumstance where a wav file actually
stores length, just sample size and then audio data length, so the data you
are getting is basically

dataSize/(sampleSize*samplesPerSecond*channels)

or something similar and getting a float out of it. Does the COM object do
anything other than parse the wav file?

[Jay B. Harlow [MVP - Outlook]]

csmba

can you comment on my other quesion? See my original post, Jon does a significantly better job then I at
explaining it. Reading Jon's & the other responses I believe the "why" has
been covered.
I know about Decimal, it is not the first choice becuase of performance. It sounds like you are worrying about the wrong end of performance. Have you
profiled the difference between calculating the value in Decimal & Double or
even Long or Integer, have you proven via Profiling (see below) in your
program that using Decimal is a performance problem in this specific
routine?

Remember the 80/20 rule. That is 80% of the execution time of your program
is spent in 20% of your code. I will optimize (worry about performance) the
20% once that 20% has been identified & proven to be a performance problem
via profiling (CLR Profiler is one profiling tool).

For info on the 80/20 rule & optimizing only the 20% see Martin Fowler's
article "Yet Another Optimization Article" at
http://martinfowler.com/ieeeSoftware...timization.pdf
Info on the CLR Profiler:
http://msdn.microsoft.com/library/de...nethowto13.asp

http://msdn.microsoft.com/library/de...anagedapps.asp

If you want "3 digits to the right precision" then I would recommend
Decimal, if you want performance then I would recommend Double. If you
really want you cake (precision) & eat it also (performance), then you may
want to consider not using Floating Point (Decimal, Single or Double), as
Jon and other have demonstrated, Single & Double floating point is inexact.
Decimal is more precise, however has this perceived performance problem. (I
say perceived, as I suspect using it in most cases will not cause any
significant performance issues).

> this number represents the length in seconds of a file, and since it is

"Length in seconds of a file", really sounds like a Performance Counter. If
I needed "length/time" I would use Performance Counters...
Hope this helps
Jay

"csmba" <cs***@nowhere.com> wrote in message
news:e4**************@TK2MSFTNGP15.phx.gbl...I know about Decimal, it is not the first choice becuase of performance.
can you comment on my other quesion? which was why this works: > Dim xSng As Single = 6547.972
> Dim xDbl As Double = 6547.972
>
> Dim d As Double = xDbl * CDbl(yInt)
> Dim s As Single = xSng * CSng(yInt)
> both d and s are calculated "correctly". acording to what you said, the double version should not have worked...
but
it does.
h.

"Jay B. Harlow [MVP - Outlook]" <Ja************@msn.com> wrote in message
news:On****************@TK2MSFTNGP12.phx.gbl...

csmba,

> all I need actually is 3 digits to the right precision.
> this number represents the length in seconds of a file, and since it is

a > 8000 samples/sec, this is all the precision I need.

Consider using Decimal.

Dim x As Decimal = 6547.972D
Dim y As Integer = 8000

Dim d As Decimal = x * y

Hope this helps
Jay

<<snip>>

[Jon Skeet [C# MVP]]

csmba <cs***@nowhere.com> wrote:

yes, this 6547.972 is the length in seconds of a wav file.
and because of the calculation I am using (8000 is the sampling rate), all
that is actually true is the .xxx digits. anything that the double adds to
it like the .xxx167... is just wrong. and was never there to begin with (the
number is a perfect factor of 8000).
so since you said that "actually" the number is stored as 6547.972164... if
I can ignore the digits behind the first 3 I get exactly what I want.... (I
can do it of course by multiplying by 1000, then flooring the number to a
int, then dividing by 1000 again)--> but if I end up storing it ina single,
and he again adds those random 167.. it will do no good.
You haven't said *why* you need it to be exact. What are you doing with
the result? If you treat it as only being exact to 3 significant
figures, it should be fine. (I would use double rather than single
though, as 6547.972 is getting close to the number of significant
digits you can rely on in a single.)

Alternatively, why not just keep it as an int, scaled up by 8000? It
really sounds like using binary floating point is definitely the wrong
way of proceeding.
Decimal is out of the question, I am doing hundred of thousands of such
calculation per file.
Hundreds of thousands of decimal operations still doesn't take very
long. Have you actually *tried* it?
while you said it is luck that
dim d as double = 6547.972
works, I find it more then strange. it can not be luck since I tested many
different numbers, all work the same: they all work when set like above. but
some return "strange" values when cast from single to double.
it is as if the problem is in the casting.
No, the problem is that the multiplication gives a result which is
accurate to the number of significant figures that single can
represent, but not accurate to the number of significant figures that
double can represent. I suspect this is still luck though, and it will
*certainly* become a problem if you deal with longer files where the
number of significant digits that single can cope with gets too small.
as for rounding the float: what I wanted is a way to guarantee that
6547.972 * 8000 gives the right answer. it seems like a fair thing to do.
Not when 6547.972 can't be exactly represented as a binary floating
point number to start with.
I am surprised that except for the suggestion of using decimal, I did not find
any real answer to this.
again, it actually does work, as long as the 6547.972 value is stored in a
double or in a single. it FAILS only when it is stored in a single and CAST
to a double as in CDbl(sing)*8000 where sing is a single parameter with
6547.972 as a value.

Yes - we've explained why. When you cast the single to a double, you're
using the less accurate representation of 6547.972, but then
multiplying that and keeping many significant digits of the result.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[csmba]

>>You haven't said *why* you need it to be exact
Well, to make all of this make sense... (this is for both Jon and Daniel
last posts)
the COM holds the wav file, and its data. my "client" wants to process a
portion of the wav file. in this case he wants to process the window 6000
sec to end of file, which he gets from the COM as a float 6547.972. now we
can say that it all boils to the fact the COM should have never return such
a float. I am not sure what are the COM alternatives... but that wasn't my
game either.

I then create a new array of samples to do computations on (calculate energy
for example), the size of that array has to be calculated from the length of
the window and the sample rate (8000). I then fill this "window" array with
values from the original array of samples returned by the COM. the COM holds
m_samples which is of correct size: 52383776.
when I fill up my window array, I have to put in its last cell the value
from the last cell on m_samples.
the "bad" calculation caused me to try and get the cell value 52383777
instead of 52383776, which is pass the last sample in the wav file...

You can't ask for number of samples or parse it yourself?

no. I, as the client have my own reasons to which windows on the file I need
to process, and they are ruled by TIME. for example, I am told that in the
last 10 seconds of the file something happens, or that between the 15 and 55
seconds of the call... so I need to make a copy of the m_samples, but only
the size of (timeEnd-TimeStart)*8000 and here is where I got the wrong
calculation (not exactly, check the first posts for the exact line of code).
Does the COM object do anything other than parse the wav file well, all I need from it is the array of samples, and the length of the call
and sample rate.
are you going to suggest I just get array, sample rate and maybe if he is so
kind the length of the array? I will then have to calculate the length in
seconds anyway... so I don't see how is that different (unless I store it in
a decimal of course).

OK, to focused this, and since everyone have been very helpful: I think we
all got the issue about double and single and precision. I also got the fact
Decimal is better (yet not supported in c++ or at least in COM).

I "solved" my problem (maybe badly I assume Jon will claim) by forcing the
calculation to be in singles. avoiding the cast to double solved my problem
in this case. I understand that if the number is much bigger, it may not
(Jon's position).
the only advice then I ask is, how should I "correctly" handle this case?

h.

as for the 80/20 rules: Jon you are absolutely right.
dataSize/(sampleSize*samplesPerSecond*channels)

or something similar and getting a float out of it. Does the COM object do
anything other than parse the wav file?

[Jon Skeet [C# MVP]]

csmba <cs***@nowhere.com> wrote:

You haven't said *why* you need it to be exact Well, to make all of this make sense... (this is for both Jon and Daniel
last posts)
the COM holds the wav file, and its data. my "client" wants to process a
portion of the wav file. in this case he wants to process the window 6000
sec to end of file, which he gets from the COM as a float 6547.972.
Do you have to get it in that form? Can you not work it out accurately
(using only integers) from other information? That would be the best
bet.

<snip>
Does the COM object do anything other than parse the wav file

well, all I need from it is the array of samples, and the length of the call
and sample rate.
are you going to suggest I just get array, sample rate and maybe if he is so
kind the length of the array? I will then have to calculate the length in
seconds anyway... so I don't see how is that different (unless I store it in
a decimal of course).

If you ask for the sample rate and the total number of samples, and you
know (as an integer) the number of seconds you want to start at, I
don't see where any floating point maths needs to be involved.

<snip>
I "solved" my problem (maybe badly I assume Jon will claim) by forcing the
calculation to be in singles. avoiding the cast to double solved my problem
in this case. I understand that if the number is much bigger, it may not
(Jon's position).
the only advice then I ask is, how should I "correctly" handle this
case?

Try to avoid using floating point at all. If you've basically got a
floating point number somewhere which is some integer / 8000, and you
want to get back to the same integer, then rather than multiplying the
floating point number by 8000, you should try to get the original
integer directly.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too