457,877 Members | 1,064 Online Need help? Post your question and get tips & solutions from a community of 457,877 IT Pros & Developers. It's quick & easy.

# Strange BUG in teh Framework?

 P: n/a hello. when doing the simple following computation, the value put into the variable numMinusOne is NOT the same as what the computation is showed to be in the Watch window!! here is the code: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim num As Integer = CInt(Math.Floor(xSng * yInt)) Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1) After running these 4 lines, numMinusOne will be:52383776 while the Watch window will claim that the value of : CInt(Math.Floor(xSng * yInt) - 1) = 52383775 BTW, the Watch window is correct. how come the numMinusOne does not get the correct value? Jul 21 '05 #1
24 Replies

 P: n/a David, I don't believe its a bug per se, its due to the inexact nature of how floating point numbers are stored. If you change xSng to a Double or Decimal, you get the correct result. For a complete discussion (and explaination) of how floating point numbers work in .NET see: http://www.yoda.arachsys.com/csharp/floatingpoint.html http://www.yoda.arachsys.com/csharp/decimal.html Hope this helps Jay "David" wrote in message news:ep**************@TK2MSFTNGP10.phx.gbl... hello. when doing the simple following computation, the value put into the variable numMinusOne is NOT the same as what the computation is showed to be in the Watch window!! here is the code: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim num As Integer = CInt(Math.Floor(xSng * yInt)) Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1) After running these 4 lines, numMinusOne will be:52383776 while the Watch window will claim that the value of : CInt(Math.Floor(xSng * yInt) - 1) = 52383775 BTW, the Watch window is correct. how come the numMinusOne does not get the correct value? Jul 21 '05 #2

 P: n/a David wrote: when doing the simple following computation, the value put into the variable numMinusOne is NOT the same as what the computation is showed to be in the Watch window!! here is the code: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim num As Integer = CInt(Math.Floor(xSng * yInt)) Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1) After running these 4 lines, numMinusOne will be:52383776 while the Watch window will claim that the value of : CInt(Math.Floor(xSng * yInt) - 1) = 52383775 BTW, the Watch window is correct. how come the numMinusOne does not get the correct value? See http://www.pobox.com/~skeet/csharp/floatingpoint.html You're actually calculating 6547.97216796875 * 8000 and then taking the floor of that. The watch window is probably doing computation in a slightly different way - e.g. using a Double instead of a Single. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #3

 P: n/a > You're actually calculating 6547.97216796875 * 8000 and then taking the floor of that. hemm... why is that? Dim xSng As Single = 6547.972 means I want to use... 6547.972, not 6547.97216796875 . are you telling me that the best representation for 6547.972 is 6547.97216796875 ?? Jul 21 '05 #4

 P: n/a I also think that it has to do with .NET, and not a general floating point issue. because of the following: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim d As Double = xSng * yInt Dim s As Single = xSng * yInt xSng * yInt should be : 52383776.0 . I still don't see how this can be any different, even after reading the problems with floating point. in any case, it either can or can not store 6547.972 as 6547.972. how can it be that d = 52383777.34375 s = 52383776.0 the runtime multiplies the numbers differently according to the left side of the equation? this is against anything I know about order of operations. if not, if the runtime does the same calculation, then why would the storing of the same result give such a different number in d and in s ? such a casting can make a small difference, not a 1.3x difference.... "csmba" wrote in message news:uk**************@TK2MSFTNGP12.phx.gbl... You're actually calculating 6547.97216796875 * 8000 and then taking the floor of that. hemm... why is that? Dim xSng As Single = 6547.972 means I want to use... 6547.972, not 6547.97216796875 . are you telling me that the best representation for 6547.972 is 6547.97216796875 ?? Jul 21 '05 #5

 P: n/a "csmba" wrote in message news:%2****************@TK2MSFTNGP11.phx.gbl...I also think that it has to do with .NET, and not a general floating point issue. because of the following: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim d As Double = xSng * yInt Dim s As Single = xSng * yInt xSng * yInt should be : 52383776.0 . I still don't see how this can be any different, even after reading the problems with floating point. in any case, it either can or can not store 6547.972 as 6547.972. how can it be that d = 52383777.34375 s = 52383776.0 the runtime multiplies the numbers differently according to the left side of the equation? this is against anything I know about order of operations. No it's not. You are forgetting about the data type conversion operations. tRemember that you usually only have arithmetic operations defined as binary operations taking two inputs of the same type and returning that same type. There is no multiplication operator coded to take an Integer and a Single and return a Double. If there were, it would probably give the right answer. Look at it this way. How would you compute: Dim d as Double = xSng * yInt There are two possible ways, and both have problems. First you could do it like this A: Dim d as Double = CDbl(CSng(xSng * CSng(yInt))) Or like this B: Dim d as Double = CDbl(xSng) * CDbl(yInt) For some values of xSng and yInt A will be more accurate, but for some values the result of xSng * yInt won't even fit in a Single. If you use A there, then you won't even get an answer! EG Dim xSng As Single = Single.MaxValue Dim yInt As Integer = 8000 So the compiler chooses B as the default order of operations. If you don't like that order, force your own. David Jul 21 '05 #6

 P: n/a In article <#d**************@TK2MSFTNGP11.phx.gbl>, cs***@nowhere.com says... I also think that it has to do with .NET, and not a general floating point issue. because of the following: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim d As Double = xSng * yInt What happens when you do Dim d As Double = xSng * Cdbl(yInt) Dim s As Single = xSng * yInt and Dim s As Single= xSng * CSng(yInt) ? -- Alan LeHun Jul 21 '05 #7

 P: n/a thanks David. what you say is true... so the compiler decided to cast it to double for the valid reason you mentioned... so the only technical thing I guess I was missing was: how come casting 6547.972 to double gives me a LESS accurate number? I guessed that double is MORE accurate then single. however, a simple program shows that all my problems are caused by the simple fact that: Dim xSng As Single = 6547.972 dim d as double = CDbl(xSng) --> gives that d's value is : 6547.97216796875 so that little cast from single to double causes a very significant difference in value. I didn't expect a double to not be able to keep a value when it comes from a single (float)... did you know that this can happen? Hernan. Jul 21 '05 #8

 P: n/a as David.B pointed out, the issue is how the compiler decides to cast the variables... it turns out that as long as it does it in Single, it works. if it does it in Doubles, it will not. looking into it, it all boils down to the fact that : Dim xSng As Single = 6547.972 dim d as double = CDbl(xSng) --> gives that d's value is : 6547.97216796875 in short, for some reason Double can not represent 6547.972, but adds that 167... to it which kills me in the calculation. so to your question,: Dim d As Double = xSng * Cdbl(yInt) d= 52383777.34375 Dim s As Single= xSng * CSng(yInt) ? d=52383776.0 However, this still makes little sense, since a DOUBLE can easily save 6547.972 as in : Dim xDbl As Double = 6547.972 check out xDbl value, it is 6547.972 so I still have no idea why this happens. Jul 21 '05 #9

 P: n/a BTW, I just checked, and a double CAN save the value 6547.972 just fine. as in Dim xDbl As Double= 6547.972 check out xDbl value, it is 6547.972 and not 6547.972167... so I still don't understand why the problem happens just when casting... Jul 21 '05 #10

 P: n/a csmba wrote: You're actually calculating 6547.97216796875 * 8000 and then taking the floor of that. hemm... why is that? Dim xSng As Single = 6547.972 means I want to use... 6547.972, not 6547.97216796875 . are you telling me that the best representation for 6547.972 is 6547.97216796875 ?? Yes. If you can find a closer floating point representation in 32 bit IEEE 754, please provide it. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #11

 P: n/a csmba wrote: as David.B pointed out, the issue is how the compiler decides to cast the variables... it turns out that as long as it does it in Single, it works. if it does it in Doubles, it will not. looking into it, it all boils down to the fact that : Dim xSng As Single = 6547.972 dim d as double = CDbl(xSng) --> gives that d's value is : 6547.97216796875 in short, for some reason Double can not represent 6547.972, but adds that 167... to it which kills me in the calculation. so to your question,: Dim d As Double = xSng * Cdbl(yInt) d= 52383777.34375 Dim s As Single= xSng * CSng(yInt) ? d=52383776.0 However, this still makes little sense, since a DOUBLE can easily save 6547.972 as in : Dim xDbl As Double = 6547.972 check out xDbl value, it is 6547.972 so I still have no idea why this happens. Rather than take each of your posts in turn, here's what's going on: 1) 6547.972 is *not* exactly representable as either a double or a single precision floating point value. Just because it shows it as 6547.972 in the debugger doesn't mean that's the exact value - it just means that the debugger is doing a bit of rounding on the display. Trust my DoubleConverter here (which is downloadable - see the page I referred you to before). Don't rely on what the debugger or framework displays - there's no simple way of getting the framework to display the exact value without something like DoubleConverter. 2) The framework is free to perform calculations at a higher precision than the operands, and that's why you're seeing the difference you are. (Chances are that the operation is actually being carried out in 80 bits, not 64.) It's not that the single version is being more accurate - it's being *less* accurate, but in a way that happens to be in the opposite direction from the way that the original numbers are inaccurately stored, so the two inaccuracies happen to cancel out. 3) If this is mucking your application up, you need to fix your application so it doesn't rely on floating point values being infinitely precise. They're not, and they're not going to be. There's no bug here - what you're seeing *is* a perfectly normal set of floating point results. If anything, you're actually being lucky - the double result is the exact value of the multiplication of the two exact values involved (8000 and 6547.97216796875). That certainly isn't always the case. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #12

 P: n/a thanks Jon, I appreciate your time. your answers are long and informative. I understand everything you said and I am not claiming that anything should have been different. the behavior which I could not understand was not the fact that single and double are not a perfect representation, but the one summarized in the next simple example: Dim xSng As Single = 6547.972 Dim xDbl As Double = 6547.972 Dim d As Double = xDbl * CDbl(yInt) Dim s As Single = xSng * CSng(yInt) both d and s are calculated "correctly". so I don't see how to explain this with your previous post: you say double can not represent 6547.972 and neither can single (and that I was lucky that a single gave me a right result). by why then would I get the correct answer if I start from a double, and only face the problem of 6547.97216796875 when it is a cast from single to double? as for the business use in my case: I do not as you say need infinitely precise numbers, all I need actually is 3 digits to the right precision. this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. I know I can probably treat it all as a integer counting in some small unit such as milliseconds, but the data I get is in float so transforming it to INT might have the same problems has I have here... hernan. Jul 21 '05 #13

 P: n/a Further to the other replies if this is 'Really' a problem consider using a Decimal instead of Double or Single hth guy "David" wrote: hello. when doing the simple following computation, the value put into the variable numMinusOne is NOT the same as what the computation is showed to be in the Watch window!! here is the code: Dim xSng As Single = 6547.972 Dim yInt As Integer = 8000 Dim num As Integer = CInt(Math.Floor(xSng * yInt)) Dim numMinusOne As Integer = CInt(Math.Floor(xSng * yInt) - 1) After running these 4 lines, numMinusOne will be:52383776 while the Watch window will claim that the value of : CInt(Math.Floor(xSng * yInt) - 1) = 52383775 BTW, the Watch window is correct. how come the numMinusOne does not get the correct value? Jul 21 '05 #14

 P: n/a csmba, all I need actually is 3 digits to the right precision. this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. Consider using Decimal. Dim x As Decimal = 6547.972D Dim y As Integer = 8000 Dim d As Decimal = x * y Hope this helps Jay "csmba" wrote in message news:e7**************@TK2MSFTNGP11.phx.gbl... thanks Jon, I appreciate your time. your answers are long and informative. I understand everything you said and I am not claiming that anything should have been different. the behavior which I could not understand was not the fact that single and double are not a perfect representation, but the one summarized in the next simple example: Dim xSng As Single = 6547.972 Dim xDbl As Double = 6547.972 Dim d As Double = xDbl * CDbl(yInt) Dim s As Single = xSng * CSng(yInt) both d and s are calculated "correctly". so I don't see how to explain this with your previous post: you say double can not represent 6547.972 and neither can single (and that I was lucky that a single gave me a right result). by why then would I get the correct answer if I start from a double, and only face the problem of 6547.97216796875 when it is a cast from single to double? as for the business use in my case: I do not as you say need infinitely precise numbers, all I need actually is 3 digits to the right precision. this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. I know I can probably treat it all as a integer counting in some small unit such as milliseconds, but the data I get is in float so transforming it to INT might have the same problems has I have here... hernan. Jul 21 '05 #15

 P: n/a csmba wrote: thanks Jon, I appreciate your time. your answers are long and informative. So long as they're actually helping - there's nothing worse than writing a long answer which only makes things *less* clear! I understand everything you said and I am not claiming that anything should have been different. the behavior which I could not understand was not the fact that single and double are not a perfect representation, but the one summarized in the next simple example: Dim xSng As Single = 6547.972 Dim xDbl As Double = 6547.972 Dim d As Double = xDbl * CDbl(yInt) Dim s As Single = xSng * CSng(yInt) both d and s are calculated "correctly". so I don't see how to explain this with your previous post: you say double can not represent 6547.972 and neither can single (and that I was lucky that a single gave me a right result). by why then would I get the correct answer if I start from a double, and only face the problem of 6547.97216796875 when it is a cast from single to double? Just chance. The error in truncating the result of the multiplication happened to go the other way from the error in the original representation - two "wrongs" happened to make a "right". as for the business use in my case: I do not as you say need infinitely precise numbers, all I need actually is 3 digits to the right precision. this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. I know I can probably treat it all as a integer counting in some small unit such as milliseconds, but the data I get is in float so transforming it to INT might have the same problems has I have here... How are you getting it as a float though? From a database, or parsing it from a file, or what? So long as you don't do very many operations, you'll certainly keep 3 significant digits correctly, but you might want to consider using decimal instead, as others have suggested. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #16

 P: n/a I know about Decimal, it is not the first choice becuase of performance. can you comment on my other quesion? which was why this works: Dim xSng As Single = 6547.972 Dim xDbl As Double = 6547.972 Dim d As Double = xDbl * CDbl(yInt) Dim s As Single = xSng * CSng(yInt) both d and s are calculated "correctly". acording to what you said, the double version should not have worked... but it does. h. "Jay B. Harlow [MVP - Outlook]" wrote in message news:On****************@TK2MSFTNGP12.phx.gbl... csmba, all I need actually is 3 digits to the right precision. this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. Consider using Decimal. Dim x As Decimal = 6547.972D Dim y As Integer = 8000 Dim d As Decimal = x * y Hope this helps Jay "csmba" wrote in message news:e7**************@TK2MSFTNGP11.phx.gbl... thanks Jon, I appreciate your time. your answers are long and informative. I understand everything you said and I am not claiming that anything should have been different. the behavior which I could not understand was not the fact that single and double are not a perfect representation, but the one summarized in the next simple example: Dim xSng As Single = 6547.972 Dim xDbl As Double = 6547.972 Dim d As Double = xDbl * CDbl(yInt) Dim s As Single = xSng * CSng(yInt) both d and s are calculated "correctly". so I don't see how to explain this with your previous post: you say double can not represent 6547.972 and neither can single (and that I was lucky that a single gave me a right result). by why then would I get the correct answer if I start from a double, and only face the problem of 6547.97216796875 when it is a cast from single to double? as for the business use in my case: I do not as you say need infinitely precise numbers, all I need actually is 3 digits to the right precision. this number represents the length in seconds of a file, and since it is a 8000 samples/sec, this is all the precision I need. I know I can probably treat it all as a integer counting in some small unit such as milliseconds, but the data I get is in float so transforming it to INT might have the same problems has I have here... hernan. Jul 21 '05 #17

 P: n/a csmba wrote: well... I get it from a COM object (BLHaaA) Oh dear :( On the other hand, it means you've probably done the floating point conversion already, so any attempts to make it "accurate" again (by using Decimal) are likely to be fruitless. Maybe I'm wrong though... it depends on how exactly the COM object is doing things, I guess. it is a c++ object, that loads a wav file to memory and I need to ask it what is the length of the file. it returns a float. That's the length of the file in seconds? Decimal just seems soooo slow. Well, it's significantly slower than float - but is this actually the bottleneck in your app? is there a trick to use a single but tell it to keep only 3 points precision? in that case, I will not have a problem when I multiply by 8000. No, but you could always round the float afterwards. I'm not sure how I see why that's a good thing though, to be honest. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #19

 P: n/a yes, this 6547.972 is the length in seconds of a wav file. and because of the calculation I am using (8000 is the sampling rate), all that is actually true is the .xxx digits. anything that the double adds to it like the .xxx167... is just wrong. and was never there to begin with (the number is a perfect factor of 8000). so since you said that "actually" the number is stored as 6547.972164... if I can ignore the digits behind the first 3 I get exactly what I want.... (I can do it of course by multiplying by 1000, then flooring the number to a int, then dividing by 1000 again)--> but if I end up storing it ina single, and he again adds those random 167.. it will do no good. Decimal is out of the question, I am doing hundred of thousands of such calculation per file. while you said it is luck that dim d as double = 6547.972 works, I find it more then strange. it can not be luck since I tested many different numbers, all work the same: they all work when set like above. but some return "strange" values when cast from single to double. it is as if the problem is in the casting. as for rounding the float: what I wanted is a way to guarantee that 6547.972 * 8000 gives the right answer. it seems like a fair thing to do. I am surprised that except for the suggestion of using decimal, I did not find any real answer to this. again, it actually does work, as long as the 6547.972 value is stored in a double or in a single. it FAILS only when it is stored in a single and CAST to a double as in CDbl(sing)*8000 where sing is a single parameter with 6547.972 as a value. H. well... I get it from a COM object (BLHaaA) Oh dear :( On the other hand, it means you've probably done the floating point conversion already, so any attempts to make it "accurate" again (by using Decimal) are likely to be fruitless. Maybe I'm wrong though... it depends on how exactly the COM object is doing things, I guess. it is a c++ object, that loads a wav file to memory and I need to ask it what is the length of the file. it returns a float. That's the length of the file in seconds? Decimal just seems soooo slow. Well, it's significantly slower than float - but is this actually the bottleneck in your app? is there a trick to use a single but tell it to keep only 3 points precision? in that case, I will not have a problem when I multiply by 8000. No, but you could always round the float afterwards. I'm not sure how I see why that's a good thing though, to be honest. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #20

 P: n/a "csmba" wrote in message news:up**************@TK2MSFTNGP11.phx.gbl... yes, this 6547.972 is the length in seconds of a wav file. and because of the calculation I am using (8000 is the sampling rate), all that is actually true is the .xxx digits. anything that the double adds to it like the .xxx167... is just wrong. and was never there to begin with (the number is a perfect factor of 8000). You can't ask for number of samples or parse it yourself? It seems to me you are basically doing a sample to length to sample conversion. I do not believe that there is ever a circumstance where a wav file actually stores length, just sample size and then audio data length, so the data you are getting is basically dataSize/(sampleSize*samplesPerSecond*channels) or something similar and getting a float out of it. Does the COM object do anything other than parse the wav file? Jul 21 '05 #21

 P: n/a csmba can you comment on my other quesion? See my original post, Jon does a significantly better job then I at explaining it. Reading Jon's & the other responses I believe the "why" has been covered. I know about Decimal, it is not the first choice becuase of performance. It sounds like you are worrying about the wrong end of performance. Have you profiled the difference between calculating the value in Decimal & Double or even Long or Integer, have you proven via Profiling (see below) in your program that using Decimal is a performance problem in this specific routine? Remember the 80/20 rule. That is 80% of the execution time of your program is spent in 20% of your code. I will optimize (worry about performance) the 20% once that 20% has been identified & proven to be a performance problem via profiling (CLR Profiler is one profiling tool). For info on the 80/20 rule & optimizing only the 20% see Martin Fowler's article "Yet Another Optimization Article" at http://martinfowler.com/ieeeSoftware...timization.pdf Info on the CLR Profiler: http://msdn.microsoft.com/library/de...nethowto13.asp http://msdn.microsoft.com/library/de...anagedapps.asp If you want "3 digits to the right precision" then I would recommend Decimal, if you want performance then I would recommend Double. If you really want you cake (precision) & eat it also (performance), then you may want to consider not using Floating Point (Decimal, Single or Double), as Jon and other have demonstrated, Single & Double floating point is inexact. Decimal is more precise, however has this perceived performance problem. (I say perceived, as I suspect using it in most cases will not cause any significant performance issues). > this number represents the length in seconds of a file, and since it is "Length in seconds of a file", really sounds like a Performance Counter. If I needed "length/time" I would use Performance Counters... Hope this helps Jay "csmba" wrote in message news:e4**************@TK2MSFTNGP15.phx.gbl...I know about Decimal, it is not the first choice becuase of performance. can you comment on my other quesion? which was why this works: > Dim xSng As Single = 6547.972 > Dim xDbl As Double = 6547.972 > > Dim d As Double = xDbl * CDbl(yInt) > Dim s As Single = xSng * CSng(yInt) > both d and s are calculated "correctly". acording to what you said, the double version should not have worked... but it does. h. "Jay B. Harlow [MVP - Outlook]" wrote in message news:On****************@TK2MSFTNGP12.phx.gbl... csmba, > all I need actually is 3 digits to the right precision. > this number represents the length in seconds of a file, and since it is a > 8000 samples/sec, this is all the precision I need. Consider using Decimal. Dim x As Decimal = 6547.972D Dim y As Integer = 8000 Dim d As Decimal = x * y Hope this helps Jay <> Jul 21 '05 #22

 P: n/a csmba wrote: yes, this 6547.972 is the length in seconds of a wav file. and because of the calculation I am using (8000 is the sampling rate), all that is actually true is the .xxx digits. anything that the double adds to it like the .xxx167... is just wrong. and was never there to begin with (the number is a perfect factor of 8000). so since you said that "actually" the number is stored as 6547.972164... if I can ignore the digits behind the first 3 I get exactly what I want.... (I can do it of course by multiplying by 1000, then flooring the number to a int, then dividing by 1000 again)--> but if I end up storing it ina single, and he again adds those random 167.. it will do no good. You haven't said *why* you need it to be exact. What are you doing with the result? If you treat it as only being exact to 3 significant figures, it should be fine. (I would use double rather than single though, as 6547.972 is getting close to the number of significant digits you can rely on in a single.) Alternatively, why not just keep it as an int, scaled up by 8000? It really sounds like using binary floating point is definitely the wrong way of proceeding. Decimal is out of the question, I am doing hundred of thousands of such calculation per file. Hundreds of thousands of decimal operations still doesn't take very long. Have you actually *tried* it? while you said it is luck that dim d as double = 6547.972 works, I find it more then strange. it can not be luck since I tested many different numbers, all work the same: they all work when set like above. but some return "strange" values when cast from single to double. it is as if the problem is in the casting. No, the problem is that the multiplication gives a result which is accurate to the number of significant figures that single can represent, but not accurate to the number of significant figures that double can represent. I suspect this is still luck though, and it will *certainly* become a problem if you deal with longer files where the number of significant digits that single can cope with gets too small. as for rounding the float: what I wanted is a way to guarantee that 6547.972 * 8000 gives the right answer. it seems like a fair thing to do. Not when 6547.972 can't be exactly represented as a binary floating point number to start with. I am surprised that except for the suggestion of using decimal, I did not find any real answer to this. again, it actually does work, as long as the 6547.972 value is stored in a double or in a single. it FAILS only when it is stored in a single and CAST to a double as in CDbl(sing)*8000 where sing is a single parameter with 6547.972 as a value. Yes - we've explained why. When you cast the single to a double, you're using the less accurate representation of 6547.972, but then multiplying that and keeping many significant digits of the result. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #23

 P: n/a >>You haven't said *why* you need it to be exact Well, to make all of this make sense... (this is for both Jon and Daniel last posts) the COM holds the wav file, and its data. my "client" wants to process a portion of the wav file. in this case he wants to process the window 6000 sec to end of file, which he gets from the COM as a float 6547.972. now we can say that it all boils to the fact the COM should have never return such a float. I am not sure what are the COM alternatives... but that wasn't my game either. I then create a new array of samples to do computations on (calculate energy for example), the size of that array has to be calculated from the length of the window and the sample rate (8000). I then fill this "window" array with values from the original array of samples returned by the COM. the COM holds m_samples which is of correct size: 52383776. when I fill up my window array, I have to put in its last cell the value from the last cell on m_samples. the "bad" calculation caused me to try and get the cell value 52383777 instead of 52383776, which is pass the last sample in the wav file... You can't ask for number of samples or parse it yourself? no. I, as the client have my own reasons to which windows on the file I need to process, and they are ruled by TIME. for example, I am told that in the last 10 seconds of the file something happens, or that between the 15 and 55 seconds of the call... so I need to make a copy of the m_samples, but only the size of (timeEnd-TimeStart)*8000 and here is where I got the wrong calculation (not exactly, check the first posts for the exact line of code). Does the COM object do anything other than parse the wav file well, all I need from it is the array of samples, and the length of the call and sample rate. are you going to suggest I just get array, sample rate and maybe if he is so kind the length of the array? I will then have to calculate the length in seconds anyway... so I don't see how is that different (unless I store it in a decimal of course). OK, to focused this, and since everyone have been very helpful: I think we all got the issue about double and single and precision. I also got the fact Decimal is better (yet not supported in c++ or at least in COM). I "solved" my problem (maybe badly I assume Jon will claim) by forcing the calculation to be in singles. avoiding the cast to double solved my problem in this case. I understand that if the number is much bigger, it may not (Jon's position). the only advice then I ask is, how should I "correctly" handle this case? h. as for the 80/20 rules: Jon you are absolutely right. dataSize/(sampleSize*samplesPerSecond*channels) or something similar and getting a float out of it. Does the COM object do anything other than parse the wav file? Jul 21 '05 #24

 P: n/a csmba wrote:You haven't said *why* you need it to be exact Well, to make all of this make sense... (this is for both Jon and Daniel last posts) the COM holds the wav file, and its data. my "client" wants to process a portion of the wav file. in this case he wants to process the window 6000 sec to end of file, which he gets from the COM as a float 6547.972. Do you have to get it in that form? Can you not work it out accurately (using only integers) from other information? That would be the best bet. Does the COM object do anything other than parse the wav file well, all I need from it is the array of samples, and the length of the call and sample rate. are you going to suggest I just get array, sample rate and maybe if he is so kind the length of the array? I will then have to calculate the length in seconds anyway... so I don't see how is that different (unless I store it in a decimal of course). If you ask for the sample rate and the total number of samples, and you know (as an integer) the number of seconds you want to start at, I don't see where any floating point maths needs to be involved. I "solved" my problem (maybe badly I assume Jon will claim) by forcing the calculation to be in singles. avoiding the cast to double solved my problem in this case. I understand that if the number is much bigger, it may not (Jon's position). the only advice then I ask is, how should I "correctly" handle this case? Try to avoid using floating point at all. If you've basically got a floating point number somewhere which is some integer / 8000, and you want to get back to the same integer, then rather than multiplying the floating point number by 8000, you should try to get the original integer directly. -- Jon Skeet - http://www.pobox.com/~skeet If replying to the group, please do not mail me too Jul 21 '05 #25

### This discussion thread is closed

Replies have been disabled for this discussion. 