457,884 Members | 1,348 Online Need help? Post your question and get tips & solutions from a community of 457,884 IT Pros & Developers. It's quick & easy.

# Rounding Problem

 P: n/a I am having a rounding problem in a value i am trying to display in VB.NET. If I have the following code: Dim x As Single = 2726.795 Dim y As Single = Math.Round(2726.795, 2) Dim s as String = String.Format("{0:0.00}", y) The value of s is 2726.79 If I have the following code: Dim y As Single = Math.Round(2726.795, 2) Dim s as String = String.Format("{0:0.00}", y) The value of s is 2726.80 I need to use the first example as I read the value of x in as a string from an xml file then convert it to a single in order to round and format it to 2 decimal places for a report display. Any ideas why the first example is not rounding to 2726.80? May 30 '06 #1
5 Replies

 P: n/a [the short answer is at the end] Jason wrote: I am having a rounding problem in a value i am trying to display in VB.NET. If I have the following code: Dim x As Single = 2726.795 Dim y As Single = Math.Round(2726.795, 2) Dim s as String = String.Format("{0:0.00}", y) The value of s is 2726.79 If I have the following code: Dim y As Single = Math.Round(2726.795, 2) Dim s as String = String.Format("{0:0.00}", y) The value of s is 2726.80 The two lines of the second example are the same as the second two lines of the first example. I suspect you have a typo somewhere. Incidentally, I see you are running without the benefit of Option Strict (Math.Round returns a Double which cannot be implicitly converted to a Single). I suggest turning it on - the additional code sometimes required is a small price to pay for the increased compile-time checking. My guess is that you meant the first example to be: Dim x As Single = 2726.795 Dim y As Single = Math.Round(x, 2) Dim s As String = String.Format("{0:0.00}", y) which *does* produce 2726.795. Assuming that is the right correction: Any ideas why the first example is not rounding to 2726.80? The first parameter to the particular overload of Math.Round being used is a Double. That means that when a Single is supplied (as in the corrected first example), it must be converted to a Double. This is OK to be done implicitly, since it is a widening conversion. Let's have a look at the actual exact values involved: Dim x As Single = 2726.795 Dim xAsDouble As Double = x Console.WriteLine(x.ToString("G9")) Console.WriteLine(xAsDouble.ToString("R")) The output is 2726.79492 2726.794921875 So we see that trying to put 2726.795 into a Single actually puts in a value slightly *less* than 2726.795, which is why it gets rounded to ....79. With a double, on the other hand (as in the second example): Dim x As Double = 2726.795 Console.WriteLine(x.ToString("G17")) 2726.7950000000001 So trying to put 2726.295 into a Double actually puts in a value slightly *more* than 2726.295, which is why it gets rounded to ...80. That's the why. The next question is the what-to-do. I'd say the first recommendation is this: don't use floating-point types to deal with exact values. Actually it's more than a recommendation, it's a golden rule. It's customary for me to give my usual example at this point: Debug.Print(0.1+0.1+0.1-0.3) 'guess output before running! The next recommendation would be, don't use Single at all unless you have a very good reason. Single only has about *seven* decimal digits of precision, which we run into pretty quickly in the real world. Any memory gains you think you might be getting by using Singles over Doubles are almost certainly illusory. Any performance gains you think you are getting are _wrong_: "Double is the most efficient of the fractional data types, because the processors on current platforms perform floating-point operations in double precision", to quote the docs. In short, if you are going to use floating point (with the above caveat that you shouldn't if... you shouldn't), use Double not Single. And now for the short answer: I need to use the first example as I read the value of x in as a string from an xml file then convert it to a single in order to round and format it to 2 decimal places for a report display. Use Decimal as the intermediate data type to avoid loss of precision. -- Larry Lard Replies to group please May 30 '06 #2

 P: n/a Larry, Using Decimal cuts right to the chase: Dim myDec As Decimal = 2726.795 MsgBox(Math.Round(myDec, 2)) This code produces 2726.80, which the OP does not want. Of course, the reason that Math.Round produces this value is very clearly laid out in the docs on Math.Round (IEEE Standard 754, section 4 or banker's rounding). The real question is probably "How to do rounding that rounds 2726.795 to 2726.79. Maybe using INT or FIX or FLOOR, etc is the required approach. Kerry Moorman "Larry Lard" wrote: Use Decimal as the intermediate data type to avoid loss of precision. May 30 '06 #3

 P: n/a > Larry, Using Decimal cuts right to the chase: Dim myDec As Decimal = 2726.795 MsgBox(Math.Round(myDec, 2)) This code produces 2726.80, which the OP does not want. Of course, the reason that Math.Round produces this value is very clearly laid out in the docs on Math.Round (IEEE Standard 754, section 4 or banker's rounding). The real question is probably "How to do rounding that rounds 2726.795 to 2726.79. Maybe using INT or FIX or FLOOR, etc is the required approach. Round includes an additional overloaded version which allows you to specify the rounding method you want: banker's round or away from 0 rounding. Unfortunately the framework isn't necessarily consistant in rounding approaches. Math.Round uses bankers rounding by default, but Format and ToString(c2) use away from 0 rounding. See my write-up at http://devauthority.com/blogs/jwoole...03/24/806.aspx for more. Jim Wooley http://devauthority.com/blogs/jwooley/default.aspx May 30 '06 #4

 P: n/a Jim That information, about MidpointRounding.AwayFromZero, is good to know and may go a ways to help clear up the confusion that has always surrounded the Math.Round method. I had not seen that before you pointed it out. Unfortunately, its a .Net 2.0-only feature. Kerry Moorman "Jim Wooley" wrote: Round includes an additional overloaded version which allows you to specify the rounding method you want: banker's round or away from 0 rounding. Unfortunately the framework isn't necessarily consistant in rounding approaches. Math.Round uses bankers rounding by default, but Format and ToString(c2) use away from 0 rounding. See my write-up at http://devauthority.com/blogs/jwoole...03/24/806.aspx for more. Jim Wooley http://devauthority.com/blogs/jwooley/default.aspx May 30 '06 #5

 P: n/a Cheers for the reply. I will switch from using Single to Double as that has solved my problem. The reason I have been using Single is that was the type I get back from my DataSet schema when connecting to the Access database I am working with. I will explicitly convert to Double in code for rounding functions. "Larry Lard" wrote: [the short answer is at the end] Jason wrote: I am having a rounding problem in a value i am trying to display in VB.NET. If I have the following code: Dim x As Single = 2726.795 Dim y As Single = Math.Round(2726.795, 2) Dim s as String = String.Format("{0:0.00}", y) The value of s is 2726.79 If I have the following code: Dim y As Single = Math.Round(2726.795, 2) Dim s as String = String.Format("{0:0.00}", y) The value of s is 2726.80 The two lines of the second example are the same as the second two lines of the first example. I suspect you have a typo somewhere. Incidentally, I see you are running without the benefit of Option Strict (Math.Round returns a Double which cannot be implicitly converted to a Single). I suggest turning it on - the additional code sometimes required is a small price to pay for the increased compile-time checking. My guess is that you meant the first example to be: Dim x As Single = 2726.795 Dim y As Single = Math.Round(x, 2) Dim s As String = String.Format("{0:0.00}", y) which *does* produce 2726.795. Assuming that is the right correction: Any ideas why the first example is not rounding to 2726.80? The first parameter to the particular overload of Math.Round being used is a Double. That means that when a Single is supplied (as in the corrected first example), it must be converted to a Double. This is OK to be done implicitly, since it is a widening conversion. Let's have a look at the actual exact values involved: Dim x As Single = 2726.795 Dim xAsDouble As Double = x Console.WriteLine(x.ToString("G9")) Console.WriteLine(xAsDouble.ToString("R")) The output is 2726.79492 2726.794921875 So we see that trying to put 2726.795 into a Single actually puts in a value slightly *less* than 2726.795, which is why it gets rounded to ....79. With a double, on the other hand (as in the second example): Dim x As Double = 2726.795 Console.WriteLine(x.ToString("G17")) 2726.7950000000001 So trying to put 2726.295 into a Double actually puts in a value slightly *more* than 2726.295, which is why it gets rounded to ...80. That's the why. The next question is the what-to-do. I'd say the first recommendation is this: don't use floating-point types to deal with exact values. Actually it's more than a recommendation, it's a golden rule. It's customary for me to give my usual example at this point: Debug.Print(0.1+0.1+0.1-0.3) 'guess output before running! The next recommendation would be, don't use Single at all unless you have a very good reason. Single only has about *seven* decimal digits of precision, which we run into pretty quickly in the real world. Any memory gains you think you might be getting by using Singles over Doubles are almost certainly illusory. Any performance gains you think you are getting are _wrong_: "Double is the most efficient of the fractional data types, because the processors on current platforms perform floating-point operations in double precision", to quote the docs. In short, if you are going to use floating point (with the above caveat that you shouldn't if... you shouldn't), use Double not Single. And now for the short answer: I need to use the first example as I read the value of x in as a string from an xml file then convert it to a single in order to round and format it to 2 decimal places for a report display. Use Decimal as the intermediate data type to avoid loss of precision. -- Larry Lard Replies to group please May 31 '06 #6

### This discussion thread is closed

Replies have been disabled for this discussion. 