By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,344 Members | 1,139 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,344 IT Pros & Developers. It's quick & easy.

Problem With Precision

P: n/a
When I loop through a datatable rows and add Amt field content (which
is float in database) sometimes it adds upto High Precision value
leaving a difference of -1.81898940354586E-12 though I am only adding
Currency Format numbers i.e., 23.43,56.99 etc,So I am not able to
directly compare the total(Computer value) to a value in text field.



Here is the Code
If Session("DT") Is Nothing Then
Dim SQL As String = "SELECT AcctNum, Amount FROM
Accts where 1=2"
Dim Dbase As New DBInterface(DBConnString)
Session("DT") = Dbase.GetDataTable(SQL)
End If
Dim DT As DataTable
DT = Session("DT")
Dim Drow As DataRow = DT.NewRow
Drow("AcctNum") = txtAcct.Text
Drow("Amount") = txtAmt.Text
DT.Rows.Add(Drow)
Dim Total As Double
Dim EachRow As DataRow
For Each EachRow In DT.Rows
Total += EachRow("Amount")
Next

dgrd.Columns.Item(0).FooterText = "Total"
dgrd.Columns.Item(1).FooterText = Total '.ToString("c")
txtTotal.Text = Total
Response.Write(Total - CDbl(txtTotal.Text))
Session("DT") = DT
dgrd.DataSource = DT
dgrd.DataBind()

Nov 21 '05 #1
Share this Question
Share on Google+
2 Replies


P: n/a
Instead of using Double, try using Decimal instead. Your problem
appears because a floating point number cannot accurately be
represented in binary. There have been numerous posts on these
newsgroups which explains it.

Nov 21 '05 #2

P: n/a
Great looks like working...Thanks

Nov 21 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.