473,756 Members | 8,110 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Small Math Errors when Converting Text to Single

TC
I am getting small math errors when I use a query to convert a column's
data type from Text to Single. I am using a calculated column with the
following expression.

HighSchoolGPA: IIf(IsNull([GPA]),Null,CSng([GPA]))

When GPA = "3.10", HighSchoolGPA := 3.0999999046325 7, not 3.1. For this
particular application, the difference between 3.0999999046325 7 and 3.1
is unacceptable.

A couple of notes: When I evaluate
IIf(IsNull("3.1 0"),Null,CSng(" 3.10")) in the immediate window, I get
the correct result of 3.1. Also, the table containing GPA is linked via
ODBC to a SQL Server database.

I can easily workaround the problem by using the Round() function, but
I'd prefer to understand why I'm getting the problem in the first
place. Can anyone offer advice?
-TC

Oct 28 '06 #1
1 2077
TC wrote:
I am getting small math errors when I use a query to convert a column's
data type from Text to Single.

http://msdn2.microsoft.com/en-us/lib...em.single.aspx

the page includes:

"Remember that a floating-point number can only approximate a decimal
number, and that the precision of a floating-point number determines
how accurately that number approximates a decimal number. By default, a
Single value contains only 7 decimal digits of precision, although a
maximum of 9 digits is maintained internally. The precision of a
floating-point number has several consequences:

Two floating-point numbers that appear equal for a particular precision
might not compare equal because their least significant digits are
different.

A mathematical or comparison operation that uses a floating-point
number might not yield the same result if a decimal number is used
because the floating-point number might not exactly approximate the
decimal number.

A value might not roundtrip if a floating-point number is involved. A
value is said to roundtrip if an operation converts an original
floating-point number to another form, an inverse operation transforms
the converted form back to a floating-point number, and the final
floating-point number is equal to the original floating-point number.
The roundtrip might fail because one or more least significant digits
are lost or changed in a conversion."

Oct 28 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
2495
by: Tom | last post by:
Has anyone ever seen a IComparer for floats the returns magnitude. i.e. instead of returning -1, it would return -5. To let you know HOW different the two numbers are. obviously for int it is a - b. But for float the results would have to be normalize some how to a 32bit range. I understand there would be percision errors. Thanks Tom
7
2602
by: Wernfried Schwenkner | last post by:
I've found the discussion about Math.Log and the error with Math.Log(8,2) on Google. Unfortunatly the full thread isn't on my news server, so I can't reply. The problem doesn't only depend on representation of floating numbers. When You execute
16
4985
by: lawrence k | last post by:
I've a file upload script on my site. I just now used it to upload a small text document (10k). Everything worked fine. Then I tried to upload a 5.3 meg Quicktime video. Didn't work. I've set the POST limit in php.ini to 8 megs. What reasons, other than the POST limit, would a large upload fail?
11
7364
by: Sambo | last post by:
I have the following module: ------------------------------- import math def ac_add_a_ph( amp1, ph1, amp2, ph2 ): amp3 = 0.0 ph3 = 0.0 ac1 = ( 0, 0j ) ac2 = ( 0, 0j )
16
1774
by: Bill Nguyen | last post by:
I'm running into a very weird problem regarding subtraction. Subtraction behaves as if it's an addition in the below sub txtJacoCost.Text = Format(mRackc - (mDisc + mJaEc), "0.#####0") txtWfCost.Text = Format(mRackc - mDisc + mWfEC, "0.#####0")
2
2044
by: astrogirl77 | last post by:
Hi, I'm new to Python and am hoping to find help with coding a Python script, applet. I code in an old version of Visual Basic 4.0, I have a simple app that is about 3 and a half pages of code long it does some relatively simple math additions and subtractions The problem I have is that some numbers get to be very large integers and VB automatically converts this to scientifc notation, what I need is to have all numbers added and...
1
2075
by: astrogirl77 | last post by:
I'm new to C++ and am hoping to find help with coding a simple C program, am wanting to obtain code and functioning exe's. I code in an old version of Visual Basic 4.0, I have a simple app that is about 3 and a half pages of code long it does some relatively simple math additions and subtractions The problem I have is that some numbers get to be very large integers and VB automatically converts this to scientifc notation, what I need is...
17
3921
by: Lenni | last post by:
Hi, I'm currently writing a web application for bicycle wheel builders that calculate the spoke length for all sorts of variations of hubs and rims. I have translated the formula from an Excel spreadsheet and in JavaScript it looks like this: var sll=Math.sqrt(Math.pow(((fdl/2*Math.sin((2*Math.PI*cross)))/
0
9456
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9275
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10040
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
9846
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
7248
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6534
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5142
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3806
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2666
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.