469,287 Members | 2,770 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,287 developers. It's quick & easy.

Server producing incorrect math?

Here's a wierd one...

I have a simple loop that adds values.

It seems that after thousands of instances using this loop with no
problems whatsoever, when the loop tries the following, it messes up.

strTotal = strTotal + strNewLabor

When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
strTotal = 4.900001

Why in the world would somthing like this happen?

--
Sugapablo
------------------------------------
http://www.sugapablo.com <--music
http://www.sugapablo.net <--personal

Jul 19 '05 #1
13 1669
"Sugapablo" wrote:

I have a simple loop that adds values.

It seems that after thousands of instances using this loop with no
problems whatsoever, when the loop tries the following, it messes up.

strTotal = strTotal + strNewLabor

When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
strTotal = 4.900001

Why in the world would somthing like this happen?


What is the binary representation of 3.2? How about 1.7? Each is an
approximation, so there is no expectation that adding them will improve
their precision. See the following:

http://support.microsoft.com/default...;EN-US;q244699
This applies to every language that uses floating point numbers.
--
Dave Anderson

Unsolicited commercial email will be read at a cost of $500 per message. Use
of this email address implies consent to these terms. Please do not contact
me directly or ask me to contact you directly for assistance. If your
question is worth asking, it's worth posting.
Jul 19 '05 #2
Sugapablo wrote:
Here's a wierd one...

I have a simple loop that adds values.

It seems that after thousands of instances using this loop with no
problems whatsoever, when the loop tries the following, it messes up.

strTotal = strTotal + strNewLabor

When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
strTotal = 4.900001

Why in the world would somthing like this happen?


http://support.microsoft.com/default...kb;EN-US;42980

HTH,
Bob Barrows
Jul 19 '05 #3
Dave Anderson wrote:
http://support.microsoft.com/default...;EN-US;q244699
This applies to every language that uses floating point numbers.


Ok, so what would the fix be?
--
Sugapablo
------------------------------------
http://www.sugapablo.com <--music
http://www.sugapablo.net <--personal

Jul 19 '05 #4
Here's another article that is more applicable to vbscript:
http://support.microsoft.com/default...b;en-us;165373

HTH,
Bob Barrows
Jul 19 '05 #5
How much accuracy do you want/need?

Single decimal

strTotal=int(strTotal*10)
strNewLabor=int(strNewLabor*10)
strTotal=(strTotal+strNewLabor)/10
"Sugapablo" <ru********@sugapablo.com> wrote in message
news:vm************@corp.supernews.com...
Dave Anderson wrote:
http://support.microsoft.com/default...;EN-US;q244699
This applies to every language that uses floating point numbers.


Ok, so what would the fix be?
--
Sugapablo
------------------------------------
http://www.sugapablo.com <--music
http://www.sugapablo.net <--personal

Jul 19 '05 #6
Intel Math:

2 + 2 = 5 for extremely large values of 2

- Wm

--
William Morris
Product Development, Seritas LLC
"Sugapablo" <ru********@sugapablo.com> wrote in message
news:vm************@corp.supernews.com...
Here's a wierd one...

I have a simple loop that adds values.

It seems that after thousands of instances using this loop with no
problems whatsoever, when the loop tries the following, it messes up.

strTotal = strTotal + strNewLabor

When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
strTotal = 4.900001

Why in the world would somthing like this happen?

--
Sugapablo
------------------------------------
http://www.sugapablo.com <--music
http://www.sugapablo.net <--personal

Jul 19 '05 #7
..9 repeating = 1, but that's true. :]

Ray at work

"WIlliam Morris" <se**************@hotmail.com> wrote in message
news:bk************@ID-205671.news.uni-berlin.de...
Intel Math:

2 + 2 = 5 for extremely large values of 2

- Wm

--
William Morris
Product Development, Seritas LLC

Jul 19 '05 #8
"Tom B" wrote:

How much accuracy do you want/need?

Single decimal

strTotal=int(strTotal*10)
strNewLabor=int(strNewLabor*10)
strTotal=(strTotal+strNewLabor)/10


Division by 10 is just another way to introduce computational error. Unless
the divisor is a power of two (or a whole divisor of the numerator), the
result will necessarily represent an *approximation*.

This is not a solution. It "works" most of the time, but not all of the
time.
--
Dave Anderson

Unsolicited commercial email will be read at a cost of $500 per message. Use
of this email address implies consent to these terms. Please do not contact
me directly or ask me to contact you directly for assistance. If your
question is worth asking, it's worth posting.
Jul 19 '05 #9
"Sugapablo" wrote:

Ok, so what would the fix be?


There is no fix to computational error if you want to continue to represent
numbers in binary any more than there is a fix to the general problem that
some numbers cannot be represented with a finite number of decimals.

If you are asking what you can do for your particular problem, then the
answer depends on what you want. If, for example, you only care about what
is being displayed, then use one of the appropriate methods/functions. Note
that these both return STRINGS:

JScript:
http://msdn.microsoft.com/library/en...mthtofixed.asp
vbscript:
http://msdn.microsoft.com/library/en...rmatnumber.asp

On the other hand, if you are trying to test for equality (or some other
relation), you will have to redefine those terms. Pick an arbitrary error
margin, such as 0.0001, and define equality to be "within the margin of
error", less than to be "NOT greater than by more than the margin of error",
etc. Your new system should work something like this:

JScript:
OLD NEW
A == B Math.abs(A-B) < e
A != B Math.abs(A-B) > e
A < B A < B+e
A > B A > B-e

vbscript:
OLD NEW
A = B Abs(A-B) < e
A <> B Abs(A-B) > e
A < B A < B+e
A > B A > B-e

Because of computational error, there is little difference in meaning
between > and >= or < and <=. The endpoints are simply too unreliable to
make that distinction.

--
Dave Anderson

Unsolicited commercial email will be read at a cost of $500 per message. Use
of this email address implies consent to these terms. Please do not contact
me directly or ask me to contact you directly for assistance. If your
question is worth asking, it's worth posting.
Jul 19 '05 #10
"WIlliam Morris" wrote:

Intel Math:

2 + 2 = 5 for extremely large values of 2


In VBScript, Round(5/2) = 2, while in JScript, Math.round(5/2) == 3.

So it would appear 3 + 3 = 5 for extremely small values of 3, as well.

--
Dave Anderson

Unsolicited commercial email will be read at a cost of $500 per message. Use
of this email address implies consent to these terms. Please do not contact
me directly or ask me to contact you directly for assistance. If your
question is worth asking, it's worth posting.
Jul 19 '05 #11
THAT got an out loud laugh. Great...now my partners are ~convinced~ I'm out
of my mind.

- Wm

--
William Morris
Product Development, Seritas LLC

"Dave Anderson" <GT**********@spammotel.com> wrote in message
news:#Z**************@TK2MSFTNGP10.phx.gbl...
"WIlliam Morris" wrote:

Intel Math:

2 + 2 = 5 for extremely large values of 2
In VBScript, Round(5/2) = 2, while in JScript, Math.round(5/2) == 3.

So it would appear 3 + 3 = 5 for extremely small values of 3, as well.

--
Dave Anderson

Unsolicited commercial email will be read at a cost of $500 per message.

Use of this email address implies consent to these terms. Please do not contact me directly or ask me to contact you directly for assistance. If your
question is worth asking, it's worth posting.

Jul 19 '05 #12
> Ok, so what would the fix be?

One work around would be to use formatnumber to limit the number of
significant digits after the decimal place. I added this as a FAQ today:

http://www.aspfaq.com/2477
Jul 19 '05 #13
Aaron Bertrand [MVP] wrote on 20 sep 2003 in
microsoft.public.inetserver.asp.general:
Ok, so what would the fix be?


One work around would be to use formatnumber to limit the number of
significant digits after the decimal place. I added this as a FAQ today:

http://www.aspfaq.com/2477


Since BCD [binary coded decimal] seems to be out of use in this time of
hardcoded floatingpointnumbercrunching pentiums, te best way to code money
is to use integers trough-out and define those values as cents.

Only when writing, you should use:

Function WriteCents(CentsIntegerValue)
CentsIntegerValue = cInt(CentsIntegerValue) ' to be sure
Response.Write FormatNumber( CentsIntegerValue/100, 2 )
End Function

--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)
Jul 19 '05 #14

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

2 posts views Thread by Phil Powell | last post: by
5 posts views Thread by Christoph | last post: by
42 posts views Thread by (PeteCresswell) | last post: by
4 posts views Thread by garyusenet | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by suresh191 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.