I am having problems adding two numbers. I am trying to add 1.005 and
1.007 to come up with 2.012. Should be easy enough right? Problem is that
I keep getting 2.0119999999999996 as my result. Why is this happening? Is
there a way that I can get it to return the right result? Any help is
appreciated.
Thanks,
David J. Ricker II 10 1426
What data types are you using?
"David Ricker" <da*************@hotmail.com> wrote in message
news:eW**************@TK2MSFTNGP12.phx.gbl... I am having problems adding two numbers. I am trying to add 1.005 and 1.007 to come up with 2.012. Should be easy enough right? Problem is
that I keep getting 2.0119999999999996 as my result. Why is this happening?
Is there a way that I can get it to return the right result? Any help is appreciated.
Thanks, David J. Ricker II
Doubles. I have also tried simply typing 1.005+1.007 into the watch window
and it gives the same result.
Thanks,
David J. Ricker II
"CJ Taylor" <no****@blowgoats.com> wrote in message
news:vs************@corp.supernews.com... What data types are you using?
"David Ricker" <da*************@hotmail.com> wrote in message news:eW**************@TK2MSFTNGP12.phx.gbl... I am having problems adding two numbers. I am trying to add 1.005
and 1.007 to come up with 2.012. Should be easy enough right? Problem is that I keep getting 2.0119999999999996 as my result. Why is this happening? Is there a way that I can get it to return the right result? Any help is appreciated.
Thanks, David J. Ricker II
What the he.....
Try a decimal. see what happens.
"David Ricker" <da*************@hotmail.com> wrote in message
news:uX**************@TK2MSFTNGP10.phx.gbl... Doubles. I have also tried simply typing 1.005+1.007 into the watch
window and it gives the same result.
Thanks, David J. Ricker II "CJ Taylor" <no****@blowgoats.com> wrote in message news:vs************@corp.supernews.com... What data types are you using?
"David Ricker" <da*************@hotmail.com> wrote in message news:eW**************@TK2MSFTNGP12.phx.gbl... I am having problems adding two numbers. I am trying to add 1.005 and 1.007 to come up with 2.012. Should be easy enough right? Problem is that I keep getting 2.0119999999999996 as my result. Why is this
happening? Is there a way that I can get it to return the right result? Any help is appreciated.
Thanks, David J. Ricker II
Strange, a decimal does work. Does anyone have any idea why a double won't
work in this situation... is it a bug?
Thanks CJ, I can use the decimals to get the desired results in my program.
Thanks Again,
David J. Ricker II
"CJ Taylor" <no****@blowgoats.com> wrote in message
news:vs************@corp.supernews.com... What the he.....
Try a decimal. see what happens.
"David Ricker" <da*************@hotmail.com> wrote in message news:uX**************@TK2MSFTNGP10.phx.gbl... Doubles. I have also tried simply typing 1.005+1.007 into the watch window and it gives the same result.
Thanks, David J. Ricker II "CJ Taylor" <no****@blowgoats.com> wrote in message news:vs************@corp.supernews.com... What data types are you using?
"David Ricker" <da*************@hotmail.com> wrote in message news:eW**************@TK2MSFTNGP12.phx.gbl... > I am having problems adding two numbers. I am trying to add
1.005 and > 1.007 to come up with 2.012. Should be easy enough right? Problem
is that > I keep getting 2.0119999999999996 as my result. Why is this
happening? Is > there a way that I can get it to return the right result? Any help
is > appreciated. > > Thanks, > David J. Ricker II > >
Hi David,
I think that this will help you.
I asume your decimal separator is a dot and not a comma http://msdn.microsoft.com/library/de...roundtopic.asp
I hope this helps a little bit.
Cor I am having problems adding two numbers. I am trying to add 1.005 and 1.007 to come up with 2.012. Should be easy enough right? Problem is that I keep getting 2.0119999999999996 as my result. Why is this happening?
Isthere a way that I can get it to return the right result? Any help is appreciated.
One would think so. But I don't know the honest answer. My best guess
would be back to high school chemistry days and understanding the use of Sig
Figs... Could be something to do with that, even though we only have 3 to
work with so I don't know what the big deal is.
I've had the same issue before, and something to do with decimal having a
precision scale is the reason I've heard before. But I quesiton everything.
=)
Then again... it could be a bug, but I'm probably wrong... but if it was,
that sounds like it would be processor level instead of code level.
-CJ
"David Ricker" <da*************@hotmail.com> wrote in message
news:ed**************@tk2msftngp13.phx.gbl... Strange, a decimal does work. Does anyone have any idea why a double won't work in this situation... is it a bug?
Thanks CJ, I can use the decimals to get the desired results in my
program. Thanks Again, David J. Ricker II
"CJ Taylor" <no****@blowgoats.com> wrote in message news:vs************@corp.supernews.com... What the he.....
Try a decimal. see what happens.
"David Ricker" <da*************@hotmail.com> wrote in message news:uX**************@TK2MSFTNGP10.phx.gbl... Doubles. I have also tried simply typing 1.005+1.007 into the watch window and it gives the same result.
Thanks, David J. Ricker II "CJ Taylor" <no****@blowgoats.com> wrote in message news:vs************@corp.supernews.com... > What data types are you using? > > "David Ricker" <da*************@hotmail.com> wrote in message > news:eW**************@TK2MSFTNGP12.phx.gbl... > > I am having problems adding two numbers. I am trying to add 1.005 and > > 1.007 to come up with 2.012. Should be easy enough right?
Problem is > that > > I keep getting 2.0119999999999996 as my result. Why is this happening? > Is > > there a way that I can get it to return the right result? Any
help is > > appreciated. > > > > Thanks, > > David J. Ricker II > > > > > >
http://www.math.grin.edu/~stone/cour...EEE-reals.html
Bob
"David Ricker" <da*************@hotmail.com> wrote in message
news:eW**************@TK2MSFTNGP12.phx.gbl... I am having problems adding two numbers. I am trying to add 1.005 and 1.007 to come up with 2.012. Should be easy enough right? Problem is that I keep getting 2.0119999999999996 as my result. Why is this happening? Is there a way that I can get it to return the right result? Any help is appreciated.
Thanks, David J. Ricker II
David,
In addition to CJ & Cor's posts. http://msdn.microsoft.com/library/de...egralTypes.asp
1.007 & 2.012 are by default a Double. Double & Single are base 2 floating
point numbers.
Decimal is a base 10 floating point number. (technally base 2, but think of
it as base 10 ;-))
Decimals are stored as a whole number (1234.) that is scaled, where as
Single & Double are stored as a fraction number (1.1234) that is scaled.
Seeing as both are represented as binary number internally, a binary whole
number can store all decimal numbers, where as a binary fraction cannot
store all decimal fractions. Hence the rounding issue you are seeing.
Hope this helps
Jay
"David Ricker" <da*************@hotmail.com> wrote in message
news:eW**************@TK2MSFTNGP12.phx.gbl... I am having problems adding two numbers. I am trying to add 1.005 and 1.007 to come up with 2.012. Should be easy enough right? Problem is
that I keep getting 2.0119999999999996 as my result. Why is this happening?
Is there a way that I can get it to return the right result? Any help is appreciated.
Thanks, David J. Ricker II
Thanks Jay,
At least now I know that it is by design.
"Jay B. Harlow [MVP - Outlook]" <Ja************@msn.com> wrote in message
news:%2***************@tk2msftngp13.phx.gbl... David, In addition to CJ & Cor's posts.
http://msdn.microsoft.com/library/de...egralTypes.asp 1.007 & 2.012 are by default a Double. Double & Single are base 2 floating point numbers.
Decimal is a base 10 floating point number. (technally base 2, but think
of it as base 10 ;-))
Decimals are stored as a whole number (1234.) that is scaled, where as Single & Double are stored as a fraction number (1.1234) that is scaled. Seeing as both are represented as binary number internally, a binary whole number can store all decimal numbers, where as a binary fraction cannot store all decimal fractions. Hence the rounding issue you are seeing.
Hope this helps Jay
"David Ricker" <da*************@hotmail.com> wrote in message news:eW**************@TK2MSFTNGP12.phx.gbl... I am having problems adding two numbers. I am trying to add 1.005
and 1.007 to come up with 2.012. Should be easy enough right? Problem is that I keep getting 2.0119999999999996 as my result. Why is this happening? Is there a way that I can get it to return the right result? Any help is appreciated.
Thanks, David J. Ricker II
* "David Ricker" <da*************@hotmail.com> scripsit: I am having problems adding two numbers. I am trying to add 1.005 and 1.007 to come up with 2.012. Should be easy enough right? Problem is that I keep getting 2.0119999999999996 as my result. Why is this happening? Is there a way that I can get it to return the right result? Any help is appreciated.
<http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<http://support.microsoft.com/default.aspx?scid=kb;de;42980>
--
Herfried K. Wagner [MVP]
<http://www.mvps.org/dotnet> This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Tom |
last post by:
A question for gui application programmers. . .
I 've got some GUI programs, written in Python/wxPython, and I've got
a help button and a help menu item. Also, I've got a compiled file
made with...
|
by: Sarir Khamsi |
last post by:
Is there a way to get help the way you get it from the Python
interpreter (eg, 'help(dir)' gives help on the 'dir' command) in the
module cmd.Cmd? I know how to add commands and help text to...
|
by: Sudheer Kareem |
last post by:
Dear All
Please tell me how to assosiate help files with my Vb.net
Project.
Regards
Sudheer
|
by: wukexin |
last post by:
Help me, good men. I find mang books that introduce bit "mang header
files",they talk too bit,in fact it is my too fool, I don't learn it, I have
do a test program, but I have no correct doing...
|
by: Colin J. Williams |
last post by:
Python advertises some basic service:
C:\Python24>python
Python 2.4.1 (#65, Mar 30 2005, 09:13:57) on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
With...
|
by: Corepaul |
last post by:
Missing Help Files
When I enter "recordset" as the keyword and search the Visual Basic Help index,
I get many topics of interest in the resulting list. But there isn't any
information available...
|
by: Steve |
last post by:
I have written a help file (chm) for a DLL and referenced it using Help.ShowHelp
My expectation is that a developer using my DLL would be able to access this help file during his development time...
|
by: Mark |
last post by:
I have loaded Visual Studio .net on my home computer and my laptop, but my
home computer has an abbreviated help screen not 2% of the help on my laptop.
All the settings look the same on both...
|
by: JonathanOrlev |
last post by:
Hello everybody,
I wrote this comment in another message of mine, but decided to post it
again as a standalone message.
I think that Microsoft's Office 2003 help system is horrible, probably...
|
by: hitencontractor |
last post by:
I am working on .NET Version 2003 making an SDI application that calls MS Excel 2003.
I added a menu item called "MyApp Help" in the end of the menu bar to show Help-> About.
The application...
|
by: ryjfgjl |
last post by:
ExcelToDatabase: batch import excel into database automatically...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: Vimpel783 |
last post by:
Hello!
Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
|
by: jfyes |
last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
|
by: CloudSolutions |
last post by:
Introduction:
For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
|
by: Defcon1945 |
last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
|
by: Shællîpôpï 09 |
last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome former...
| |