What Algorithm Do Single and Double Use when Storing Imprecise Values?

[Nathan Sokalski]

I know that Single and Double can only store values that can be stored as
x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use to
determine what value to store? Thanks.
--
Nathan Sokalski
nj********@hotmail.com
http://www.nathansokalski.com/

[Alberto Poblacion]

"Nathan Sokalski" <nj********@hotmail.comwrote in message
news:OL**************@TK2MSFTNGP03.phx.gbl...

>I know that Single and Double can only store values that can be stored as
x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use
to determine what value to store? Thanks.

They use a mantissa and an exponent, so that a number is expressed as
(mantissa)*10^(exponent). We do this frequently in everyday engineering work
when we say for instance, that something took "1.23*10^-6 seconds". However,
when speaking about Single and Double, the "10" has to be interpreted in
base 2 (meaning that it is a "2" in base 10). The exponent and mantissa are
also in base 2. The precission of the mantissa is limited, because the
mantissa and exponent and their signs have to fit into 4 bytes (Single) or 8
bytes (Double). The numbers that can be represented exactly are those that
can be converted into base 2 within that number of bits. For instance, 0.5
(base10) is 0.1 (base 2), but some numbers that have a small number of
decimals in base 10 have an infinite number of decimal places once converted
to base 2, so they will be truncated when assigned to the mantissa, and
therefore they will not be "exact".

[Armin Zingler]

"Nathan Sokalski" <nj********@hotmail.comschrieb

>I know that Single and Double can only store values that can be stored as
x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use
to determine what value to store? Thanks.

As the documentation
http://msdn.microsoft.com/en-us/libr...em.single.aspx
says, the types comply with IEEE 754 which also defines rounding rules.
Search for it; you'll find wikipedia etc.

Armin

[Ken Halter]

"Nathan Sokalski" <nj********@hotmail.comwrote in message
news:OL**************@TK2MSFTNGP03.phx.gbl...

>I know that Single and Double can only store values that can be stored as
x/2^y, such as 0.5 and 0.125. But when they attempt to store values such as
0.1 or 0.2 whose exact value cannot be stored, what algorithm do they use
to determine what value to store? Thanks.
--
Nathan Sokalski
nj********@hotmail.com
http://www.nathansokalski.com/

What Every Computer Scientist Should Know About Floating-Point Arithmetic
http://docs.sun.com/source/806-3568/ncg_goldberg.html

Here's a PDF version
http://www.physics.ohio-state.edu/~d...point_math.pdf

[Rudy Velthuis]

Ken Halter wrote:

What Every Computer Scientist Should Know About Floating-Point
Arithmetic http://docs.sun.com/source/806-3568/ncg_goldberg.html

Here's a PDF version

http://www.physics.ohio-state.edu/~d...point_math.pdf

Interesting link, thanks. I can use such a .pdf from time to time. It
is amazing that CS people know so little about it.

--
Rudy Velthuis http://rvelthuis.de

"I once heard two ladies going on and on about the pains of
childbirth and how men don't seem to know what real pain is. I
asked if either of them ever got themselves caught in a zipper."
-- Emo Philips.

[Nathan Sokalski]

I understand how they are stored, that is very simple. My question was what
algorithm they use to determine what value to store. Let me rephrase what I
meant by this. When a value is assigned to a variable that cannot be stored
as an exact value, such as the following:

Dim x As Byte = 1.2

Since the exact value cannot be stored, what algorithm is used to determine
what value will be stored? Since the value is obviously not exact, it must
be rounded up or down, so how do I know which it is?
--
Nathan Sokalski
nj********@hotmail.com
http://www.nathansokalski.com/

"Alberto Poblacion" <ea******************************@poblacion.orgwro te
in message news:%2****************@TK2MSFTNGP02.phx.gbl...

"Nathan Sokalski" <nj********@hotmail.comwrote in message
news:OL**************@TK2MSFTNGP03.phx.gbl...

>>I know that Single and Double can only store values that can be stored as
x/2^y, such as 0.5 and 0.125. But when they attempt to store values such
as 0.1 or 0.2 whose exact value cannot be stored, what algorithm do they
use to determine what value to store? Thanks.

They use a mantissa and an exponent, so that a number is expressed as
(mantissa)*10^(exponent). We do this frequently in everyday engineering
work when we say for instance, that something took "1.23*10^-6 seconds".
However, when speaking about Single and Double, the "10" has to be
interpreted in base 2 (meaning that it is a "2" in base 10). The exponent
and mantissa are also in base 2. The precission of the mantissa is
limited, because the mantissa and exponent and their signs have to fit
into 4 bytes (Single) or 8 bytes (Double). The numbers that can be
represented exactly are those that can be converted into base 2 within
that number of bits. For instance, 0.5 (base10) is 0.1 (base 2), but some
numbers that have a small number of decimals in base 10 have an infinite
number of decimal places once converted to base 2, so they will be
truncated when assigned to the mantissa, and therefore they will not be
"exact".

[Steve Thackery]

Since the exact value cannot be stored, what algorithm is used to

determine what value will be stored? Since the value is obviously not
exact, it must be rounded up or down, so how do I know which it is?

But you've already been told: IEEE 754.

SteveT