On Thu, 13 Nov 2008 07:02:12 -0800 (PST), Tim wrote:
>Folks,
Can anyone thow some clarifying light on the following?
Hi Tim,
In addition to Erland's reply, here are a few othe rpoints to keep in
mind:
1) What you see on your screen is how the front end chooses to render
the numbers. Not all clients use the same number of digits when
displaying a floating point number. One way to get a reliable look at
the real precision is to cast the number to e.g. DECIMAL(38,30) (unless
the number is too big or too small to fit is the range of fixed point
numbers.
2) Precision of a floating point number is measured in "significant
figures", i.e. it's calculated from the first non-zero number. So the
numbers 123000000, 123, 1.23, and 0.0000123 all have a precision of
three significant figures.
3) Floating point numbers are actually stored in some binary format.
Some numbers that can be represented as an exact number in decimal can
not be in binary (just as 1/3 can not be represented exactly in
decimal). The actual precision is based on the internals of the actual
storage; the precision quoted in Books Online is an approximation (and I
believe it's a "safe" approximation, i.e. you will get at least that
precision for any number - but don't take my word for this, I'm not 100%
sure).
Does this answer all your questions?
--
Hugo Kornelis, SQL Server MVP
My SQL Server blog:
http://sqlblog.com/blogs/hugo_kornelis