Hi All,
A very simple but confusing problem, I've now a lot of calculations in stored procedures that are all inaccurate!, e.g
1234567890 * (1.0 - 0.5/14.0)
the result is actually : 1,190,476,179.~
but SQL gives me: 1,190,476,532.3 7.~ [MSSQL 2005/SP2]
Is any way to force SQLServer to use maximum precision in constants and division results in a formula?
1  5517  deepuv04 227
Recognized Expert New Member
Hi All,
A very simple but confusing problem, I've now a lot of calculations in stored procedures that are all inaccurate!, e.g
1234567890 * (1.0 - 0.5/14.0)
the result is actually : 1,190,476,179.~
but SQL gives me: 1,190,476,532.3 7.~ [MSSQL 2005/SP2]
Is any way to force SQLServer to use maximum precision in constants and division results in a formula?
Hi,
you can achieve this in two ways
for example if you have the datatypes as follows
declare @a numeric(5,2)
declare @b numeric(5,2)
declare @c numeric(5,2)
set @a = 10.0
set @b = 5.0
set @c = @a/@b
select @a/@b,@c -- see the differencein the the output of two columns
another way is calculate the values first and convert to some round value
or limit the precision value
SELECT CONVERT(NUMERIC (12,2), 1234567890 * (1.0 - 0.5/14.0) )
SELECT 1234567890 * (1.0 - 0.5/14.0)
thanks
Sign in to post your reply or Sign up for a free account.
Similar topics | |
by: Brad Tilley |
last post by:
Hello,
What is the proper way to limit the results of division to only a few
spaces after the decimal? I don't need rocket-science like precision.
Here's an example:
1.775 is as exact as I need to be and normally, 1.70 will do.
Thank you,
Brad
|
by: Dave Brown |
last post by:
Hi all,
I'm having trouble understanding which data type to use to get results I
expect.
using float, decimal, and double I have tried the following, ..
float test = 55/60;
I always get 0 as the results.
I expect 0.916666
|
by: John C Kirk |
last post by:
I've come across an odd situation, where doing a floating point
division produces different results for the same numbers. Basically,
there are 4 ways to run this application:
A) Debug build, inside the IDE
B) Debug build, outside the IDE (e.g. launched from Explorer)
C) Release build, inside the IDE
D) Release build, outside the IDE
Using methods A-C produces one result, and using method D produces a
|
by: nessuno |
last post by:
I can't find any discussion of this question in this NG.
I'd like to implement some variable precision integer arithmetic in C,
and do it efficiently. A problem arises with the divide/remainder
operations. I'm using gcc on x86, but a lot of what I say applies to
other architectures.
First, I assume the div and ldiv functions are more efficient than
using / and % (obviously it depends on the compiler and level of
optimization, but...
|
by: Pakna |
last post by:
Why do I have a zero result on this query for all non-zero rows? Do I
have to declare specifically the precision of ratio division? How do I
do that? And is there a way to make DB2 'ignore' division by zero and
just punch up a 0 result where such division is encountered so that I
can remove the where-clause?
Here's the query, I think it's very clear what I want to do:
select substr(tablespace_name, 1, 120) as TBSPC_NAME, used_pages,...
| | |
by: jamesonang |
last post by:
Supposed unsigned int(32 bits) is the largest number that computer can
represent with a single variable.
Now, i have a big integer ( less than 64 bit, but great than 32 bit) .
i represent it by this way:
unsigned int dividend :
divident store low 32 bits of big integer, dividend store high 32
bits of big integer.
|
by: krypto.wizard |
last post by:
Last month I appeared for an interview with EA sports and they asked
me this question.
How would you divide a number by 7 without using division operator ?
I did by doing a subtraction and keeping a counter that kept a tab on
how many times I subtracted.
Later, the EA sport guy told me that of course there are can be better
technique by using bit operator.
|
by: Artemio |
last post by:
Hello all!
I just stumbled across a weird problem with precision of a division
operation. I am on Mac OS X, GCC 4.0.1.
Say I have two float or double numbers, and I want to divide one by
another, e.g.
float A, B, C;
C = A / B;
|
by: Mike |
last post by:
I'm running DB2 v7 for z/OS. When I use SPUFI, SELECT CAST(6.0 AS
FLOAT)/CAST(10.0 AS FLOAT) FROM SYSIBM.SYSDUMMY1 returns
0.6000000000000000E+00. When I use DSNTIAUL,DSNTEP2, or DSNALI (call
attach facility), the same statement returns 0.59999999999999999E 00.
The only reason I$B!G(Bve heard to explain this behavior is that float
stores too much precision, but I$B!G(Bve used double-precision floating-
point data types in SQL...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
