473,383 Members | 1,829 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,383 software developers and data experts.

Precision Problem

Hi,

I can't seem to get a stored procedure to return decimal places. Here are the steps to re-create the problem:

--first create the following procedure
CREATE PROCEDURE test_precision
AS
BEGIN
RETURN 5.2
END

--then run the following code
DECLARE @x DECIMAL(18,2)
EXECUTE @x = test_precision
SELECT @x

I would like this code to return 5.20, but instead it returns 5.00. Any assistance would be greatly appreciated. Thanks.
Oct 30 '06 #1
2 1301
scripto
143 100+
if you are using SQL 2005 that is the problem. Your stored Procedure would work correctly in SQL 2000.

The work around
declare @x FLOAT(18,2)

give that a try.
Oct 30 '06 #2
scripto
143 100+
ooops!! allow me to take that back - wrong thought process. sorry.

you need to edit your proc to read 5.20
Oct 30 '06 #3

Sign in to post your reply or Sign up for a free account.

Similar topics

0
by: John Benson | last post by:
I got the multiplication precision rule wrong: relative precision is defined as the uncertainty divided by the magnitude of the measured quantity, e.g. 10 meters +/- 1 meter gives an...
1
by: Tariq | last post by:
I've a SQL view performing a number of very precise calculations ie. 495/(1.112 - (.00043499*((Col1 + Col2 + Col3))) + ( .00000055 * ((Col1 + Col2 + Col3)) * ((Col1 + Col2 + Col3))) - (.00028826...
5
by: DAVID SCHULMAN | last post by:
I've been trying to perform a calculation that has been running into an underflow (insufficient precision) problem in Microsoft Excel, which calculates using at most 15 significant digits. For this...
21
by: syntax | last post by:
hi, i need to get high presion float numbers. say, i need pi = 22/7.0 = 3.142857....(upto 80 digits) is it possible ? does gcc/g++ compiler can give such type of high precision?? plz...
15
by: michael.mcgarry | last post by:
Hi, I have a question about floating point precision in C. What is the minimum distinguishable difference between 2 floating point numbers? Does this differ for various computers? Is this...
3
by: Madan | last post by:
Hi all, I had problem regarding float/double arithmetic only with + and - operations, which gives inaccurate precisions. I would like to know how the arithmetic operations are internally handled...
5
by: rocknbil | last post by:
Hello everyone! I'm new here but have been programming for the web in various languages for 15 years or so. I'm certainly no "expert" but can keep myself out of trouble (or in it?) most of the time....
10
by: Artemio | last post by:
Hello all! I just stumbled across a weird problem with precision of a division operation. I am on Mac OS X, GCC 4.0.1. Say I have two float or double numbers, and I want to divide one by...
3
by: Gugale at Lincoln | last post by:
Hi, I am working on an application which uses date as a primary key. All my records are at least a few millisecond apart and are in the form "20070630T12:50:24.207". SQL Server has a precision...
137
by: mathieu.dutour | last post by:
Dear all, I want to do multiprecision floating point, i.e. I want to go beyond single precision, double precision and have quadruple precision, octuple precision and the like, and possibly with...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.