Hi, I am using Sybase 12.5 dataserver and ASP.NET
I am calling a stored procedure from my asp.net page, in this stored
procedure, I have 2 float fields that are returned. One is directly
taken from one float column of a table, and the other one is a
substraction of 2 float columns of a table. The first field is all the
time ok, whereas the calculated field is almost all the time wrong,
that is it returns things like :
1,332294 - 1,334709 = -0,0024149999999 9983 or
1,334 - 1,334093 = -9,2999999999898 7E-5 or
1,3281 - 1,328256 = -0,0001560000000 00045
Sometimes it's ok like :
1,8774 - 1,864693 = 0,012707 or
103,96 - 103,8564 = 0,1036
When I launch this stored procedure with isql, which is the basic
inline tool of Sybase, I get all the time the right calculations. How
is that? What can I set in my .Net environment to do the right
calculation all the time?
Thank you for your help,
JULIEN
2  1407 
see http://www.yoda.arachsys.com/csharp/floatingpoint.html
julien wrote: Hi, I am using Sybase 12.5 dataserver and ASP.NET
I am calling a stored procedure from my asp.net page, in this stored
procedure, I have 2 float fields that are returned. One is directly
taken from one float column of a table, and the other one is a
substraction of 2 float columns of a table. The first field is all the
time ok, whereas the calculated field is almost all the time wrong,
that is it returns things like :
1,332294 - 1,334709 = -0,0024149999999 9983 or
1,334 - 1,334093 = -9,2999999999898 7E-5 or
1,3281 - 1,328256 = -0,0001560000000 00045
Sometimes it's ok like :
1,8774 - 1,864693 = 0,012707 or
103,96 - 103,8564 = 0,1036
When I launch this stored procedure with isql, which is the basic
inline tool of Sybase, I get all the time the right calculations. How
is that? What can I set in my .Net environment to do the right
calculation all the time?
Thank you for your help,
JULIEN
Hi Hans,
That is an excellent resource! I would recommend it to almost any
programmer.
Another possibility in this particular case might be the use of types. By
types I mean the type of the numbers being computed and the default types of
the .Net platform. In .Net operators are overloaded for different numeric
types. For example, multiplying 2 floats results in a float result.
Multiplying 2 doubles returns a double result. But what happens when you
multiply a float by a double, or an integer by a float? What data type is
returned? And what happens when you perform math on a variable of a certain
type and a numeric literal (e.g. 1.5 or 23)? If you don't cast the types
correctly and/or specify the types of literals, you can get unexpected
results, depending upon the defaults used by the .Net platform. So that
could also be an issue here.
The .Net SDK details all of the default types, the return types for
differing types in a math operation, and the types of the overloaded
operators.
--
HTH,
Kevin Spencer
Microsoft MVP
..Net Developer
What You Seek Is What You Get.
"Hans Kesting" <ne***********@ spamgourmet.com > wrote in message
news:uA******** *****@TK2MSFTNG P12.phx.gbl... see http://www.yoda.arachsys.com/csharp/floatingpoint.html
julien wrote: Hi, I am using Sybase 12.5 dataserver and ASP.NET
I am calling a stored procedure from my asp.net page, in this stored
procedure, I have 2 float fields that are returned. One is directly
taken from one float column of a table, and the other one is a
substraction of 2 float columns of a table. The first field is all the
time ok, whereas the calculated field is almost all the time wrong,
that is it returns things like :
1,332294 - 1,334709 = -0,0024149999999 9983 or
1,334 - 1,334093 = -9,2999999999898 7E-5 or
1,3281 - 1,328256 = -0,0001560000000 00045
Sometimes it's ok like :
1,8774 - 1,864693 = 0,012707 or
103,96 - 103,8564 = 0,1036
When I launch this stored procedure with isql, which is the basic
inline tool of Sybase, I get all the time the right calculations. How
is that? What can I set in my .Net environment to do the right
calculation all the time?
Thank you for your help,
JULIEN
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: Bengt Richter |
last post by:
Peculiar boundary cases:
>>> 2.0**31-1.0
2147483647.0
>>> int(2147483647.0)
2147483647L
>>> int(2147483647L )
2147483647
>>>
>>> -2.0**31
|
by: Bryan R. Meyer |
last post by:
I am a relatively new C++ programmer and am attempting to write a
function that takes a number of type float and adds commas to it in
the appropriate places. In order to manipulate the number to add the
commas, I convert it to a string (specifically a char rather than an
actual string object) using the
gcvt function as shown below.
char amt;
gcvt(amount,50,amt);
|
by: Andy |
last post by:
Hi,
I don't know if this is the correct group to post this, but
when I multiply a huge floating point value by a really
small (non-zero) floating point value, I get 0 (zero) for the result.
This creates a big hole in a 32-bit timer routine I wrote.
Questions.
1. Why does this happen?
2. Is there C macros/functions I can call to tell me
when two non-zero numbers are multiplied and the
|
by: vijay |
last post by:
Hello,
What happens to float variable in loops. For example,
float f=8.7;
if(f<8.7)
printf("less");
else if(f==8.7)
printf("equal");
else if(f>8.7)
|
by: James Thurley |
last post by:
According to the docs, floats are 32 bit and doubles are 64 bit. So
using floats should be faster than using doubles on a 32 bit processor,
and my tests confirm this. However, most of the Math methods deal with
doubles and I'm having problems with casting them back to floats.
For example:
(double)0.1f = 0.10000000149011612
I don't need the accuracy of doubles, but the speed of floats would be
benificial, but it seems I'm either...
| | |
by: Partho |
last post by:
I have a float variable which I need to add to a short variable. How do
I do this? Do I have to typecast or is there a way around? I tried
typecasting the float to a short, but that gives me a 0 or +/-1, which
is unacceptable.
Any pointers to how to get over this?
|
by: iulian.ilea |
last post by:
I have this functions, c3 and c2 that computes and return a float
number. The problem is that wen I pass the result of c2 (float number)
to c3(), in function apears to be an integer. Ex.:
$calc2=c2(params); (float number)
$calc3=c3($calc2); ($calc2 - float number)
echo $calc3; (is not the number that supposed to be, because c4
computes $calc2 as if it were an integer: if $calc=39.96 it wil be
converted in 39).
What is the problem? I...
|
by: Hank Stalica |
last post by:
I'm having this weird problem where my code does the following
conversion from string to float:
27000000.0 -27000000.00
2973999.99 -29740000.00
2989999.13 -2989999.25
The number on the left is the string I get after tokenizing a bigger
string. The number on the right is the number I get after the conversion.
|
by: docmur |
last post by:
Okay heres my problem I have a problem called Calc which the user enters 2 numbers and a sign to be done with what is entered
EX:
./calc 200 + 200
or
./calc 200.20 + 200
Okay thats easy where I can't seem to get, my prof wants the input checked to see if the char *argv or char *argv contains a float or int. so I have to check for a decimal point.
I know I can search the number like a string because it's in an array of char but...
|
by: DirtyRasa |
last post by:
Here is what my professor told me to to.
Write a function that converts a string to a float and returns a float.
Test your function by entering f4 and with 4f. Both should say
Data entered was not a number. Try again.
and ask the user to type in another number.
Here's what I have so far.
Code:
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| | |
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
