473,804 Members | 4,518 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

precision loss??

does anyone know why this expression:
str(csng(18333. 332))

equals:
"18333.33"

?

why am i losing precision from this operation?


Nov 20 '05 #1
4 1197
"hunt n peck" <x> schrieb
does anyone know why this expression:
str(csng(18333. 332))

equals:
"18333.33"

?

why am i losing precision from this operation?


Welcome to the precision of a single value. ;-) To store more precise
values, you can use the Double data type.
--
Armin

http://www.plig.net/nnq/nquote.html
http://www.netmeister.org/news/learn2quote.html

Nov 20 '05 #2
But of course. Thanks.
It is confusing when in the command window you type:
? csng(18333.332)

and you get:
18333.332
Another question:

When should I use Decimal? Is it better than Double?



"Armin Zingler" <az*******@free net.de> wrote in message
news:u0******** ******@TK2MSFTN GP12.phx.gbl...
"hunt n peck" <x> schrieb
does anyone know why this expression:
str(csng(18333. 332))

equals:
"18333.33"

?

why am i losing precision from this operation?


Welcome to the precision of a single value. ;-) To store more precise
values, you can use the Double data type.
--
Armin

http://www.plig.net/nnq/nquote.html
http://www.netmeister.org/news/learn2quote.html

Nov 20 '05 #3
"hunt n peck" <x> schrieb
does anyone know why this expression:
str(csng(18333. 332))

equals:
"18333.33"

?

why am i losing precision from this operation?

In addition, a link taken from a recent post from Herfried:

http://research.microsoft.com/~holla...ieeefloat.html
--
Armin

Nov 20 '05 #4
"hunt n peck" <x> schrieb
But of course. Thanks.
It is confusing when in the command window you type:
? csng(18333.332)

and you get:
18333.332
Good question. Hmmm... currently I don't know. Maybe it does not really
convert it to a single value like it is done by the compiler.

If I write

dim s as single
s = 18333.332
msgbox s

the msgbox returns "18333.33" whereas

? s

in the immediate window displays "18333.332" . Seems to use a different
conversion mechanism.
Another question:

When should I use Decimal? Is it better than Double?


It depends on the purpose. The data types are described in the docs.
--
Armin

http://www.plig.net/nnq/nquote.html
http://www.netmeister.org/news/learn2quote.html

Nov 20 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
7854
by: Roger Leigh | last post by:
Hello, I'm writing a fixed-precision floating point class, based on the ideas in the example fixed_pt class in the "Practical C++ Programming" book by Steve Oualline (O' Reilly). This uses a long int to store the value, and the precision (number of decimal points) is variable (it's a templated class): template <size_t _decimal_places = 4> class FixedFloat {
2
2805
by: Brian van den Broek | last post by:
Hi all, I guess it is more of a maths question than a programming one, but it involves use of the decimal module, so here goes: As a self-directed learning exercise I've been working on a script to convert numbers to arbitrary bases. It aims to take any of whole numbers (python ints, longs, or Decimals), rational numbers (n / m n, m whole) and floating points (the best I can do for reals), and convert them to any base between 2 and...
5
7998
by: Bryan R. Meyer | last post by:
I am a relatively new C++ programmer and am attempting to write a function that takes a number of type float and adds commas to it in the appropriate places. In order to manipulate the number to add the commas, I convert it to a string (specifically a char rather than an actual string object) using the gcvt function as shown below. char amt; gcvt(amount,50,amt);
16
6269
by: BigMan | last post by:
How can I check if assignment of a float to a double (or vice versa) will result in loss of precision?
13
5352
by: Sebastian | last post by:
Hi, yesterday I discovered something that won't come as a surprise to more experienced C++ - programmers - but I was genuinely amazed. I had this line of code: float x = 0.9f; and when I had a look at it in the VS.NET 2003 debugger, it said x had avalue of 0.89999998 ! When I printf'ed x to a logfile, it was 0.899999976158142090000000000000. Now, why on earth is that? I understand that the results of floatingpoint operations will always...
1
2684
by: Nishant Deshpande | last post by:
On Solaris 64-bit: sizeof(long) = 8 sizeof(double) = 8 !!! So i figure i can't store a long with the same precision in a double. But, when i create a long and assign its largest value to it, then assign this to a double, then subtract the double from the long, I get 0.
15
2940
by: giff | last post by:
Hi all, I have a doubt, I'll try to expose it to you as clearly as I can, maybe it is not completely in topic, sorry about that. I have a few vectors of doubles (output of some calculations) and I check the norm and the dot product between couples of them before storing them in a txt file, obtaining the expected results. The vectors are orthonormal so the norm=1 and the dot product gives
10
8863
by: Artemio | last post by:
Hello all! I just stumbled across a weird problem with precision of a division operation. I am on Mac OS X, GCC 4.0.1. Say I have two float or double numbers, and I want to divide one by another, e.g. float A, B, C; C = A / B;
6
17095
by: Matthew | last post by:
Hi, I want to change the precision level of floating point variables and calculations which is done in php.ini. However the server I rent for my domain does not give me access to php.ini, they say 'for security reasons'. Can the precision level be changed by PHP code as needed?
0
9579
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10332
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
10077
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9150
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7620
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5521
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5651
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4299
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3820
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.