473,387 Members | 1,545 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,387 software developers and data experts.

Numeric precision

I'm just learning C# so for practice, I decided to try and duplicate some of
the functionality of the Microsoft Calculator from scratch.

I used double precision (64 bit) types I.E.

private double window;

but I find that it doesn't count as high as Microsoft Calculator. I
eventuallu get to a point where the window displays "infinity" but the
Microsoft Calculator seems to go on forever displaying numbers with an
exponent.

I tried using decimal types which should give me 128 bits of precision but I
cet a compile error with the following:

private decimal window = 4;

window = Math.Sqrt(window);

// compiler error The best overloaded method match for
'System.Math.Sqrt(double)' has some invalid arguments

I assume that the decimal type is for money only and won't do what I want.
So how can I cet more precision?

Edwin
Nov 17 '05 #1
1 2792
Edwin,
// compiler error The best overloaded method match for
'System.Math.Sqrt(double)' has some invalid arguments
You need casts in order for the code to compile:

window = (decimal) Math.Sqrt((double) window);

This won't give you the precision you need.
I assume that the decimal type is for money only and won't do what I want.
So how can I get more precision?


There's no implementation of Sqrt that works on a decimal argument. You
could build your own using Newton-Raphson method for instance.

Regards - Octavio
Nov 17 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
by: Matt Feinstein | last post by:
Hi all-- I'm new to Python, and was somewhat taken aback to discover that the core language lacks some basic numerical types (e.g., single-precision float, short integers). I realize that there...
1
by: Tariq | last post by:
I've a SQL view performing a number of very precise calculations ie. 495/(1.112 - (.00043499*((Col1 + Col2 + Col3))) + ( .00000055 * ((Col1 + Col2 + Col3)) * ((Col1 + Col2 + Col3))) - (.00028826...
6
by: JP | last post by:
Why in the world is this happening: double MyCount=0.00; MyCount = (100 * 2611508) / 37792454; Response.Write(MyCount.ToString("F2"); Always return a whole # (in this case 6). The value...
5
by: David Garamond | last post by:
In Interbase and Firebird, NUMERIC is implemented as 64-bit integer. This limits the range to NUMERIC(18, *) but for many uses that's adequate. And moreover it's fast and efficient. Is there a...
6
by: M.A. Oude Kotte | last post by:
Hi All, I hope this is the correct mailing list for this question. But neither postgresql.org nor google could help me out on this subject. I did find one disturbing topic on the mailing list...
2
by: mdeaver2003 | last post by:
I'm trying to output a double using a precision that varies, governed by the value of a precision variable. In C I can do it like this: double pi = 3.14159; int prec = 4; printf( "%.*f",...
11
by: Pieter | last post by:
Hi, I'm having some troubles with my numeric-types in my VB.NET 2005 application, together with a SQL Server 2000. - I first used Single in my application, and Decimal in my database. But a...
0
by: neojhz | last post by:
Hi all, I've been encountering this error "numeric value out of range" during update on a .Net Web Service function and found out it was due to excess digits on a field's value on the xml. What I...
8
by: Frank Swarbrick | last post by:
My DBA says that a column defined, for instance, as DECIMAL(11,2) and containing a value of 1.00 takes up no more space on the database disk than a column defined as DECIMAL(7,2) and containing a...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.