Please consider this code:
----------
Dim i As Decimal
Dim p As Decimal = 31D
i = (2D ^ (p - 1D)) * (2D ^ p - 1D)
----------
Variable "i" should be equal to "2305843008139952128" but, for some reason,
variable "i" becomes "2305843008139950000".
Am I doing something wrong, am I missing something or is this a bug? 2 1242
"xfx" <xf*@discussions.microsoft.com> schrieb Please consider this code:
----------
Dim i As Decimal Dim p As Decimal = 31D
i = (2D ^ (p - 1D)) * (2D ^ p - 1D)
----------
Variable "i" should be equal to "2305843008139952128" but, for some reason, variable "i" becomes "2305843008139950000".
Am I doing something wrong, am I missing something or is this a bug?
Another reason why Option Strict should always be On. You can enable it in
the project properties (General -> Build: Option Strict [On].
After switching it on, split your calculations into multiple steps to follow
the calculation of the final value.
Armin
xfx,
Are you using VS 2003 or VS 2005?
You might consider using a Shift operator instead of the Exponentiation
operator. Something like:
Dim i As Long
Dim p As Integer = 31
i = (1L << (p - 1)) * ((1L << p) - 1)
Remember the Exponentiation operator is defined for type Double only. So
although your operands are all Decimal, they are being implicitly being
converted to & from Double for the expression.
Hope this helps
Jay
"xfx" <xf*@discussions.microsoft.com> wrote in message
news:E8**********************************@microsof t.com...
| Please consider this code:
|
| ----------
|
| Dim i As Decimal
| Dim p As Decimal = 31D
|
| i = (2D ^ (p - 1D)) * (2D ^ p - 1D)
|
| ----------
|
| Variable "i" should be equal to "2305843008139952128" but, for some
reason,
| variable "i" becomes "2305843008139950000".
|
| Am I doing something wrong, am I missing something or is this a bug? This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Suvodip Mukherjee |
last post by:
Hi,
I am facing a problem using the DBLIBRARY API dbconvert().
When dbconvert() is used to convert a MS SQL Server MONEY data type to
DECIMAL and then to STRING, the scale is getting lost.
eg....
|
by: swhite76 |
last post by:
I have a section of code that adds some double values and gives an incorrect
result. This occurs with data that isn't really waht I would call high
precision. An example is the following code...
|
by: teddysnips |
last post by:
Application is a Work Tracking/Timesheet database.
The increments of work are stored in the TimesheetItem table. This
contains, inter alia, the Work Code, the Start and the Duration that
the...
|
by: Carl |
last post by:
Can you tell me what is wrong with this syntax ?
string select = "UPDATE .. " +
"(,,,,,,,
,,,,
,,, ,
, , ) " +
" VALUES
(@id,@clientid,@total,@tps,@tvq,@gtotal,@datefac,@datepay,
|
by: Brian Tkatch |
last post by:
I think the VB .NET parameter has incorrect automatic argument when it
comes to DECIMAL. That is, precision is never set.
Example:
CREATE a TABLE with a DECIMAL in it.
Start a VB Project...
|
by: RipperT |
last post by:
Don't know if this group covers web apps, but here goes. In VS 2005, I am
trying to get variables to hold thier values during postback from the
server. I convert a text box's user-keyed value to an...
|
by: =?Utf-8?B?RmFiaW8=?= |
last post by:
Hi
Looking at the following code ...
double a = 100.1;
double b = 0.1;
double c = a + b;
I would like to know why the value of c is 100.19999999999999 and not 100.2?
|
by: roamnet |
last post by:
hi
i created database file with .mdf extention ,sql server as a source
and use grid view to display data
there're no problem in data retrieve and display,but i want to edit it
or insert new...
|
by: Elroy Flynn |
last post by:
Decimal a = 42.485M;
Decimal b = 48M;
Decimal c = a / b;
Console.WriteLine(c.ToString());
gives the result 0.8851041666666666666666666667
The correct, exact answer is .885.
I know that...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome former...
|
by: ryjfgjl |
last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
| |