Overflow with 'long' variable type?  | Familiar Sight | | Join Date: Mar 2007 Location: igirisu~
Posts: 184
| |
Hi all, I'm relatively new to VB but I know that an overflow occurs when you try to make a variable be too high or low for what you've prepared it for, e.g. 300 when you've prepared it for use as a byte.
Here's what happening - I am trying to make a function which 'decodes' a color number (I mean, the kind of values object.BackColor can be, and which the RGB() function gives back) into its separate R, G and B values each from 0 to 255.
But here's the problem. To do that, I use this code: -
Dim ColumnValue As Long
-
Dim CurrentColumn As Integer
-
-
For CurrentColumn = 255 To 0 Step -1
-
-
ColumnValue = 255 * 255 * CurrentColumn 'This is where it overflows
-
(This is the part of it which is causing an overflow)
CurrentColumn is sometimes needed to be higher than 255, that's why I didn't make it a 'byte' type.
When I test this (i.e. F5, not compiling to an EXE) VB says it overflows. Even when I type into the Immediate Window:
it still overflows.
According to this site, a 'long' data type in VB can be up to 2,147,483,647 (which is 10 digits long), and yet 255^3 is only 16,581,375 (8 digits long).
Why does VB overflow when it tries to do this? (Especially why does it also overflow in the Immediate Window?)
| | Moderator | | Join Date: Oct 2006 Location: Australia
Posts: 7,748
| | | re: Overflow with 'long' variable type?
The problem is that VB is converting things internally to the smallest format that it thinks it can get away with. And it seems to make very poor choices in this regard. Most likely it is converting to Integer in this case.
Try this example in the immediate window...
? 255! * 255! * 255!
Then try this modified line... - ColumnValue = 255! * 255! * CurrentColumn 'This used to overflow
What we are doing is using the data type definition character "!" to ensure that all of the values used in the calculation are Long values. Then VB doesn't convert down.
| | Familiar Sight | | Join Date: Mar 2007 Location: igirisu~
Posts: 184
| | | re: Overflow with 'long' variable type?
Ah, I see. Thank you, that makes it work fine!
Out of interest though, you said that sort of 'forces' VB to keep it a Long data type. Can you force it into keeping it as other data types?
Just wondering if it might be useful to know in the future.
| | Moderator | | Join Date: Oct 2006 Location: Australia
Posts: 7,748
| | | re: Overflow with 'long' variable type?
Quote:
Originally Posted by Robbie Ah, I see. Thank you, that makes it work fine!
Out of interest though, you said that sort of 'forces' VB to keep it a Long data type. Can you force it into keeping it as other data types?
Just wondering if it might be useful to know in the future. Sure.
It's not actually forcing VB to do anything in particular about the calculation, just specifying that the 255 (in this case) is a Long value. We could have put 255# to make it a double-precision decimal value. (This would be less efficient and make things slower, but unless you were doing the calculation millions of times the difference would be too small to notice.)
It's just that when setting up a calculation, VB takes into account all of the data types involved (or maybe just the "smallest" - I forget). So ensuring your data is the correct type can affect the way it works, as you've seen. You could have achieved the same effect by, say, declaring something like: Public Const TwoFiftyFive As Long = 255
and then using that constant in your calculation.
| | Familiar Sight | | Join Date: Mar 2007 Location: igirisu~
Posts: 184
| | | re: Overflow with 'long' variable type?
Quote:
Originally Posted by Killer42 Sure.
It's not actually forcing VB to do anything in particular about the calculation, just specifying that the 255 (in this case) is a Long value. We could have put 255# to make it a double-precision decimal value. (This would be less efficient and make things slower, but unless you were doing the calculation millions of times the difference would be too small to notice.)
It's just that when setting up a calculation, VB takes into account all of the data types involved (or maybe just the "smallest" - I forget). So ensuring your data is the correct type can affect the way it works, as you've seen. You could have achieved the same effect by, say, declaring something like:
Public Const TwoFiftyFive As Long = 255
and then using that constant in your calculation. Okay, I see now. But actually, I am doing it about 200,000 times so the speed is quite important (because it needs to do it for every pixel in an image). Although it's managing it now at about quarter of a second for a thumbnail-sized image, which is OK for how I'm using it.
I also managed to make it work by doing: -
ColumnValue = (255 ^ 1) * 255
-
...which also seems kinda weird...
But thanks for the info. :)
| | Moderator | | Join Date: Oct 2006 Location: Australia
Posts: 7,748
| | | re: Overflow with 'long' variable type?
Quote:
Originally Posted by Robbie Okay, I see now. But actually, I am doing it about 200,000 times so the speed is quite important (because it needs to do it for every pixel in an image). Although it's managing it now at about quarter of a second for a thumbnail-sized image, which is OK for how I'm using it.
I also managed to make it work by doing:
-
ColumnValue = (255 ^ 1) * 255
-
...which also seems kinda weird...
But thanks for the info. :) No problem.
Performing a calculation 200,000 times is really not much for a modern processor. I doubt you'll see a big difference, though it may be enough to notice. Might be interesting to find out, though.
I always try to use Long wherever possible, for processing efficiency. Unless I'm short of memory or need a really huge array, in which case I might try to save space by using Integer or Byte if possible. On a 32 bit processor, Long is (supposedly) the fastest size to process.
|  | Site Addict | | Join Date: Feb 2007
Posts: 579
| | | re: Overflow with 'long' variable type?
ur problem is solved
try this code
see this is the game of explicit type conversions
use explicit type conversions
as shown here -
-
Dim a&
-
Private Sub Form_Load()
-
a = Clng(255) * Clng(255) * Clng(255)
-
End Sub
-
-
| | Moderator | | Join Date: Oct 2006 Location: Australia
Posts: 7,748
| | | re: Overflow with 'long' variable type?
Quote:
Originally Posted by vijaydiwakar ur problem is solved
try this code
see this is the game of explicit type conversions
use explicit type conversions
as shown here
- Dim a&
-
Private Sub Form_Load()
-
a = Clng(255) * Clng(255) * Clng(255)
-
End Sub
I have to disagree on this one.
You are executing the Clng() function at runtime. Three times! This surely has to be more work than telling the compiler to use the correct format to begin with. Or to put it more succinctly, I believe a = 255! * 255! * 255!
will be processed much more efficiently than a = Clng(255) * Clng(255) * Clng(255)
and just for once, shouldn't require any more memory (probably slightly less, in fact).
I do agree that the conversion functions such as Clng() are useful in many cases. But not for this simple example using literal values.
| | Site Addict | | Join Date: Feb 2007
Posts: 579
| | | re: Overflow with 'long' variable type?
Quote:
Originally Posted by Killer42 I have to disagree on this one.
You are executing the Clng() function at runtime. Three times! This surely has to be more work than telling the compiler to use the correct format to begin with. Or to put it more succinctly, I believe
a = 255! * 255! * 255!
will be processed much more efficiently than
a = Clng(255) * Clng(255) * Clng(255)
and just for once, shouldn't require any more memory (probably slightly less, in fact).
I do agree that the conversion functions such as Clng() are useful in many cases. But not for this simple example using literal values. this is explecit type conversions
it helps compilers to solve the problem more easily
in ur example u've used single notation and nothing else
| | Familiar Sight | | Join Date: Mar 2007 Location: igirisu~
Posts: 184
| | | re: Overflow with 'long' variable type?
Wouldn't a public const as Killer42 said near the start be fastest of all?
Because then you wouldn't need to be doing any converting data types at all because you made it a Long in the first place.
Also, is using a Long really quicker than using a Byte or Integer? Because I'm using bytes quite a lot, and any speed improvements would be welcome! ^^;
| | Moderator | | Join Date: Oct 2006 Location: Australia
Posts: 7,748
| | | re: Overflow with 'long' variable type?
Quote:
Originally Posted by Robbie Wouldn't a public const as Killer42 said near the start be fastest of all?
Because then you wouldn't need to be doing any converting data types at all because you made it a Long in the first place.
Also, is using a Long really quicker than using a Byte or Integer? Because I'm using bytes quite a lot, and any speed improvements would be welcome! ^^; Yes, in my opinion a Const defined as Long would be your best bet.
As for the conversion, that's what the disagreement is about, between myself and vijaydiwakar. I believe there's a fundamental difference in our approaches. Using the "!" notation causes the compiler to use a Long value, so that no conversion is necessary at runtime. His method uses an Integer (or Byte, or Variant (:(which I hate) or whatever VB deems appropriate), then executes the Clng( ) function at runtime to convert it, thus requiring more processing and possibly making the code slightly larger.
Also, though this may no longer apply in VB6, a Const or variable used to be faster to access than a literal. In other words, in older versions of BASIC (and I think VB6, but not certain) using MyConstValue is faster than using 255. So in this case, defining a Const of Long format will definitely prevent your overflow problem, and possibly execute faster than using the literal value. It's a win-win situation. :)
As for the difference between data types, it wouldn't be a bad idea to test it for yourself. Just set up a loop that does some arithmetic operation (or whatever) a few million times, on one specific variable. Then try it a few times with each data type, and see what happens. I think you'll find that Long is the fastest. Where speed is not an issue, of course, a shorter type allows you to save space.
But I believe, as I said, using the native data type (which is the four-byte "long integer" type on a 32 bit processor) reduces the work that must be done "behind the scenes" and executes a little faster. On the prior 16 bit processors, this applies to the two-byte integer type. Presumably it will apply to the eight-byte "even longer integer:D" type on a 64 bit processor (and software).
P.S. I normally include in my project a code module called Numbers in which I define all these sort of constants as Public Const.
|  |
Similar Visual Basic 4 / 5 / 6 bytes | | |  /bytes/about
We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights.
Get the best answers to your questions from over 226,471 network members.
|
