473,385 Members | 1,329 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,385 software developers and data experts.

Performance to convert byte[] to a value

What is the way to have best performance to copy a byte[] to a value such as
long?
I use
BitConverter.ToInt64(binary, offset)
But the performace is not good enough. I need to have the best performance
in my case.

Jun 27 '08 #1
10 4826
Roy wrote:
What is the way to have best performance to copy a byte[] to a value such as
long?
I use
BitConverter.ToInt64(binary, offset)
But the performace is not good enough. I need to have the best performance
in my case.
Do you know something about the bytes you will have that makes them
special compared to the general case Microsoft has coded for ?

If not I find it difficult to believe that it will be possible
to find something faster.

Oh - and are you sure this is really the bottleneck in your app ??

(you should be able to do 100-200 million conversions per second)

Arne

Jun 27 '08 #2
On May 4, 8:23*am, Arne Vajhj <a...@vajhoej.dkwrote:
Roy wrote:
What is the way to have best performance to copy a byte[] to a value such as
long?
I use
BitConverter.ToInt64(binary, offset)
But the performace is not good enough. I need to have the best performance
in my case.

Do you know something about the bytes you will have that makes them
special compared to the general case Microsoft has coded for ?

If not I find it difficult to believe that it will be possible
to find something faster.

Oh - and are you sure this is really the bottleneck in your app ??

(you should be able to do 100-200 million conversions per second)

Arne
As per my understanding BIrConverter is the apt option to convert byte
array to numeric. It has a property to check the endianness also -
IsLittleEndian. Do not know how bad it is from perdormance point of
view. Do you have any benchmark results?
Jun 27 '08 #3
Aneesh Pulukkul [http://dotnet-revolutions.blogspot.com] wrote:
On May 4, 8:23 am, Arne Vajhj <a...@vajhoej.dkwrote:
>Roy wrote:
>>What is the way to have best performance to copy a byte[] to a value such as
long?
I use
BitConverter.ToInt64(binary, offset)
But the performace is not good enough. I need to have the best performance
in my case.
Do you know something about the bytes you will have that makes them
special compared to the general case Microsoft has coded for ?

If not I find it difficult to believe that it will be possible
to find something faster.

Oh - and are you sure this is really the bottleneck in your app ??

(you should be able to do 100-200 million conversions per second)

As per my understanding BIrConverter is the apt option to convert byte
array to numeric. It has a property to check the endianness also -
IsLittleEndian. Do not know how bad it is from perdormance point of
view. Do you have any benchmark results?
The endianess check is a single if statement (I looked with reflector).

It uses shift and or to create the long.

I can not right away come up with any ideas that I would expect
to be faster.

I did test BitConverter against a union (struct with field offset) and
BitConverter was much faster.

If it was a huge array that needed to be converted, then possible
a call to native code could be worth considering.

Arne
Jun 27 '08 #4
I do have a List<byte[]with one million of byte[] entries (each of tem has
3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
for my case. What is the native code to convert it with faster performance? I
don't mind how ugly the code looks like as long as the performance is best in
my case.

"Arne Vajhøj" wrote:
Aneesh Pulukkul [http://dotnet-revolutions.blogspot.com] wrote:
On May 4, 8:23 am, Arne Vajhøj <a...@vajhoej.dkwrote:
Roy wrote:
What is the way to have best performance to copy a byte[] to a value such as
long?
I use
BitConverter.ToInt64(binary, offset)
But the performace is not good enough. I need to have the best performance
in my case.
Do you know something about the bytes you will have that makes them
special compared to the general case Microsoft has coded for ?

If not I find it difficult to believe that it will be possible
to find something faster.

Oh - and are you sure this is really the bottleneck in your app ??

(you should be able to do 100-200 million conversions per second)
As per my understanding BIrConverter is the apt option to convert byte
array to numeric. It has a property to check the endianness also -
IsLittleEndian. Do not know how bad it is from perdormance point of
view. Do you have any benchmark results?

The endianess check is a single if statement (I looked with reflector).

It uses shift and or to create the long.

I can not right away come up with any ideas that I would expect
to be faster.

I did test BitConverter against a union (struct with field offset) and
BitConverter was much faster.

If it was a huge array that needed to be converted, then possible
a call to native code could be worth considering.

Arne
Jun 27 '08 #5
Roy wrote:
I do have a List<byte[]with one million of byte[] entries (each of tem has
3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
for my case. What is the native code to convert it with faster performance? I
don't mind how ugly the code looks like as long as the performance is best in
my case.
How exactly do you *obtain* a list with one million byte arrays? Wouldn't it
be much faster (and more memory-efficient) to change whatever is producing
such a list to produce the list you need in the first place?

What's the result of the conversion? Another list? Keep in mind that
allocating memory takes time too. The conversion of individual entries is
not the bottleneck, in the sense that you will not be able to speed that up
much. You'll want to focus on minimizing the conversion effort.

--
J.
http://symbolsprose.blogspot.com
Jun 27 '08 #6
I have a basic item consisting of 3 values (I know the types and they alway
together). An item consisted of any number of the basic items can be stored
as a record in database as varbinary(max). After I retrieve the records
(items) from database, I need to
(1) split the record (item) into base items (because the client want to see
the basic item)
(2) restore each basic item from the binary to the original types (3 values)
And I have about 1 million basic items. That's why the performance is very
important for me. I stored multiple basic items together as an item
(varbinary(max)) in database to improve the performace for query. Now I need
to improve the performace for (1) and (2). Any suggestions?
Thanks.

"Jeroen Mostert" wrote:
Roy wrote:
I do have a List<byte[]with one million of byte[] entries (each of tem has
3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
for my case. What is the native code to convert it with faster performance? I
don't mind how ugly the code looks like as long as the performance is best in
my case.
How exactly do you *obtain* a list with one million byte arrays? Wouldn't it
be much faster (and more memory-efficient) to change whatever is producing
such a list to produce the list you need in the first place?

What's the result of the conversion? Another list? Keep in mind that
allocating memory takes time too. The conversion of individual entries is
not the bottleneck, in the sense that you will not be able to speed that up
much. You'll want to focus on minimizing the conversion effort.

--
J.
http://symbolsprose.blogspot.com
Jun 27 '08 #7
Roy <Ro*@discussions.microsoft.comwrote:
I have a basic item consisting of 3 values (I know the types and they alway
together). An item consisted of any number of the basic items can be stored
as a record in database as varbinary(max). After I retrieve the records
(items) from database, I need to
(1) split the record (item) into base items (because the client want to see
the basic item)
(2) restore each basic item from the binary to the original types (3 values)
And I have about 1 million basic items. That's why the performance is very
important for me. I stored multiple basic items together as an item
(varbinary(max)) in database to improve the performace for query. Now I need
to improve the performace for (1) and (2). Any suggestions?
Thanks.
As Arne said, BitConverter is already very fast - on my laptop it goes
through 100 million arrays in a second.

How long is it taking you to get the data from the database in the
first place? I'd expect that to utterly dwarf the conversion time.

--
Jon Skeet - <sk***@pobox.com>
Web site: http://www.pobox.com/~skeet
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com
Jun 27 '08 #8
Roy wrote:
I do have a List<byte[]with one million of byte[] entries (each of tem has
3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
for my case. What is the native code to convert it with faster performance? I
don't mind how ugly the code looks like as long as the performance is best in
my case.
On my PC it takes about 0.043 seconds to convert each byte[3] to long
in a list with one million elements.

Actually I had to test with ten million elements and divide by 10,
because I could not really measure with so small data as one
million elements.

Your 1-2 seconds must be spend on something else.

I think you need to analyze the app again.

List<byte[]is not suitable for native code I think.

But because there are just 3 bytes you may be able to
do it faster that BitConverter that has to handle all
8 bytes.

The 0.043 seconds are with:

private static long Cvt1(byte[] b)
{
byte[] tmp = new byte[8];
tmp[0] = b[0];
tmp[1] = b[1];
tmp[2] = b[2];
return BitConverter.ToInt64(tmp, 0);
}

but using:

private static long Cvt2(byte[] b)
{
return (b[2] << 16) | (b[1] << 8) | b[0];
}

then I can get it down to 0.006 seconds.

But if it is probably not relevant. If your existing code takes 1.000
seconds then the difference will just reduce it to 0.963 seconds - noone
will notice the difference.

Arne
Jun 27 '08 #9

"Arne Vajhj" <ar**@vajhoej.dkwrote in message
news:48***********************@news.sunsite.dk...
Roy wrote:
>I do have a List<byte[]with one million of byte[] entries (each of tem
has 3 values) to convert. It takes 1 - 2 seconds to convert. And it is
too slow for my case. What is the native code to convert it with faster
performance? I don't mind how ugly the code looks like as long as the
performance is best in my case.
How are you iterating through the list? Are you using any virtual calls
(through an IList or IEnumerable interface, for example)? Retrieving an
item from the List is probably several times slower than the BitConverter
call.
>
On my PC it takes about 0.043 seconds to convert each byte[3] to long
in a list with one million elements.
What? According to the code for ToInt64, that would throw.

if (startIndex (value.Length - 8))
{
ThrowHelper.ThrowArgumentException(ExceptionResour ce.Arg_ArrayPlusOffTooSmall);
}

>
Actually I had to test with ten million elements and divide by 10,
because I could not really measure with so small data as one
million elements.

Your 1-2 seconds must be spend on something else.

I think you need to analyze the app again.

List<byte[]is not suitable for native code I think.

But because there are just 3 bytes you may be able to
do it faster that BitConverter that has to handle all
8 bytes.
To start with, just use ToInt32 instead...
>
The 0.043 seconds are with:

private static long Cvt1(byte[] b)
{
byte[] tmp = new byte[8];
tmp[0] = b[0];
tmp[1] = b[1];
tmp[2] = b[2];
return BitConverter.ToInt64(tmp, 0);
}
Ahh, that's not a byte[3] after all.
>
but using:

private static long Cvt2(byte[] b)
{
return (b[2] << 16) | (b[1] << 8) | b[0];
}

then I can get it down to 0.006 seconds.

But if it is probably not relevant. If your existing code takes 1.000
seconds then the difference will just reduce it to 0.963 seconds - noone
will notice the difference.

Arne


Jun 27 '08 #10
Ben Voigt [C++ MVP] wrote:
"Arne Vajhj" <ar**@vajhoej.dkwrote in message
news:48***********************@news.sunsite.dk...
>Roy wrote:
>>I do have a List<byte[]with one million of byte[] entries (each of tem
has 3 values) to convert. It takes 1 - 2 seconds to convert. And it is
too slow for my case. What is the native code to convert it with faster
performance? I don't mind how ugly the code looks like as long as the
performance is best in my case.

How are you iterating through the list? Are you using any virtual calls
(through an IList or IEnumerable interface, for example)? Retrieving an
item from the List is probably several times slower than the BitConverter
call.
I would expect retrieving a ref to the byte[] from a List<byte[]to
be much faster than BitCoverter.ToInt64, but I have not measured it
and I can be completely wrong.
>On my PC it takes about 0.043 seconds to convert each byte[3] to long
in a list with one million elements.

What? According to the code for ToInt64, that would throw.

if (startIndex (value.Length - 8))
{
ThrowHelper.ThrowArgumentException(ExceptionResour ce.Arg_ArrayPlusOffTooSmall);
}
>The 0.043 seconds are with:

private static long Cvt1(byte[] b)
{
byte[] tmp = new byte[8];
tmp[0] = b[0];
tmp[1] = b[1];
tmp[2] = b[2];
return BitConverter.ToInt64(tmp, 0);
}

Ahh, that's not a byte[3] after all.
Let us just say that I got an exception in first attempt ...

:-)

Arne
Jun 27 '08 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: CSharpener | last post by:
This should be *so* easy! How do I convert a Byte or int to a binary string representation in C# In JavaScript, it goes like this for an int: javascript:(123).toString(2 or...
4
by: John Smith | last post by:
Hello, Suppose I have the following C# code which I want to convert to VB: for (int i = 0; i < nFieldLength; i++) Console.Write((char) sValue); sValue is a byte array. The problem is...
3
by: Alexander Levchenko | last post by:
Hello All, I try to convert hex-value "&HAA" in character and after that in byte Dim sTest As String Dim Encoder As New System.Text.ASCIIEncoding Dim bByte() As Byte sTest = Chr(&HAA)...
2
by: Tedmond | last post by:
Can anyone tell me how to convert a byte to bit pattern? e.g. Byte b = 1; after conversion = 00000001 Tedmond
1
by: fsc2006 | last post by:
i have created a program for WMI which retrieves information about the hard disk FreeSpace and SIze, but the value that i am getting is in bztes and iw ant that to be in gigabytes and also how can i...
6
by: | last post by:
Hi, How do I convert a byte to an object? I am trying to use IADs PutEx and the last parameter required a variant array. Thanks, Alex
6
by: Bob Altman | last post by:
Hi all, I'm looking for the fastest way to convert an array of bytes to String. I also need to convert a String back to its original Byte() representation. Convert.ToBase64String and...
18
by: MrVS | last post by:
Hi, I have a C++ CLR class method that takes System::Byte *b as parameter argument. I want the CSharp caller pass a byte * to this function. But in the CSharp prorgram, I only managed to create a...
2
by: GauravRashinkar | last post by:
How to convert byte array into a image? Byte photo = Convert.FromBase64String(value); //System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding(); //Byte...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.