473,326 Members | 2,175 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

littleEndian/bigEndian

Hi,

as someone pulling to C# over from C++ I stumbled over something today for
which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and vice
versa?

In C++ I have this:

#pragma intrinsic(_byteswap_ushort)
#pragma intrinsic(_byteswap_ulong)

*myword = _byteswap_ulong(mybigendianword);

which results in a wonderful single assembler instruction.

The only documentation I found about endianess is on handling UTF8/16
strings, but not with binary stuff.

Can someone give me a hint how I perform such things FAST and not via my own
clumsy version?

thanks
doc

Jun 29 '06 #1
19 7989
docschnipp wrote:
as someone pulling to C# over from C++ I stumbled over something today for
which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and vice
versa?


Have a look at http://www.pobox.com/~skeet/csharp/miscutil
In particular, the EndianBinaryReader/Writer and EndianBitConverter.

Jon

Jun 29 '06 #2
"Jon Skeet [C# MVP]" wrote:
docschnipp wrote:
as someone pulling to C# over from C++ I stumbled over something today for
which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and vice
versa?


Have a look at http://www.pobox.com/~skeet/csharp/miscutil
In particular, the EndianBinaryReader/Writer and EndianBitConverter.

"Every so often, someone on the newsgroup asks about something which doesn't
exist in the framework, but which I think should. Sometimes, I'll write some
code for them to plug the gap. "

Well, fits perfect I guess :)

I'll take a look into it, thanks.

doc

Jun 29 '06 #3
Hello docschnipp,

Here's what I currently use (in VB):

Public Function ReadBigEndianInt32() As Int32

Dim tReturn As Int32 = 0
Dim tBuffer(3) As Byte

oStream.Read(tBuffer, 0, tBuffer.Length)

Array.Reverse(tBuffer)
tReturn = BitConverter.ToInt32(tBuffer, 0)

Return tReturn

End Function

And the inverse of course for Writing..

-Boo
Hi,

as someone pulling to C# over from C++ I stumbled over something today
for which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and
vice versa?

In C++ I have this:

#pragma intrinsic(_byteswap_ushort)
#pragma intrinsic(_byteswap_ulong)
*myword = _byteswap_ulong(mybigendianword);

which results in a wonderful single assembler instruction.

The only documentation I found about endianess is on handling UTF8/16
strings, but not with binary stuff.

Can someone give me a hint how I perform such things FAST and not via
my own clumsy version?

thanks
doc

Jun 29 '06 #4
GhostInAK <gh*******@gmail.com> wrote:
Here's what I currently use (in VB):

Public Function ReadBigEndianInt32() As Int32

Dim tReturn As Int32 = 0
Dim tBuffer(3) As Byte

oStream.Read(tBuffer, 0, tBuffer.Length)

Array.Reverse(tBuffer)
tReturn = BitConverter.ToInt32(tBuffer, 0)

Return tReturn

End Function


That's not guaranteed to work - calling Read on a stream isn't
guaranteed to read all the bytes you ask for. It's likely to, but it's
not guaranteed.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jun 29 '06 #5
IPAddress.NetworkToHostOrder
IPAddress.HostToNetworkOrder

(not sure why these are in the IPAddress class but they're static anyway)

/claes

"docschnipp" <do********@newsgroup.nospam> wrote in message
news:BC**********************************@microsof t.com...
Hi,

as someone pulling to C# over from C++ I stumbled over something today for
which I was not able to find an answer:

HOW do I convert a byte block with a given endianess to an int32 and vice
versa?

In C++ I have this:

#pragma intrinsic(_byteswap_ushort)
#pragma intrinsic(_byteswap_ulong)

*myword = _byteswap_ulong(mybigendianword);

which results in a wonderful single assembler instruction.

The only documentation I found about endianess is on handling UTF8/16
strings, but not with binary stuff.

Can someone give me a hint how I perform such things FAST and not via my
own
clumsy version?

thanks
doc

Jun 29 '06 #6
Hello Jon Skeet [C# MVP],

Jon,

Thanks for the comment. I became aware of that fact the other day and just
hadn't got around to updating the routine at the time of the post.

Thanks,
-Boo
GhostInAK <gh*******@gmail.com> wrote:
Here's what I currently use (in VB):

Public Function ReadBigEndianInt32() As Int32

Dim tReturn As Int32 = 0
Dim tBuffer(3) As Byte
oStream.Read(tBuffer, 0, tBuffer.Length)

Array.Reverse(tBuffer)
tReturn = BitConverter.ToInt32(tBuffer, 0)
Return tReturn

End Function

That's not guaranteed to work - calling Read on a stream isn't
guaranteed to read all the bytes you ask for. It's likely to, but it's
not guaranteed.

Jun 29 '06 #7
"Claes Bergefall" wrote:
IPAddress.NetworkToHostOrder
IPAddress.HostToNetworkOrder

(not sure why these are in the IPAddress class but they're static anyway)

/claes


Usually because the libs connected to sockets, tcp/ip etc. are early cross
platform libs and need all the translation of network to local endianess.

I now decided to use this variant until I find something better:

----------
UInt32 myBigEndian = 0xdeadbeef;
UInt32 mLittleEndian = ( (myBigEndian& 0xFF000000) >> 24) | ((myBigEndian&
0x00FF0000) >>8) | (( myBigEndian& 0x0000FF00) <<8 ) | ((myBigEndian&
0x000000FF) << 24);
----------

The big advantage is: it works for all scales of integers and is not too
clumsy. Unfortunately I found no source for something that allows the native
compiler to take use of the bswap mnemonic. I would really appreciate a way
to make this very fast. bswap takes 2 ( 1 cycle plus prefix ) cycles only,
the code above ~34 cycles.

I have to read a _lot_ of data, several dozends of megabytes so
investigation on this topic is a worthwhile goal for me.
thanks for input,

doc

Jul 1 '06 #8
Hi Guys,

I've made some tests, the Reverse() method seems to be the slowest of all.

The code is found below, results were:

Time 1: 3765 ( the .Reverse/BitConverter method )
Time 2: 1322 ( the manual reindexing/BitConverter method )
Time 3: 1071 ( the Bitconverter/reshift method )
(if UInt is not needed, only Int:)
Time 4: 191 ( the shift the bytes directly/OR method )
So I think I will stick to method 3 and 4. Does anyone knows how I can cast
a byte to uint? :)

Thanks for all your input.

UInt32 mLittleEndian;
int loopcount = 20000000;

s1 = Environment.TickCount;
Array.Copy(myOrg,0,myArr,0,4);
for (i = 0; i < loopcount; i++)
{
if ( BitConverter.IsLittleEndian )
{
Array.Reverse(myNew,0,4);
}
mLittleEndian = BitConverter.ToUInt32(myNew,0);
}
s2 = Environment.TickCount;
Array.Copy(myOrg,0,myArr,0,4);
for ( i = 0; i< loopcount ; i++)
{
if ( BitConverter.IsLittleEndian )
{

myNew[3] = myArr[0];
myNew[2] = myArr[1];
myNew[1] = myArr[2];
myNew[0] = myArr[3];
mLittleEndian = BitConverter.ToUInt32(myNew,0);
}
}
s3 = Environment.TickCount;

// UInt32 mLittleEndian;
for ( i = 0; i< loopcount ; i++)
{
// myBigEndian = BitConverter.ToUInt32(myNew,0);
myBigEndian = BitConverter.ToUInt32(myOrg,0);
if ( BitConverter.IsLittleEndian)
{
mLittleEndian = ( (myBigEndian & 0xFF000000) >24) |
((myBigEndian & 0x00FF0000) >>8) | (( myBigEndian &
0x0000FF00) <<8 ) | ((myBigEndian & 0x000000FF) << 24);
}
}
s4 = Environment.TickCount;
Int32 mSLittleEndian;
for ( i = 0; i< loopcount ; i++)
{
if ( BitConverter.IsLittleEndian)
{
mSLittleEndian = myArr[0] | (myArr[1] << 8) | (myArr[2] << 16) | (myArr[3]
<< 24);
}
}
s5 = Environment.TickCount;
s1 = s2-s1;
s2 = s3-s2;
s3 = s4-s3;
s4 = s5-s4;
Console.WriteLine("Time 1: {0} Time 2: {1} Time 3: {2} Time 4:
{3}",s1.ToString(),s2.ToString(),s3.ToString(),s4. ToString());

Jul 2 '06 #9
Hi Docschnipp,

Thanks for your feedback!

Yes, I see your concern, BSWAP instruction should be faster than manually
change the byte order. However, based on my experience, C# does not
contains the code statement to generate this specific instruction.(I think
JIT compiler takes control of the instructions generated)

To workaround this issue, I think you may use C/C++ to write a dll which
internally uses inline ASM to call BSWAP instruction. Then in C# code, you
may p/invoke the exported function of the C/C++ dll to call BSWAP
indirectly. You can also declare the C/C++ function as naked to eliminate
the function prolog and epilog.

Hope this helps!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Jul 3 '06 #10
docschnipp <do********@newsgroup.nospamwrote:

<snip>
I have to read a _lot_ of data, several dozends of megabytes so
investigation on this topic is a worthwhile goal for me.
Are you really sure? I've just done a test with my MiscUtil library,
and I was able to read 100MB of data in BigEndian format (as Int32
values) in 2.5 seconds on my laptop. Do you really need to do it faster
than that?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jul 3 '06 #11
"Jon Skeet [C# MVP]" wrote:
docschnipp <do********@newsgroup.nospamwrote:

<snip>
I have to read a _lot_ of data, several dozends of megabytes so
investigation on this topic is a worthwhile goal for me.

Are you really sure? I've just done a test with my MiscUtil library,
and I was able to read 100MB of data in BigEndian format (as Int32
values) in 2.5 seconds on my laptop. Do you really need to do it faster
than that?
Assume you are running in a background process that is low prioritised and
the whole system is quite active due displaying graphics and controls in
various ways. I found that such threads can run for 1ms and be held by the
scheduler for 10ms, and the system load is not that high.

So you have a performance going down by factor 10.

An algorithm I did that made something in suitable time on simple testing,
was not so nice when I was using it in the final product. Therefore I just
want to investigate which version is best, and as you might have seen in my
other post, things are really different depending on how I code it.

I've taken a look at that library and it is really nice, but I just wanted
to take the opportunity to learn a bit more of C# interna than just use a
library.

thanks for your input
doc

p.s.: a nice read about my motivation:
http://www.joelonsoftware.com/Articl...000000319.html

Jul 5 '06 #12
Hi Jeffrey,
Hi Docschnipp,

Yes, I see your concern, BSWAP instruction should be faster than manually
change the byte order. However, based on my experience, C# does not
contains the code statement to generate this specific instruction.(I think
JIT compiler takes control of the instructions generated)
Yes. I just thought, there is a kind of "hinting" for this, like in the C++
variant (the pragma allows the compiler to reduce the function call to a
single BSWAP instruction.

But I guess, a "byteswap" type instruction is not defined in the MSIL.

Anyway, the code I found is sufficient enough, just took the opportunity to
dive a bit into C# interna.

Thanks
doc

Jul 5 '06 #13
docschnipp <do********@newsgroup.nospamwrote:
I have to read a _lot_ of data, several dozends of megabytes so
investigation on this topic is a worthwhile goal for me.
Are you really sure? I've just done a test with my MiscUtil library,
and I was able to read 100MB of data in BigEndian format (as Int32
values) in 2.5 seconds on my laptop. Do you really need to do it faster
than that?

Assume you are running in a background process that is low prioritised and
the whole system is quite active due displaying graphics and controls in
various ways. I found that such threads can run for 1ms and be held by the
scheduler for 10ms, and the system load is not that high.
If it's a background process, does it matter that much?
So you have a performance going down by factor 10.
Not really - if the system as a whole is busy, it just means that your
background process isn't disrupting the rest of the system.
An algorithm I did that made something in suitable time on simple testing,
was not so nice when I was using it in the final product. Therefore I just
want to investigate which version is best, and as you might have seen in my
other post, things are really different depending on how I code it.
That suggests that your simple testing wasn't representative. That's a
valuable lesson to learn, but you shouldn't go straight to the other
extreme and assume that every instruction counts. The lesson is to try
to *get* representative data for testing.

What's your actual situation going to be? How much data are you going
to have to process, and how long is it acceptable for it to take? Until
you know that, you won't know what your target is.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jul 5 '06 #14
Hi Docschnipp,

Thanks for your feedback!

Yes, I see. C# is built upon the CLR, the performance is not its
specialist, C/C++/ASM will be better on this subject. So if we usually
place the performance sensitive code snippet in the C/C++ dll and use
p/invoke to invoke it.

To learn C# internals, we can divide it into 3 parts:
1. C# language
2. .Net Framework Class Library
3. .Net CLR

1. C# language is built upon the MSIL language, CLR and it leverages a
subset feature of MSIL language. "Inside Microsoft .NET IL Assembler" and
"CLR via C#, Second Edition" are 2 recommended books on this subject.
2. I think .Net Reflector is an invaluable tool to inspect the source code
of .Net FCL via different languages.
3. The 2 recommended books in #1 and "Essential .Net" have covered CLR to a
good point.

Hope this helps!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Jul 6 '06 #15
"Jon Skeet [C# MVP]" wrote:
docschnipp <do********@newsgroup.nospamwrote:
Assume you are running in a background process that is low prioritised and
the whole system is quite active due displaying graphics and controls in
various ways. I found that such threads can run for 1ms and be held by the
scheduler for 10ms, and the system load is not that high.

If it's a background process, does it matter that much?
I use it to perform things triggered by an user. So I want to keep response
to the user tight, but the background task should, of course, load the data
as fast as it can.
So you have a performance going down by factor 10.
Not really - if the system as a whole is busy, it just means that your
background process isn't disrupting the rest of the system.
Yes, you are right. It is not 10. But it was quite puzzling to see it slow
down THAT much. The foreground work is a kind of realtime process with
elevated thread priority (15).
An algorithm I did that made something in suitable time on simple testing,
was not so nice when I was using it in the final product. Therefore I just
want to investigate which version is best, and as you might have seen in my
other post, things are really different depending on how I code it.

That suggests that your simple testing wasn't representative. That's a
valuable lesson to learn, but you shouldn't go straight to the other
Indeed :-) The simple testing was okay for correctness but obviously not for
performance. I guess these are the things you won't learn in school and this
kind of knowledge makes you a bit of a better programmer.
extreme and assume that every instruction counts. The lesson is to try
to *get* representative data for testing.

What's your actual situation going to be? How much data are you going
to have to process, and how long is it acceptable for it to take? Until
you know that, you won't know what your target is.
This current thing shall load DICOM streams (used in medical imaging), they
can get quite large and can change their endianess within the stream. They
should be loaded in the background while the user itself works with already
loaded files (or parts of it). It also should run on mobile devices and that
is where the whole thing should at least avoid unfortunate coding style like
the (sorry GhostInAK) Array.Reverse() method.

I am surely not going to the extreme, I just wanted to explore a bit since I
am new to C# and the .NET framework. The solution I have now is totally
sufficient for the sake of 'clean' code.

Btw.: I appreciated the reading of your other articles about C#, nice writing.

best regards & thanks
doc

Jul 6 '06 #16
""Jeffrey Tan[MSFT]"" wrote:
Hi Docschnipp,

Thanks for your feedback!

Yes, I see. C# is built upon the CLR, the performance is not its
specialist, C/C++/ASM will be better on this subject. So if we usually
place the performance sensitive code snippet in the C/C++ dll and use
p/invoke to invoke it.

1. (...)"Inside Microsoft .NET IL Assembler" and
"CLR via C#, Second Edition" are 2 recommended books on this subject.
Will take a look into it. In general I have the feeling that certain parts
of programming are kept out for the sake of cleaness and compatibility. And
maybe a (JIT-) Compiler would be clever enough to find out what I am doing
and may translate the shift/or commands into a bswap in future. This might be
really the future way, same way as inlining functions is automated by the
compiler. I was hoping there is something I can trigger in that way, but
currently I leave code optimisation out of the bill.
2. I think .Net Reflector is an invaluable tool to inspect the source code
of .Net FCL via different languages.
I took a look into the (x86) disassembly of my code which is quite
interesting, how things turn out later. But taking a look into the meta form
(MSIL) will be interesting, too.
3. The 2 recommended books in #1 and "Essential .Net" have covered CLR to a
good point.
Thanks for the book tips.

best
doc

Jul 6 '06 #17
Hi Docschnipp,

Thanks for your feedback!

Yes, I agree with that this is the key point. If JIT compiler is smart
enough to detect the running platform and the CPU instruction set, it
should be able to emit the most effective instruction to get the work done.
It seems .Net CLR compiler did not get to such degree currently.

Personally I have below suspicious: since .Net FCL does not provide a bswap
class and C# language does not provide a bswap statement, this may mean
that .Net does not include bswap function, so JIT compiler does not take
into emiting bswap instruction into plan/design. Also, this design has gone
out of my control, thanks for your understanding.

If you want to feedback this issue to our CLR/JIT team, it is a good
opporunity to submit request in the MSDN feedback center below:
http://connect.microsoft.com/Main/co...ContentID=2220

Our product team will receive your request and follow up with you. Hope
this helps.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Jul 7 '06 #18
""Jeffrey Tan[MSFT]"" wrote:
Yes, I agree with that this is the key point. If JIT compiler is smart
enough to detect the running platform and the CPU instruction set, it
should be able to emit the most effective instruction to get the work done.
It seems .Net CLR compiler did not get to such degree currently.
I imagine it is not too easy finding such an optimisation from a set of
SHIFTs and ORs ;)
Personally I have below suspicious: since .Net FCL does not provide a bswap
class and C# language does not provide a bswap statement, this may mean
that .Net does not include bswap function, so JIT compiler does not take
into emiting bswap instruction into plan/design. Also, this design has gone
out of my control, thanks for your understanding.
Of course. Never mind.
If you want to feedback this issue to our CLR/JIT team, it is a good
opporunity to submit request in the MSDN feedback center below:
http://connect.microsoft.com/Main/co...ContentID=2220
Done that. I tried to consider where I would expect such a thing and where a
framework/compiler designer could possible take advantage of the hint and my
idea is now, that BitConverter should have static functions that read a given
endianess and convert it to the platform endianess.

Example:
Uint32 myHeaderSize = BitConverter.ReadUInt32BigEndian(buffer);

Before reading a file or stream one usually determines the format and would
just use this and would not need to care about the platform (which might
transition somewhere in the future [I would love to see this] ).

I know that there is the NetWorkToHost and vice versa functions in the
network namespace but this is just for 32bit integers, BitConverter should
provide it for
all multibyte integers (and maybe even floats).
Our product team will receive your request and follow up with you. Hope
this helps.
Looking forward to this.

Thanks
doc

Jul 7 '06 #19
Yes, based on my review, your feedback and statement makes sense, it is a
good idea to add a such useful method in BitConverter class.

Just as a reminder: since there is no service guarantee to this feedback
request and due to the time limit of developer team, it may cost some time
for them to give you a useful reply.

Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Jul 10 '06 #20

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: TF | last post by:
Hi, I've a byte array with 4 elements in BigEndian format i.e. {0, 0, 0, 12} and i want to convert it to 4-byte integer (Int32). I am trying Encoding.BigEndianUnicode property but it has no effect...
55
by: Sharon | last post by:
hi all, I generate binary text file (in Windows using Dev-C++) Is the generated binary file using BigEndian? Are there any method to generate binary file using LittleEndian? Because I want the...
14
by: Laszlo Szijarto | last post by:
Can BinaryReader be forced to read a stream, say a TCP/IP stream or memory stream or even file stream in big endian order or do I have to write something custom to reverse the byte order? So, for...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.