I'm faced with endian issues during communication across different
platforms.
So I've recommendations/tips (from one of these newsgroups - cant
recall which) on an approach (or two or three) that seems reasonable
and is based on issues that were encountered and how to handle them in
this situation.
One of which - the one I'm currently investigating - is as follows:
For discussion purposes consider atruct that'll be transmitted between
two plaforms (a and b). Platform a is little endian. b is big endian.
A sample struct is akin to.
//
typedef double tag_word;
struct t2_data
{
tag_word tag_word_;
bool lvt_enable : 1;
bool range : 1;
bool valid_mst : 1;
// more
};
struct t2_status
{
tag_word tag_word_;
bool lvt_status : 1;
bool range_status : 1;
bool msg_is_valid : 1;
unsigned int : 2;
unsigned int : 4;
// more
};
struct msg_header
{
unsigned int msg_size;
unsigned int msg_id;
unsigned int msg_count;
unsigned int msg_chksum;
};
// later
struct platform_a_to_b // little endian (platform a) to big endian
(platform b)
{
msg_header header;
t2_data t2_data_;
};
// later
struct platform_b_to_a // big endian (platform b) to little endian
(platform a)
{
msg_header header;
t2_status t2_stat;
};
The approach under investigation involves allocation of a buffer large
enough to hold the message being transmitted/received. So now: For
the receipt of message (IOW a receives from b or vice versa), I'll:
1. Allocate an unsigned char buffer large enough to hold the message.
2. Read the message into the buffer.
3. Convert the endianism of the data(if necessary).
4. Load each member of the model struct from the buffer.
Now I'm told that transmittal of messages is the reverse order of items
1 to 4, but that puzzles me. Why? With a 'few' tweaks heres the
reverse order as I see it:
1a. Allocate a unsinged char buffer large enough to hold the message.
This assumes a buffer hasn't already been allocated.
2a. Load each member of the model struct INTO (in this case lets take
the information from the struct and put into the buffer) the buffer.
3a. Convert the endianism of the data.
Item 2a seems flawed to me. The idea that I could just dump a struct
ino a byte buffer and send it across a wire seems like a bad idea, for
obvious reasons. Namely, I'm not guaranteed that the two plarforms
pack structs the same way and that would cause more headaches than it
will solve. That said, I'm puzzled about how this 'byte' buffer
approach works.
For those who've used this approach and have code that works, if its
possible to share (conversion code + sample struct) I'd appreaciate it.
Perusing _ACTUAL_ code assists me in 'getting my head around things'.
One other thing:
While it can be said that the requirements per the number of bytes for
a field is 'fixed' in a document, the types, - namely bool - specified
in the structs above need revisiting. I've been advised to follow
protocol below when dealing with structs and bit fields:
1. use only double, unsigned int (32 bits) and unsigned char.
2. if I need bool take and int, if I have to consider platforms using
different bool type sizes.
3. use double int and char (arrays) in that order.
4. make char arr's size a multiple of 4. (better multiple of 8).
I'll be re-visiting/investigating this approach. 13 1643 ma******@pegasus.cc.ucf.edu wrote: For discussion purposes consider atruct that'll be transmitted between two plaforms (a and b). Platform a is little endian. b is big endian. A sample struct is akin to.
Bitfield packing is implementation defined. I've seen the order vary
even machines of the same byte ordering.
4. Load each member of the model struct from the buffer.
I guess that depends on what you mean by this.
3a. Convert the endianism of the data.
I would combine this with step with 4 as I'm not sure it's valid to try
to do them separately. Item 2a seems flawed to me. The idea that I could just dump a struct ino a byte buffer and send it across a wire seems like a bad idea, for obvious reasons. Namely, I'm not guaranteed that the two plarforms pack structs the same way and that would cause more headaches than it will solve. That said, I'm puzzled about how this 'byte' buffer approach works.
You're correct, but it's all dealt with by step 4 above. If you always
convert a double to 8 bytes in a known encoding, and the bools into a
certain sequence of bytes, then it should work...you're essentially
not just "copying the struct to the bytestream" but enforcing your own
encoding on it. ma******@pegasus.cc.ucf.edu wrote: I'm faced with endian issues during communication across different platforms.
So I've recommendations/tips (from one of these newsgroups - cant recall which) on an approach (or two or three) that seems reasonable and is based on issues that were encountered and how to handle them in this situation.
One of which - the one I'm currently investigating - is as follows:
For discussion purposes consider atruct that'll be transmitted between two plaforms (a and b). Platform a is little endian. b is big endian. A sample struct is akin to.
// typedef double tag_word;
struct t2_data { tag_word tag_word_; bool lvt_enable : 1; bool range : 1; bool valid_mst : 1; // more };
struct t2_status { tag_word tag_word_; bool lvt_status : 1; bool range_status : 1; bool msg_is_valid : 1; unsigned int : 2; unsigned int : 4; // more };
struct msg_header { unsigned int msg_size; unsigned int msg_id; unsigned int msg_count; unsigned int msg_chksum; };
// later struct platform_a_to_b // little endian (platform a) to big endian (platform b) { msg_header header; t2_data t2_data_; };
// later struct platform_b_to_a // big endian (platform b) to little endian (platform a) { msg_header header; t2_status t2_stat; };
The approach under investigation involves allocation of a buffer large enough to hold the message being transmitted/received. So now: For the receipt of message (IOW a receives from b or vice versa), I'll:
1. Allocate an unsigned char buffer large enough to hold the message.
2. Read the message into the buffer. 3. Convert the endianism of the data(if necessary). 4. Load each member of the model struct from the buffer.
Now I'm told that transmittal of messages is the reverse order of items 1 to 4, but that puzzles me. Why? With a 'few' tweaks heres the reverse order as I see it: 1a. Allocate a unsinged char buffer large enough to hold the message. This assumes a buffer hasn't already been allocated. 2a. Load each member of the model struct INTO (in this case lets take the information from the struct and put into the buffer) the buffer. 3a. Convert the endianism of the data.
Item 2a seems flawed to me. The idea that I could just dump a struct ino a byte buffer and send it across a wire seems like a bad idea, for obvious reasons. Namely, I'm not guaranteed that the two plarforms pack structs the same way and that would cause more headaches than it will solve. That said, I'm puzzled about how this 'byte' buffer approach works.
For those who've used this approach and have code that works, if its possible to share (conversion code + sample struct) I'd appreaciate it. Perusing _ACTUAL_ code assists me in 'getting my head around things'.
One other thing: While it can be said that the requirements per the number of bytes for a field is 'fixed' in a document, the types, - namely bool - specified in the structs above need revisiting. I've been advised to follow protocol below when dealing with structs and bit fields:
1. use only double, unsigned int (32 bits) and unsigned char. 2. if I need bool take and int, if I have to consider platforms using different bool type sizes. 3. use double int and char (arrays) in that order. 4. make char arr's size a multiple of 4. (better multiple of 8).
I'll be re-visiting/investigating this approach.
Or just send/rcv the data in ascii; or use xml.
If each struct has read/write methods (serialize(), etc)
that write/read the structs contents as text it greatly
eases data xfer across platforms.
Transferring formatted binary data (structs, etc) from
one arch to another is almost never a good idea.
Regards,
Larry
--
Anti-spam address, change each 'X' to '.' to reply directly.
Ron, thanks for the info. Let me make sure I've got my head around
this. Lets assume playform 'a' native tongue is 32 bits. Lets assume
platform 'b's' native tongue is 16 bits.
Same story: a is little endian b is big. Lets further assume we
have a
struct test {
unsinged int some_value;
};
Lets assume // later :
test my_test; my_test.some_value = 655394;
Now a transmits to b, but prior to transmitting. a will need to 'load'
some_value into a byte buffer. ( Now I wish I could draw in usenet
world). We have 4 bytes to represent some_value.
So now:
Byte Buffer
-------------- -------------- --------------
--------------
byte 1 byte 2 byte 3 byte 4
-------------- -------------- --------------
--------------
Endian conversion
Transmit
b receives. b's native tongue is 16 bits. I think I see what needs to
happen here but what would 'b' need to do to map the 4 bytes sent from
a into the test struct? Your answer will perhaps help me 'answer' all
the scenarios I've generated in my head. ma******@pegasus.cc.ucf.edu wrote: Ron, thanks for the info. Let me make sure I've got my head around this. Lets assume playform 'a' native tongue is 32 bits. Lets assume platform 'b's' native tongue is 16 bits. Same story: a is little endian b is big. Lets further assume we have a
struct test { unsinged int some_value; };
Lets assume // later : test my_test; my_test.some_value = 655394;
Now a transmits to b, but prior to transmitting. a will need to 'load' some_value into a byte buffer. ( Now I wish I could draw in usenet world). We have 4 bytes to represent some_value. So now:
Byte Buffer -------------- -------------- -------------- -------------- byte 1 byte 2 byte 3 byte 4 -------------- -------------- -------------- -------------- Endian conversion Transmit
b receives. b's native tongue is 16 bits. I think I see what needs to happen here but what would 'b' need to do to map the 4 bytes sent from a into the test struct? Your answer will perhaps help me 'answer' all the scenarios I've generated in my head.
Read up on the library functions htonl(), htons(), ntohl(),
and ntohs(). They convert 16 and 32 bit ints to/from the
current host's endian and 'network' endian. For example:
on the sending side
uint32_t n = htonl(some_32bit_int); // 'n' now holds the
// 'network' endian version of 'some_32bit_int'
now send 'n' to the remote (put it in the sendbuffer, etc)
on the receiving side
uint32_t n = read 'n' from the network (from the rcvbuffer, etc)
uint32_t some_32bit_int = ntohl(n); // 'some_32bit_int' is now
// the correct endian for THIS host
Regards,
Larry
--
Anti-spam address, change each 'X' to '.' to reply directly.
Larry I Smith wrote: ma******@pegasus.cc.ucf.edu wrote: Ron, thanks for the info. Let me make sure I've got my head around this. Lets assume playform 'a' native tongue is 32 bits. Lets
assume platform 'b's' native tongue is 16 bits. Same story: a is little endian b is big. Lets further assume we have a
struct test { unsinged int some_value; };
Lets assume // later : test my_test; my_test.some_value = 655394;
Now a transmits to b, but prior to transmitting. a will need to
'load' some_value into a byte buffer. ( Now I wish I could draw in usenet world). We have 4 bytes to represent some_value. So now:
Byte Buffer -------------- -------------- -------------- -------------- byte 1 byte 2 byte 3 byte 4 -------------- -------------- -------------- -------------- Endian conversion Transmit
b receives. b's native tongue is 16 bits. I think I see what needs
to happen here but what would 'b' need to do to map the 4 bytes sent
from a into the test struct? Your answer will perhaps help me 'answer'
all the scenarios I've generated in my head.
Read up on the library functions htonl(), htons(), ntohl(), and ntohs(). They convert 16 and 32 bit ints to/from the current host's endian and 'network' endian. For example:
on the sending side
uint32_t n = htonl(some_32bit_int); // 'n' now holds the // 'network' endian version of 'some_32bit_int' now send 'n' to the remote (put it in the sendbuffer, etc)
on the receiving side
uint32_t n = read 'n' from the network (from the rcvbuffer, etc) uint32_t some_32bit_int = ntohl(n); // 'some_32bit_int' is now // the correct endian for THIS host
Regards, Larry
-- Anti-spam address, change each 'X' to '.' to reply directly.
Interesting that you brought up ntohl and htonl. When I opted to use
them I was told:
These functions do not always swap endian-ness and are _NOT_ always
portable. Interesting. The alternatives:
// 1
inline void endian_swap(unsigned short& x)
{
x = (x>>8) |
(x<<8);
}
inline void endian_swap(unsigned int& x)
{
x = (x>>24) |
((x<<8) & 0x00FF0000) |
((x>>8) & 0x0000FF00) |
(x<<24);
}
// __int64 for MSVC, "long long" for gcc
inline void endian_swap(unsigned __int64& x)
{
x = (x>>56) |
((x<<40) & 0x00FF000000000000) |
((x<<24) & 0x0000FF0000000000) |
((x<<8) & 0x000000FF00000000) |
((x>>8) & 0x00000000FF000000) |
((x>>24) & 0x0000000000FF0000) |
((x>>40) & 0x000000000000FF00) |
(x<<56);
}
/// 2 ( I prefer this route)
#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))
void ByteSwap(unsigned char * b, int n)
{
register int i = 0;
register int j = n-1;
while (i<j)
{
std::swap(b[i], b[j]);
i++, j--;
}
} ma******@pegasus.cc.ucf.edu wrote: Ron, thanks for the info. Let me make sure I've got my head around this. Lets assume playform 'a' native tongue is 32 bits. Lets assume platform 'b's' native tongue is 16 bits.
If you want binary transfers of data, forget native tongue. You must decide
if you want to use 16, 32 or whatever number of bits data size. Use fixed
size data types if your compiler has it, or use a type that the standard
guarantees have at least that size, and check at runtime that values out of
range are not used.
And you can also forget endianess, just always transfer the numbers to a
unsigned char array byte by byte using >> and & operators. That way you can
use the same code in both platforms without using conditional compilation.
Take care with signed numbers, that may require additional measures if you
want platform independence in that aspect.
--
Salu2
Bye the way. My REAL issue is trying to understand how the buffer of
bytes approach solves the current problem. IOW. Notwithstanding the
fact that the endian issue is what drove me to this, I'm more
interested in undertanding how a buffer of bytes - on a much LARGER
scale - assists when dealing with communications across platforms whose
native tongue (16 bit ints on one, 32 bits on the other) might might be
different. Even more interesting is the fact that the native tongue
(32 bit integers) might be the same yet the compiler could wreak
havoc.
To be on the safe side. Again just throw in a buffer of bytes and ..
well!!
Bye the way. My REAL issue is trying to understand how the buffer of
bytes approach solves the current problem. IOW. Notwithstanding the
fact that the endian issue is what drove me to this, I'm more
interested in undertanding how a buffer of bytes - on a much LARGER
scale - assists when dealing with communications across platforms whose
native tongue (16 bit ints on one, 32 bits on the other) might might be
different. Even more interesting is the fact that the native tongue
(32 bit integers) might be the same yet the compiler could wreak
havoc.
To be on the safe side. Again just throw in a buffer of bytes and ..
well!! ma******@pegasus.cc.ucf.edu wrote: Larry I Smith wrote:ma******@pegasus.cc.ucf.edu wrote:Ron, thanks for the info. Let me make sure I've got my head around this. Lets assume playform 'a' native tongue is 32 bits. Lets assumeplatform 'b's' native tongue is 16 bits. Same story: a is little endian b is big. Lets further assume we have a
struct test { unsinged int some_value; };
Lets assume // later : test my_test; my_test.some_value = 655394;
Now a transmits to b, but prior to transmitting. a will need to 'load'some_value into a byte buffer. ( Now I wish I could draw in usenet world). We have 4 bytes to represent some_value. So now:
Byte Buffer -------------- -------------- -------------- -------------- byte 1 byte 2 byte 3 byte 4 -------------- -------------- -------------- -------------- Endian conversion Transmit
b receives. b's native tongue is 16 bits. I think I see what needs tohappen here but what would 'b' need to do to map the 4 bytes sent froma into the test struct? Your answer will perhaps help me 'answer' allthe scenarios I've generated in my head. Read up on the library functions htonl(), htons(), ntohl(), and ntohs(). They convert 16 and 32 bit ints to/from the current host's endian and 'network' endian. For example:
on the sending side
uint32_t n = htonl(some_32bit_int); // 'n' now holds the // 'network' endian version of 'some_32bit_int' now send 'n' to the remote (put it in the sendbuffer, etc)
on the receiving side
uint32_t n = read 'n' from the network (from the rcvbuffer, etc) uint32_t some_32bit_int = ntohl(n); // 'some_32bit_int' is now // the correct endian for THIS host
Regards, Larry
-- Anti-spam address, change each 'X' to '.' to reply directly.
Interesting that you brought up ntohl and htonl. When I opted to use them I was told:
These functions do not always swap endian-ness and are _NOT_ always portable. Interesting. The alternatives:
I don't believe that's the case. Their whole purpose is to
handle the endian problem.
Anyway, as I said in my other post, XML and the likes are
most often used these days to xfr data host-to-host because
that elminates the numerous problems associated with binary
xfrs. // 1 inline void endian_swap(unsigned short& x) { x = (x>>8) | (x<<8); }
inline void endian_swap(unsigned int& x) { x = (x>>24) | ((x<<8) & 0x00FF0000) | ((x>>8) & 0x0000FF00) | (x<<24); }
// __int64 for MSVC, "long long" for gcc inline void endian_swap(unsigned __int64& x) { x = (x>>56) | ((x<<40) & 0x00FF000000000000) | ((x<<24) & 0x0000FF0000000000) | ((x<<8) & 0x000000FF00000000) | ((x>>8) & 0x00000000FF000000) | ((x>>24) & 0x0000000000FF0000) | ((x>>40) & 0x000000000000FF00) | (x<<56); }
/// 2 ( I prefer this route)
#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))
void ByteSwap(unsigned char * b, int n) { register int i = 0; register int j = n-1; while (i<j) { std::swap(b[i], b[j]); i++, j--; } }
--
Anti-spam address, change each 'X' to '.' to reply directly. >>>> If you want binary transfers of data, forget native tongue.
You must decide
if you want to use 16, 32 or whatever number of bits data size. Use
fixed
size data types if your compiler has it, or use a type that the
standard
guarantees have at least that size, and check at runtime that values
out of
range are not used.
Ah!!! This is perhaps where the encoding factors in with Ron's post.
With regard to my earlier post, where I higlighted
test my_test; my_test.some_value = 655394;
I could do.
byte 0 ______ byte 2 ______ byte 3 ______ byte 4 ______
byte 5
start ___________________ 655349 ____________________ stop
Where start and stop a char or ........ The premise: The
sender/receiver upon receipt of the data knows when to start and stop
(loosely speaking). > Take care with signed numbers, that may require additional
measures if you
want platform independence in that aspect.
Got it!!
Thanks
Thanks. I'll look into the XML and ascii.. Add another approach to my
arsenal :) if/most likely when I get confused, I'll be back
Larry I Smith wrote: ma******@pegasus.cc.ucf.edu wrote:
snipped Anyway, as I said in my other post, XML and the likes are most often used these days to xfr data host-to-host because that elminates the numerous problems associated with binary xfrs.
snipped
Whilst its true that some middleware technologies use XML as the data
transfer type (SOAP), I would not say its 'most'.
XML as a means of marshaling data for transmission greatly increases the
total amount of data transfered - as everything is a String.
CORBA or ICE would be better alternative for the OP to consider seeing
as there is communicates between two (or more) different types of
machines. These middleware technologies are designed for this very
problem. They also have the benefit that either side can be written in
any number programming languages.
Andrew McDonagh wrote: Larry I Smith wrote: ma******@pegasus.cc.ucf.edu wrote:
snipped
Anyway, as I said in my other post, XML and the likes are most often used these days to xfr data host-to-host because that elminates the numerous problems associated with binary xfrs.
snipped
Whilst its true that some middleware technologies use XML as the data transfer type (SOAP), I would not say its 'most'.
XML as a means of marshaling data for transmission greatly increases the total amount of data transfered - as everything is a String.
CORBA or ICE would be better alternative for the OP to consider seeing as there is communicates between two (or more) different types of machines. These middleware technologies are designed for this very problem. They also have the benefit that either side can be written in any number programming languages.
Well, there are many xfr tech's. CORBA is a PITA and often
requires proprietary solutions (we used to use ORBIX from IONA).
The nice thing about XML is that it's platfrom portable, is easy
to use, can easily be done with C/C++/Java, has free implemenations
on most OS'es, etc, etc.
There is no 'silver bullet', but these days many companies are
choosing XML for multi-platform data exchange - mine did.
Regards,
Larry
--
Anti-spam address, change each 'X' to '.' to reply directly. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Scott Robert Ladd |
last post by:
I've posted my revised C++ implementation of the Mersenne Twister at:
http://www.coyotegulch.com/libcoyote/TwistedRoad/TwistedRoad.html
This is "free-as-in-liberty" and "free-as-in-beer" code....
|
by: Ken Brady |
last post by:
I'm on a team building some class libraries to be used by many other
projects.
Some members of our team insist that "All public methods should be virtual"
just in case "anything needs to be...
|
by: Jordan Abel |
last post by:
I have written this function and was wondering if anyone else could
point out if there's anything wrong with it. The purpose is to
substitute for printf when in a situation where low-level I/O has...
|
by: Kevin Frey |
last post by:
Consider this simple example:
interface IReader
{
bool Read( );
};
class MyReader : IReader
{
bool Read( ); // or should it be bool IReader.Read( ) ?
|
by: Soumyadip Rakshit |
last post by:
Hi,
Could anyone direct me to a very good 1D DCT Implementation in C/C++
My data is in sets of 8 numbers.
Thanks a ton,
Soumyadip.
|
by: Christof Warlich |
last post by:
Hi,
consider this:
template<typename Tclass X {
public:
void doSomething(T t);
};
int main(void) {
X<intx;
|
by: Luc Kumps |
last post by:
(Sorry about the previous post, it got transmitted before it was complete)
We try to separate implementation and interface defintions, but we run into
a problem. I hope the guru's can solve this,...
|
by: TEK |
last post by:
Hello
I'm having some questions regarding the implementation of
IPropertyChanged interface.
Status:
I have the basic stuff up and running, and on every property change I
get a call to my...
|
by: Jason Doucette |
last post by:
I am converting a C-style unit into a C++ class. I have an
implementation function that was defined in the .cpp file (so it was
hidden from the interface that exists in the .h file). It uses a...
|
by: Ankur Arora |
last post by:
Hi All,
I'm building a sample application that uses a custom auto_ptr
implementation.
The program crashes with the following output:-
Output (Debug Assertion failed)
----------
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |