prob with struct and byte order

[nephish]

hello there, all.

i have a difficult app that connects to a server to get information for
our database here.
this server is our access point to some equipment in the field that we
monitor.

the messages come in over a socket connection. And according to their
(very limited) documentation, are set up in a particular order. like
this

STX [length indicator] [message type] [message body] ENX

length indicator = length of the message body in 4 bytes
message type = the code for what type of message is being sent
for example 200 = login
message body = the actual command to send to the server

STX and ENX are the values assigned to start and stop a message.
they are literally 'STX' and 'ENX'
when i capture a message i can print it to the screen and the 'STX' at
the beginning and the 'ENX' at the end are discernable, but the rest of
the message looks like this..

\x00\x00\x00,\x00\x00\x00\e17758\x00\x00 and so on and so forth...
with some commas and other symbols in there. (the e before the 17758
has a little hat on it)
If i print it out to a text file, i get this...
STXNULNULNULea17758NULLNULL etc..

now the 17758 is significant because it is the actual unit number of
the machine we want to monitor. I just dont know how to extract the
real values out of this message.

anyone have an idea where to start? i have experimented with struct,
but do not know enough about it. Does anyone know a good tutorial about
learning byte codes and such. The docs on pythons website explain the
module well, but i still do not know what to do with it. Or how to
generate a message or pull apart these values to get a message out of
what the server sends us.

thanks
sk

[Steve Holden]

ne*****@xit.net wrote:

hello there, all.

i have a difficult app that connects to a server to get information for
our database here.
this server is our access point to some equipment in the field that we
monitor.

the messages come in over a socket connection. And according to their
(very limited) documentation, are set up in a particular order. like
this

STX [length indicator] [message type] [message body] ENX

length indicator = length of the message body in 4 bytes
message type = the code for what type of message is being sent
for example 200 = login
message body = the actual command to send to the server

STX and ENX are the values assigned to start and stop a message.
they are literally 'STX' and 'ENX'

I believe you, but this is clearly a perversion of the single-character
"start of transmission" and "end of transmission" that are part of the
ASCII alphabet. So the protocol designer may have been inexperienced,
and the documentation may be confused ...

>
when i capture a message i can print it to the screen and the 'STX' at
the beginning and the 'ENX' at the end are discernable, but the rest of
the message looks like this..

\x00\x00\x00,\x00\x00\x00\e17758\x00\x00 and so on and so forth...
with some commas and other symbols in there. (the e before the 17758
has a little hat on it)
If i print it out to a text file, i get this...
STXNULNULNULea17758NULLNULL etc..

The "\x00" is the Python repr() of a single-byte string containing a
null byte (i.e. a byte whose decimal value is zero).

now the 17758 is significant because it is the actual unit number of
the machine we want to monitor. I just dont know how to extract the
real values out of this message.

anyone have an idea where to start? i have experimented with struct,
but do not know enough about it. Does anyone know a good tutorial about
learning byte codes and such. The docs on pythons website explain the
module well, but i still do not know what to do with it. Or how to
generate a message or pull apart these values to get a message out of
what the server sends us.

Well if the string you are seeing (when you print it from Python) looks like

"STX\x00\x00\x00,\x00\x00\x00\e17758\x00\x00 ... ETX"

then it looks like the bytes you want can be obtained, supposing the
string is stored in variable s, as s[12:17].

>>s = "STX\x00\x00\x00,\x00\x00\x00\e17758\x00\x00 ... ETX"
s[12:17]

'17758'

>>>

You may need to juggle about with the indexes a little, but one of the
joys of Python is that the interactive interpreter is so cooperative!

Good luck.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Grant Edwards]

On 2006-07-24, Steve Holden <st***@holdenweb.comwrote:

ne*****@xit.net wrote:

>hello there, all.

i have a difficult app that connects to a server to get information for
our database here.
this server is our access point to some equipment in the field that we
monitor.

the messages come in over a socket connection. And according to their
(very limited) documentation, are set up in a particular order. like
this

STX [length indicator] [message type] [message body] ENX

length indicator = length of the message body in 4 bytes
message type = the code for what type of message is being sent
for example 200 = login
message body = the actual command to send to the server

STX and ENX are the values assigned to start and stop a message.
they are literally 'STX' and 'ENX'

I believe you, but this is clearly a perversion of the
single-character "start of transmission" and "end of
transmission" that are part of the ASCII alphabet. So the
protocol designer may have been inexperienced, and the
documentation may be confused ...

I had the same reaction: surely he means the frame is delmited
at the beginning by the ASCII STX character (0x02) and the end
by the ETX character (0x03).

If somebody is indeed sending the three character string "STX"
to mark the beginning of a frame and "ENX" to mark the end,
then they're seriously confused (and can't spell).

>when i capture a message i can print it to the screen and the 'STX' at
the beginning and the 'ENX' at the end are discernable, but the rest of
the message looks like this..

\x00\x00\x00,\x00\x00\x00\e17758\x00\x00 and so on and so forth...
with some commas and other symbols in there. (the e before the 17758
has a little hat on it)

So the unit number is an ASCII string. Firstly, I'd recommend
printing the message in hex:

print ' '.join(["%02.2x" % ord(b) for b in message])

That should make it easier to figure how which byte indexes
contain the info you're looking for.

>If i print it out to a text file, i get this...
STXNULNULNULea17758NULLNULL etc..

I'm a but baffled how the string shown above would get printed
like that.

The "\x00" is the Python repr() of a single-byte string containing a
null byte (i.e. a byte whose decimal value is zero).

>now the 17758 is significant because it is the actual unit number of
the machine we want to monitor. I just dont know how to extract the
real values out of this message.

What is meant by "real values"?

>anyone have an idea where to start? i have experimented with
struct, but do not know enough about it. Does anyone know a
good tutorial about learning byte codes and such.

What are "byte codes"?

>The docs on pythons website explain the module well, but i
still do not know what to do with it.

You've got to figure out the format and location within the
message of the objects you care about. Once you know that, you
use struct to pull out the appropriate bytes and convert them
into Python objects.

If you know (or suspect) there are IEEE 754 32-bit floating
point values in the message, then start converting all of the
possible 4-byte chunks into Python floats (using both endian
conventions) until you see numbers you recognize.

Same for 64-bit floats and interger values of various sizes.

--
Grant Edwards grante Yow! HUMAN REPLICAS are
at inserted into VATS of
visi.com NUTRITIONAL YEAST...

[nephish]

ok, i did this print ' '.join(["%02.2x" % ord(b) for b in message])
and i got this in the text file
5354580000002c000000ea3137353834363638353500000000 000000000000000000000000000000000000d6090d54000000 00454e58
so, yes, more of the info seems discernable now.

according to their docs, all of their variables are sent as 32bit long
int (prefferably unsigned long int).

the developers at the server we are trying to talk to have released an
api for C (maybe C++) that comes in an osx module that one is supposed
to import into visual basic. I am not developing in visual basic, or
windows or C or C++. So my challenge is to get some of the same
functionality out of python on debian linux.

the osx module is what they give out to developers who have apps that
need to talk with this server.

the starting and ending deliminators are ASCII 'STX' and 'ENX'. i
wondered why they did it like that also. But they did.

the message length is a 4 byte integer indicating how long the message
body is.
the message type is also 4 bytes. There is a table in the docs that
describe what each message type is. for example 200 = login,
etc...etc...

the big deal in the docs goes on about how the clients need the byte
order to match that of the arch that the server runs on 'Sun
UltraSPARC' , i think they run Solaris.

if i sound like i do not know what i am talking about... there is a
reason... i am very new to this type of communication with the server.
I have about 6 months with python ( my first language ).

thanks for your help, gents .

-sk
Grant Edwards wrote:

On 2006-07-24, Steve Holden <st***@holdenweb.comwrote:

ne*****@xit.net wrote:

hello there, all.

i have a difficult app that connects to a server to get information for
our database here.
this server is our access point to some equipment in the field that we
monitor.

the messages come in over a socket connection. And according to their
(very limited) documentation, are set up in a particular order. like
this

STX [length indicator] [message type] [message body] ENX

length indicator = length of the message body in 4 bytes
message type = the code for what type of message is being sent
for example 200 = login
message body = the actual command to send to the server

STX and ENX are the values assigned to start and stop a message.
they are literally 'STX' and 'ENX'

I believe you, but this is clearly a perversion of the
single-character "start of transmission" and "end of
transmission" that are part of the ASCII alphabet. So the
protocol designer may have been inexperienced, and the
documentation may be confused ...

I had the same reaction: surely he means the frame is delmited
at the beginning by the ASCII STX character (0x02) and the end
by the ETX character (0x03).

If somebody is indeed sending the three character string "STX"
to mark the beginning of a frame and "ENX" to mark the end,
then they're seriously confused (and can't spell).

when i capture a message i can print it to the screen and the 'STX' at
the beginning and the 'ENX' at the end are discernable, but the rest of
the message looks like this..

\x00\x00\x00,\x00\x00\x00\e17758\x00\x00 and so on and so forth...
with some commas and other symbols in there. (the e before the 17758
has a little hat on it)

So the unit number is an ASCII string. Firstly, I'd recommend
printing the message in hex:

print ' '.join(["%02.2x" % ord(b) for b in message])

That should make it easier to figure how which byte indexes
contain the info you're looking for.

If i print it out to a text file, i get this...
STXNULNULNULea17758NULLNULL etc..

I'm a but baffled how the string shown above would get printed
like that.

The "\x00" is the Python repr() of a single-byte string containing a
null byte (i.e. a byte whose decimal value is zero).

now the 17758 is significant because it is the actual unit number of
the machine we want to monitor. I just dont know how to extract the
real values out of this message.

What is meant by "real values"?

anyone have an idea where to start? i have experimented with
struct, but do not know enough about it. Does anyone know a
good tutorial about learning byte codes and such.

What are "byte codes"?

The docs on pythons website explain the module well, but i
still do not know what to do with it.

You've got to figure out the format and location within the
message of the objects you care about. Once you know that, you
use struct to pull out the appropriate bytes and convert them
into Python objects.

If you know (or suspect) there are IEEE 754 32-bit floating
point values in the message, then start converting all of the
possible 4-byte chunks into Python floats (using both endian
conventions) until you see numbers you recognize.

Same for 64-bit floats and interger values of various sizes.

--
Grant Edwards grante Yow! HUMAN REPLICAS are
at inserted into VATS of
visi.com NUTRITIONAL YEAST...

[Grant Edwards]

On 2006-07-24, ne*****@xit.net <ne*****@xit.netwrote:

ok, i did this print ' '.join(["%02.2x" % ord(b) for b in message])
and i got this in the text file

5354580000002c000000ea3137353834363638353500000000 000000000000000000000000000000000000d6090d54000000 00454e58

No, I don't think so. If you did what you said you did, you
would have gotten something with spaces in it:

>>' '.join(["%02.2x" % ord(b) for b in "hi there, how are you"])

'68 69 20 74 68 65 72 65 2c 20 68 6f 77 20 61 72 65 20 79 6f 75'

It's very important that you cut-paste things into postings
exactly as they appear in your program and output. We can
sometimes guess what you did, but not always. In this case,
I'm guessing you did ''.join rather than ' '.join.

so, yes, more of the info seems discernable now.

according to their docs, all of their variables are sent as 32bit long
int (prefferably unsigned long int).

Then you can use the struct module to pull those values out of
the message:

def tohex(bytes):
return ' '.join(['%02.2x' % ord(b) for b in bytes])

startDelimiter,msglen = struct.unpack('>3sI', message[:7])
endDelimiter = struc.unpack('>3s',message[-3:])
assert startDelimiter == 'STX'
assert endDelimiter == 'ENX'
payload = message[7:-3]
assert msglen == len(payload)

while len(payload) >= 4:
print struct.unpack('>I',payload[:4])
payload = payload[4:]

if payload:
print "extra bytes", tohex(payload)

.... or whatever. You'll probably want to pull the message type
out first, and then look up a format string using the message
type as the key.

NB: The above code is off-the-cuff and untested. It almost
certainly contains typos and maybe even a thinko.

the big deal in the docs goes on about how the clients need the byte
order to match

That's what the '>' at the beginning of the struct format
string does. Your message appears to be big-endian, so you use
the '>' specifier to tell the struct module to interprent the
data as big-endian.

that of the arch that the server runs on 'Sun UltraSPARC'

Which is indeed big-endian.

i think they run Solaris.

Which doesn't actually matter -- you'd see the same thing if
they were running Linux, BSD, or some other OS.

--
Grant Edwards grante Yow! Are we live or
at on tape?
visi.com

[John Machin]

ne*****@xit.net wrote:

the starting and ending deliminators are ASCII 'STX' and 'ENX'. i
wondered why they did it like that also. But they did.

the message length is a 4 byte integer indicating how long the message
body is.
the message type is also 4 bytes. There is a table in the docs that
describe what each message type is. for example 200 = login,
etc...etc...

the big deal in the docs goes on about how the clients need the byte
order to match that of the arch that the server runs on 'Sun
UltraSPARC' , i think they run Solaris.

if i sound like i do not know what i am talking about... there is a
reason... i am very new to this type of communication with the server.
I have about 6 months with python ( my first language ).

Perhaps if you could post
(a) the *FULL* docs for one record type -- it's not apparent whether
you have these or not; Grant is trying to show you how to do elementary
data detective work ...
(b) the Python repr() of an example or two of that record type
(c) what you expect (or are told) that example data means,
we could post back some sample complete code with a bit of explanation

Cheers,
John

[John Machin]

Grant Edwards wrote:

On 2006-07-24, Steve Holden <st***@holdenweb.comwrote:

ne*****@xit.net wrote:

now the 17758 is significant because it is the actual unit number of
the machine we want to monitor. I just dont know how to extract the
real values out of this message.

What is meant by "real values"?

:-)
I guess he means "real" as opposed to unreal/surreal/virtual/imagined,
not as in the FORTRAN programmer's credo:
"""
GOD is REAL
JESUS is INTEGER
"""
(-:

[nephish]

ok. indeed i did do '' instead of ' '.
here is an example of a message
'STX\x00\x00\x004\x00\x00\x00\xc8stateman\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00state1man\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00ENX'

tohex gave me '53 54 58 00 00 00 34 00 00 00 c8 70 69 76 6f 74 72 61 63
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 72 61 63 31 70 69 76
6f 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58'

this is the login message (message type 200)

i will try out your code sample when i get to work in the morning.

thanks, gents, for your time and attention on this

sk

John Machin wrote:

Grant Edwards wrote:

On 2006-07-24, Steve Holden <st***@holdenweb.comwrote:

ne*****@xit.net wrote:
>
>now the 17758 is significant because it is the actual unit number of
>the machine we want to monitor. I just dont know how to extract the
>real values out of this message.

What is meant by "real values"?

:-)
I guess he means "real" as opposed to unreal/surreal/virtual/imagined,
not as in the FORTRAN programmer's credo:
"""
GOD is REAL
JESUS is INTEGER
"""
(-:

[John Machin]

ne*****@xit.net wrote:

ok. indeed i did do '' instead of ' '.
here is an example of a message
'STX\x00\x00\x004\x00\x00\x00\xc8stateman\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00state1man\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00ENX'

tohex gave me '53 54 58 00 00 00 34 00 00 00 c8 70 69 76 6f 74 72 61 63
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 72 61 63 31 70 69 76
6f 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58'

It may well have given you that, but *NOT* from the 'STXetcetc' string
you quote above.

>
this is the login message (message type 200)

Perhaps you might consider giving examples of some other message
type(s) e.g. ones that have more data and less possibly sensitive info
than what are presumably usernames and passwords.

>
i will try out your code sample when i get to work in the morning.

Do you actually have documentation of the individual fields in each
message type (like what is the maximum length of each of those
NUL-padded text fields in the login message")?
How many different message types are there?
Note that given a reasonably well laid out soft copy of the docs for
each message type, it would be quite within reason to write a fairly
short script to *generate* correct legible runnable Python code ...
it's scarcely a novel concept.

Cheers,
John

[Marc 'BlackJack' Rintsch]

In <11*********************@b28g2000cwb.googlegroups. com>, nephish wrote:

tohex gave me
'53 54 58

S T X

00 00 00 34

Length!? Decimal 57.

00 00 00 c8

Type!? Decimal 200.

70 69 76 6f 74 72 61 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
74 72 61 63 31 70 69 76 6f 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Payload!?

45 4e 58'

E N X

this is the login message (message type 200)

import struct

data = ('STX'
'\x00\x00\x004'
'\x00\x00\x00\xc8'
'stateman\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00'
'\x00\x00\x00state1man\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00'
'\x00\x00\x00\x00ENX')

def split_message(message):
if not (message.startswith('STX') or message.endswith('ENX')):
raise ValueError('missing start or end sequence')
length, message_type = struct.unpack('>II', message[3:11])
return length, message_type, message[11:-3]

print 'length: %d\ntype: %d\n%r' % split_message(data)

The problem I see is the length. The payload without STX, ENX and the two
numbers in front is 47 bytes so there's a 5 byte difference. You have to
look at some more messages to get an idea how the length corresponds to
the actual payloads length.

Ciao,
Marc 'BlackJack' Rintsch

[John Machin]

Marc 'BlackJack' Rintsch wrote:

In <11*********************@b28g2000cwb.googlegroups. com>, nephish wrote:

tohex gave me
'53 54 58

S T X

00 00 00 34

Length!? Decimal 57.

3 * 16 + 4 -52 where I come from -- assuming hex means hexadecimal
and not witchcraft :-)

>

00 00 00 c8

Type!? Decimal 200.

70 69 76 6f 74 72 61 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
74 72 61 63 31 70 69 76 6f 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Payload!?

45 4e 58'

E N X

this is the login message (message type 200)

The problem I see is the length. The payload without STX, ENX and the two
numbers in front is 47 bytes so there's a 5 byte difference.

I don't think so.

You have to
look at some more messages to get an idea how the length corresponds to
the actual payloads length.

Yes, but not because of the 5-difference problem. The OP has favoured
us with 3 messages (in 3 different formats), 2 x login and 1 of some
sort of data. See below.

8<--- script start
import struct

def unhex(s, spaced):
return ''.join([chr(int(s[x:x+2], 16)) for x in xrange(0, len(s), 2
+ spaced)]) # gasp

hex1 =
"5354580000002c000000ea313735383436363835350000000 0000000000000000000000000000000000000d6090d5400000 000454e58"
txt1 = unhex(hex1, 0)

txt2 =
'STX\x00\x00\x004\x00\x00\x00\xc8stateman\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00state1man\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00ENX'
hex3 = '53 54 58 00 00 00 34 00 00 00 c8 70 69 76 6f 74 72 61 63 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 72 61 63 31 70 69 76 6f 74
00 00 00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58'
txt3 = unhex(hex3, 1)

for msgno, msg in enumerate((txt1, txt2, txt3)):
print "\nMessage %d: length of actual string is %d" % (msgno + 1,
len(msg))
print "Has STX/ENX:", msg.startswith("STX") and msg.endswith("ENX")
print "repr:", repr(msg)
print "hex :", ' '.join(["%02x" % ord(x) for x in msg])
msg_len, msg_type = struct.unpack('>II', msg[3:11])
print "Internal len: %d; type: %d" % (msg_len, msg_type)
8<--- end script, start output

Message 1: length of actual string is 54
Has STX/ENX: True
repr:
'STX\x00\x00\x00,\x00\x00\x00\xea1758466855\x00\x0 0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\xd6\t\rT\x00\x00\x 00\x00ENX'
hex : 53 54 58 00 00 00 2c 00 00 00 ea 31 37 35 38 34 36 36 38 35 35 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d6 09 0d
54 00 00 00 00 45 4e 58
Internal len: 44; type: 234

Message 2: length of actual string is 61
Has STX/ENX: True
repr:
'STX\x00\x00\x004\x00\x00\x00\xc8stateman\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00state1man\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00ENX'
hex : 53 54 58 00 00 00 34 00 00 00 c8 73 74 61 74 65 6d 61 6e 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 73 74 61 74 65 31 6d 61 6e 00 00
00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58
Internal len: 52; type: 200

Message 3: length of actual string is 62
Has STX/ENX: True
repr:
'STX\x00\x00\x004\x00\x00\x00\xc8pivotrac\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00trac1pivot\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00\x00\x00\x00\x00ENX'
hex : 53 54 58 00 00 00 34 00 00 00 c8 70 69 76 6f 74 72 61 63 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 74 72 61 63 31 70 69 76 6f 74 00
00 00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58
Internal len: 52; type: 200
8<---
Messages 1 and 3 tend to indicate that external_len == internal_len +
10 ... maybe there's been a copy/paste problem somewhere along the line
with message 2; perhaps the OP could check this.

Cheers,
John

[nephish]

ok. here is how i got a message in the first place. The data server
developers released this one windows app that simulates a real app. In
that it logs into the server and sends and receives commands. instead
of putting in the data servers ip though, i put in the ip address of
the linux box i am building this on. when a command got sent, i had a
listening socket open that would receive the command from the
simulation program and dump it into a mysql table. So far its worked
becuase the same command works with the real server. But now, i have to
learn how to contruct messages for real because of the control we want
to have over the field units.

i put this in

def split_message(message):
if not (message.startswith('STX') or message.endswith('ENX')):
raise ValueError('missing start or end sequence')
length, message_type = struct.unpack('>II', message[3:11])
return length, message_type, message[11:-3]

print 'length: %d\ntype: %d\n%r' % split_message(data)

and this is what was in the txt file. ( i am using a text file for
sys.stdout becuase my terminal app does not do copy and paste.)

length: 52
type: 200
'stateman\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00stat1manet\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

i used the same snippit for a query message
length: 44
type: 234
'1758466855\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ xd6\t\rT\x00\x00\x00\x00'

now, the message type 234 is correct according to the docs. Here is
what it has to say.
query_ANSI:
used to request information about a particular data system in the field
unsigned long int messageType = 234;
unsigned char primaryMIN(32); # this is the serial number of
the unit
unsigned long int ESN # serial number of the communicator
unsigned long int userID
thanks again guys.. i think we're getting closer.

-sk

John Machin wrote:

Marc 'BlackJack' Rintsch wrote:

In <11*********************@b28g2000cwb.googlegroups. com>, nephish wrote:

tohex gave me
'53 54 58

S T X

00 00 00 34

Length!? Decimal 57.

3 * 16 + 4 -52 where I come from -- assuming hex means hexadecimal
and not witchcraft :-)

00 00 00 c8

Type!? Decimal 200.

70 69 76 6f 74 72 61 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
74 72 61 63 31 70 69 76 6f 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Payload!?

45 4e 58'

E N X

this is the login message (message type 200)

The problem I see is the length. The payload without STX, ENX and the two
numbers in front is 47 bytes so there's a 5 byte difference.

I don't think so.

You have to
look at some more messages to get an idea how the length corresponds to
the actual payloads length.

Yes, but not because of the 5-difference problem. The OP has favoured
us with 3 messages (in 3 different formats), 2 x login and 1 of some
sort of data. See below.

8<--- script start
import struct

def unhex(s, spaced):
return ''.join([chr(int(s[x:x+2], 16)) for x in xrange(0, len(s), 2
+ spaced)]) # gasp

hex1 =
"5354580000002c000000ea313735383436363835350000000 0000000000000000000000000000000000000d6090d5400000 000454e58"
txt1 = unhex(hex1, 0)

txt2 =
'STX\x00\x00\x004\x00\x00\x00\xc8stateman\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00state1man\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00ENX'
hex3 = '53 54 58 00 00 00 34 00 00 00 c8 70 69 76 6f 74 72 61 63 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 72 61 63 31 70 69 76 6f 74
00 00 00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58'
txt3 = unhex(hex3, 1)

for msgno, msg in enumerate((txt1, txt2, txt3)):
print "\nMessage %d: length of actual string is %d" % (msgno + 1,
len(msg))
print "Has STX/ENX:", msg.startswith("STX") and msg.endswith("ENX")
print "repr:", repr(msg)
print "hex :", ' '.join(["%02x" % ord(x) for x in msg])
msg_len, msg_type = struct.unpack('>II', msg[3:11])
print "Internal len: %d; type: %d" % (msg_len, msg_type)
8<--- end script, start output

Message 1: length of actual string is 54
Has STX/ENX: True
repr:
'STX\x00\x00\x00,\x00\x00\x00\xea1758466855\x00\x0 0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\xd6\t\rT\x00\x00\x 00\x00ENX'
hex : 53 54 58 00 00 00 2c 00 00 00 ea 31 37 35 38 34 36 36 38 35 35 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d6 09 0d
54 00 00 00 00 45 4e 58
Internal len: 44; type: 234

Message 2: length of actual string is 61
Has STX/ENX: True
repr:
'STX\x00\x00\x004\x00\x00\x00\xc8stateman\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00state1man\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00ENX'
hex : 53 54 58 00 00 00 34 00 00 00 c8 73 74 61 74 65 6d 61 6e 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 73 74 61 74 65 31 6d 61 6e 00 00
00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58
Internal len: 52; type: 200

Message 3: length of actual string is 62
Has STX/ENX: True
repr:
'STX\x00\x00\x004\x00\x00\x00\xc8pivotrac\x00\x00\ x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00trac1pivot\x00\x00\x00\x00\x00\x00\x00\x00\x0 0\x00\x00\x00\x00\x00ENX'
hex : 53 54 58 00 00 00 34 00 00 00 c8 70 69 76 6f 74 72 61 63 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 74 72 61 63 31 70 69 76 6f 74 00
00 00 00 00 00 00 00 00 00 00 00 00 00 45 4e 58
Internal len: 52; type: 200
8<---
Messages 1 and 3 tend to indicate that external_len == internal_len +
10 ... maybe there's been a copy/paste problem somewhere along the line
with message 2; perhaps the OP could check this.

Cheers,
John