473,322 Members | 1,510 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

problem with logic in reading a binary file

Hello,

I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.

Any help would be great.

Here is my function that takes in a string.

def parseSequence(data, start):

group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
pos = start+8+length
element = (group_num+element_num)

if element == '\xfe\xff\x00\xe0':
data = value

while start < length:
group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
start = start+8+length
element = (group_num+element_num)

if element == '\xfe\xff\x00\xe0':
data = value

while start < length:
group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
start = start+8+length
element = (group_num+element_num)
return element, start, value

else:
return element, start, value

else:
return element, pos, value

And, here is a sample string (I have split up and indented for
readability). There is an identifier (\xfe\xff\x00\xe0) followed by
the length of the nested values.
'\xfe\xff\x00\xe0\x18\x02\x00\x00 -length=536
\n0q\x00\x02\x00\x00\x001
\n0x\x00\x02\x00\x00\x0010
\n0\x80\x00\x02\x00\x00\x004
\n0\xa0\x00\x02\x00\x00\x000
\x0c0\x04\x00\xe8\x01\x00\x00
\xfe\xff\x00\xe0p\x00\x00\x00 -length=112
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x0c\x00\x00\x008.9617062e-1
\n0\x86\x00\x10\x00\x00\x00127.378510918301
\x0c0\x06\x00\x02\x00\x00\x001
\xfe\xff\x00\xe0p\x00\x00\x00 -length=112
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x0c\x00\x00\x001.629998e-1
\n0\x86\x00\x10\x00\x00\x0023.159729257873
\x0c0\x06\x00\x02\x00\x00\x004
\xfe\xff\x00\xe0t\x00\x00\x00 -length=116
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x10\x00\x00\x001.26285318894435
\n0\x86\x00\x10\x00\x00\x00227.690980638769
\x0c0\x06\x00\x02\x00\x00\x003
\xfe\xff\x00\xe0t\x00\x00\x00 -length=116
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x10\x00\x00\x001.52797639111557
\n0\x86\x00\x10\x00\x00\x00263.433384670643
\x0c0\x06\x00\x02\x00\x00\x002 ')
Mar 29 '08 #1
8 1918
Br***********@gmail.com wrote:
Hello,

I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.

Any help would be great.
Without having looked at your code an any detail, may I humbly suggest
that you throw it all out and use the struct module:

http://docs.python.org/lib/module-struct.html

It is meant to solve this kind of problem, and it is quite easy to use.

Gary Herron
Here is my function that takes in a string.

def parseSequence(data, start):

group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
pos = start+8+length
element = (group_num+element_num)

if element == '\xfe\xff\x00\xe0':
data = value

while start < length:
group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
start = start+8+length
element = (group_num+element_num)

if element == '\xfe\xff\x00\xe0':
data = value

while start < length:
group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
start = start+8+length
element = (group_num+element_num)
return element, start, value

else:
return element, start, value

else:
return element, pos, value

And, here is a sample string (I have split up and indented for
readability). There is an identifier (\xfe\xff\x00\xe0) followed by
the length of the nested values.
'\xfe\xff\x00\xe0\x18\x02\x00\x00 -length=536
\n0q\x00\x02\x00\x00\x001
\n0x\x00\x02\x00\x00\x0010
\n0\x80\x00\x02\x00\x00\x004
\n0\xa0\x00\x02\x00\x00\x000
\x0c0\x04\x00\xe8\x01\x00\x00
\xfe\xff\x00\xe0p\x00\x00\x00 -length=112
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x0c\x00\x00\x008.9617062e-1
\n0\x86\x00\x10\x00\x00\x00127.378510918301
\x0c0\x06\x00\x02\x00\x00\x001
\xfe\xff\x00\xe0p\x00\x00\x00 -length=112
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x0c\x00\x00\x001.629998e-1
\n0\x86\x00\x10\x00\x00\x0023.159729257873
\x0c0\x06\x00\x02\x00\x00\x004
\xfe\xff\x00\xe0t\x00\x00\x00 -length=116
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x10\x00\x00\x001.26285318894435
\n0\x86\x00\x10\x00\x00\x00227.690980638769
\x0c0\x06\x00\x02\x00\x00\x003
\xfe\xff\x00\xe0t\x00\x00\x00 -length=116
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x10\x00\x00\x001.52797639111557
\n0\x86\x00\x10\x00\x00\x00263.433384670643
\x0c0\x06\x00\x02\x00\x00\x002 ')
Mar 29 '08 #2
On Mar 29, 3:44 pm, "Bryan.Fodn...@gmail.com"
<Bryan.Fodn...@gmail.comwrote:
Hello,

I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.

Any help would be great.
I'm too lazy to debug your binary string, but I suggest that you
completely throw away the binary file and restart with a database or
structured text. See, for example:

http://pyyaml.org/wiki/PyYAML

If you have some legacy binary file that you need to process, try
creating a C program that freads the binary file and printfs a text
equivalent.

If the decision of using binary files is not yours, then

>
Here is my function that takes in a string.

def parseSequence(data, start):

group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
pos = start+8+length
element = (group_num+element_num)

if element == '\xfe\xff\x00\xe0':
data = value

while start < length:
group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
start = start+8+length
element = (group_num+element_num)

if element == '\xfe\xff\x00\xe0':
data = value

while start < length:
group_num = data[start:start+2]
element_num = data[start+2:start+4]
vl_field = data[start+4:start+8]
length = struct.unpack('hh', vl_field)[0]
value = data[start+8:(start+8+length)]
start = start+8+length
element = (group_num+element_num)
return element, start, value

else:
return element, start, value

else:
return element, pos, value

And, here is a sample string (I have split up and indented for
readability). There is an identifier (\xfe\xff\x00\xe0) followed by
the length of the nested values.

'\xfe\xff\x00\xe0\x18\x02\x00\x00 -length=536
\n0q\x00\x02\x00\x00\x001
\n0x\x00\x02\x00\x00\x0010
\n0\x80\x00\x02\x00\x00\x004
\n0\xa0\x00\x02\x00\x00\x000
\x0c0\x04\x00\xe8\x01\x00\x00
\xfe\xff\x00\xe0p\x00\x00\x00 -length=112
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x0c\x00\x00\x008.9617062e-1
\n0\x86\x00\x10\x00\x00\x00127.378510918301
\x0c0\x06\x00\x02\x00\x00\x001
\xfe\xff\x00\xe0p\x00\x00\x00 -length=112
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x0c\x00\x00\x001.629998e-1
\n0\x86\x00\x10\x00\x00\x0023.159729257873
\x0c0\x06\x00\x02\x00\x00\x004
\xfe\xff\x00\xe0t\x00\x00\x00 -length=116
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x10\x00\x00\x001.26285318894435
\n0\x86\x00\x10\x00\x00\x00227.690980638769
\x0c0\x06\x00\x02\x00\x00\x003
\xfe\xff\x00\xe0t\x00\x00\x00 -length=116
\n0\x82\x002\x00\x00\x0042.9068704277562\\-392.3545926477\
\189.182112099444
\n0\x84\x00\x10\x00\x00\x001.52797639111557
\n0\x86\x00\x10\x00\x00\x00263.433384670643
\x0c0\x06\x00\x02\x00\x00\x002 ')
Mar 30 '08 #3
On Mar 30, 3:58 pm, hdante <hda...@gmail.comwrote:
On Mar 29, 3:44 pm, "Bryan.Fodn...@gmail.com"

<Bryan.Fodn...@gmail.comwrote:
Hello,
I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.
Any help would be great.

I'm too lazy to debug your binary string, but I suggest that you
completely throw away the binary file and restart with a database or
structured text. See, for example:

http://pyyaml.org/wiki/PyYAML

If you have some legacy binary file that you need to process, try
creating a C program that freads the binary file and printfs a text
equivalent.
.... and that couldn't be done faster and better in Python??

Mar 30 '08 #4
On Mar 30, 4:31 am, John Machin <sjmac...@lexicon.netwrote:
On Mar 30, 3:58 pm, hdante <hda...@gmail.comwrote:
On Mar 29, 3:44 pm, "Bryan.Fodn...@gmail.com"
<Bryan.Fodn...@gmail.comwrote:
Hello,
I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.
Any help would be great.
I'm too lazy to debug your binary string, but I suggest that you
completely throw away the binary file and restart with a database or
structured text. See, for example:
http://pyyaml.org/wiki/PyYAML
If you have some legacy binary file that you need to process, try
creating a C program that freads the binary file and printfs a text
equivalent.

... and that couldn't be done faster and better in Python??
No. A C struct is done faster and better than python (thus, the
correctness check is faster in C). Also, chances are high that there's
already an include file with the binary structure.
Mar 30 '08 #5
hdante schrieb:
On Mar 30, 4:31 am, John Machin <sjmac...@lexicon.netwrote:
>On Mar 30, 3:58 pm, hdante <hda...@gmail.comwrote:
>>On Mar 29, 3:44 pm, "Bryan.Fodn...@gmail.com"
<Bryan.Fodn...@gmail.comwrote:
Hello,
I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.
Any help would be great.
I'm too lazy to debug your binary string, but I suggest that you
completely throw away the binary file and restart with a database or
structured text. See, for example:
http://pyyaml.org/wiki/PyYAML
If you have some legacy binary file that you need to process, try
creating a C program that freads the binary file and printfs a text
equivalent.
... and that couldn't be done faster and better in Python??

No. A C struct is done faster and better than python (thus, the
correctness check is faster in C). Also, chances are high that there's
already an include file with the binary structure.
That is utter nonsense. There is no "correctness check" in C. and using
printf & thus creating strings that you then need to parse in python
just doubles the effort needlessly.

The standard-lib module "struct" is exactly what you need, nothing else.
it sure is faster than any parsing of preprocessed data, doesn't
introduce a language-mixture and is prototyped/tested much faster
because of it being python - and not C-compiler and C-debugger.

Alternatively, *IF* there were C-structure-declarations available for
the binary format, the usage of ctypes would allow for roughly the same,
even reducing the effort to create the structure definition a great deal.

Diez
Mar 30 '08 #6
On Mar 30, 9:23 am, "Diez B. Roggisch" <de...@nospam.web.dewrote:
hdante schrieb:
On Mar 30, 4:31 am, John Machin <sjmac...@lexicon.netwrote:
On Mar 30, 3:58 pm, hdante <hda...@gmail.comwrote:
>On Mar 29, 3:44 pm, "Bryan.Fodn...@gmail.com"
<Bryan.Fodn...@gmail.comwrote:
Hello,
I am having trouble writing the code to read a binary string. I would
like to extract the values for use in a calculation.
Any help would be great.
I'm too lazy to debug your binary string, but I suggest that you
completely throw away the binary file and restart with a database or
structured text. See, for example:
http://pyyaml.org/wiki/PyYAML
If you have some legacy binary file that you need to process, try
creating a C program that freads the binary file and printfs a text
equivalent.
... and that couldn't be done faster and better in Python??
No. A C struct is done faster and better than python (thus, the
correctness check is faster in C). Also, chances are high that there's
already an include file with the binary structure.

That is utter nonsense. There is no "correctness check" in C. and using
printf & thus creating strings that you then need to parse in python
just doubles the effort needlessly.

The standard-lib module "struct" is exactly what you need, nothing else.
it sure is faster than any parsing of preprocessed data, doesn't
introduce a language-mixture and is prototyped/tested much faster
because of it being python - and not C-compiler and C-debugger.

Alternatively, *IF* there were C-structure-declarations available for
the binary format, the usage of ctypes would allow for roughly the same,
even reducing the effort to create the structure definition a great deal.

Diez
Whatever you say.
Mar 30 '08 #7
>
Whatever you say.
Can't express what your approval means to me!

Diez
Mar 30 '08 #8
On Sun, 30 Mar 2008 04:48:59 -0700 (PDT), hdante <hd****@gmail.comwrote:
On Mar 30, 4:31 am, John Machin <sjmac...@lexicon.netwrote:
>On Mar 30, 3:58 pm, hdante <hda...@gmail.comwrote:
If you have some legacy binary file that you need to process, try
creating a C program that freads the binary file and printfs a text
equivalent.

... and that couldn't be done faster and better in Python??

No. A C struct is done faster and better than python (thus, the
correctness check is faster in C). Also, chances are high that there's
already an include file with the binary structure.
If a C struct defines the file format, he is probably screwed already.
There are no guarantees that even different compilers on the same
machine have the same struct layout.

I have never seen this done by a serious program.

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.se R'lyeh wgah'nagl fhtagn!
Mar 30 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Brandon McCombs | last post by:
This may be the wrong group but I didn't see anything for VC++ so I'm trying here. I have a C++ book by Deitel and Deitel that says I can use fstream File("data.dat", ios::in | ios::out |...
6
by: | last post by:
I am rewriting a C++ application in C#. This file has a combination of Text and Binary data. I used CFile before to read the text. If I hit a certain string that denotes the following data is...
20
by: ishmael4 | last post by:
hello everyone! i have a problem with reading from binary file. i was googling and searching, but i just cant understand, why isnt this code working. i could use any help. here's the source code:...
11
by: Abhishek | last post by:
I have a problem transfering files using sockets from pocket pc(.net compact c#) to desktop(not using .net just mfc and sockets 2 API). The socket communication is not a issue and I am able to...
2
by: Mad Scientist Jr | last post by:
i'm trying to read a file byte by byte (and later alter the data and write it to a 2nd file byte by byte) and running into a problem where it seems to keep reading the same byte over and over again...
6
by: arne.muller | last post by:
Hello, I've come across some problems reading strucutres from binary files. Basically I've some strutures typedef struct { int i; double x; int n; double *mz;
6
by: efrenba | last post by:
Hi, I came from delphi world and now I'm doing my first steps in C++. I'm using C++builder because its ide is like delphi although I'm trying to avoid the vcl. I need to insert new features...
5
by: Neil Crighton | last post by:
I'm using the zipfile library to read a zip file in Windows, and it seems to be adding too many newlines to extracted files. I've found that for extracted text-encoded files, removing all instances...
27
by: Jeff | last post by:
Im trying to figure out why I cant read back a binary file correctly. I have the following union: #define BITE_RECORD_LEN 12 typedef union { unsigned char byte; struct { unsigned char type;...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.