473,320 Members | 2,083 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Struggling with struct.unpack() and "p" format specifier

Hope someone can help.
I am trying to read data from a file binary file and then unpack the
data into python variables. Some of the data is store like this;

xbuffer: '\x00\x00\xb9\x02\x13EXCLUDE_CREDIT_CARD'
# the above was printed using repr(xbuffer).
# Note that int(0x13) = 19 which is exactly the length of the visible
text
#

In the code I have the following statement;
x = st.unpack('>xxBBp',xbuffer)

This throws out the following error;

x = st.unpack('>xxBBp',xbuffer)
error: unpack str size does not match format

As I read the documentation the "p" format string seems to address
this situation, where the number bytes of the string to read is the
first byte of the stored value but I keep getting this error.

Am I missing something ?
Can the "p" format character be used to unpack this type of data ?

As I mentioned, I can parse the string and read it with multiple
statements, I am just looking for a more efficient solution.

Thanks.
Jul 18 '05 #1
5 13698
[Geoffrey <ge********@hotmail.com>]
I am trying to read data from a file binary file and then unpack the
data into python variables. Some of the data is store like this;

xbuffer: '\x00\x00\xb9\x02\x13EXCLUDE_CREDIT_CARD'
# the above was printed using repr(xbuffer).
# Note that int(0x13) = 19 which is exactly the length of the visible
text
#

In the code I have the following statement;
x = st.unpack('>xxBBp',xbuffer)

This throws out the following error;

x = st.unpack('>xxBBp',xbuffer)
error: unpack str size does not match format

As I read the documentation the "p" format string seems to
address this situation, where the number bytes of the string to
read is the first byte of the stored value but I keep getting this
error.


....

Well, the docs mean it when they say:

Note that for unpack(), the "p" format character consumes count
bytes

You don't have an explicit count in front of your "p" code, so count
defaults to 1, so only one byte of xbuffer will get consumed.

This works, telling struct that this particular p field consumes 20
bytes (including the string-length byte):
struct.unpack('>xxBB20p',xbuffer) (185, 2, 'EXCLUDE_CREDIT_CARD')

Or, a bit more generally, assuming your p field is always at the end,
and is preceded by 4 bytes:
struct.unpack('>xxBB%dp' % (len(xbuffer) - 4), xbuffer)

(185, 2, 'EXCLUDE_CREDIT_CARD')

Note that there's no direct support for any kind of variable-width
data in struct. The number of bytes involved has to be deducible from
the format string alone.
Jul 18 '05 #2
Geoffrey wrote:
I am trying to read data from a file binary file and then unpack the
data into python variables. Some of the data is store like this; .... As I read the documentation the "p" format string seems to address
this situation, where the number bytes of the string to read is the
first byte of the stored value but I keep getting this error.

Am I missing something ?
Can the "p" format character be used to unpack this type of data ?


I've tried experimenting with "p" and cannot get any meaningful
results. In all cases pack() returns '\x00' while unpack()
with anything other than a one-byte string returns an exception
(unpack str size does not match format) while with a one-byte
string it always returns ('',).

I would be inclined to say that the "p" format in struct (using
Python 2.4rc1 or Python 2.3.3) does not act as documented on
Windows XP SP2, at least...

I hope we've both just missed something obvious.

-Peter
Jul 18 '05 #3
Peter Hansen wrote:
I would be inclined to say that the "p" format in struct (using
Python 2.4rc1 or Python 2.3.3) does not act as documented on
Windows XP SP2, at least...

I hope we've both just missed something obvious.


Okay, we were certainly missing something, but I don't believe
I would call it obvious.

I can't deduce from the documentation the fact that the "p"
format requires a length *in front of the p in the format string*.

Furthermore, it assumes a length of 1 if one is not specified.

And there is no example that shows how to do it correctly.

(I did Google searches and found examples, but by then I
was looking for a bug report and didn't even think to look
at the examples themselves. :-( )

Doc bug? Did anyone else find the documentation on "p"
to be clear and effective?

-Peter
Jul 18 '05 #4
Geoffrey wrote:
As I mentioned, I can parse the string and read it with multiple
statements, I am just looking for a more efficient solution.


This looks like about the best you can do, using the information
from Tim's reply:
buf = '\0\0\xb9\x02\x13EXCLUDE_CREDIT_CARD'
import struct
x = struct.unpack('>xxBB%sp' % (ord(buf[4])+1), buf)
x

(185, 2, 'EXCLUDE_CREDIT_CARD')

If you wanted to avoid hard-coding the 4, you would
be most correct to do this:

header = '>xxBB'
lenIndex = struct.calcsize(header)
x = struct.unpack('%s%dp' % (header, ord(buf[lenIndex])+1), buf)

.... though that doesn't exactly make it all that readable.

-Peter
Jul 18 '05 #5
Thanks for your response.

I guess the documentation on the p format wasn't clear to me ... or
perhaps I was just hoping to much for an easy solution !

The data is part of a record structure that is written to a file with
a few "int"'s and "longs" mixed in. The pattern repeats through the
file with sometime up to 2500 repititions.

Clearly I can create a subroutine to read the records and extract out
the fields. I was just hoping I could use the "struct" module and
create a pattern like 'LLHpHLpppH' which would unpack the date and
automatically give me the strings without needing to first determine
their lengths as the length is already embedded in the data.

Any suggestion on how to go about proposing the ability to read
variable length strings based on the preceeding byte value to the
struct module ? It seems it would be a valuable addition, helping
with code clarity, readability and saving quite a few lines of code -
well atleast me anyways !

Thanks again.

Peter Hansen <pe***@engcorp.com> wrote in message news:<co**********@utornnr1pp.grouptelecom.net>...
Geoffrey wrote:
As I mentioned, I can parse the string and read it with multiple
statements, I am just looking for a more efficient solution.


This looks like about the best you can do, using the information
from Tim's reply:
>>> buf = '\0\0\xb9\x02\x13EXCLUDE_CREDIT_CARD'
>>> import struct
>>> x = struct.unpack('>xxBB%sp' % (ord(buf[4])+1), buf)
>>> x

(185, 2, 'EXCLUDE_CREDIT_CARD')

If you wanted to avoid hard-coding the 4, you would
be most correct to do this:

header = '>xxBB'
lenIndex = struct.calcsize(header)
x = struct.unpack('%s%dp' % (header, ord(buf[lenIndex])+1), buf)

... though that doesn't exactly make it all that readable.

-Peter

Jul 18 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Matthew Barnes | last post by:
I was wondering if there would be any interest in extending the struct.unpack format notation to be able to express groups of data with parenthesis. For example: >>> data =...
5
by: grant | last post by:
Hi All, I am pretty new to python and am having a problem intepreting binary data using struct.unpack. I am reading a file containing binary packed data using open with "rb". All the values are...
6
by: g.franzkowiak | last post by:
Hello Everybody, I've read a pipe and store it in a object. My next step was the separation from 4 bytes with obj = string.join(list(dataObject) ==> '\x16 \x00 \x00 \x00' and the converting by...
3
by: Eric Jacoboni | last post by:
Hi, To experiment with unpacking, i've written a little C code which stores one record in a file. Then, i try to reread this file to unpack the record. Here's the struct of a record: ...
2
by: Nadav Samet | last post by:
Hi, I am trying to unpack a 32-bit unsigned integer from a string using struct.unpack. so using string.unpack('L', data) would work fine on 32-bit systems, But apparently, on 64-bit...
4
by: OhKyu Yoon | last post by:
Hi! I have a really long binary file that I want to read. The way I am doing it now is: for i in xrange(N): # N is about 10,000,000 time = struct.unpack('=HHHH', infile.read(8)) # do...
2
by: brnstrmrs | last post by:
If I run: testValue = '\x02\x00' junk = struct.unpack('h', testValue) Everything works but If I run testValue = raw_input("Enter Binary Code..:") inputting at the console '\x02\x00' junk...
0
by: Ping Zhao | last post by:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I am writing a small program to decode MS bitmap image. When I use statements as follow, it works fine: header = str(struct.unpack('2s',...
2
by: Heikki Toivonen | last post by:
M2Crypto has some old code that gets and sets socket timeouts in http://svn.osafoundation.org/m2crypto/trunk/M2Crypto/SSL/Connection.py, for example: def get_socket_read_timeout(self): return...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.