470,641 Members | 1,511 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,641 developers. It's quick & easy.

struct unpack newline

Hi All,

I am pretty new to python and am having a problem
intepreting binary data using struct.unpack.
I am reading a file containing binary packed data
using open with "rb". All the values are coming through
fine when using (integer1,) = struct.unpack('l', line[86:90])
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values. Error = unpack
string size dows not match format. It seems that
struct, instead of reading 4 bytes for line[86:90]
only reads 2 bytes if the second byte is CR or LF.

Thanks
Grant

Jul 19 '05 #1
5 5201
gr***@idscape.co.za wrote:
I am pretty new to python and am having a problem
intepreting binary data using struct.unpack.
I am reading a file containing binary packed data
using open with "rb". All the values are coming through
fine when using (integer1,) = struct.unpack('l', line[86:90])
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values. Error = unpack
string size dows not match format. It seems that
struct, instead of reading 4 bytes for line[86:90]
only reads 2 bytes if the second byte is CR or LF.


verifying that struct doesn't care about newlines is of course
pretty trivial:
import struct
struct.unpack("l", "\0\0\0\0") (0,) struct.unpack("l", "\0\r\0\0") (3328,) struct.unpack("l", "\0\n\0\0") (2560,)

have you verified that len(line) really is what you think?
struct.unpack("l", "\0\r\n\0") (658688,) struct.unpack("l", "\0\n\0")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
struct.error: unpack str size does not match format

</F>

Jul 19 '05 #2

<gr***@idscape.co.za> wrote in message news:11**********************@g14g2000cwa.googlegr oups.com...
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values.


You probably don't want to be reading binary data a
line at a time, if that's what you're doing.
Jul 19 '05 #3
Hi ,

Thanks for the tip regarding checking the length of line. I discovered
that on the problem record it was short by a few bytes. After changing
the read method from "for line in.." to "infile.read(n)" my problem was
solved,
what concerns me though is that although the file is opened in binary
mode,
"for line.." has a problem reading the file correctly.

Thanks
Grant
Fredrik Lundh wrote:
gr***@idscape.co.za wrote:
I am pretty new to python and am having a problem
intepreting binary data using struct.unpack.
I am reading a file containing binary packed data
using open with "rb". All the values are coming through
fine when using (integer1,) = struct.unpack('l', line[86:90])
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values. Error = unpack
string size dows not match format. It seems that
struct, instead of reading 4 bytes for line[86:90]
only reads 2 bytes if the second byte is CR or LF.


verifying that struct doesn't care about newlines is of course
pretty trivial:
import struct
struct.unpack("l", "\0\0\0\0") (0,) struct.unpack("l", "\0\r\0\0") (3328,) struct.unpack("l", "\0\n\0\0") (2560,)

have you verified that len(line) really is what you think?
struct.unpack("l", "\0\r\n\0") (658688,) struct.unpack("l", "\0\n\0")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
struct.error: unpack str size does not match format

</F>


Jul 19 '05 #4
gr***@idscape.co.za wrote:
what concerns me though is that although the file is opened in binary
mode,
"for line.." has a problem reading the file correctly.


There is _no_ correct way of splitting a file containing binary data in
lines because binary data may contain newline bytes that do not indicate a
new line. E. g.
struct.unpack("l", "\r\n\r\n")

(168626701,)

how should Python know whether it just encountered two empty lines or the
integer 168626701?

Peter

Jul 19 '05 #5
Good point. Hadn't thouhgt of that.

Thanks
Grant

Jul 19 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Josiah Carlson | last post: by
10 posts views Thread by Giovanni Bajo | last post: by
3 posts views Thread by David Bear | last post: by
4 posts views Thread by OhKyu Yoon | last post: by
2 posts views Thread by brnstrmrs | last post: by
reply views Thread by Ping Zhao | last post: by
???
1 post views Thread by Stoney L | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.