struct unpack newline

[grant]

Hi All,

I am pretty new to python and am having a problem
intepreting binary data using struct.unpack.
I am reading a file containing binary packed data
using open with "rb". All the values are coming through
fine when using (integer1,) = struct.unpack('l', line[86:90])
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values. Error = unpack
string size dows not match format. It seems that
struct, instead of reading 4 bytes for line[86:90]
only reads 2 bytes if the second byte is CR or LF.

Thanks
Grant

[Fredrik Lundh]

gr***@idscape.co.za wrote:

I am pretty new to python and am having a problem
intepreting binary data using struct.unpack.
I am reading a file containing binary packed data
using open with "rb". All the values are coming through
fine when using (integer1,) = struct.unpack('l', line[86:90])
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values. Error = unpack
string size dows not match format. It seems that
struct, instead of reading 4 bytes for line[86:90]
only reads 2 bytes if the second byte is CR or LF.

verifying that struct doesn't care about newlines is of course
pretty trivial:

import struct
struct.unpack("l", "\0\0\0\0") (0,) struct.unpack("l", "\0\r\0\0") (3328,) struct.unpack("l", "\0\n\0\0") (2560,)

have you verified that len(line) really is what you think?
struct.unpack("l", "\0\r\n\0") (658688,) struct.unpack("l", "\0\n\0")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
struct.error: unpack str size does not match format

</F>

[Richard Brodie]

<gr***@idscape.co.za> wrote in message news:11**********************@g14g2000cwa.googlegr oups.com...

except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values.

You probably don't want to be reading binary data a
line at a time, if that's what you're doing.

[grant]

Hi ,

Thanks for the tip regarding checking the length of line. I discovered
that on the problem record it was short by a few bytes. After changing
the read method from "for line in.." to "infile.read(n)" my problem was
solved,
what concerns me though is that although the file is opened in binary
mode,
"for line.." has a problem reading the file correctly.

Thanks
Grant
Fredrik Lundh wrote:

gr***@idscape.co.za wrote:

I am pretty new to python and am having a problem
intepreting binary data using struct.unpack.
I am reading a file containing binary packed data
using open with "rb". All the values are coming through
fine when using (integer1,) = struct.unpack('l', line[86:90])
except when line[86:90] contains "carriage-return" "linefeed"
which are valid binary packed values. Error = unpack
string size dows not match format. It seems that
struct, instead of reading 4 bytes for line[86:90]
only reads 2 bytes if the second byte is CR or LF.

verifying that struct doesn't care about newlines is of course
pretty trivial:

import struct
struct.unpack("l", "\0\0\0\0") (0,) struct.unpack("l", "\0\r\0\0") (3328,) struct.unpack("l", "\0\n\0\0") (2560,)

have you verified that len(line) really is what you think?
struct.unpack("l", "\0\r\n\0") (658688,) struct.unpack("l", "\0\n\0")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
struct.error: unpack str size does not match format

</F>

[Peter Otten]

gr***@idscape.co.za wrote:

what concerns me though is that although the file is opened in binary
mode,
"for line.." has a problem reading the file correctly.

There is _no_ correct way of splitting a file containing binary data in
lines because binary data may contain newline bytes that do not indicate a
new line. E. g.

struct.unpack("l", "\r\n\r\n")

(168626701,)

how should Python know whether it just encountered two empty lines or the
integer 168626701?

Peter

[grant]

Good point. Hadn't thouhgt of that.

Thanks
Grant