struct.calcsize problem

[Chandu]

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

[Larry Bates]

Chandu wrote:

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

Maybe this will help:

import struct
hdrFormats = ['16s','32s', '32s','B','8s','H','8s','H','4s','H','H',
'20s','64s','64s','64s','32s','32s','64s','L','L', 'B',
'B','B','B','B','64s','64s']

sumofcalcsize=0
i=0
for fmt in hdrFormats:
l=struct.calcsize(fmt)
sumofcalcsize+=l
print "fmt='%s', calcsize=%i, sumofcalcsize=%i, calcsize=%i" % \
(fmt, l, sumofcalcsize, struct.calcsize(' '.join(hdrFormats[:i+1])))
i+=1
Outputs:

fmt='16s', calcsize=16, sumofcalcsize=16, calcsize=16
fmt='32s', calcsize=32, sumofcalcsize=48, calcsize=48
fmt='32s', calcsize=32, sumofcalcsize=80, calcsize=80
fmt='B', calcsize=1, sumofcalcsize=81, calcsize=81
fmt='8s', calcsize=8, sumofcalcsize=89, calcsize=89
fmt='H', calcsize=2, sumofcalcsize=91, calcsize=92 <====
fmt='8s', calcsize=8, sumofcalcsize=99, calcsize=100
fmt='H', calcsize=2, sumofcalcsize=101, calcsize=102
fmt='4s', calcsize=4, sumofcalcsize=105, calcsize=106
fmt='H', calcsize=2, sumofcalcsize=107, calcsize=108
fmt='H', calcsize=2, sumofcalcsize=109, calcsize=110
fmt='20s', calcsize=20, sumofcalcsize=129, calcsize=130
fmt='64s', calcsize=64, sumofcalcsize=193, calcsize=194
fmt='64s', calcsize=64, sumofcalcsize=257, calcsize=258
fmt='64s', calcsize=64, sumofcalcsize=321, calcsize=322
fmt='32s', calcsize=32, sumofcalcsize=353, calcsize=354
fmt='32s', calcsize=32, sumofcalcsize=385, calcsize=386
fmt='64s', calcsize=64, sumofcalcsize=449, calcsize=450
fmt='L', calcsize=4, sumofcalcsize=453, calcsize=456 <====
fmt='L', calcsize=4, sumofcalcsize=457, calcsize=460
fmt='B', calcsize=1, sumofcalcsize=458, calcsize=461
fmt='B', calcsize=1, sumofcalcsize=459, calcsize=462
fmt='B', calcsize=1, sumofcalcsize=460, calcsize=463
fmt='B', calcsize=1, sumofcalcsize=461, calcsize=464
fmt='B', calcsize=1, sumofcalcsize=462, calcsize=465
fmt='64s', calcsize=64, sumofcalcsize=526, calcsize=529
fmt='64s', calcsize=64, sumofcalcsize=590, calcsize=593

Larry Bates

[Bengt Richter]

On 7 Nov 2005 15:27:06 -0800, "Chandu" <ch****@trillian.us> wrote:

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

Looks to me like you are getting default native byte order and _alignment_
and some pad bytes are getting added in. For native order with no padding,
try prefixing the format string with '='

hdrFormat '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s 64s L L B B B B B 64s 64s'

If you add up the individual sizes, padding doesn't happen, apparently:
map(struct.calcsize, hdrFormat.split()) [16, 32, 32, 1, 8, 2, 8, 2, 4, 2, 2, 20, 64, 64, 64, 32, 32, 64, 4, 4, 1, 1, 1, 1, 1, 64, 64] sum(map(struct.calcsize, hdrFormat.split())) 590

But default: struct.calcsize(hdrFormat) 593
Apparently is native, with native alignment & I get the same as you: struct.calcsize('@'+hdrFormat) 593
Whereas native order standard (no pad) alignment is: struct.calcsize('='+hdrFormat) 590
Little endian, standard alignment: struct.calcsize('<'+hdrFormat) 590
Big endian, standard alignment struct.calcsize('>'+hdrFormat) 590
Network (big endian), standard alignment: struct.calcsize('!'+hdrFormat)

590

I guess if you want alignment for anything non-native, you have to specify pad bytes
where you need them (with x format character).

Regards,
Bengt Richter

[Chandu]

Thanks for the helpful feedback. I guessed it was the alignment issue,
but could not find the exact format for changing the default. It is not
mystery any more!
Bengt Richter wrote:

On 7 Nov 2005 15:27:06 -0800, "Chandu" <ch****@trillian.us> wrote:

In using the following struct format I get the size as 593. The same C
struct is 590 if packed on byte boundary and 596 when using pragma
pack(4). I am using pack(4) and added 3 spares at the end to get by.
hdrFormat = '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s
64s L L B B B B B 64s 64s'

Any ideas on what I am doing wrong?

Looks to me like you are getting default native byte order and _alignment_
and some pad bytes are getting added in. For native order with no padding,
try prefixing the format string with '='

>>> hdrFormat '16s 32s 32s B 8s H 8s H 4s H H 20s 64s 64s 64s 32s 32s 64s L L B B B B B 64s 64s'

If you add up the individual sizes, padding doesn't happen, apparently:
>>> map(struct.calcsize, hdrFormat.split()) [16, 32, 32, 1, 8, 2, 8, 2, 4, 2, 2, 20, 64, 64, 64, 32, 32, 64, 4, 4, 1, 1, 1, 1, 1, 64, 64] >>> sum(map(struct.calcsize, hdrFormat.split())) 590

But default: >>> struct.calcsize(hdrFormat) 593
Apparently is native, with native alignment & I get the same as you: >>> struct.calcsize('@'+hdrFormat) 593
Whereas native order standard (no pad) alignment is: >>> struct.calcsize('='+hdrFormat) 590
Little endian, standard alignment: >>> struct.calcsize('<'+hdrFormat) 590
Big endian, standard alignment >>> struct.calcsize('>'+hdrFormat) 590
Network (big endian), standard alignment: >>> struct.calcsize('!'+hdrFormat)

590

I guess if you want alignment for anything non-native, you have to specify pad bytes
where you need them (with x format character).

Regards,
Bengt Richter