473,320 Members | 1,982 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

FAQ: how to vary the byte offset of a field of a ctypes.Structure

How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.

The errors I get as I try techniques that don't work include:

AttributeError: '_fields_' must be a sequence of pairs
AttributeError: _fields_ is final
ValueError: Memory cannot be resized because this object doesn't own
it
TypeError: incompatible types, c_ubyte_Array_2 instance instead of
c_ubyte_Array_1 instance

How do I change the offset of a field of a ctypes.Structure?

Is the answer to contribute to the next version of CTypes? Or is this
feature already supported somehow?

Curiously yours, thank in advance,

http://www.google.com/search?q=ctypes+variable+size
http://www.google.com/search?q=ctypes+variable+length
http://www.google.com/search?q=ctypes+variable+offset
http://www.google.com/search?q=ctype...+of+strings%22
http://www.google.com/search?q=ctype...byte+offset%22
http://www.google.com/search?q=ctype...byte+offset%22

May 31 '07 #1
7 6941
p.*******@ieee.org wrote:
How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.

The errors I get as I try techniques that don't work include:

AttributeError: '_fields_' must be a sequence of pairs
AttributeError: _fields_ is final
ValueError: Memory cannot be resized because this object doesn't own
it
TypeError: incompatible types, c_ubyte_Array_2 instance instead of
c_ubyte_Array_1 instance

How do I change the offset of a field of a ctypes.Structure?

Is the answer to contribute to the next version of CTypes? Or is this
feature already supported somehow?

Curiously yours, thank in advance,
How about something like:

class fooStruct(ctypes.Structure):
_pack_ = 1
_fields_=[]
def __init__(self, fields):
self._fields_=fields
ctypes.Structure.__init__(self)

a=fooStruct([('first', 3*ctypes.c_ubyte),
('second', 4*ctypes.c_ubyte)])

print a._fields_
-Larry
May 31 '07 #2
p.*******@ieee.org schrieb:
How do I vary the byte offset of a field of a ctypes.Structure?

How do I "use the dynamic nature of Python, and (re-)define the data
type after the required size is already known, on a case by case
basis"?

\\\

For example, suppose sometimes I receive the value '\x03hi' + \x04bye'
for the struct:

class Struct34(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 3 * ctypes.c_ubyte),
('second', 4 * ctypes.c_ubyte)]

but then sometimes instead I receive the value '\x05left' + \x06right'
for the struct:

class Struct56(ctypes.Structure):
_pack_ = 1
_fields_ = [('first', 5 * ctypes.c_ubyte),
('second', 6 * ctypes.c_ubyte)]

Thus in general I receive (0xFF ** 2) possible combinations of field
lengths.

///

How do I declare all those hugely many simply regular combinations as
one CTypes.structure?

I also need to do series of 3 or 4 or 5 strings, not just 2 strings.
But always the byte offsets of the subsequent fields vary when the
byte sizes of the preceding fields vary. The byte size of the
enclosing packed struct varies as the length of the packed bytes it
contains.
Often it helps to ask yourself the question: How would I do this in C?

IMO, the answer to this question, applied to your problem, would be:
*Not* by using a structure. A structure is fine if the definition is fixed,
or at most has *one* variable sized field at the very end. Nothing
is true for your problem.

Thomas

May 31 '07 #3
ctypes.sizeof(a) is still zero, as if ctypes.Structure.__init__
fetches a.__class__._fields_ rather than a._fields_

May 31 '07 #4
""" Thomas,

Ouch ouch I must have misunderstood what you meant by "use the dynamic
nature of Python, and (re-)define the data type after the required
size is already known, on a case by case basis".

Do you have an example of what you meant? I searched but did not find.
Are those your words?

Yes, to declare strings of strings in Python would be to express a
familiar idea that I don't know how to say well in C.

These are standard structs that I exchange with third parties, e.g.,
me editing Class files read later by a Java interpreter, so I can't
avoid having to deal with this complexity designed into them. For
discussion I've simplified the problem: back in real life I have a few
dozen variations of structs like this to deal with.

The following Python mostly works, but only at the cost of rewriting
the small part of CTypes that I need for this app.
"""

import binascii
import struct

class Struct:

def __init__(self, declarations = []):

"""Add initial values to self and list the names declared."""

names = []
for (initial, name) in declarations:
names += [name]
python = ' '.join(['self.' + name, '=',
'initial'])
exec python
self._names_ = names

def _fields_(self):

"""List the fields."""

fields = []
for name in self._names_:
python = 'self.' + name
fields += [eval(python)]
return fields

def _pack_(self):

"""Pack a copy of the fields."""

packs = ''
for field in self._fields_():
packs += field._pack_()
return packs

def _size_(self, bytes = None):

"""Count the bytes of the fields."""

return len(self._pack_())

def _unpack_(self, bytes):

"""Count the bytes of a copy of the fields."""

offset = 0
for field in self._fields_():
size = field._size_(bytes[offset:])
field._unpack_(bytes[offset:][:size])
offset += size
if offset != len(bytes):
why = ('_unpack_ requires a string argument'
'of length %d' % offset)
raise struct.error(why)

class Field(Struct):

"""Contain one value."""

def __init__(self, value = 0):
self._value_ = value

def _pack_(self):
raise AttributeError('abstract')

def _unpack_(self, bytes):
raise AttributeError('abstract')

class Byte(Field):

"""Contain one byte."""

def _pack_(self):
return struct.pack('B', self._value_)

def _unpack_(self, bytes):
self._value_ = struct.unpack('B', bytes)[0]

class ByteArray(Field):

"""Contain the same nonnegative number of bytes always."""

def _pack_(self):
return self._value_

def _unpack_(self, bytes):
if len(bytes) == len(self._value_):
self._value_ = bytes
else:

why = ('_unpack_ requires a string argument'
'of length %d' % len(self._value_))
raise struct.error(why)

class Symbol(Struct):

"""Contain a count of bytes."""

def __init__(self, value = ''):
Struct.__init__(self, [
(Byte(), 'length'),
(ByteArray(value), 'bytes')])
self._pack_()
def _size_(self, bytes = None):
return ord(bytes[0])

def _pack_(self):
self.length = Byte(self.length._size_() +
self.bytes._size_())
return Struct._pack_(self)

class TwoSymbols(Struct):

"""Contain two Symbols."""

def __init__(self, values = ['', '']):
Struct.__init__(self, [
(Symbol(values[0]), 'first'),
(Symbol(values[1]), 'second')])

zz = Symbol()
print binascii.hexlify(zz._pack_()).upper()
# 01

zz = Symbol()
zz.bytes = ByteArray('left')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x05left'

zz = Symbol()
zz.bytes = ByteArray('right')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x06right'

zz = TwoSymbols()
zz.first.bytes = ByteArray('hi')
zz.second.bytes = ByteArray('bye')
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())
# '\x03hi\x04bye'

print zz._size_()
# 7

yy = '''
def yyFunc(self):
return [self.length, self.bytes]
'''
exec yy
Symbol._fields_ = yyFunc
zz = TwoSymbols(['alef', 'bet'])
zz._unpack_(zz._pack_()) ; print repr(zz._pack_())

May 31 '07 #5
I see that changing self._fields_ doesn't change ctypes.sizeof(self).

I guess ctypes.Structure.__init__(self) fetches
self.__class__._fields_ not self._fields_.

Jun 1 '07 #6
Often it helps to ask yourself the question: How would I do this in C? ...
>
*Not* by using a structure. A structure is fine if the definition is fixed,
or at most has *one* variable sized field at the very end.
http://docs.python.org/lib/module-pickle.html
might be near where I'll find concise Python ways of pickling and
unpickling the (0xFF ** N) possible ways of packing N strings of byte
lengths of 0..xFE together ...

I notice by now I need what ctypes does well often enough that I have
to make my corrupt copy of a subset of ctypes coexist with ctypes per
se. So I have two different BYTE definitions. One is the
ctypes.c_ubyte. The other is my corrupt copy. And likewise for a big-
endian BIG_WORD and so on. So I end up naming the one BYTE and the
other BITE, the one BIG_WORD and the other BIG_WURD, yuck.

Jun 2 '07 #7
http://docs.python.org/lib/module-pickle.html
... concise Python ways of pickling and unpickling
the (0xFF ** N) possible ways of
packing N strings of byte lengths of 0..xFE together ...
Aye, looks like an exercise left open for the student to complete:
>>pickle.dumps("")
"S''\np0\n."
>>>
pickle.dumps("abc")
"S'abc'\np0\n."
>>>
pickle.loads(pickle.dumps("abc"))
'abc'
>>>
pickle.dumps(ctypes.c_ubyte(0))
....
TypeError: abstract class
>>>
Jun 4 '07 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: Vince | last post by:
Hi, I am starting to play with with C++ and I have some questions. I need to parse a XML file that describes a smart card file structure and to initialize my data structure. First I chose this...
11
by: Bradford Chamberlain | last post by:
I work a lot with multidimensional arrays of dynamic size in C, implementing them using a single-dimensional C array and the appropriate multipliers and offsets to index into it appropriately. I...
7
by: War Eagle | last post by:
I have two byte arrays and a char (the letter S) I was to concatenate to one byte array. Here is what code I have. I basically want to send this in a one buffer (byte array?) through a socket. ...
5
by: Olaf Baeyens | last post by:
I have another problem, maybe it is simple to fix. I have this: byte Test=new byte; But I now want to have a second pointer Test2 to point to a location inside this Test. But with no...
14
by: Ronodev.Sen | last post by:
i have a C# program that is sending data in a byte array through a socket. the VC++ application server receives data in teh following format.... typedef struct advice { header sHdr; char ...
5
by: moni | last post by:
Hey, My buffer contains a short int, some char, and a structure in form of a byte array. Read the string as: TextBox4.Text = System.Text.Encoding.ASCII.GetString(buffer1, 0, 31); Read...
6
by: Jack | last post by:
I'm not able to build IP2Location's Python interface so I'm trying to use ctypes to call its C interface. The functions return a pointer to the struct below. I haven't been able to figure out how...
3
by: Andrew Lentvorski | last post by:
Basically, I'd like to use the ctypes module as a much more descriptive "struct" module. Is there a way to take a ctypes.Structure-based class and convert it to/from a binary string? Thanks,...
5
by: castironpi | last post by:
Hi all, I have a mmap and a data structure in it. I know the structure's location in the mmap and what structure it is. It has a ctypes definition. I want to initialize a ctypes object to...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.