472,364 Members | 2,055 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,364 software developers and data experts.

proposed struct module format code addition

Good day everyone,

I have produced a patch against the latest CVS to add support for two
new formatting characters in the struct module. It is currently an RFE,
which I include a link to at the end of this post. Please read the
email before you respond to it.

Generally, the struct module is for packing and unpacking of binary
data. It includes support to pack and unpack the c types:
byte, char, short, long, long long, char[], *, and certain variants of
those (signed/unsigned, big/little endian, etc.)
I had proposed two new formatting characters, 'g' and 'G' (for biGint or
lonG int).

There was one primary purpose, to offer users the opportunity to specify
their own integer lengths (very useful for cryptography, and real-world
applications that involve non-standard sized integers). Current
solutions involve shifting, masking, and multiple passes over data.

There is a secondary purpose, and that is that future n-byte integers
(like 16-byte/128-bit integers as supported by SSE2) are already taken
care of.

It also places packing and unpacking of these larger integers in the
same module as packing and packing of other integers, floats, etc. This
makes documentation easy.

Functionality-wise, it merely uses the two C functions
_PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to
handle interfacing with the pack and unpack functions in the struct module.

An example of use is as follows:
struct.pack('>3g', -1) '\xff\xff\xff' struct.pack('>3g', 2**23-1) '\x7f\xff\xff' struct.pack('>3g', 2**23) Traceback (most recent call last):
File "<stdin>", line 1, in ?
OverflowError: long too big to convert struct.pack('>3G', 2**23)


It follows the struct module standard 'lowercase for signed, uppercase
for unsigned'.
There seem to be a few arguments against its inclusion into

The size specifier is variable, so you must know the size/magnitude
of the thing you are (un)packing before you (un)pack it.

My Response:
All use cases I have for this particular mechanism involve not using
'variable' sized structs, but fixed structs with integers of
non-standard byte-widths. Specifically, I have a project in which I use
some 3 and 5 byte unsigned integers. One of my (un)pack format
specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these
are the most concise).
Certainly this does not fit the pickle/cPickle long (un)packing
use-case, but that problem relies on truely variable long integer
lengths, of which this specifier does not seek to solve.
Really, the proposed 'g' and 'G' format specifiers are only as
variable as the previously existing 's' format specifier.
The new specifiers are not standard C types.

My Response:
Certainly they are not standard C types, but they are flexible
enough to subsume all current integer C type specifiers. The point was
to allow a user to have the option of specifying their own integer
lengths. This supports use cases involving certain kinds of large
dataset processing (my use case, which I may discuss after we release)
and cryptography, specifically in the case of PKC...
while 1:
blk = get_block()
iblk = struct.unpack('>128G', blk)[0]
uiblk = pow(iblk, power, modulous)
write_block(struct.pack('>128G', uiblk))

The 'p' format specifier is also not a standard C type, and yet it
is included in struct, specifically because it is useful.
You can already do the same thing with:
and some likely soon-to-be included additions to the binascii module.

My Response:
That is not the same. Nontrivial problems require multiple passes
over your data with multiple calls. A simple:
struct.unpack('H3G3G', st)
pickle.decode_long(st[:2]) #or an equivalent struct call
And has no endian or sign options, or requires the status quo using of
masks and shifts to get the job done. As previously stated, one point
of the module is to reduce the amount of bit shifting and masking required.
We could just document a method for packing/unpacking these kinds of
things in the struct module, if this really is where people would look
for such a thing.

My Response:
I am not disputing that there are other methods of doing this, I am
saying that the struct module includes a framework and documentation
location that can include this particular modification with little
issue, which is far better than any other proposed location for
equivalent functionality.
Note that functionality equivalent to pickle.encode/decode_long is
NOT what this proposed enhancement is for.
The struct module has a steep learning curve already, and this new
format specifier doesn't help it.

My Response:
I can't see how a new format specifier would necessarily make the
learning curve any more difficult, if it was even difficult in the first

Why am I even posting
Raymond has threatened to close this RFE due to the fact that only I
have been posting to state that I would find such an addition useful.

If you believe this functionality is useful, or even if you think that I
am full of it, tell us: http://python.org/sf/1023290

- Josiah
Jul 18 '05 #1
0 2171

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: Angelo Secchi | last post by:
I'm trying to use the unpack method in the struct module to parse a binary file without success. I have a binary file with records that include many fields for a total length of 1970. Few days ago...
by: Raymond Hettinger | last post by:
Comments are invited on the following proposed PEP. Raymond Hettinger ------------------------------------------------------- PEP: 329
by: Geoffrey | last post by:
Hope someone can help. I am trying to read data from a file binary file and then unpack the data into python variables. Some of the data is store like this; xbuffer:...
by: Bryan Olson | last post by:
The Python slice type has one method 'indices', and reportedly: This method takes a single integer argument /length/ and computes information about the extended slice that the slice object would...
by: Richard Cornford | last post by:
Anyone who has taken a look at the online FAQ today may have noticed that I have updated it. The majority of the changes are the updating of broken links and the implementation of that extensive...
by: Giovanni Bajo | last post by:
Hello, given the ongoing work on struct (which I thought was a dead module), I was wondering if it would be possible to add an API to register custom parsing codes for struct. Whenever I use it...
by: nephish | last post by:
hello there, all. i have a difficult app that connects to a server to get information for our database here. this server is our access point to some equipment in the field that we monitor. ...
by: Jansson Christer | last post by:
Hi all, I have discovered that in my Python 2.4.1 installation (on Solaris 8), struct.pack handles things in a way that seems inconsistent to me. I haven't found any comprehensible...
by: homostannous | last post by:
I want to import some binary data with the struct module, edit it, then export it again. The problem is that I can't find a shorthand way to represent my list of data without having to write it...
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was proposed, which integrated multiple engines and...
by: Arjunsri | last post by:
I have a Redshift database that I need to use as an import data source. I have configured the DSN connection using the server, port, database, and credentials and received a successful connection...
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web server and have made sure to enable curl. I get a...
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand. Background colors can be used to highlight important...
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS starter kit that's not only easy to use but also...
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python has gained popularity among beginners and experts...
by: Ricardo de Mila | last post by:
Dear people, good afternoon... I have a form in msAccess with lots of controls and a specific routine must be triggered if the mouse_down event happens in any control. Than I need to discover what...
by: Johno34 | last post by:
I have this click event on my form. It speaks to a Datasheet Subform Private Sub Command260_Click() Dim r As DAO.Recordset Set r = Form_frmABCD.Form.RecordsetClone r.MoveFirst Do If...
by: ezappsrUS | last post by:
Hi, I wonder if someone knows where I am going wrong below. I have a continuous form and two labels where only one would be visible depending on the checkbox being checked or not. Below is the...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.