473,320 Members | 1,861 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

struct.Struct random access

I'd like to seriously nominate this idea and get a considered opinion
on it.

struct.Struct lets you encode Python objects into structured memory.
It accepts a format string, and optionally a buffer and offset to/from
which to read/write the structure. What do you think of random access
to the results? To avoid Marc's concern about meaningless anonymous
record entries, model the constructor after 'namedtuple' type.

(unproduced)
>>packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
packer.unpack_from( buf, off, 'i3' )
30
>>packer.unpack_from( buf, off )
( 10, 20, 30, 0.5, 'abc' )
>>packer.pack_into( buf, off, 'i1', 12 )
Even in sequential access speed benefits, by avoiding the construction
of n-1 objects.

PS: If you don't have time or don't want to consider it seriously, or
want to see more specifics, just say so.
Aug 26 '08 #1
4 1710
castironpi <ca********@gmail.comwrote:
>
I'd like to seriously nominate this idea and get a considered opinion
on it.

struct.Struct lets you encode Python objects into structured memory.
It accepts a format string, and optionally a buffer and offset to/from
which to read/write the structure. What do you think of random access
to the results? To avoid Marc's concern about meaningless anonymous
record entries, model the constructor after 'namedtuple' type.

(unproduced)
>>>packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
packer.unpack_from( buf, off, 'i3' )
30
>>>packer.unpack_from( buf, off )
( 10, 20, 30, 0.5, 'abc' )
>>>packer.pack_into( buf, off, 'i1', 12 )
What data type would you expect "buf" to be? Your design requires it to be
both an input and an output parameter. Today's "struct" converts into and
out of a string, and strings are immutable. So, to continue with that
standard, you would have to pass a string into "pack_into" and have the
modified result returned as an output:

buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
buf = packer.pack_into( buf, 0, 'i1', 12 )

In the end, I'm not sure you are really saving anything.
>Even in sequential access speed benefits, by avoiding the construction
of n-1 objects.
This is a fairly major change, because it requires a "struct.Struct" object
to maintain state, something that the "struct" module currently does not
do.

In my personal opinion, this is a rather large and confusing change, for
very little benefit.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Aug 28 '08 #2
On Aug 28, 1:59*am, Tim Roberts <t...@probo.comwrote:
castironpi <castiro...@gmail.comwrote:
I'd like to seriously nominate this idea and get a considered opinion
on it.
struct.Struct lets you encode Python objects into structured memory.
It accepts a format string, and optionally a buffer and offset to/from
which to read/write the structure. *What do you think of random access
to the results? *To avoid Marc's concern about meaningless anonymous
record entries, model the constructor after 'namedtuple' type.
(unproduced)
>>packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
packer.unpack_from( buf, off, 'i3' )
30
>>packer.unpack_from( buf, off )
( 10, 20, 30, 0.5, 'abc' )
>>packer.pack_into( buf, off, 'i1', 12 )

What data type would you expect "buf" to be? *Your design requires it to be
both an input and an output parameter. *Today's "struct" converts into and
out of a string, and strings are immutable. *So, to continue with that
standard, you would have to pass a string into "pack_into" and have the
modified result returned as an output:

* * buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
* * buf = packer.pack_into( buf, 0, 'i1', 12 )

In the end, I'm not sure you are really saving anything.
Even in sequential access speed benefits, by avoiding the construction
of n-1 objects.

This is a fairly major change, because it requires a "struct.Struct" object
to maintain state, something that the "struct" module currently does not
do.

In my personal opinion, this is a rather large and confusing change, for
very little benefit.
--
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
CMIIW correct me if I'm wrong, I don't think that pack_into returns a
value the way that pack does.

The syntax is ambiguous in the case of two-element structs: are you
packing a new struct or assigning a field? Maybe the field name needs
a keyword. A field name can be a third argument in unpack_from
regardless, however. Or use the field name as keyword:
strt.pack_into( buf, off, i1= 32 ).

I just tried an experiment with an Mmap and PyObject_AsWriteBuffer.
Mmaps could work but you would not be able to use arbitrary strings.
You would have to specially allocate a ctypes buffer with a size to
match the packed size of the structure.
Aug 28 '08 #3
castironpi <ca********@gmail.comwrote:
>
CMIIW correct me if I'm wrong, I don't think that pack_into returns a
value the way that pack does.
Sorry, I was not aware that struct.pack_into and struct.unpack_from already
existed (they were added in Python 2.5). I thought you were proposing them
as new additions.

Because of that, the rest of my post doesn't make much sense.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Aug 30 '08 #4
On Aug 29, 10:30*pm, Tim Roberts <t...@probo.comwrote:
castironpi <castiro...@gmail.comwrote:
CMIIW correct me if I'm wrong, I don't think that pack_into returns a
value the way that pack does.

Sorry, I was not aware that struct.pack_into and struct.unpack_from already
existed (they were added in Python 2.5). *I thought you were proposing them
as new additions.

Because of that, the rest of my post doesn't make much sense.
--
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
I was a lot less enthusiastic too when I discovered
'PyObject_AsWriteBuffer'. Nevertheless I think it could be
instructive as well as a benefit. The ctypes conversion,
cast( buffer_p + offset ), is a bit of a kludge. With the
introduction of 'namedtuple' type, I think it falls right in line with
the direction Python is going in.

It might suffice to merely expose the 'offset' member of the items in
the array of 'formatcode'. Though, the addition of the dictionary to
lookup offsets by field name could offset the benefit somewhat.
Aug 30 '08 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Thomas | last post by:
What's the quickest way to write and read 10.000 integer values ( or more ) to and from a file? Using struct somehow? The example in the docs shows how to handle to or three arguments, but is the...
4
by: James Harris | last post by:
Having updated my Debian system it now complains that I am using an incompatible pointer type in warnings such as "passing arg 2 of 'bind' from incompatible pointer type" from code, struct...
23
by: myth.drannon | last post by:
lets say I have a header file : struct AAAA { blabla..... }; typedef struct AAAA A; typedef struct BBB
5
by: joseph.paterson | last post by:
Hello, I'm having some trouble with accessing members of a structure i've defined... I've got something like this: #define bool short int #define TRUE 1 #define FALSE 0 #define...
3
by: Peter Olcott | last post by:
What is the correct syntax for the last line??? #include <vector> #include <stdio.h> struct TestType { double ONE; double TWO; };
2
by: kl | last post by:
Hi, I'm trying to learn some STL using map or hash_map would maybe even better to use but I don't really know how to find a specific struct depending on a DWORD value that is in range of two...
0
by: Cameron Simpson | last post by:
On 25Jun2008 22:38, Steven Clark <steven.p.clark@gmail.comwrote: | On Wed, Jun 25, 2008 at 7:03 PM, John Machin <sjmachin@lexicon.netwrote: | On Jun 26, 9:00 am, "Steven Clark"...
6
by: castironpi | last post by:
struct.Struct lets you encode Python objects into structured memory. It accepts a format string, and optionally a buffer and offset to/from which to read/write the structure. What do you think of...
4
by: Tzury Bar Yochay | last post by:
Hi, I can't find in the documentation the way to use these two functions. can someone share a simple code that utilize these two functions?
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.