473,597 Members | 2,113 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

An idiom for code generation with exec

Hello,

In a Python program I'm writing I need to dynamically generate
functions[*] and store them in a dict. eval() can't work for me
because a function definition is a statement and not an expression, so
I'm using exec. At the moment I came up with the following to make it
work:

def build_func(args ):
code """def foo(...)..."""
d = {}
exec code in globals(), d
return d['foo']

My question is, considering that I really need code generation[*] -
"is there a cleaner way to do this ?" Also, what happens if I replace
globals() by None ?
Additionally, I've found indentation to be a problem in such
constructs. Is there a workable way to indent the code at the level of
build_func, and not on column 0 ?

Thanks in advance
Eli
[*] I know that each time a code generation question comes up people
suggest that there's a better way to achieve this, without using exec,
eval, etc. But in my case, for reasons too long to fully lay out, I
really need to generate non-trivial functions with a lot of hard-coded
actions for performance. And there's no problem of security
whatsoever. If someone is very interested in the application, I will
elaborate more.


Jun 27 '08 #1
31 2443
eliben a écrit :
Hello,

In a Python program I'm writing I need to dynamically generate
functions[*]
(snip)
[*] I know that each time a code generation question comes up people
suggest that there's a better way to achieve this, without using exec,
eval, etc.
Just to make things clear: you do know that you can dynamically build
functions without exec, do you ?
But in my case, for reasons too long to fully lay out, I
really need to generate non-trivial functions with a lot of hard-coded
actions for performance.
Just out of curiousity : could you tell a bit more about your use case
and what makes a simple closure not an option ?
And there's no problem of security
whatsoever. If someone is very interested in the application, I will
elaborate more.
Jun 27 '08 #2
On Jun 20, 9:17 am, Bruno Desthuilliers <bruno.
42.desthuilli.. .@websiteburo.i nvalidwrote:
eliben a écrit :Hello,
In a Python program I'm writing I need to dynamically generate
functions[*]

(snip)
[*] I know that each time a code generation question comes up people
suggest that there's a better way to achieve this, without using exec,
eval, etc.

Just to make things clear: you do know that you can dynamically build
functions without exec, do you ?
Yes, but the other options for doing so are significantly less
flexible than exec.
But in my case, for reasons too long to fully lay out, I
really need to generate non-trivial functions with a lot of hard-coded
actions for performance.

Just out of curiousity : could you tell a bit more about your use case
and what makes a simple closure not an option ?
Okay.

I work in the field of embedded programming, and one of the main uses
I have for Python (and previously Perl) is writing GUIs for
controlling embedded systems. The communication protocols are usually
ad-hoc messages (headear, footer, data, crc) built on top of serial
communication (RS232).

The packets that arrive have a known format. For example (YAMLish
syntax):

packet_length: 10
fields:
- name: header
offset: 0
length: 1
- name: time_tag
offset: 1
length: 1
transform: val * 2048
units: ms
- name: counter
offset: 2
length: 4
bytes-msb-first: true
- name: bitmask
offset: 6
length: 1
bit_from: 0
bit_to: 5
...

This is a partial capability display. Fields have defined offsets and
lengths, can be only several bits long, can have defined
transformations and units for convenient display.

I have a program that should receive such packets from the serial port
and display their contents in tabular form. I want the user to be able
to specify the format of his packets in a file similar to above.

Now, in previous versions of this code, written in Perl, I found out
that the procedure of extracting field values from packets is very
inefficient. I've rewritten it using a dynamically generated procedure
for each field, that does hard coded access to its data. For example:

def get_counter(pac ket):
data = packet[2:6]
data.reverse()
return data

This gave me a huge speedup, because each field now had its specific
function sitting in a dict that quickly extracted the field's data
from a given packet.

Now I'm rewriting this program in Python and am wondering about the
idiomatic way to use exec (in Perl, eval() replaces both eval and exec
of Python).

Eli

Jun 27 '08 #3
eliben wrote:
Additionally, I've found indentation to be a problem in such
constructs. Is there a workable way to indent the code at the level of
build_func, and not on column 0 ?
exec "if 1:" + code.rstrip()

Peter

Jun 27 '08 #4
On Jun 20, 8:03 am, eliben <eli...@gmail.c omwrote:
On Jun 20, 9:17 am, Bruno Desthuilliers <bruno.

42.desthuilli.. .@websiteburo.i nvalidwrote:
eliben a écrit :Hello,
In a Python program I'm writing I need to dynamically generate
functions[*]
(snip)
[*] I know that each time a code generation question comes up people
suggest that there's a better way to achieve this, without using exec,
eval, etc.
Just to make things clear: you do know that you can dynamically build
functions without exec, do you ?

Yes, but the other options for doing so are significantly less
flexible than exec.
But in my case, for reasons too long to fully lay out, I
really need to generate non-trivial functions with a lot of hard-coded
actions for performance.
Just out of curiousity : could you tell a bit more about your use case
and what makes a simple closure not an option ?

Okay.

I work in the field of embedded programming, and one of the main uses
I have for Python (and previously Perl) is writing GUIs for
controlling embedded systems. The communication protocols are usually
ad-hoc messages (headear, footer, data, crc) built on top of serial
communication (RS232).

The packets that arrive have a known format. For example (YAMLish
syntax):

packet_length: 10
fields:
- name: header
offset: 0
length: 1
- name: time_tag
offset: 1
length: 1
transform: val * 2048
units: ms
- name: counter
offset: 2
length: 4
bytes-msb-first: true
- name: bitmask
offset: 6
length: 1
bit_from: 0
bit_to: 5
...

This is a partial capability display. Fields have defined offsets and
lengths, can be only several bits long, can have defined
transformations and units for convenient display.

I have a program that should receive such packets from the serial port
and display their contents in tabular form. I want the user to be able
to specify the format of his packets in a file similar to above.

Now, in previous versions of this code, written in Perl, I found out
that the procedure of extracting field values from packets is very
inefficient. I've rewritten it using a dynamically generated procedure
for each field, that does hard coded access to its data. For example:

def get_counter(pac ket):
data = packet[2:6]
data.reverse()
return data

This gave me a huge speedup, because each field now had its specific
function sitting in a dict that quickly extracted the field's data
from a given packet.
It's still not clear why the generic version is so slower, unless you
extract only a few selected fields, not all of them. Can you post a
sample of how you used to write it without exec to clarify where the
inefficiency comes from ?

George
Jun 27 '08 #5
On Jun 20, 5:03*am, eliben <eli...@gmail.c omwrote:
I've rewritten it using a dynamically generated procedure
for each field, that does hard coded access to its data. For example:

def get_counter(pac ket):
* data = packet[2:6]
* data.reverse()
* return data

This gave me a huge speedup, because each field now had its specific
function sitting in a dict that quickly extracted the field's data
from a given packet.

Now I'm rewriting this program in Python and am wondering about the
idiomatic way to use exec (in Perl, eval() replaces both eval and exec
of Python).
FWIW, when I had a similar challenge for dynamic coding, I just
generated a py file and then imported it. This technique was nice
because can also work with Pyrex or Psyco.

Also, the code above can be simplified to: get_counter = lambda
packet: packet[5:1:-1]

Since function calls are expensive in python, you can also gain speed
by parsing multiple fields at a time:

header, timetag, counter = parse(packet)
Raymond
Jun 27 '08 #6
eliben a écrit :
On Jun 20, 9:17 am, Bruno Desthuilliers <bruno.
42.desthuilli.. .@websiteburo.i nvalidwrote:
>eliben a écrit :Hello,
>>In a Python program I'm writing I need to dynamically generate
functions[*]
(snip)
>>[*] I know that each time a code generation question comes up people
suggest that there's a better way to achieve this, without using exec,
eval, etc.
Just to make things clear: you do know that you can dynamically build
functions without exec, do you ?

Yes, but the other options for doing so are significantly less
flexible than exec.
Let's see...
>>But in my case, for reasons too long to fully lay out, I
really need to generate non-trivial functions with a lot of hard-coded
actions for performance.
Just out of curiousity : could you tell a bit more about your use case
and what makes a simple closure not an option ?

Okay.

I work in the field of embedded programming, and one of the main uses
I have for Python (and previously Perl) is writing GUIs for
controlling embedded systems. The communication protocols are usually
ad-hoc messages (headear, footer, data, crc) built on top of serial
communication (RS232).
ok
The packets that arrive have a known format. For example (YAMLish
syntax):

packet_length: 10
fields:
- name: header
offset: 0
length: 1
- name: time_tag
offset: 1
length: 1
transform: val * 2048
units: ms
- name: counter
offset: 2
length: 4
bytes-msb-first: true
- name: bitmask
offset: 6
length: 1
bit_from: 0
bit_to: 5
...

This is a partial capability display. Fields have defined offsets and
lengths, can be only several bits long, can have defined
transformations and units for convenient display.
ok
I have a program that should receive such packets from the serial port
and display their contents in tabular form. I want the user to be able
to specify the format of his packets in a file similar to above.
ok
Now, in previous versions of this code, written in Perl, I found out
that the procedure of extracting field values from packets is very
inefficient. I've rewritten it using a dynamically generated procedure
for each field, that does hard coded access to its data. For example:

def get_counter(pac ket):
data = packet[2:6]
data.reverse()
return data

This gave me a huge speedup, because each field now had its specific
function sitting in a dict that quickly extracted the field's data
from a given packet.
ok. So if I get it right, you build the function's code as a string
based on the YAML specification.

If so, well, I can't think of anything really better[1] - at least *if*
dynamically generated procedures are really better performance wise,
which may *or not* be the case in Python.

[1] except using compile to build a code object with the function's
body, then instanciate a function object using this code, but I'm not
sure whether it will buy you much more performance-wise. I'd personnaly
prefer this because I find it more explicit and readable, but YMMV.
Now I'm rewriting this program in Python and am wondering about the
idiomatic way to use exec (in Perl, eval() replaces both eval and exec
of Python).
Well... So far, the most pythonic way to use exec is to avoid using it -
unless it's the right tool for the job !-)
Jun 27 '08 #7
FWIW, when I had a similar challenge for dynamic coding, I just
generated a py file and then imported it. This technique was nice
because can also work with Pyrex or Psyco.
I guess this is not much different than using exec, at the conceptual
level. exec is perhaps more suitable when you really need just one
function at a time and not a whole file of related functions.
Also, the code above can be simplified to: get_counter = lambda
packet: packet[5:1:-1]
OK, but that was just a demonstration. The actual functions are
complex enough to not fit into a single expression.

Eli
Jun 27 '08 #8
[1] except using compile to build a code object with the function's
body, then instanciate a function object using this code, but I'm not
sure whether it will buy you much more performance-wise. I'd personnaly
prefer this because I find it more explicit and readable, but YMMV.
How is compiling more readable than exec - doesn't it require an extra
step ? You generate code dynamically anyway.

Eli
Jun 27 '08 #9
On Jun 20, 3:19 pm, George Sakkis <george.sak...@ gmail.comwrote:
On Jun 20, 8:03 am, eliben <eli...@gmail.c omwrote:
On Jun 20, 9:17 am, Bruno Desthuilliers <bruno.
42.desthuilli.. .@websiteburo.i nvalidwrote:
eliben a écrit :Hello,
In a Python program I'm writing I need to dynamically generate
functions[*]
(snip)
[*] I know that each time a code generation question comes up people
suggest that there's a better way to achieve this, without using exec,
eval, etc.
Just to make things clear: you do know that you can dynamically build
functions without exec, do you ?
Yes, but the other options for doing so are significantly less
flexible than exec.
But in my case, for reasons too long to fully lay out, I
really need to generate non-trivial functions with a lot of hard-coded
actions for performance.
Just out of curiousity : could you tell a bit more about your use case
and what makes a simple closure not an option ?
Okay.
I work in the field of embedded programming, and one of the main uses
I have for Python (and previously Perl) is writing GUIs for
controlling embedded systems. The communication protocols are usually
ad-hoc messages (headear, footer, data, crc) built on top of serial
communication (RS232).
The packets that arrive have a known format. For example (YAMLish
syntax):
packet_length: 10
fields:
- name: header
offset: 0
length: 1
- name: time_tag
offset: 1
length: 1
transform: val * 2048
units: ms
- name: counter
offset: 2
length: 4
bytes-msb-first: true
- name: bitmask
offset: 6
length: 1
bit_from: 0
bit_to: 5
...
This is a partial capability display. Fields have defined offsets and
lengths, can be only several bits long, can have defined
transformations and units for convenient display.
I have a program that should receive such packets from the serial port
and display their contents in tabular form. I want the user to be able
to specify the format of his packets in a file similar to above.
Now, in previous versions of this code, written in Perl, I found out
that the procedure of extracting field values from packets is very
inefficient. I've rewritten it using a dynamically generated procedure
for each field, that does hard coded access to its data. For example:
def get_counter(pac ket):
data = packet[2:6]
data.reverse()
return data
This gave me a huge speedup, because each field now had its specific
function sitting in a dict that quickly extracted the field's data
from a given packet.

It's still not clear why the generic version is so slower, unless you
extract only a few selected fields, not all of them. Can you post a
sample of how you used to write it without exec to clarify where the
inefficiency comes from ?

George
The generic version has to make a lot of decisions at runtime, based
on the format specification.
Extract the offset from the spec, extract the length. Is it msb-
first ? Then reverse. Are specific bits required ? If so, do bit
operations. Should bits be reversed ? etc.

A dynamically generated function doesn't have to make any decisions -
everything is hard coded in it, because these decisions have been done
at compile time. This can save a lot of dict accesses and conditions,
and results in a speedup.

I guess this is not much different from Lisp macros - making decisions
at compile time instead of run time and saving performance.

Eli
Jun 27 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
2030
by: Carlo v. Dango | last post by:
Hello there. I have found a need to runtime generate a method and instert it into an object instance. The code is a simple forwarding mechanism like def foo(self, *args, **kwargs): self.i.foo(*args, **kwargs) method.. however, it is only at runtime that I know the name "foo" so I cannot gerenate such a method any sooner. I have tried the compile(str) method but I haven't succeeded. I've tried using the __getattribute__ but there is a...
28
3569
by: Fábio Mendes | last post by:
I'm sorry if it's an replicate. Either my e-mail program is messing with things or the python-list sent my msg to /dev/null. I couldn't find anything related in previous PEP's, so here it goes a very early draft for a new "assert" syntax: This was inspired in Ruby's assert syntax. I'm not familiar with Ruby at all, so the chances are that this piece of code is broken, but I think the idea is very obvious. In Ruby, assert is simply a...
8
2613
by: Frohnhofer, James | last post by:
My initial problem was to initialize a bunch of dictionaries at the start of a function. I did not want to do def fn(): a = {} b = {} c = {} . . . z = {}
14
1418
by: Mac | last post by:
Is there a nice Python idiom for constructors which would expedite the following? class Foo: def __init__(self, a,b,c,d,...): self.a = a self.b = b self.c = c self.d = d ...
5
2845
by: _BNC | last post by:
I've been adapting an older unmanaged C++ legacy app to C#---with limited success. The original app made use of an older, but straightforward, C DLL that worked efficiently under the VC++ 6 model. To adapt to C#, I've wrapped the older DLL calls in an unmanaged C++ class which pretty much just parallels the original function calls and encapsulates the more ragged aspects (handles, etc). Then in turn, I wrapped the unmanaged C++ class...
8
3027
by: Paul Cochrane | last post by:
Hi all, I've got an application that I'm writing that autogenerates python code which I then execute with exec(). I know that this is not the best way to run things, and I'm not 100% sure as to what I really should do. I've had a look through Programming Python and the Python Cookbook, which have given me ideas, but nothing has gelled yet, so I thought I'd put the question to the community. But first, let me be a little more detailed...
0
7969
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8272
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8381
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8035
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8258
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
5847
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
3927
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2404
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1494
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.