473,381 Members | 1,555 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,381 software developers and data experts.

Coroutines and argument tupling

Hi,

I'm trying to write a decorator which allows one to produce simple
coroutines by just writing a function as a generator expression which
re-receives it's arguments as a tuple from each yield. For example:

@coroutine
def nextn(n=1):
values = []
for i in itertools.count():
while n <= 0:
(n,) = (yield values)
values = []
values.append(i)
n = n - 1

print nextn() # =[0]
print nextn(3) # =[1, 2, 3]
print nextn(n=2) # =[4, 5]
print nextn() # =[6]

I've got this working, but have two questions. First, is there a better
way to generically transform function arguments into a tuple than I'm
doing now? That way being with this pretty hideous chunk of code:

class ArgPacker(object):
def __init__(self, function):
args, varargs, varkw, defaults = inspect.getargspec(function)
self.args = args or []
self.varargs = (varargs is not None) and 1 or 0
self.varkw = (varkw is not None) and 1 or 0
self.nargs = len(self.args) + self.varargs + self.varkw
defaults = defaults or []
defargs = self.args[len(self.args) - len(defaults):]
self.defaults = dict([(k, v) for k, v in izip(defargs, defaults)])

def pack(self, *args, **kwargs):
args = list(args)
result = [None] * self.nargs
for i, arg in izip(xrange(len(self.args)), self.args):
if args:
result[i] = args.pop(0)
elif arg in kwargs:
result[i] = kwargs[arg]
del kwargs[arg]
elif arg in self.defaults:
result[i] = self.defaults[arg]
else:
return None
if self.varargs:
result[len(self.args)] = args
elif args:
return None
if self.varkw:
result[-1] = kwargs
elif kwargs:
return None
return tuple(result)

I also tried a version using exec, which was much tighter, but used
exec.

Second, am I trying to hammer a nail with a glass bottle here? The
ugliness of the ArgPacker class makes me suspect that I should perhaps
just manually create and track a generator when I need a function with
generator-like properties.

Thanks!

-Marshall

Aug 15 '07 #1
6 1599
Marshall T. Vandegrift wrote:
I'm trying to write a decorator which allows one to produce simple
coroutines by just writing a function as a generator expression
which re-receives it's arguments as a tuple from each yield.
May I ask why? Passing it the same arguments over and over is no
use; and there is the send method.
The ugliness of the ArgPacker class makes me suspect that I should
perhaps just manually create and track a generator when I need a
function with generator-like properties.
What do you mean? I don't quite understand why you'd have to "track"
a generator for getting generator-like properties.

Regards,
Björn

--
BOFH excuse #103:

operators on strike due to broken coffee machine

Aug 15 '07 #2
Bjoern Schliessmann <us**************************@spamgourmet.comwrite s:
>I'm trying to write a decorator which allows one to produce simple
coroutines by just writing a function as a generator expression
which re-receives it's arguments as a tuple from each yield.

May I ask why? Passing it the same arguments over and over is no
use; and there is the send method.
That's what I meant. The wrapper produced by the decorator passes the
arguments back into the generator as a tuple via the `send' method.
>The ugliness of the ArgPacker class makes me suspect that I should
perhaps just manually create and track a generator when I need a
function with generator-like properties.

What do you mean? I don't quite understand why you'd have to "track"
a generator for getting generator-like properties.
Using the trivial `nextn' example from my original post with my
decorator lets you do just:

print nextn(2) # =[0, 1]
print nextn(3) # =[2, 3, 4]
print nextn() # =[5]

Without the decorator that becomes:

gen = nextn(2)
print gen.next() # =[0, 1]
print gen.send(3) # =[2, 3, 4]
print gen.send(1) # =[5]

The former is just that smidgen nicer, and allows you to continue to
make use of argument defaults and varadic arguments if so desired.

-Marshall
Aug 15 '07 #3
On Aug 15, 3:37 pm, "Marshall T. Vandegrift" <llas...@gmail.com>
wrote:
Bjoern Schliessmann <usenet-mail-0306.20.chr0n...@spamgourmet.comwrites:
I'm trying to write a decorator which allows one to produce simple
coroutines by just writing a function as a generator expression
which re-receives it's arguments as a tuple from each yield.
May I ask why? Passing it the same arguments over and over is no
use; and there is the send method.

That's what I meant. The wrapper produced by the decorator passes the
arguments back into the generator as a tuple via the `send' method.
The ugliness of the ArgPacker class makes me suspect that I should
perhaps just manually create and track a generator when I need a
function with generator-like properties.
What do you mean? I don't quite understand why you'd have to "track"
a generator for getting generator-like properties.

Using the trivial `nextn' example from my original post with my
decorator lets you do just:

print nextn(2) # =[0, 1]
print nextn(3) # =[2, 3, 4]
print nextn() # =[5]

Without the decorator that becomes:

gen = nextn(2)
print gen.next() # =[0, 1]
print gen.send(3) # =[2, 3, 4]
print gen.send(1) # =[5]

The former is just that smidgen nicer, and allows you to continue to
make use of argument defaults and varadic arguments if so desired.

Do you really need a generator or co-routine to do this? Maybe
you can just use a closure:

import itertools
class Foo(object):
it = itertools.count(0)

def __call__(self):
return self.y

def __init__(self, n=1):
self.y = [ Foo.it.next() for each in xrange(n) ]

def nextn(n=1):
return Foo(n)()

print nextn()
print nextn(3)
print nextn(n=2)
print nextn()

--
Hope this helps,
Steven

Aug 16 '07 #4
Marshall T. Vandegrift wrote:
Without the decorator that becomes:

gen = nextn(2)
print gen.next() # =[0, 1]
print gen.send(3) # =[2, 3, 4]
print gen.send(1) # =[5]

The former is just that smidgen nicer, and allows you to continue
to make use of argument defaults and varadic arguments if so
desired.
The solution I'd use is a decorator that calls next automatically
one time after instantiation. Then you can use send normally, and
don't have to care about any initial parameters, which makes the
code clearer (initial parameters should be used for setup purposes,
but not for the first iteration, IMHO). It'd look like this (from
PEP 342, http://www.python.org/dev/peps/pep-0342/):

def consumer(func):
def wrapper(*args,**kw):
gen = func(*args, **kw)
gen.next()
return gen
wrapper.__name__ = func.__name__
wrapper.__dict__ = func.__dict__
wrapper.__doc__ = func.__doc__
return wrapper

@consumer
def nextn():
...

gen = nextn()
print gen.send(2) # =[0, 1]
print gen.send(3) # =[2, 3, 4]
print gen.send(1) # =[5]

Regards,
Björn

--
BOFH excuse #60:

system has been recalled

Aug 16 '07 #5
Bjoern Schliessmann <us**************************@spamgourmet.comwrite s:
The solution I'd use is a decorator that calls next automatically one
time after instantiation. Then you can use send normally, and don't
have to care about any initial parameters, which makes the code
clearer (initial parameters should be used for setup purposes, but not
for the first iteration, IMHO). It'd look like this (from PEP 342,
http://www.python.org/dev/peps/pep-0342/):
I'd seen the consumer decorator, and it certainly is cleaner than just
using a generator. I don't like how it hides the parameter signature in
the middle of the consumer function though, and it also doesn't provide
for argument default values. It's the difference between:

...
def __init__(self, ...):
...
self.consumer = self._function(value)
...

def function(self, first, second=3, *args, **kwargs):
self.consumer.send((first, second, args, kwargs))

@consumer
def _function(self, setup):
...
first, second, args, kwargs = yield # initial 'next'
while condition:
...
first, second, args, kwargs = yield retval

Versus just:

@coroutine
def function(self, first, second=3, *args, **kwargs):
...
while condition:
...
first, second, args, kwargs = yield retval

Thanks in any case for the replies! Since I've apparently decided my
ArgPacker is worth it, the complete code for my coroutine decorator
follows.

-Marshall
import inspect
import types
import functools
from itertools import izip

__all__ = [ 'coroutine' ]

class ArgPacker(object):
def __init__(self, function):
args, varargs, varkw, defaults = inspect.getargspec(function)
self.args = args or []
self.varargs = (varargs is not None) and 1 or 0
self.varkw = (varkw is not None) and 1 or 0
self.nargs = len(self.args) + self.varargs + self.varkw
defaults = defaults or []
defargs = self.args[len(self.args) - len(defaults):]
self.defaults = dict([(k, v) for k, v in izip(defargs, defaults)])

def pack(self, *args, **kwargs):
args = list(args)
result = [None] * self.nargs
for i, arg in izip(xrange(len(self.args)), self.args):
if args:
result[i] = args.pop(0)
elif arg in kwargs:
result[i] = kwargs[arg]
del kwargs[arg]
elif arg in self.defaults:
result[i] = self.defaults[arg]
else:
return None
if self.varargs:
result[len(self.args)] = args
elif args:
return None
if self.varkw:
result[-1] = kwargs
elif kwargs:
return None
return tuple(result)

class coroutine(object):
"""Convert a function to be a simple coroutine.

A simple coroutine is a generator bound to act as much as possible like a
normal function. Callers call the function as usual while the coroutine
produces new return values and receives new arguments with `yield'.
"""
def __init__(self, function):
self.function = function
self.gname = ''.join(['__', function.__name__, '_generator'])
self.packer = ArgPacker(function)
coroutine = self
def method(self, *args, **kwargs):
return coroutine.generate(self, self, *args, **kwargs)
self.method = method
functools.update_wrapper(self, function)
functools.update_wrapper(method, function)

def __get__(self, obj, objtype=None):
return types.MethodType(self.method, obj, objtype)

def __call__(self, *args, **kwargs):
return self.generate(self, *args, **kwargs)

def generate(self, obj, *args, **kwargs):
try:
generator = getattr(obj, self.gname)
except AttributeError:
generator = self.function(*args, **kwargs)
setattr(obj, self.gname, generator)
retval = generator.next()
else:
packed = self.packer.pack(*args, **kwargs)
if packed is None:
self.function(*args, **kwargs) # Should raise TypeError
raise RuntimeError("ArgPacker reported spurious error")
retval = generator.send(packed)
return retval

Aug 16 '07 #6
Marshall T. Vandegrift wrote:
I'd seen the consumer decorator, and it certainly is cleaner than
just using a generator. I don't like how it hides the parameter
signature in the middle of the consumer function though, and it
also doesn't provide for argument default values.
Mh, that may be. :( I didn't use it much yet.
Thanks in any case for the replies! Since I've apparently decided
my ArgPacker is worth it, the complete code for my coroutine
decorator follows.
Thanks for reporting back, and happy coding.

Regards,
Björn

--
BOFH excuse #198:

Post-it Note Sludge leaked into the monitor.

Aug 16 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Timothy Fitz | last post by:
It seems to me that in python, generators are not truly coroutines. I do not understand why. What I see is that generators are used almost exclusively for generation of lists just-in-time. Side...
5
by: Booted Cat | last post by:
I've seen lots of discussions on the proposed inclusion of "function call with named arguments" to C/C++ on these newsgroups. My proposal is slightly different in that: * No ANSI approval is...
2
by: Ken | last post by:
Does anyone know if it's possible to use Thread.GetCompressedStack and Thread.SetCompressedStack as the basis for a coroutine library? There doesn't seem to be a lot of documentation on these...
11
by: Deiter | last post by:
State Machines and Coroutines The other thread on goto: Lead me to want to ask... In the spirit of state machines and coroutines, This n00b to C would like to know if setjmp/longjmp are the only...
8
by: A. Anderson | last post by:
Howdy everyone, I'm experiencing a problem with a program that I'm developing. Take a look at this stack report from GDB - #0 0xb7d782a3 in strlen () from /lib/tls/i686/cmov/libc.so.6 #1 ...
4
by: robert | last post by:
On a server the binary (red hat) installed python2.4 and also a fresh compiled python2.5 spits "sem_post: Invalid argument". What is this and how can this solved? Robert ============== ...
3
by: rocco.rossi | last post by:
I would really like to know more about python 2.5's new generator characteristics that make them more powerful and analogous to coroutines. Is it possible for instance to employ them in situations...
0
by: Jean-Paul Calderone | last post by:
On Wed, 23 Apr 2008 07:17:46 -0700 (PDT), rocco.rossi@gmail.com wrote: They're not coroutines. The difference between generators in Python 2.4 and in Python 2.5 is that in Python 2.5, `yield´...
4
by: ig | last post by:
First off, I'm a python n00b, so feel free to comment on anything if I'm doing it "the wrong way." I'm building a discrete event simulation tool. I wanted to use coroutines. However, I want to know...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.