By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
424,856 Members | 1,986 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 424,856 IT Pros & Developers. It's quick & easy.

itertools candidate: warehouse()

P: n/a
def warehouse(stock, factory=None):
"""warehouse(stock, factory=None) -> iavailable, iremainder.

Iterate over stock, yielding each value. Once the 'stock' sequence
is
exhausted, the factory function (or any callable, such as a class)
is
called to produce a new valid object upon each subsequent call to
next().

If factory is None, the class of the first item in the sequence is
used
as a constructor. If the factory function is not a bound method, it
does
not receive any arguments, so if your class has mandatory arguments
to
__init__, wrap the class in a function which can supply those.

A common use for warehouse is to reuse a set of existing objects,
often
because object creation and/or destruction is expensive. The
warehouse
function returns the second iterable ('iremainder') to allow
consumers to
"clean up" any items from the initial sequence which did not get
re-used.

For example, given a homogeneous iterable named 'i':

available, remainder = warehouse(i)
for thing in some_other_sequence:
thing.use(available.next())
for item in remainder:
item.close()
"""

if not hasattr(stock, 'next'):
stock = iter(stock)

def pull():
"""An inner generator from itertools.warehouse()."""
for item in stock:
yield item

if factory is None:
try:
local_factory = item.__class__
except NameError:
raise ValueError("Empty sequence and no factory
supplied.")
else:
local_factory = factory

while True:
yield local_factory()

return pull(), stock
What do you all think? I've been using a class-based variant of this in
production code (business app) for six months now, and have found it
extremely helpful. I saw a pointer to itertools today and figured a
function-based version might be nice to include in that module someday.
Robert Brewer
MIS
Amor Ministries
fu******@amor.org
Jul 18 '05 #1
Share this Question
Share on Google+
6 Replies


P: n/a
Robert Brewer wrote:

What do you all think? I've been using a class-based variant of this in
production code (business app) for six months now, and have found it
extremely helpful. I saw a pointer to itertools today and figured a
function-based version might be nice to include in that module someday.


Could you give a real-world example of its use?
--
Michael Hoffman
Jul 18 '05 #2

P: n/a
On Tue, 19 Oct 2004 00:19:43 +0100, Michael Hoffman
<m.*********************************@example.com > wrote:
Robert Brewer wrote:

What do you all think? I've been using a class-based variant of this in
production code (business app) for six months now, and have found it
extremely helpful. I saw a pointer to itertools today and figured a
function-based version might be nice to include in that module someday.


Could you give a real-world example of its use?


I think Robert is referring to a conversation where I took part
earlier today. My problem was to evaluate some alternatives for a
recursive generator. The problem with the recursive generator is that
one has to loop over the nested generators simply to yield back the
results to the original caller; for example, if you are at depth three
on the recursive generator, each yield in the inner generator will in
turn generate two more yields on as you go up the recursion stack,
before finally yielding the value to the original caller.

The warehouse may be useful in that situation; the solution is not
exactly the same, but it allows to express similar ideas and may be a
good alternative when designing generator-based code for complex
structures.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: ca********@gmail.com
mail: ca********@yahoo.com
Jul 18 '05 #3

P: n/a
Robert Brewer wrote:
def warehouse(stock, factory=None):
[snip documentation]
if not hasattr(stock, 'next'):
stock = iter(stock)
Note that unconditional use of iter() is usually harmless:
it = iter("abc")
it is iter(it) True
def pull():
"""An inner generator from itertools.warehouse()."""
for item in stock:
yield item

if factory is None:
try:
local_factory = item.__class__
except NameError:
raise ValueError("Empty sequence and no factory
supplied.")
else:
local_factory = factory

while True:
yield local_factory()

return pull(), stock
What do you all think? I've been using a class-based variant of this in
production code (business app) for six months now, and have found it
extremely helpful. I saw a pointer to itertools today and figured a
function-based version might be nice to include in that module someday.


Most of the building blocks for the warehouse() are already there, but you
didn't use them, oddly enough.
So here comes a variant written in terms of current (2.4) itertools:
from itertools import *
def peek(iterable): .... a, b = tee(iterable)
.... try:
.... return a, b.next()
.... except StopIteration:
.... raise ValueError("cannot peek into an empty iterable")
.... iterable = iter("abc")
iterable, first = peek(iterable)
factory = first.__class__
data = range(10)
for a, b in izip(data, chain(iterable, starmap(factory, repeat(())))):

.... print a, repr(b)
....
0 'a'
1 'b'
2 'c'
3 ''
4 ''
5 ''
6 ''
7 ''
8 ''
9 ''

Ok, I see I'm suffering from a functional overdose :-)
I would have suggested chain(iterable, starmap(factory, repeat(()))) for
inclusion in the collection of recipes in the documentation, but upon
checking the development docs I see that Raymond Hettinger has already been
there, done that with the starmap() part.

So now you can do

chain(iterable, repeatfunc(factory))

after putting a copy of these recipes into your site-packages. Why aren't
they already there, btw?

The only extra your warehouse() has to offer is the (last) item's class as
the default factory for the padding items. I don't think that is needed
often enough to warrant the inclusion in the itertools.

Peter

Jul 18 '05 #4

P: n/a
Peter Otten <__*******@web.de> wrote:
...
iavailable = chain(stock, starmap(random.random, repeat(())))
... No, you cannot pass a callable to repeat() and have it called. In the above
line repeat(()) yields the same empty tuple ad infinitum. The trick is that
starmap() calls random.random() with that empty tuple as the argument list,


Stylistically, I prefer
iter(random.random, None)
using the 2-args form of the built-in iter, to
itertools.starmap(random.random, itertools.repeat(()))

However, itertools IS a speed demon...:

kallisti:~/cb alex$ python -m timeit -s 'import random, itertools as it'
\ > 'list(it.islice(iter(random.random, None), 666))'
1000 loops, best of 3: 884 usec per loop

kallisti:~/cb alex$ python -m timeit -s 'import random, itertools as it'
\ > 'list(it.islice(it.starmap(random.random, it.repeat(())), 666))'
1000 loops, best of 3: 407 usec per loop
Alex
Jul 18 '05 #5

P: n/a
On Tue, Oct 19, 2004 at 09:30:31PM +0200, Alex Martelli wrote:
Peter Otten <__*******@web.de> wrote:
...
>iavailable = chain(stock, starmap(random.random, repeat(()))) ...
No, you cannot pass a callable to repeat() and have it called. In the above
line repeat(()) yields the same empty tuple ad infinitum. The trick is that
starmap() calls random.random() with that empty tuple as the argument list,
Stylistically, I prefer
iter(random.random, None)
using the 2-args form of the built-in iter, to
itertools.starmap(random.random, itertools.repeat(()))

kallisti:~/cb alex$ python -m timeit -s 'import random, itertools as it'

\ > 'list(it.islice(it.starmap(random.random, it.repeat(())), 666))'

unrelated, I also often import itertools as 'it'. It bothers me because
I do it frequently (grep says 25 of 97 modules). Moving all of itertools into
builtins seems like overkill, but could we hang them off the 'iter' builtin?
Making the above:
kallisti:~/cb alex$ python -m timeit -s 'import random'

\ > 'list(iter.islice(iter.starmap(random.random, iter.repeat(())), 666))'

My life would definitely get easier, no more assuming 'it' is imported and
then adding the import when I find out differently.

A quick dir() of iter shows no conflicts.
dir(iter)

['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__str__']
-Jack

Jul 18 '05 #6

P: n/a
[Raymond Hettinger]
FWIW, I prefer loading the itertools recipes so I can write:

repeatfunc(random.random)

IMO, that is plainer than both the iter() and starmap() versions.

[Alex Martelli] I agree, but I cannot distribute Python code containing that, since the
itertools recipes aren't part of the stdlib.
Hogwash (see below).
--- in a separate note ---
[Robert Brewer] But in general, I don't write scripts for my own limited use;
I'm writing frameworks, which shouldn't depend upon little
recipes scattered hither and yon. :/


Double hogwash ;-)

Just paste the relevant recipe in your code and be done. No need for code
scattered hither and yon. Just apply a pre-made solution ready for re-use.

It is not a sin to write little helper functions to make the rest of your code
more readable. Recipes are perfect for this use because they have been tested
and refined for generic re-use.

Since when did a design pattern or code technique have to be in the standard
library to be useful?
Raymond
Jul 18 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.