empty lists vs empty generators

[Brian Roberts]

I'm using using generators and iterators more and more intead of
passing lists around, and prefer them. However, I'm not clear on the
best way to detect an empty generator (one that will return no items)
when some sort of special case handling is required.

Typical code for handling an empty list:
if somelist:
for x in somelist:
something(x)
else:
empty_list_special_case

But this doesn't work with iterators -- a generator is "true"
regardless of whether its going to return any items. (I understand
why).

The closest equivalent I know of is:
n = 0
for n, x in enumerate(somegenerator()):
something(x)
if n == 0:
empty_list_special_case

Which seems rather awkward -- doesn't read as easily for me, and
introduces another variable.

Q1: Is there a better or alternate way to handle this?
Q2: Is there a way that handles both lists and generators, so I don't
have to worry about which one I've got?

Thanks,
Brian.

[jfj]

Brian Roberts wrote:

I'm using using generators and iterators more and more intead of
passing lists around, and prefer them. However, I'm not clear on the
best way to detect an empty generator (one that will return no items)
when some sort of special case handling is required.

Usually it will be the job of the generator to signal something like
this. I think a possible way might be:

class GeneratorEmpty: pass

def generator():
if not X:
raise GeneratorEmpty
for i in X:
yield i

try:
for x in generator
something (x)
except GeneratorEmpty:
generator_special_case

The trick is that when generators raise exceptions they terminate.
Although this is probably not what you want. The thing is that you
cannot know if a generator will return any elements until you call
its next() method.

Q2: Is there a way that handles both lists and generators, so I don't
have to worry about which one I've got?

I don't think this is possible. A generator must be called (with
next()) in order for its code to take over and see if it is empty or
not. Unlike the list.
jfj

[Roy Smith]

In article <50*************************@posting.google.com> ,
br***@mirror.org (Brian Roberts) wrote:

I'm using using generators and iterators more and more intead of
passing lists around, and prefer them. However, I'm not clear on the
best way to detect an empty generator (one that will return no items)
when some sort of special case handling is required.

The best I can come up with is to depend on the fact that

for item in foo:
pass

only defines item if foo yields any items. Assuming item is not defined
before you execute the for loop, you can check to see if it's defined after
the loop, and use that to tell if foo was an empty list or generator.
Here's a demo. Unfortunately, I'm not sure if it's really any cleaner than
your way (but at least it doesn't add any extraneous variables)
# Creates an iterator which yields n items.
class gen:
def __init__(self, n):
self.n = n

def __iter__(self):
for i in range(self.n):
yield None

def checkEmpty(genOrList):
for item in genOrList:
pass

try:
item
print "%s had items" % genOrList
except NameError:
print "%s was empty" % genOrList

checkEmpty(gen(0))
checkEmpty(gen(1))
checkEmpty([])
checkEmpty([1])

--------------

Roy-Smiths-Computer:play$ ./gen.py
<__main__.gen instance at 0x36c620> was empty
<__main__.gen instance at 0x36c620> had items
[] was empty
[1] had items

[Jeremy Bowers]

On Mon, 02 May 2005 16:14:57 -0700, Brian Roberts wrote:

Q1: Is there a better or alternate way to handle this? Q2: Is there a way
that handles both lists and generators, so I don't have to worry about
which one I've got?

Are you in control of your generators? You could put a method on them that
tells if there is anything in them by manually implementing the .next()
call.

The other thing you could do is a generator wrapper that can tell for you,
but you'll lose some performance:

class EmptyGeneratorDetector(object):
"""Provides a method you can call to detect an empty
generator. You should probably name this class something
shorter.

Check if the generator is empty after construction by looking at
the isEmpty property."""

def __init__(self, generator):
self.generator = generator

self.isEmpty = False
self.givenFirst = False
try:
self.firstItem = generator.next()
except StopIteration:
self.isEmpty = True

def next(self):
if self.isEmpty:
raise StopIteration

if not self.givenFirst:
self.givenFirst = True
return self.firstItem
else:
return self.generator.next()

def __iter__(self):
return self

In action:

Python 2.3.5 (#1, Mar 3 2005, 17:32:12)
[GCC 3.4.3 (Gentoo Linux 3.4.3, ssp-3.4.3-0, pie-8.7.6.6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from genwrap import *
def emptyGenerator(): .... raise StopIteration
.... yield None
.... def nonEmptyGenerator(): .... yield 1
.... yield 2
.... yield 3
.... e = emptyGenerator()
n = nonEmptyGenerator()
E = EmptyGeneratorDetector(e)
N = EmptyGeneratorDetector(n)
E.isEmpty True N.isEmpty False for i in E: .... print i
.... for i in N: .... print i
....
1
2
3

It is tested as much as you see it above :-)

(I recall a lengthy discussion of the best way to create an empty iterator
a while back, and that was not the winner. But it will do for now.)

[Bengt Richter]

On 2 May 2005 16:14:57 -0700, br***@mirror.org (Brian Roberts) wrote:

I'm using using generators and iterators more and more intead of
passing lists around, and prefer them. However, I'm not clear on the
best way to detect an empty generator (one that will return no items)
when some sort of special case handling is required.

Typical code for handling an empty list:
if somelist:
for x in somelist:
something(x)
else:
empty_list_special_case

But this doesn't work with iterators -- a generator is "true"
regardless of whether its going to return any items. (I understand
why).

The closest equivalent I know of is:
n = 0
for n, x in enumerate(somegenerator()):
something(x)
if n == 0:
empty_list_special_case

Which seems rather awkward -- doesn't read as easily for me, and
introduces another variable. And, if I understood the intent, doesn't work ;-)

n = 0
for n, x in enumerate(c for c in 'a'): ... print 'something', x
...
something a if n == 0: ... print 'empty list special case ??'
...
empty list special case ??

You could have used n = -1 as a sentinel that enumerate would not set,
but using a guaranteed-unique sentinel, you don't need enumerate, e.g.,
x = sentinel = object()
for x in (c for c in 'a'): ... print 'something', x
...
something a if x is sentinel: ... print 'empty list special case ??'
...

(nothing printed there)
and for the actually empty sequence
x = sentinel = object()
for x in (c for c in ''): ... print 'something', x
... if x is sentinel:

... print 'empty list special case ??'
...
empty list special case ??

Q1: Is there a better or alternate way to handle this?
Q2: Is there a way that handles both lists and generators, so I don't
have to worry about which one I've got?

UIAM this should work for any iterable. You don't have to manufacture
a locally bound sentinel as above. You could pick anything to preset
the for-target that you know is not going to be produced by the iterable,
though you might need to use '==' instead of 'is' depending on your choice.
But e.g., I don't think I'd write

x = Exception # weird sentinel choice
for x in mystring:
print x, ord(x)
if x is Exception:
print 'null sequence'

None probably works well a lot of the time, but not always.
Similarly ''. Seems like a builtin sentinel binding like sentinel = object()
might be handy to standardize usage.

Regards,
Bengt Richter

[Michele Simionato]

Starting from Python 2.4 we have tee in the itertools
module, so you can define the following:

from itertools import tee

def is_empty(it):
it_copy = tee(it)[1]
try:
it_copy.next()
except StopIteration:
return True
else:
return False

It works with generic iterables too.

Michele Simionato

[Terry Reedy]

"Brian Roberts" <br***@mirror.org> wrote in message
news:50*************************@posting.google.co m...

I'm using using generators and iterators more and more intead of
passing lists around, and prefer them. However, I'm not clear on the
best way to detect an empty generator (one that will return no items)
when some sort of special case handling is required.

If you write an iterator class instead of the abbreviated generator form,
and you can tell from the initialization parameters whether there will be
any data, then you can give the class a __nonzero__ method. You can also
have an initially nonempty iterator flag when it becomes empty.

My point is that writing an iterator as a generator is a convenience, not a
necessity, and that one gives up the full flexibility of an iterator class
when one does so, but that one is not required to do so.

I quite understanding wanting to have your cake and eat it too. The
convenience is sometimes major.

Terry J. Reedy

[Leif K-Brooks]

Jeremy Bowers wrote:

def __init__(self, generator):
self.generator = generator

You'll want to use iter(generator) there in order to handle reiterables.

[Jeremy Bowers]

On Wed, 04 May 2005 13:45:00 +0000, Leif K-Brooks wrote:

Jeremy Bowers wrote:

def __init__(self, generator):
self.generator = generator

You'll want to use iter(generator) there in order to handle reiterables.

Can you expand that explanation a bit? I'm not certain what you mean. I'm
just trusting what the user passes in; maybe the user should pass it
iter(generator) when it's a "reiterable"? (Honest question.)

What definition of "re-iterable" are you using? (A quick google for
"Python reiterabile" just turns up some Python dev list entries from 2003.)

[Leif K-Brooks]

Jeremy Bowers wrote:

On Wed, 04 May 2005 13:45:00 +0000, Leif K-Brooks wrote:

Jeremy Bowers wrote:

def __init__(self, generator):
self.generator = generator

You'll want to use iter(generator) there in order to handle reiterables.

Can you expand that explanation a bit? I'm not certain what you mean. I'm
just trusting what the user passes in; maybe the user should pass it
iter(generator) when it's a "reiterable"? (Honest question.)

What definition of "re-iterable" are you using? (A quick google for
"Python reiterabile" just turns up some Python dev list entries from 2003.)

Reiterable is generally defined as an object which can be iterated over
multiple times (i.e. is iterable but isn't an iterator). The simplest
example is a list, but a few other built-in types (set and dict, for
instance) also qualify.

With the EmptyGeneratorDetector class as you defined it, lists will fail:

EmptyGeneratorDetector([])

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 15, in __init__
AttributeError: 'list' object has no attribute 'next'

Of course, the class is labeled as an empty generator detector, not an
empty iterable detector, so it's doing what it says it will, but a
little bit of extra generalism can't hurt.

[Jeremy Bowers]

On Wed, 04 May 2005 20:33:31 +0000, Leif K-Brooks wrote:

With the EmptyGeneratorDetector class as you defined it, lists will fail:

>>> EmptyGeneratorDetector([])

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 15, in __init__
AttributeError: 'list' object has no attribute 'next'

Of course, the class is labeled as an empty generator detector, not an
empty iterable detector, so it's doing what it says it will, but a little
bit of extra generalism can't hurt.

OK, thanks, now I see what you mean. I was worried that you might be
referring to an iterator type that returned something other than itself
when you called iter on it, which I thought wasn't legal.