is parameter an iterable?

I have function which takes an argument. My code needs that argument
to be an iterable (something i can loop over)...so I dont care if its a
list, tuple, etc. So I need a way to make sure that the argument is an
iterable before using it. I know I could do...

def foo(inputVal):
if isinstance(inputVal, (list, tuple)):
for val in inputVal:
# do stuff

....however I want to cover any iterable since i just need to loop over
it.

any suggestions?

Nov 22 '05

Subscribe Post Reply

3959

Steven D'Aprano

Ah, found the right solution! Thanks to Fredrik Lundh
for so gently pointing me at this post. Just one
question, if I may:-

Alex Martelli wrote:

It's not hard...:

try:
_it = iter(whatever)
except TypeError:
print 'non-iterable'
else:
for i in _it: # etc, etc

Alas and alack, I have to write code which is backwards
compatible with older versions of Python:

Python 2.1.1 (#1, Aug 25 2001, 04:19:08)
[GCC 3.0.1] on sunos5
Type "copyright", "credits" or "license" for more
information.

iter

Traceback (most recent call last):
File "<stdin>", line 1, in ?
NameError: name 'iter' is not defined

What should I do when I can't rely on functions that
don't exist in older versions of Python?
--
Steven.

Nov 22 '05 #51

Steven D'Aprano

Ah, found the right solution! Thanks to Fredrik Lundh
for so gently pointing me at this post. Just one
question, if I may:-

Alex Martelli wrote:

It's not hard...:

try:
_it = iter(whatever)
except TypeError:
print 'non-iterable'
else:
for i in _it: # etc, etc

iter

Nov 22 '05 #52

Steven D'Aprano

Roy Smith wrote:

Now you're assuming that the message is a constant for all TypeErrors
caused by attempting to iterate over a non-iterable object.

Yes I am, and of course you are correct that that could
change and so shouldn't be relied on.

Seems that my crappy solution is not quite as
future-proof as I thought it was. Since a better
solution does exist, I agree it should be used in
preference to my crappy one, assuming you are running a
recent enough version of Python.
--
Steven.

Nov 22 '05 #53

Steven D'Aprano

Roy Smith wrote:

Now you're assuming that the message is a constant for all TypeErrors
caused by attempting to iterate over a non-iterable object.

Nov 22 '05 #54

Duncan Booth

Steven D'Aprano wrote:

What should I do when I can't rely on functions that
don't exist in older versions of Python?

Ideally:

if 'iter' not in dir(__builtins__):
import sys
sys.exit('Archaic Python not supported, please upgrade')
Alternatively fill in the blanks with appropriate fallback code:

if 'iter' not in dir(__builtins__):
def iter(obj):
... remainder of definition left as an exercise ...
__builtins__.iter = iter

where iter does roughly what the iter builtin does.

Nov 22 '05 #55

Duncan Booth

Steven D'Aprano wrote:

What should I do when I can't rely on functions that
don't exist in older versions of Python?

Nov 22 '05 #56

Roy Smith

Steven D'Aprano <st***@REMOVEMEcyber.com.au> wrote:

Alas and alack, I have to write code which is backwards
compatible with older versions of Python:
[...]
NameError: name 'iter' is not defined

What should I do when I can't rely on functions that
don't exist in older versions of Python?

It really sucks trying to support obsolete environments. Been there, done
that. But sometimes you just have to pull out the duct tape and deal with
the problem any way you can. Parsing undocumented exception strings is
sort of like screen scraping; it's good to know that it's possible as an
absolute last resort, but you really want to put some effort into finding a
better way.

You could use two nested try blocks that both catch TypeErrors, one
enclosing the entire loop, the other enclosing just the body. Anything
that gets caught by the outer try had to have been generated within the
loop control code itself:

-------------------------
Roy-Smiths-Computer:play$ cat iter.py
#!/usr/bin/env python

def doSomething (obj):
sum = 0
try:
for i in obj:
try:
sum += i
except TypeError, ex:
print "exception (%s) in the inner block" % ex
return
except TypeError, ex:
print "non iterable object (%s)" % ex
return
print "sum = %d" % sum

doSomething ([1, 2, 3])
doSomething (["one", "two", "three"])
doSomething (0)
Roy-Smiths-Computer:play$ ./iter.py
sum = 6
exception (unsupported operand type(s) for +=: 'int' and 'str') in the
inner block
non iterable object (iteration over non-sequence)
-------------------------

This is pretty ugly, but I think it does what you want. It could be made a
bit less ugly by refactoring the body of the for loop into another function.

BTW, if you do resort to trying to parse the exception messages, and it
breaks in some future code, you might have me to blame. I submitted a bug
report a while back which, if it's ever acted on, would break your code!

http://sourceforge.net/tracker/index...37&group_id=54
70&atid=105470

Nov 22 '05 #57

Roy Smith

Steven D'Aprano <st***@REMOVEMEcyber.com.au> wrote:

Alas and alack, I have to write code which is backwards
compatible with older versions of Python:
[...]
NameError: name 'iter' is not defined

What should I do when I can't rely on functions that
don't exist in older versions of Python?

Nov 22 '05 #58

Micah Elliott

On Nov 17, Duncan Booth wrote:

Steven D'Aprano wrote:
What should I do when I can't rely on functions that
don't exist in older versions of Python?
sys.exit('Archaic Python not supported, please upgrade')

+1 QOTW.

I recently gave up on trying to support (backport to) pre-2.2 in my
projects. It's been ~3 years since 2.2 released and that seem like a
pretty reasonable support window. I wound up with HACK and
PORTABILITY comments all over the place and too much boilerplate. Now
I go with a startup check on sys.hexversion, and exit with similar
wording.

--
_ _ ___
|V|icah |- lliott <>< md*@micah.elliott.name
" " """

Nov 22 '05 #59

Micah Elliott

On Nov 17, Duncan Booth wrote:

Steven D'Aprano wrote:
What should I do when I can't rely on functions that
don't exist in older versions of Python?
sys.exit('Archaic Python not supported, please upgrade')

Nov 22 '05 #60

Roy Smith

Micah Elliott <md*@micah.elliott.name> wrote:

I recently gave up on trying to support (backport to) pre-2.2 in my
projects. It's been ~3 years since 2.2 released and that seem like a
pretty reasonable support window.

It depends on what you're doing. If you're developing for in-house
use, you have control over what you run and can upgrade whenever you
feel you can justify devoting the resources to the upgrade effort. If
you're a one-man band, upgrading to a new version may be an hour's
work to download and install the latest and greatest. If you're a cog
in a large organization, it may be quite a bit more hassle.

If you're developing for outside customers, it's a whole different
story. Lots of people are running operating systems which are several
years old (heck, I'm typing this on a Windows-2000 box), and those
operating systems may have shipped with versions of langauges which
were several years old at the time they shipped.

Consider the following conversation with a prospective customer:

Prospect: "I really love your product, the price is OK, and I've got
budget approval for the purchase. Now, what do we need to run it?"

You: "All you need is a box running Python 2.2".

Prospect: "Oh, bummer, we run 2.1 [or 2.0, or 1.5.2] in-house on all
our machines".

You: "No problem, it's easy to upgrade to 2.2"

Prospect: "Unfortunately, not around here it isn't. I.T. owns the
servers, and they won't do the upgrade. Looks like we won't be able
to use your product after all. Sorry."

I'm not saying you shouldn't use new stuff, but don't fool yourself
about how long a lifetime old versions have in the field. And, the
old adage that "The customer is always right" is a good one.

Nov 22 '05 #61

Roy Smith

Micah Elliott <md*@micah.elliott.name> wrote:

I recently gave up on trying to support (backport to) pre-2.2 in my
projects. It's been ~3 years since 2.2 released and that seem like a
pretty reasonable support window.

Nov 22 '05 #62

Steve Holden

Micah Elliott wrote:

On Nov 17, Duncan Booth wrote:
Steven D'Aprano wrote:
What should I do when I can't rely on functions that
don't exist in older versions of Python?

sys.exit('Archaic Python not supported, please upgrade')

+1 QOTW.

I recently gave up on trying to support (backport to) pre-2.2 in my
projects. It's been ~3 years since 2.2 released and that seem like a
pretty reasonable support window. I wound up with HACK and
PORTABILITY comments all over the place and too much boilerplate. Now
I go with a startup check on sys.hexversion, and exit with similar
wording.

We could all take an example from Fredrik Lundh, who still regularly
releases products that run on everything back to Python 1.5.2. Kudos to
the effbot!

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Nov 22 '05 #63

Steve Holden

Micah Elliott wrote:

On Nov 17, Duncan Booth wrote:
Steven D'Aprano wrote:
What should I do when I can't rely on functions that
don't exist in older versions of Python?

sys.exit('Archaic Python not supported, please upgrade')

+1 QOTW.

I recently gave up on trying to support (backport to) pre-2.2 in my
projects. It's been ~3 years since 2.2 released and that seem like a
pretty reasonable support window. I wound up with HACK and
PORTABILITY comments all over the place and too much boilerplate. Now
I go with a startup check on sys.hexversion, and exit with similar
wording.

Nov 22 '05 #64

Steven D'Aprano

Fredrik Lundh wrote:

Steven D'Aprano wrote:

Alas and alack, I have to write code which is backwards
compatible with older versions of Python:
[snip]
What should I do when I can't rely on functions that
don't exist in older versions of Python?

python 2.1 doesn't support iterators, so that question doesn't
make much sense.

The _question_ doesn't make much sense? I could
understand you saying that backwards-compatibility is
"not important [to me]" but to say that the very
question of how to maintain backwards compatibility
makes little sense is a bit extreme, don't you think?

Fredrik, I bow to your superior knowledge about Python,
no sarcasm intended, and I've learnt a lot from your
posts, thank you. But this is not one of your shining
moments. Your attitude was utterly dismissive: the
"right way" to solve the problem of recognising
iterables was to use iter, and that's all that needs to
be said.

The problem of how to recognise iterables did not come
into existence with version 2.2, and backwards
compatibility is sometimes a real requirement. A few
months back I had to mangle some Py2.4 code so that it
would run under version 2.0, and wasn't that fun.

if you want to write code that runs under 2.1, you have to write
your code in terms of what 2.1 supports.

Do you think I don't know this?

I never imagined for an instant that Python 2.1 would
somehow magically be able to use features that didn't
exist in Python 2.1. But then it wasn't me saying that
there was nothing to discuss, the "right way" is to use
iter(), end of story.

If I have to write code that can't rely on iter()
existing in the language, what should I do?

Are there practical idioms for solving the metaproblem
"solve problem X using the latest features where
available, otherwise fall back on older, less powerful
features"?

For instance, perhaps I might do this:

try:
built_in_feature
except NameError:
# fall back on a work-around
from backwards_compatibility import \
feature as built_in_feature

Do people do this or is it a bad idea?

Are there other techniques to use? Obviously refusing
to run is a solution (for some meaning of "solution"),
it may even be a practical solution for some cases, but
is it the only one?

In the specific case of iter(), are there good
alternative ways of detecting an iterable without
consuming it?

--
Steven.

Nov 22 '05 #65

Alex Martelli

Steven D'Aprano <st***@REMOVEMEcyber.com.au> wrote:
...

In the specific case of iter(), are there good
alternative ways of detecting an iterable without
consuming it?

Not a problem I've had often, but when I did, if I recall correctly, I
did something like:

try:
iter
except NameError:
def isiterable(x):
try: x[0]
except Exception: return 0
else: return 1
else:
def isiterable(x):
try: iter(x)
except TypeError: return 0
else: return 1

Not True/False because they wouldn't be available in 2.0 any more than
iter would. "Consuming" didn't really come into consideration for the
backwards compatibility part because only objects indexable with
integers, 0 and up (and raising IndexError at some point) were usable in
for statements in old Pythons, there was no "consuming". The tests here
are not 100% reliable, of course -- in 2.0, a dict which just happens to
have a 0 key would erroneously pass; but short of actually trying the
loop, there's no infallible way that I know of.

One possibility (that I haven't tried in real life) to deal with the
hellish situation of having to write code that's able to loop on an
iterable but fail softly otherwise, while not hiding errors in the
loop's body, would be the suitably hellish...:

class NotATypeError(Exception):
def __init__(self, e, t):
self.e = e
self.t = t

try:
try:
for item in whatever:
try:
...loop body here...
except TypeError, e, t:
raise NotATypeError(e, t)
except TypeError:
....fail-softly code...
except NotATypeError, x:
raise TypeError, x.e, x.t
This kind of spaghetti code is what gives backwards compatibility its
bad name, of course. Be sure that you're getting paid for this in
proportion to its ugliness, and your finances should be set for life.
Alex

Nov 22 '05 #66

Steven D'Aprano

Alex Martelli wrote:

"Consuming" didn't really come into consideration for the
backwards compatibility part because only objects indexable with
integers, 0 and up (and raising IndexError at some point) were usable in
for statements in old Pythons, there was no "consuming".
Ah yes, of course. How quickly we forget: you couldn't
say "for line in file...". No generators either.

But there was still at least one object that was
consumable: xreadlines.

Still, I think using an xreadlines object is an unusual
enough case that I'm not going to lose any sleep about
it. Document it as a known issue and forget it *wink*

[snip]
This kind of spaghetti code is what gives backwards compatibility its
bad name, of course. Be sure that you're getting paid for this in
proportion to its ugliness, and your finances should be set for life.

Hah, I wish!
Thanks for the assistance, I learnt a lot. Let's hope I
don't have to use it ever again...
--
Steven.

Nov 22 '05 #67

Fredrik Lundh

Alex Martelli wrote:

In the specific case of iter(), are there good
alternative ways of detecting an iterable without
consuming it?
Not a problem I've had often, but when I did, if I recall correctly, I
did something like:

try:
iter
except NameError:
def isiterable(x):
try: x[0]
except Exception: return 0
else: return 1
else:
def isiterable(x):
try: iter(x)
except TypeError: return 0
else: return 1

Not True/False because they wouldn't be available in 2.0 any more than
iter would. "Consuming" didn't really come into consideration for the
backwards compatibility part because only objects indexable with
integers, 0 and up (and raising IndexError at some point) were usable in
for statements in old Pythons, there was no "consuming".

Before the iterator protocol was added, using the sequence protocol
to implement "forward-only iterators" weren't uncommon. e.g.

class my_file_iterator:
def __init__(self, file):
self.file = file
def __getitem__(self, index):
line = self.file.readline()
if not line:
raise IndexError
return line

(for extra points, add code that makes sure that the index matches
the line number)

To test for this, you could look for __getitem__ methods on instance
objects, and use operator.isSequenceType on everything else, but that
doesn't distinguish between sequences and mappings.
short of actually trying the loop, there's no infallible way that I know of.

Exactly. And iter() can fail too, even if it's far less likely that you
stumble upon an iterable that misbehaves if you call __iter__ one
extra time. Which is why the portable pragmatic pythonic solution
is to design your program so it doesn't depend on type/interface
testing.

If you want to loop over things, loop over things, and leave it to
Python to raise an exception if you get something that doesn't
support looping.

And unless you have very good reasons, you should treat that
exception as a bug in your program.

</F>

Nov 22 '05 #68

Tom Anderson

On Tue, 22 Nov 2005, Steven D'Aprano wrote:

Are there practical idioms for solving the metaproblem "solve problem X
using the latest features where available, otherwise fall back on older,
less powerful features"?

For instance, perhaps I might do this:

try:
built_in_feature
except NameError:
# fall back on a work-around
from backwards_compatibility import \
feature as built_in_feature

Do people do this or is it a bad idea?
From some code i wrote yesterday, which has to run under 2.2:

try:
True
except NameError:
True = 1 == 1
False = 1 == 0

Great minds think alike!

As for whether it's a bad idea, well, bad or not, it certainly seems like
the least worst.
Are there other techniques to use? Obviously refusing to run is a
solution (for some meaning of "solution"), it may even be a practical
solution for some cases, but is it the only one?
How about detecting which environment you're in, then running one of two
entirely different sets of code? Rather than trying to construct modern
features in the antique environment, write code for each, using the local
idioms. The trouble with this is that you end up with massive duplication;
you can try to factor out the common parts, but i suspect that the
differing parts will be a very large fraction of the codebase.
If I have to write code that can't rely on iter() existing in the
language, what should I do?

Can you implement your own iter()? I have no idea what python 2.0 was
like, but would something like this work:

class _iterator:
def __init__(self, x):
self.x = x
self.j = 0
def next(self):
self.j = self.j + 1
return self.x.next()
def __getitem__(self, i):
if (i != self.j):
raise ValueError, "out of order iteration"
try:
return self.next()
except StopIteration:
raise IndexError
def __iter__(self):
return self
# hopefully, we don't need this, but if we do ...
def __len__(self):
return sys.maxint # and rely on StopIteration to stop the loop

class _listiterator(_iterator):
def next(self):
try:
item = self.x[self.j]
self.j = self.j + 1
return item
except IndexError:
raise StopIteration
def __getitem__(self, i):
if (i != self.j):
raise ValueError, "out of order iteration"
self.j = self.j + 1
return self.x[i]

import types

def iter(x):
# if there's no hasattr, use explicit access and try-except blocks
# handle iterators and iterables from the future
if hasattr(x, "__iter__"):
return _iterator(x.__iter__())
# if there's no __getitem__ on lists, try x[0] and catch the exception
# but leave the __getitem__ test to catch objects from the future
if hasattr(x, "__getitem__"):
return _listiterator(x)
if type(x) == types.FileType:
return _fileiterator(x) # you can imagine the implementation of this
# insert more tests for specific types here as you like
raise TypeError, "iteration over non-sequence"

?

NB haven't actually tried to run that code.

tom

--
I'm angry, but not Milk and Cheese angry. -- Mike Froggatt

Nov 22 '05 #69

Fredrik Lundh

Tom Anderson wrote:

How about detecting which environment you're in, then running one of two
entirely different sets of code? Rather than trying to construct modern
features in the antique environment, write code for each, using the local
idioms. The trouble with this is that you end up with massive duplication;
you can try to factor out the common parts, but i suspect that the
differing parts will be a very large fraction of the codebase.

That sounds like the worst possible way of writing portable code (unless you
have huge amounts of time and money, and wasting them don't really matter
to you).

In my experience, a more practical approach is to write code that targets the
2.0/2.1 platform, and selectively (based on feature tests, not version numbers)
replace portions with more efficient mechanisms from newer versions. (most
of those are found in the standard library, not in the language itself).

(this is related to another popular myth: that code that uses "the old way" is
automatically slower than code that uses newfangled stuff, also when running
on recent Python versions. that's very seldom true; new Python versions runs
"old" code faster too.)

If you want to support 1.5.2 or Unicode-less Python builds, you need to add
some fallbacks too, but that's mostly trivial. The easiest way is to pass 8-bit
strings through "as is", leaving encoding issues to the application.

If I have to write code that can't rely on iter() existing in the
language, what should I do?

Can you implement your own iter()? I have no idea what python 2.0 was
like, but would something like this work:

Python 2.0 had sequences. Python 2.4 has sequences. Iterators don't really
add anything. Emulating them in older versions is mostly pointless.

I've said it many times, and I'll say it again: the only fundamentally new concept
that has been added since Python 1.5.2 is generators. And a lot of the stuff you
want generators for can be emulated with sequences and extra buffer layers.

All the rest is just coloured frosting; you can save a line here and there (which is
a real benefit only if your return key is broken), but that's about it.

</F>

Nov 23 '05 #70

Roy Smith

"Fredrik Lundh" <fr*****@pythonware.com> wrote:

I've said it many times, and I'll say it again: the only fundamentally
new concept that has been added since Python 1.5.2 is generators.
[...]
All the rest is just coloured frosting

In my mind, the biggest thing since 1.5.2 is string methods. They are
perhaps, as you say, just frosting, but they're very good tasting frosting
:-)

Nov 23 '05 #71

Similar topics