473,695 Members | 3,057 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

docs on for-loop with no __iter__?

Can someone point me to the documentation on what's supposed to happen
when you use the "for x in X:" syntax when X does not have an __iter__
method? I know that the code:
class S: .... def __len__(self): return 42
.... def __getitem__(sel f, i): return i
.... for x in S():

.... print x

tries to print all the non-negative integers, starting with 0. (Note
that the __len__ method doesn't stop it at 42.) Obviously, the right
way to do this is with __iter__, but presumably this behavior is
documented somewhere...

Steve
--
You can wordify anything if you just verb it.
- Bucky Katt, Get Fuzzy
Jul 18 '05 #1
21 2062
Steven Bethard wrote:
Can someone point me to the documentation on what's supposed to happen
when you use the "for x in X:" syntax when X does not have an __iter__
method?
You need to raise an IndexError
.... def __len__(self): return 42
.... def __getitem__(sel f, i): return i
Make that say

def __getitem__(sel f, i):
if i >= 42: raise IndexError, i
return i

Obviously, the right
way to do this is with __iter__, but presumably this behavior is
documented somewhere...


http://docs.python.org/ref/sequence-types.html

] Note: for loops expect that an IndexError will be
] raised for illegal indexes to allow proper detection
] of the end of the sequence.

That's all I can find in the docs (searched docs.python.org
for __getitem__ and IndexError )

Looking at the language reference from CVS, I found
http://www.python.org/dev/doc/devel/ref/for.html

It states

] The suite is then executed once for each item in
] the sequence, in the order of ascending indices.

That implies the sequence is indexed, yes? But if
the sequence implements __iter__ then there's no
possibly no underlying idea of 'index'.

Should this be fixed?

Andrew
da***@dalkescie ntific.com
Jul 18 '05 #2
Andrew Dalke <adalke <at> mindspring.com> writes:
http://docs.python.org/ref/sequence-types.html

] Note: for loops expect that an IndexError will be
] raised for illegal indexes to allow proper detection
] of the end of the sequence.


Thanks, that's what I was looking for. I knew it had to be around there
somewhere. =) Presumably there was a reason not to use len() to determine
the end of the sequence?

Steve
Jul 18 '05 #3
"Andrew Dalke" <ad****@mindspr ing.com> wrote in message
news:bM******** *********@newsr ead1.news.pas.e arthlink.net...
<snip>

Looking at the language reference from CVS, I found
http://www.python.org/dev/doc/devel/ref/for.html

It states

] The suite is then executed once for each item in
] the sequence, in the order of ascending indices.

That implies the sequence is indexed, yes? But if
the sequence implements __iter__ then there's no
possibly no underlying idea of 'index'.

Should this be fixed?

Andrew
da***@dalkescie ntific.com

Section 7.3 (from the link given above) gives the syntax for "for" as:

for_stmt ::= "for" target_list "in" expression_list ":" suite
["else" ":" suite]

and then begins describing the component syntax elements as, "The expression
list is evaluated once; it should yield a sequence." This seems to be a bit
dated, since expression_list could also be a generator or iterator.

Additionally, "for" uses an adaptive method to try to simulate an iterator
if no __iter__ method is provided, by successively calling __getitem__ until
IndexError is raised (this error gets silently absorbed within this
pseudo-iterator).

Here is a simple test class: (I also implemented __len__ thinking that it
would be used to limit the calls to __getitem__, but we can see from the
output that it is never called - instead __getitem__ gets called one time
too many, telling the pseudo-iterator that there are no more entries).

class A(object):
def __init__(self,l st):
self.list_ = lst[:]

def __len__(self):
print self.__class__. __name__+".__le n__"
return len(self.list_)

def __getitem__(sel f,i):
print self.__class__. __name__+".__ge titem__"
return self.list_[i]

class A_with_iter(A):
def __iter__(self):
print self.__class__. __name__+".__it er__"
return iter(self.list_ )
for cls in (A, A_with_iter):

a = cls([1,2,3])

print "iterate over %s" % cls.__name__
for i in a:
print i

print

output:
iterate over A
A.__getitem__
1
A.__getitem__
2
A.__getitem__
3
A.__getitem__

iterate over A_with_iter
A_with_iter.__i ter__
1
2
3
Note that this is the basis for the recently reported bugs in email.Message
and cgi.FieldStorag e. email.Message does not implement __iter__, and its
__getitem__ method assumes that items will be retrieved like keyed
dictionary lookups, not using integer sequence access. So when __getitem__
calls .lower() on the input string, Python throws an error - you can't do
lower() on an int.
em = email.message_f rom_file(open(' MAILMESSAGE'))
for i in em:

... print i
...
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/email/Message.py",
line 304, in __getitem__
File "/usr/lib/python2.3/email/Message.py",
line 370, in get
AttributeError: 'int' object has no attribute 'lower'


-- Paul
Jul 18 '05 #4
Steven Bethard wrote:
Presumably there was a reason not to use len() to determine
the end of the sequence?


Because that allows iteration over things where
you don't yet know the size. For example, reading
lines from a file. Here's how it could have been
done using Python 1.x

class FileReader:
def __init__(self, infile):
self.infile = infile
self._n = 0

def next(self):
line = self.infile.rea dline()
if not line:
return None
self._n = self._n + 1
return line

def __getitem__(sel f, i):
assert i == self._n, "forward iteration only"
x = self.next()
if x is None:
raise IndexError, i
return x

I used the _n to double check that people don't
think it's a random access container.

Support for Python 2.x iterators was easy,

def __iter__(self):
return iter(self.next, None)

Andrew
da***@dalkescie ntific.com
Jul 18 '05 #5
Andrew Dalke <adalke <at> mindspring.com> writes:
Steven Bethard wrote:
Presumably there was a reason not to use len() to determine
the end of the sequence?


Because that allows iteration over things where
you don't yet know the size.


I'm trying to imagine a situation where it makes sense to want the x[i]
behavior from __getitem__ but where you don't know the final index. x[i]
suggests random access, so if you don't really have random access, shouldn't
you be defining __iter__ instead? Are there some good examples of classes
that allow the x[i] indexing but don't support random access (e.g. you can do
only do x[2] after you do x[1])?

(Obviously your FileReader class was an example of this, but I've assumed this
was just an example of how things *could* have been written, not how they
actually were. Please correct me if I'm wrong!)

Steve

Jul 18 '05 #6
In article <ma************ *************** ***********@pyt hon.org>,
Steven Bethard <st************ @gmail.com> wrote:
Andrew Dalke <adalke <at> mindspring.com> writes:
Steven Bethard wrote:
Presumably there was a reason not to use len() to determine
the end of the sequence?


Because that allows iteration over things where
you don't yet know the size.


I'm trying to imagine a situation where it makes sense to want the x[i]
behavior from __getitem__ but where you don't know the final index. x[i]
suggests random access, so if you don't really have random access, shouldn't
you be defining __iter__ instead? Are there some good examples of classes
that allow the x[i] indexing but don't support random access (e.g. you can do
only do x[2] after you do x[1])?


You're looking at it from a wrong totally wrong direction. The iterator
protocol did not exist prior to Python 2.2, and "__getitem_ _ iteration"
was the only way to implement custom iteration behavior. Its
limitiations were (I think) a major incentive to create something
better. In fact, the "new" iteration protocol is so brilliant, obvious
and simple, that it's hard to see how we ever did without it, or even
why noone invented it earlier...

In other words: you're looking at a legacy protocol.

Just
Jul 18 '05 #7
Steven Bethard wrote:
I'm trying to imagine a situation where it makes sense to want the x[i]
behavior from __getitem__ but where you don't know the final index. x[i]
suggests random access, so if you don't really have random access, shouldn't
you be defining __iter__ instead?

The example I gave was for Python 1.x, which didn't support __iter__.
What I showed was how to support basic forward iteration (for use
in a for loop) in the old-days.

I used the assert check precisely so that the suggestion of
random access breaks when someone tries to use it that way.
Are there some good examples of classes
that allow the x[i] indexing but don't support random access (e.g. you can do
only do x[2] after you do x[1])?

(Obviously your FileReader class was an example of this, but I've assumed this
was just an example of how things *could* have been written, not how they
actually were. Please correct me if I'm wrong!)


You say "x[i] indexing" but I don't think that's the right
way to frame the question. What I wanted was forward iteration
in Python 1.x. It happened that forward iteration was
implemented only on top of indexing, so I had to hijack the
indexing mechanism to get what I wanted. But I never thought
of it as "x[i] indexing" only "the hack needed to get forward
iteration working correctly."

I wrote several classes using that technique. One was a
way to read records from a file, another to get records from
from a database. They worked, in that I could do things like

for record in FastaReader(ope n(filename)):
print record.descript ion

The new-style __iter__ is in all ways better.

Andrew
da***@dalkescie ntific.com
Jul 18 '05 #8
Andrew Dalke <adalke <at> mindspring.com> writes:
What I wanted was forward iteration
in Python 1.x. It happened that forward iteration was
implemented only on top of indexing, so I had to hijack the
indexing mechanism to get what I wanted. But I never thought
of it as "x[i] indexing" only "the hack needed to get forward
iteration working correctly."


Good, that reaffirms my intuition that you didn't really want the __getitem__
behavior (eg x[i] access) -- it was just the only way to get the __iter__
behavior too.

Would it break old code if the __getitem__ iterator checked for a __len__
method and tried to use it if it was there? It just seems like if you already
know you're creating a sequence type and you have a __len__ and a __getitem__,
then you've already provided all the necessary information for iteration. Why
should you have to define an __iter__ or throw IndexErrors in your __getitem__?

Steve

Jul 18 '05 #9
Steven Bethard <st************ @gmail.com> wrote:
Andrew Dalke <adalke <at> mindspring.com> writes:
What I wanted was forward iteration
in Python 1.x. It happened that forward iteration was
implemented only on top of indexing, so I had to hijack the
indexing mechanism to get what I wanted. But I never thought
of it as "x[i] indexing" only "the hack needed to get forward
iteration working correctly."
Good, that reaffirms my intuition that you didn't really want the __getitem__
behavior (eg x[i] access) -- it was just the only way to get the __iter__
behavior too.


Yes, it used to be.

Would it break old code if the __getitem__ iterator checked for a __len__
method and tried to use it if it was there? It just seems like if you already
Yes, it would break old code wherever the old code had a __len__ method
returning a value not congruent with the index value for which
__getitem__ raises IndexError. That's possibly weird old code, but why
should it get broken?

Consider __len__ used to be a popular way to let your instances be
usable in a boolean context -- I believe __nonzero__ was introduced
later. So, take a class which only know whether it's empty or not, it
could have a __len__ that only returns 0 (==empty) or 1(==nonempty),
and still allow proper iteration by only raising in __getitem__ when all
items have been iterated on. If loops took account of __len__ suddenly
all that old code would break. Maybe there's much and maybe there's
little, but why break ANY of it?!
know you're creating a sequence type and you have a __len__ and a __getitem__,
then you've already provided all the necessary information for iteration. Why
should you have to define an __iter__ or throw IndexErrors in your

__getitem__?

Because you used to be allowed to return from __len__ a value not
congruent with the index value for which __getitem__ raises IndexError,
and changing that might break old code. As to whether that legacy
protocol was optimal, I think not -- today's __iter__ is clearly better,
simpler and faster. If you have many classes which define __getitem__
and __len__ and want to iterate on them all, make a mixin class:

class MixinLenwiseIte rator:
def __iter__(self):
return LenwiseIterator (self)

class LenwiseIterator (object):
def __iter__(self):
return self
def __init__(self, seq):
self.seq = seq
self.i = 0
def next(self):
if self.i >= len(self.seq):
raise StopIteration
result = self.seq[self.i]
self.i += 1
return result

Just add MixinLenwiseIte rator to your sequence classes' bases and be
happy.
Alex
Jul 18 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
1781
by: 帅猛 | last post by:
-----原始邮件----- 发件人: 帅猛 发送时间: 2004年3月3日 10:41 收件人: 'pgsql-hackers@postgresql.org' 主题: docs on tsearch2 hi all Are there any other docs about tsearch2 except for the 3 docs in \contrib\tsearch2\docs i want some tech docs on tsearch2 thank all---------------------------(end of broadcast)---------------------------TIP 2: you can get off all lists at once with the unregister command (send "unregister...
5
329
by: Alan Isaac | last post by:
This is an attempt to synthesize Bill and Carsten's proposals. (I'm changing the subject line to better match the topic.) http://docs.python.org/lib/typesmapping.html: for footnote (3) Keys and values are listed in an arbitrary order. This order is indeterminate and generally depends on factors outside the scope of the containing program. However, if items(), keys(), values(), iteritems(), iterkeys(), and itervalues() are called...
12
1864
by: Alan Isaac | last post by:
This discussion ended abruptly, and I'd like to see it reach a conclusion. I will attempt to synthesize Bill and Carsten's proposals. There are two proposed patches. The first is to http://docs.python.org/lib/typesmapping.html where it is proposed for footnote (3) to state: Keys and values are listed in an arbitrary order. This order is indeterminate and generally depends on factors outside the scope
12
1711
by: 7stud | last post by:
Hi, 1) The shelve module doesn't list close() as a method. 2) The fnmatch module does not even mention translate(). I have a hard time believing I am the first one to notice those omissions. Are the docs just old and poorly maintained? Or, is there some reason those methods were omitted?
2
1229
by: Lawrence D'Oliveiro | last post by:
Been getting 403 errors all afternoon. At one time I used to assiduously download PDF files of all the documentation I wanted to refer to. These days I've grown accustomed to just having a bunch of Web browser windows semi-permanently open. Until a rude shock like this happens to hit me. Maybe I should get back into the PDF habit again...
0
8619
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we抣l explore What is ONU, What Is Router, ONU & Router抯 main usage, and What is the difference between ONU and Router. Let抯 take a closer look ! Part I. Meaning of...
0
9112
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8971
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8817
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
5831
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4336
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4571
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2258
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
1970
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.