Question about exausted iterators

[Christophe]

Is there a good reason why when you try to take an element from an
already exausted iterator, it throws StopIteration instead of some other
exception ? I've lost quite some times already because I was using a lot
of iterators and I forgot that that specific function parameter was one.

Exemple :

def f(i): .... print list(i)
.... print list(i)
.... f(iter(range(2) )) [0, 1]
[]

This is using Python 2.4.2

[Christophe]

Fredrik Lundh a écrit :

Christophe wrote:

Because I'm still waiting for a valid answer to my question. The
answer "Because it has been coded like that" or is not a valid one.
it's been coded like that because that's what the specification says:

http://www.python.org/dev/peps/pep-0234/

I didn't though I had to mention that "Because the spec has been
writen like that" wasn't a valid answer either.

so what is a valid answer?

Some valid use case for that behaviour, some example of why what I ask
could cause problems, some implementation difficulties etc ...

Saying it's like that because someone said so isn't exactly what I was
expecting as an answer :) People sometimes can be wrong you know.

[Roel Schroeven]

Fredrik Lundh schreef:

Christophe wrote:

Because I'm still waiting for a valid answer to my question.
The answer "Because it has been coded like that" or is not a
valid one.
it's been coded like that because that's what the specification
says:

http://www.python.org/dev/peps/pep-0234/

I didn't though I had to mention that "Because the spec has been
writen like that" wasn't a valid answer either.

so what is a valid answer?

I think he wants to know why the spec has been written that way.

The rationale mentions exhausted iterators:

"Once a particular iterator object has raised StopIteration, will
it also raise StopIteration on all subsequent next() calls?
Some say that it would be useful to require this, others say
that it is useful to leave this open to individual iterators.
Note that this may require an additional state bit for some
iterator implementations (e.g. function-wrapping iterators).

Resolution: once StopIteration is raised, calling it.next()
continues to raise StopIteration."

This doesn't, however, completey answer the OP's question, I think. It
is about raising or not raising StopIteration on subsequent next() calls
but doesn't say anything on possible alternatives, such as raising
another exception (I believe that's what the OP would like).

Not that I know of use cases for other exceptions after StopIteration;
just clarifying what I think the OP means.

--
If I have been able to see further, it was only because I stood
on the shoulders of giants. -- Isaac Newton

Roel Schroeven

[Christophe]

Diez B. Roggisch a écrit :

Christophe wrote:

Fredrik Lundh a écrit :

Christophe wrote:
Because I'm still waiting for a valid answer to my question. The
answer "Because it has been coded like that" or is not a valid one.
it's been coded like that because that's what the specification says:

http://www.python.org/dev/peps/pep-0234/
I didn't though I had to mention that "Because the spec has been writen
like that" wasn't a valid answer either.

The important thing is: it _is_ specified. And what about code like this:
iterable = produce_some_it erable()

for item in iterable:
if some_condition( item)
break
do_something()

for item in iterable:
do_something_wi th_the_rest()
If it weren't for StopIteration raised if the iterable was exhausted, you'd
have to clutter that code with something like

try:
for item in iterable:
do_something_wi th_the_rest()
except IteratorExhaust ed:
pass

It would be ugly but you could do that instead :

iterable = produce_some_it erable()

for item in iterable:
if some_condition( item)
break
do_something()
else:
iterable = []

for item in iterable:
do_something_wi th_the_rest()

I'll admit that the else clause in for/while loops isn't the most common
and so some people might be a little troubled by that.

There's also that :

iterable = produce_some_it erable()

for item in iterable:
if some_condition( item)
for item in iterable:
do_something_wi th_the_rest()
break
do_something()
What makes you say that this is better than the above? Just because _you_
had some cornercases that others seems not to have (at least that
frequently, I personally can't remember I've ever bitten by it) isn't a
valid reason to _not_ do it as python does.
Maybe I've used more iterables than most of you. Maybe I've been doing
that wrong. But I'd like to think that if I've made those mistakes,
others will make it too and would benefit for some help in debugging
that from the interpreter :)
Besides that: it would be a major change of semantics of iterators that I
seriously doubt it would make it into anything before P3K. So - somewhat a
moot point to discuss here I'd say.

It wouldn't be such a big semantic change I think. You could add that
easily[1] as deprecation warning at first and later on switch to a full
blown error.

[1] "Easily" provided you can easily code what I ask itself ;)

[Christophe]

Roel Schroeven a écrit :

Fredrik Lundh schreef:

so what is a valid answer?

I think he wants to know why the spec has been written that way.

The rationale mentions exhausted iterators:

"Once a particular iterator object has raised StopIteration, will
it also raise StopIteration on all subsequent next() calls?
Some say that it would be useful to require this, others say
that it is useful to leave this open to individual iterators.
Note that this may require an additional state bit for some
iterator implementations (e.g. function-wrapping iterators).

Resolution: once StopIteration is raised, calling it.next()
continues to raise StopIteration."

This doesn't, however, completey answer the OP's question, I think. It
is about raising or not raising StopIteration on subsequent next() calls
but doesn't say anything on possible alternatives, such as raising
another exception (I believe that's what the OP would like).

Exactly !
Not that I know of use cases for other exceptions after StopIteration;
just clarifying what I think the OP means.

There are no use cases yet for me. I want those exceptions as an hard
error for debuging purposes.

[Fredrik Lundh]

Christophe wrote:

Maybe I've used more iterables than most of you. Maybe I've been doing
that wrong.

your problem is that you're confusing iterables with sequences. they're
two different things.

</F>

[Christophe]

Fredrik Lundh a écrit :

Christophe wrote:

Maybe I've used more iterables than most of you. Maybe I've been doing
that wrong.

your problem is that you're confusing iterables with sequences. they're
two different things.

Yes, I know perfectly well that the bugs were my fault. But this doesn't
prevent me from asking for a feature that will have ( in my opinion ) a
negligible effect of current valid code and will help all of us catch
errors earlier.

[Erik Max Francis]

Christophe wrote:

Yes, I know perfectly well that the bugs were my fault. But this doesn't
prevent me from asking for a feature that will have ( in my opinion ) a
negligible effect of current valid code and will help all of us catch
errors earlier.

.... and apparently choosing to ask in such a way that guarantees
practically no one will take your suggestion seriously.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Chance favors the trained mind.
-- Louis Pasteur

[Terry Reedy]

"Christophe " <ch************ *@free.fr> wrote in message
news:44******** **************@ news.free.fr...

Instead of saying that all works as intended could you be a little
helpful and tell me why it was intended in such an obviously broken way
instead ?

I answered both your explicit and implied questions in good faith. But you
seem to be too attached to your pre-judgment to have benefited much, so I
won't waste my time and yours saying more. Instead I suggest that you try
this:

1. Write a specification for your an alternate, more complicated, iterator
protocol.
2. Write a simple class with .next method that implements your
specification.
3. Test your class with your example.
4. Consider how you would persuade people to add the extra machinery
needed.
5. Consider what you would do when people don't.

If you want, post a report on your experiment, and I will read it if I see
it.

Terry Jan Reedy

[Christophe]

Terry Reedy a écrit :

"Christophe " <ch************ *@free.fr> wrote in message
news:44******** **************@ news.free.fr...

Instead of saying that all works as intended could you be a little
helpful and tell me why it was intended in such an obviously broken way
instead ?

I answered both your explicit and implied questions in good faith. But you
seem to be too attached to your pre-judgment to have benefited much, so I
won't waste my time and yours saying more. Instead I suggest that you try
this:

1. Write a specification for your an alternate, more complicated, iterator
protocol.

Specification : same as now except iterators raise once StopIteration
and any subsequent call to next raises ExaustedIterato rError.
2. Write a simple class with .next method that implements your
specification.
class ExaustedIterato rError(Exceptio n):
pass

class safe_iter(objec t):
def __init__(self, seq):
self.it = iter(seq)
def __iter__(self):
return self
def next(self):
try:
return self.it.next()
except StopIteration:
del self.it
raise
except AttributeError:
raise ExaustedIterato rError
3. Test your class with your example.

it = safe_iter(range (10))
print list(it) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] print list(it)

Traceback (most recent call last):
File "safe_iter_test .py", line 20, in ?
print list(it)
File "safe_iter_test .py", line 13, in next
raise ExaustedIterato rError
__main__.Exaust edIteratorError
4. Consider how you would persuade people to add the extra machinery
needed.
Well, the main reason for such change is and will always be to catch
bugs. The fact is, using duct typing is something very common with the
Python language. And as such, considering a lot of functions which take
sequences as parameters work as well with an iterator instead, you can
say that it's an application of duct typing.

The problem is of course the same as for cases. Even if those two objets
( iterator and container ) look alike from a certain point of view, they
have some fundamental differences.

So, we have quite a few functions which take freely either a container
or an iterator, until someone changes that function a little. At that
point there are three kind errors which happen :
- the function expected a sequence and tries to access it's [] operator
which fails. Standard duct typing behaviour.
- the function uses the iterator more than once and so, sometimes it
works without errors but produces an incorrect result.
- the function uses the iterator more than once but never exhausts it's
values. Same result as above but much harder to catch.

In the sake of avoiding behaviour which lets obvious errors pass and
produces incorrect results, I propose to change the standard behaviour
of all the iterators in the standard Python. The change will be so that
they refuse to be used anymore once they have been exausted. Thus it'll
help catch the second class. The other procedure used to catch such bugs
would require explicit typing of the function parameters but this is for
some other proposal.
5. Consider what you would do when people don't.
I'm already doing it. Cleaning up the code, changing parameters names
around so that it is clear such parameter is an iterator and that other
one is not, making some functions explicitly refuse iterators etc ... It
should not that that last feature will be useful even without the
exhausted iterator guard I propose.
If you want, post a report on your experiment, and I will read it if I see
it.

I suppose I could add safe_iter to the builtins in my project and use it
around. It would be easy to catch all usages of that one once we settle
on something different then.

[yairchu]

Consider this example:

X = range(5)
Y = iter(X)
Z = iter(Y)
As you can see, X is a container, and Y is an iterator.
They are simliar in that "iter" works on them both.

Cristoph claims that this causes confusion.
Why? Because "iter" doesn't have the same meaning for both of them.
For X it always returns an iterator that yields the same set of values.
For Y it returns an iterator yielding different values each time.
Z = iter(Y)
Z.next() 0 Z = iter(Y)
Z.next() 1

Most of the uses of iterators iterate all values until exaustion.
Given that, the first call to "iter" will mean the same for an iterator
and a container, but the second one won't.

Allow me to compare it to division in python 2.4 4./2, 4/2 2.0, 2 5./2, 5/2 2.5, 2

Division sometimes works the same for integers and reals, and sometimes
doesn't.
This caused consfusion and bugs, and that's why future Pythons will
change that.

But changing "iter" to have the same meaning for containers and
iterables is impossible.
You cannot, conceptually, reiterate an iterator.
So what Cristoph is suggesting - is to add an exception for the cases
in which iterators and collections behave differently.
Somewhat similar to this:
4/2 2 5/2

Traceback (most recent call last):
File "<stdin>", line 1, in ?
IntegerDivision Error: 5 does not divide by 2