By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
435,473 Members | 3,657 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 435,473 IT Pros & Developers. It's quick & easy.

iterator clone

P: n/a
Whats is the way to clone "independent" iterator? I can't use tee(),
because I don't know how many "independent" iterators I need. copy and
deepcopy doesn't work...

--pavel
Jul 13 '08 #1
Share this Question
Share on Google+
13 Replies


P: n/a
Yosifov Pavel wrote:
Whats is the way to clone "independent" iterator? I can't use tee(),
because I don't know how many "independent" iterators I need. copy and
deepcopy doesn't work...
There is no general way. For "short" sequences you can store the items in a
list which is also the worst-case behaviour of tee().

What are you trying to do?

Peter

Jul 13 '08 #2

P: n/a
On 13 , 14:12, Peter Otten <__pete...@web.dewrote:
Yosifov Pavel wrote:
Whats is the way to clone "independent" iterator? I can't use tee(),
because I don't know how many "independent" iterators I need. copy and
deepcopy doesn't work...

There is no general way. For "short" sequences you can store the items ina
list which is also the worst-case behaviour of tee().

What are you trying to do?

Peter
I try to generate iterators (iterator of iterators). Peter, you are
right! Thank you. For example, it's possible to use something like
this:

def cloneiter( it ):
"""return (clonable,clone)"""
return tee(it)

and usage:

clonable,seq1 = cloneiter(seq)

...iter over seq1...
then clone again:

clonable,seq2 = cloneiter(clonable)

...iter over seq2...

Or in class:

class ReIter:
def __init__( self, it ):
self._it = it
def __iter__( self ):
self._it,ret = tee(self._it)
return ret

and usage:

ri = ReIter(seq)
...iter over ri...
...again iter over ri...
...and again...

But I think (I'm sure!) it's deficiency of Python iterators! They are
not very good...

--Pavel
Jul 13 '08 #3

P: n/a
Yosifov Pavel wrote:
On 13 июл, 14:12, Peter Otten <__pete...@web.dewrote:
>Yosifov Pavel wrote:
Whats is the way to clone "independent" iterator? I can't use tee(),
because I don't know how many "independent" iterators I need. copy and
deepcopy doesn't work...

There is no general way. For "short" sequences you can store the items in
a list which is also the worst-case behaviour of tee().

What are you trying to do?

Peter

I try to generate iterators (iterator of iterators). Peter, you are
right! Thank you. For example, it's possible to use something like
this:

def cloneiter( it ):
"""return (clonable,clone)"""
return tee(it)
[snip]

That is too abstract, sorry. What concrete problem are you trying to solve
with your cloned iterators? There might be a way to rearrange your setup in
a way that doesn't need them.
But I think (I'm sure!) it's deficiency of Python iterators! They are
not very good...
Well, I think Python's iterators, especially the generators, are beautiful.
More importantly, I think there is no general way to make iterators
copyable, regardless of the programming language. The problem is that most
of the useful ones depend on external state.

Peter
Jul 13 '08 #4

P: n/a
Well, I think Python's iterators, especially the generators, are beautiful.
More importantly, I think there is no general way to make iterators
copyable, regardless of the programming language. The problem is that most
of the useful ones depend on external state.

Peter
Hmm, but tee() de facto do it (clone iterator) and ignore side-effects
of iterator ("external" state). And tee() create independent
**internal** state of iterator (current position). But **external**
state - is headache of programmer. So, iterator/generator have to be
method for copy itself (the tee() implementation) or be "re-
startable". Why not?

Concrete problem was to generate iterators (iterator of slices). It
was solved with ReIter.

--Best regards,
--pavel
Jul 14 '08 #5

P: n/a
On Sun, 13 Jul 2008 18:51:19 -0700, Yosifov Pavel wrote:
>Well, I think Python's iterators, especially the generators, are beautiful.
More importantly, I think there is no general way to make iterators
copyable, regardless of the programming language. The problem is that most
of the useful ones depend on external state.

Hmm, but tee() de facto do it (clone iterator) and ignore side-effects
of iterator ("external" state). And tee() create independent
**internal** state of iterator (current position).
`tee()` doesn't copy the iterator or its internal state but just caches
it's results, so you can iterate over them again. That makes only sense
if you expect to use the two iterators in a way they don't get much out of
sync. If your usage pattern is "consume iterator 1 fully, and then
re-iterate with iterator 2" `tee()` has no advantage over building a list
of all results of the original iterator and iterate over that twice.
`tee()` would be building this list anyway.
But **external** state - is headache of programmer. So,
iterator/generator have to be method for copy itself (the tee()
implementation) or be "re- startable". Why not?
Because it's often not possible without generating a list with all
results, and the advantage of a low memory footprint is lost.

Ciao,
Marc 'BlackJack' Rintsch
Jul 14 '08 #6

P: n/a
`tee()` doesn't copy the iterator or its internal state but just caches
it's results, so you can iterate over them again. That makes only sense
if you expect to use the two iterators in a way they don't get much out of
sync. If your usage pattern is "consume iterator 1 fully, and then
re-iterate with iterator 2" `tee()` has no advantage over building a list
of all results of the original iterator and iterate over that twice.
`tee()` would be building this list anyway.
It's interesting and a concrete answer. Thanks a lot.
Because it's often not possible without generating a list with all
results, and the advantage of a low memory footprint is lost.

Ciao,
Marc 'BlackJack' Rintsch
Seems like "monada". But I think is possible to determine when there
is a bounded external state (side-effects) or not, may be is needed
some new class-protocol for it... or something else. Or another way:
iterators may be re-iterable always, but if programmer need to point
to the extra- (external) state, he has to raise some a special
exception in __iter)) method... OK, it's only fantasies about language
design :-)

--pavel
Jul 14 '08 #7

P: n/a
On 13 juil, 12:05, Yosifov Pavel <b...@ngs.ruwrote:
(snip)
def cloneiter( it ):
"""return (clonable,clone)"""
return tee(it)
This might as well be written as

cloneiter = tee

Or yet better, just remove the above code and s/cloneiter/tee/g in the
remaining...

Jul 14 '08 #8

P: n/a
On 13 Jul., 08:53, Yosifov Pavel <b...@ngs.ruwrote:
Whats is the way to clone "independent" iterator? I can't use tee(),
because I don't know how many "independent" iterators I need. copy and
deepcopy doesn't work...

--pavel
You can try generator_tools

http://pypi.python.org/pypi/generator_tools/0.3.3

Jul 15 '08 #9

P: n/a
On 14 , 23:36, "bruno.desthuilli...@gmail.com"
<bruno.desthuilli...@gmail.comwrote:
On 13 juil, 12:05, Yosifov Pavel <b...@ngs.ruwrote:
(snip)
defcloneiter( it ):
"""return (clonable,clone)"""
return tee(it)

This might as well be written as

cloneiter = tee

Or yet better, just remove the above code and s/cloneiter/tee/g in the
remaining...
Yes, sure. It was only for illustration. BUT: Marc Rintsch is right:
cloning of iterators in this manner is bad, more good is to use one,
single list(my_iter) instead of (see
http://aquagnu.blogspot.com/2008/07/...n-python.html).

Thanks to all!

--pavel
Jul 15 '08 #10

P: n/a
On 15 Jul., 08:16, Yosifov Pavel <b...@ngs.ruwrote:
cloning of iterators in this manner is bad, more good is to use one,
single list(my_iter) instead of (seehttp://aquagnu.blogspot.com/2008/07/self-repair-iterator-in-python.html).
This won't work for "big" iterators as mentioned by Peter Otten. With
this recipe you can't even clone generator objects ( which are
iterators ) that produce Fibonaccis in a lazy manner.

Regards, Kay

Jul 15 '08 #11

P: n/a
Kay, can you show example of such generator? ReIter, for example, work
with usual generators.

But for "big" iterator, I think is no any good solutions. IMHO we can
discern 2 types of iterators: re-startable (based on internal Python
objects) and not re-startable (with an external state, side-
effects)...

Best regards, Pavel
Jul 16 '08 #12

P: n/a
On Tue, 15 Jul 2008 19:54:30 -0700, Yosifov Pavel wrote:
Kay, can you show example of such generator? ReIter, for example, work
with usual generators.

But for "big" iterator, I think is no any good solutions. IMHO we can
discern 2 types of iterators: re-startable (based on internal Python
objects) and not re-startable (with an external state, side-
effects)...
Has nothing to do with internal vs. external.
Examples: ``itertools.count(1)``, ``itertools.cycle(iterable)``, or

def fib():
a, b = 0, 1
while True:
yield a
a, b = b, a + b

Ciao,
Marc 'BlackJack' Rintsch
Jul 16 '08 #13

P: n/a
On 16 , 11:32, Marc 'BlackJack' Rintsch <bj_...@gmx.netwrote:
On Tue, 15 Jul 2008 19:54:30 -0700, Yosifov Pavel wrote:
Kay, can you show example of such generator? ReIter, for example, work
with usual generators.
But for "big" iterator, I think is no any good solutions. IMHO we can
discern 2 types of iterators: re-startable (based on internal Python
objects) and not re-startable (with an external state, side-
effects)...

Has nothing to do with internal vs. external.
Examples: ``itertools.count(1)``, ``itertools.cycle(iterable)``, or

def fib():
a, b = 0, 1
while True:
yield a
a, b = b, a + b

Ciao,
Marc 'BlackJack' Rintsch
Yes. So, I'm disconcerted: what Python "means" about iterators. Why
iterator's are not clonable in default?.. ``itertools.count(it) ``
"suppose" ``it`` will be restarted after count? So ``count(it)``
"suppose" ``it`` is restarable and therefore clonable (why not?!).
Generator is only function, so can be restarted/cloned. Generator
keeps "position" and can't be iterated again. But this position can be
reseted (main rule: if generator function has not side-effects/
external state, see below)!

Iterator, I think, is more general method (for naturally serial
access). For example, you can read values from some device (RS323 or
other) and represent it in program like iterator. In this case,
``__iter__()`` make some preparation, ``next()`` read next value from
RS232. In this case, iterator can't be restarted and really cloned. It
has external state (state of device) and can't to return it at start.
But when series of values are generated (are born) in the program: no
problem to have the feature of clone/restart iterator.

And is very interesting to research Icon iterators. Is possible to
create something in Python such theirs, for non-deterministic solving
purpose... :-)

--pavel
Jul 16 '08 #14

This discussion thread is closed

Replies have been disabled for this discussion.