Please comment on the new PEP for reverse iteration methods.
Basically, the idea looks like this:
for i in xrange(10).iter_backwards(): # 9,8,7,6,5,4,3,2,1,0
<do something with i>
The HTML version is much more readable than the ReST version.
See: http://www.python.org/peps/pep-0322.html
Several interesting ideas surfaced in the pre-pep thread:
* Call it ireverse() instead of iter_backwards().
Good idea. This is much more pithy.
* Use object properties instead of methods.
I can't yet claim to understand what the author is really
proposing. It has something to do which providing
access to an object that responds to iter, getitem, and
getslice with reversed indices.
* using a single function that looks for an __riter__ magic
method and, if not found, use __getitem__ and __len__
to build a reverse iterator.
A workable version of this was posted.
It saves implementing some object methods at the
expense of adding a new builtin function and of
creating a new magic method name.
It is slower than direct access to the underlying object.
It crashes when applied to an infinite iterator.
It produces bizarre results when applied to mappings.
* special markers for infinite iterators
This idea is interesting but doesn't extend well
when the object is wrapped by another iterator.
Also, there is no automatic way to determine
which generators can be infinite.
Raymond Hettinger 59  4062 
"Raymond Hettinger" <vz******@verizon.net> wrote in message
news:95*****************@nwrdny01.gnilink.net... Please comment on the new PEP for reverse iteration methods.
# from PEP 322
"""
Rejected Alternative Ideas
[snip]
Add a builtin function, reverse() ...
"""
Do you mean add a function reverse() (or ireverse()) to the itertools
library, or to __builtins__?
I think it says what needs to be said.
+1
John Roth
"Raymond Hettinger" <vz******@verizon.net> wrote in message
news:95*****************@nwrdny01.gnilink.net... Please comment on the new PEP for reverse iteration methods.
Basically, the idea looks like this:
for i in xrange(10).iter_backwards(): # 9,8,7,6,5,4,3,2,1,0
<do something with i>
The HTML version is much more readable than the ReST version.
See:
http://www.python.org/peps/pep-0322.html
Several interesting ideas surfaced in the pre-pep thread:
* Call it ireverse() instead of iter_backwards().
Good idea. This is much more pithy.
* Use object properties instead of methods.
I can't yet claim to understand what the author is really
proposing. It has something to do which providing
access to an object that responds to iter, getitem, and
getslice with reversed indices.
* using a single function that looks for an __riter__ magic
method and, if not found, use __getitem__ and __len__
to build a reverse iterator.
A workable version of this was posted.
It saves implementing some object methods at the
expense of adding a new builtin function and of
creating a new magic method name.
It is slower than direct access to the underlying object.
It crashes when applied to an infinite iterator.
It produces bizarre results when applied to mappings.
* special markers for infinite iterators
This idea is interesting but doesn't extend well
when the object is wrapped by another iterator.
Also, there is no automatic way to determine
which generators can be infinite.
Raymond Hettinger
On Thu, 25 Sep 2003 00:58:45 GMT, "Raymond Hettinger"
<vz******@verizon.net> wrote: * Use object properties instead of methods.
I can't yet claim to understand what the author is really
proposing. It has something to do which providing
access to an object that responds to iter, getitem, and
getslice with reversed indices.
The idea is basically to have a way to get a proxy to a sequence which
behaves like the original sequence, but with indexes reversed. There
is no reason why member functions couldn't return those proxies
instead of using properties, except that when the result is sliced it
looks less ugly.
I have a demo of it for lists only (listing at end of post). It has
three properties...
backward : reversed, slicing returns a list
ibackward : reversed, slicing returns an iterator
iforward : forward, slicing returns an iterator
All properties currently support __getitem__, __setitem__,
__delitem__, __str__, __repr__, __iter__ and __len__ (they all return
an instance of the same proxy class but with different options set).
For simple iterating-over-ints, I'd use the same principles with that
object-acting-as-sliceable-set-of-all-integers idea from the PEP284
thread.
I had some problems getting the slice translation to work - there are
some little nasties with negative steps. You can write...
x [::-1]
....but trying to make it explicit...
x [len(x)-1:-1:-1]
....breaks it because of that -ve stop bound. Not being a Numeric user,
slice steps are quite a new thing to me - and that little trap just
caught me by surprise. It isn't too hard to work around it, though.
The code following isn't pretty but so far as I can tell it works.
Python 2.3 is assumed.
The question is - is the exercise worthwhile. I'm not sure even though
it's my own idea. I think it is nice in its way, and a lot more
flexible than the 'iter_backward' method proposal, but that is quite
likely a bad thing as it is probably massive overkill for the problem
being solved - and in particular the extra flexibility has costs.
What the hell, though - it was fun to play with for a bit!
Anyway, here is a quick test session... from rilist import rilist
x=rilist(range(20))
x.backward
[19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0] x.backward[:10]
[19, 18, 17, 16, 15, 14, 13, 12, 11, 10] x.backward[::2]
[19, 17, 15, 13, 11, 9, 7, 5, 3, 1] del x.backward[::2]
x.backward
[18, 16, 14, 12, 10, 8, 6, 4, 2, 0] x
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18] for i in x.backward :
.... print i
....
18
16
14
12
10
8
6
4
2
0 for i in x.backward [5:]:
.... print i
....
8
6
4
2
0 for i in x [5:]:
.... print i
....
10
12
14
16
18
I've done some 'ibackward' tests as well, and that seems to work - but
I haven't bothered with 'iforward'.
Here is the source - warning, it isn't very self-documenting ATM...
class c_List_Proxy (object) :
"""
Proxy to list class which supports a subset of methods with
modified behaviours, mainly...
- Slicing may create an iterator rather than a list.
- Subscripts and ranges may be reversed.
"""
def __init__ (self, p_List, p_Reverse=False, p_Iter_Slice=False) :
self.List = p_List
self.Reverse = p_Reverse
self.Iter_Slice = p_Iter_Slice
# need some support stuff
def __SliceIter__ (self, p_Slice) :
if p_Slice.step > 0 :
i = p_Slice.start
while i < p_Slice.stop :
yield self.List [i]
i += p_Slice.step
else :
i = p_Slice.start
while i > p_Slice.stop :
yield self.List [i]
i += p_Slice.step
def __FullIter__ (self) :
i = len (self.List) - 1
while i >= 0 :
yield self.List [i]
i -= 1
def __KeyTransformed__ (self, p_Key) :
if self.Reverse :
if p_Key < 0 :
return (-1) - p_Key
else :
return (len (self.List) - 1) - p_Key
else :
return p_Key
def __Bound__ (self, p_Given, p_Default) :
if p_Given is None :
return p_Default
elif p_Given < 0 :
return len (self.List) + p_Given
else :
return p_Given
def __SliceFixed__ (self, p_Slice) :
l_Step = p_Slice.step or 1
if l_Step == 0 :
raise IndexError, "Step must be non-zero"
elif l_Step > 0 :
l_Start = self.__Bound__ (p_Slice.start, 0)
l_Stop = self.__Bound__ (p_Slice.stop, len (self.List))
else :
l_Start = self.__Bound__ (p_Slice.start, len (self.List) - 1)
l_Stop = self.__Bound__ (p_Slice.stop, -1)
l_Count = (((l_Stop - 1) - l_Start) // l_Step) + 1
l_Stop = l_Start + l_Count * l_Step
return slice(l_Start,l_Stop,l_Step)
def __SliceTransformed__ (self, p_Slice) :
l_Slice = self.__SliceFixed__ (p_Slice)
if self.Reverse :
l_End = len(self.List) - 1
return slice (l_End - l_Slice.start, l_End - l_Slice.stop,
-l_Slice.step)
else :
return p_Slice
# some members are trivial
def __len__ (self) :
return len (self.List)
def __iter__ (self) :
return self.__FullIter__ ()
# some members need a bit more work...
def __str__ (self) :
if self.Reverse :
return str(self.List [::-1])
else :
return str(self.List)
def __repr__ (self) :
if self.Reverse :
return repr(self.List [::-1])
else :
return repr(self.List)
def __getitem__ (self, p_Item) :
if isinstance (p_Item, slice) :
if self.Iter_Slice :
return self.__SliceIter__ (self.__SliceTransformed__ (p_Item))
else :
l_Slice = self.__SliceTransformed__ (p_Item)
if l_Slice.stop == -1 : # annoying special case
return self.List [l_Slice.start::l_Slice.step]
return self.List [l_Slice.start:l_Slice.stop:l_Slice.step]
else :
return self.List [self.__KeyTransformed__ (p_Item)]
def __setitem__ (self, p_Item, p_Value) :
if isinstance (p_Item, slice) :
l_Slice = self.__SliceTransformed__ (p_Item)
if l_Slice.stop == -1 : # annoying special case
self.List [l_Slice.start::l_Slice.step] = p_Value
else :
self.List [l_Slice.start:l_Slice.stop:l_Slice.step] = p_Value
else :
self.List [self.__KeyTransformed__ (p_Item)] = p_Value
def __delitem__ (self, p_Item) :
if isinstance (p_Item, slice) :
l_Slice = self.__SliceTransformed__ (p_Item)
if l_Slice.stop == -1 : # annoying special case
del self.List [l_Slice.start::l_Slice.step]
else :
del self.List [l_Slice.start:l_Slice.stop:l_Slice.step]
else :
del self.List [self.__KeyTransformed__ (p_Item)]
class rilist (list) :
"""
Extend normal list to provide properties giving variants of normal
indexing behaviour - specifically...
indexing slicing returns
-------- ---------------
iforward forward iterator
ibackward backward iterator
backward backward list
"""
def __IFwd__ (self) :
return c_List_Proxy (self, False, True)
def __IBack__ (self) :
return c_List_Proxy (self, True, True)
def __Back__ (self) :
return c_List_Proxy (self, True, False)
iforward = property(__IFwd__ )
ibackward = property(__IBack__)
backward = property(__Back__ )
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
Raymond Hettinger: Please comment on the new PEP for reverse iteration methods.
It says:
] Also, it only works with lists (strings, for example, do not define a
reverse method):
]
] rseqn = list(seqn)
] rseqn.reverse()
] for value in rseqn:
] print value
However, the example code will work on a string, because list(string)
returns a list, which does have a reverse.
] Extending slicing minimizes the code overhead but does nothing
] for memory efficiency, beauty, or clarity.
I don't agree with all three of those. Take for example your mhlib
use case
testfolders.reverse();
for t in testfolders:
do('mh.deletefolder(%s)' % `t`)
The PEP suggests that
for t in testfolders.iter_backwards():
do('mh.deletefolder(%s)' % `t`)
is more beautiful and clearer than
for t in testfolders[::-1]:
do('mh.deletefolder(%s)' % `t`)
(It definitely is more memory efficient if there are enough
folders.)
I don't think I agree with that. If I'm not concerned about memory
efficiency then extended slicing is nice because I don't have to
learn nor explain one more thing. ("There should be one ...")
The beauty and clarity, at least to me, only come into play
when memory is a concern, which is only rarely true for most
of the times iter_backwards is expected to be used. (Estimate
based on the use cases and my own judgement.)
] Should file objects be included? Implementing reverse iteration
] may not be easy though it would be useful on occasion.
I would suggest no - the file may not be seekable.
You suggest changing
] while _exithandlers:
] func, targs, kargs = _exithandlers.pop()
to
] for func, target, kargs in _exithandlers.iter_backwards():
] . . .
] del _exithandlers
One thing to bear in mind is that the semantics are slightly
different. In the first one, the finalizers for func, targs, kargs
*may* be called from last to first. This is the case for
CPython if there are no other references to the _exithandlers
objects. But since Python-the-language makes no guarantees
on that and Jython doesn't support it, it's a bad idea to write
code that way.
That is only a very, very minor nit. Feel free to ignore it :)
In difflib, I disagree that
result.sort()
return [x for score, x in result[-n:].iter_backwards()]
is easier to understand than
result.sort() # Retain only the best n.
result = result[-n:] # Move best-scorer to head of list.
result.reverse() # Strip scores.
return [x for score, x in result]
The former is more terse, but the latter, existing code is
broken down into bite-size chunks and allows verbose
documentation of each step. That isn't possible when
nearly everything is written in one line.
Also, a variant implementation may be even more concise,
which is to replace
result.append((s.ratio(), x))
with
result.append((-s.ratio(), x))
then get rid of the reverse (since the minus sign reverses the
sort order) and simply do
result.sort()
return [x for score, x in result[:n]]
] heapq.heapify() uses for i in xrange(n//2 - 1, -1, -1)
The proposed form is
for i in xrange(n//2-1).iter_backwards():
Later in your post you mention a reason against a function
which uses __riter__ magic is that it is "slower than direct
access to the underlying object."
If that's a concern then you should also point out that using
iter_backwards for cases like this is slower than the existing
solution. (It has an extra attr lookup and function call. OTOH,
doesn't the -1, -1 require two calls of the negative unary op
code, or has that been fixed? Good, it has, says dis on
some test code.)
] platform._dist_try_harder() uses for n in range(len(verfiles)-1,-1,-1)
] because the loop deletes selected elements from verfiles but needs
] to leave the rest of the list intact for further iteration. This use case
] could be addressed with itertools.ifilter() but would require the
] selection predicate to be in a lambda expression. The net result is
] less clear and readable than the original. A better reformulation is
] to replace the first line with the proposed method.
Are you sure about that? The code is
verfiles = os.listdir('/usr/lib/setup')
for n in range(len(verfiles)-1, -1, -1):
if verfiles[n][:14] != 'slack-version-':
del verfiles[n]
I think a better reformulation is
verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]
(you could put the os.listdir in the list comp. but I've
found that gets ungainly.)
] random.shuffle() uses for i in xrange(len(x)-1, 0, -1)
This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.
list(xrange(5, 0, -1))
[5, 4, 3, 2, 1]
] Rejected Alternative Ideas
There were two intermediate 'reverse' proposals. Here's the four
I know about:
- call the magic method __riter__
- call the magic method __riter__. If it doesn't exist, use
for i in xrange(len(obj)): yield obj[i]
- call the magic method __riter__. If it doesn't exist, use
for i in itertools.count(): yield[obj[-i]]
(this has the advantage of supporting the infinite sequence
0, 1, 2, ... -3, -2, -1
)
- same as the 2nd but if len doesn't exist use
data = list(obj)
data.reverse()
for x in data: yield x
I agree with the conclusion that the last three have unexpected
interactions with mappings. However, iter *also* has unexpected
interactions with mappings.
class Dict:
.... def keys(self):
.... return [0, 1, 2, 4]
.... def __getitem__(self, key):
.... if key not in self.keys(): raise KeyError(key)
.... return key * 2
.... def values(self):
.... return [self[x] for x in self.keys()]
.... def items(self): return zip(self.keys(), self.values())
.... def len(self): return len(self.keys())
.... for x in Dict():
.... print x
....
0
2
4
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "<interactive input>", line 5, in __getitem__
KeyError: 3
While I think I was the one who originally raised this
objection, given the way Python already behaves wrt
mappings, I retract my objection.
Another possibility, which I just thought of now, is
- call the magic method __riter__. If it doesn't exist, look
for the reverse-iterator adapter, as per PEP-0246.
I don't have any experience with the PEP to judge its
appropriateness nor usefulness. The motivation section
seems to suggest that it would be appropriate.
In your post list a set of problems with the function approaches.
It saves implementing some object methods at the
expense of adding a new builtin function and of
creating a new magic method name.
It is slower than direct access to the underlying object.
It crashes when applied to an infinite iterator.
It produces bizarre results when applied to mappings.
I would like to see this list added to the PEP. I do have'
some qualifiers for the list.
- it doesn't need to be a global function; it could be in,
say, itertools. (I know I originally suggested it as a builtin,
but I'm not saying I'm right ;)
- (see comments above that xrange(5).iter_backwards()
is likely slower than xrange(5, -1, -1) because of the
extra lookup and call)
- the first variant, which only looks for __riter__ and raises
an exception if it doesn't exist, has exactly the same problems
with infinite iterators as calling the new method.
- similarly, it doesn't have additional problems with mappings.
Also, none of the use cases dealt with infinite sequences so
the lack of appropriate support for them shouldn't be considered
an outright rejection.
BTW, my current belief is that if this PEP leads to code
then it should be:
- a new function, either builtin or in itertools
- which uses __riter__ if present (unless we decide to
use the adapt PEP)
- else uses "for i in itertools.count(): yield[obj[-i]]"
Andrew da***@dalkescientific.com
Me: - else uses "for i in itertools.count(): yield[obj[-i]]"
should be
- else uses "for i in itertools.count(1): yield[obj[-i]]"
Andrew da***@dalkescientific.com
The PEP has the following sentence in the rejected alternatives: * Add a builtin function, reverse() which calls a magic method, __riter__.
I see this as more overhead for no additional benefit.
I believe that this is the better choice because it is the way forward
iteration work in python: you have the iter() function which call a
__iter__ special method.
So, I'd expect to have a riter() function which would call the
__riter__ special method.
I like riter as name to all other alternative because (iter, riter)
looks really like (find, rfind) or (index, rindex) ....
And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.
"Raymond Hettinger" <vz******@verizon.net> wrote ... Please comment on the new PEP for reverse iteration methods.
Looks good. But
[...] It crashes when applied to an infinite iterator.
[...]
Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?
regards
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/pwp/
"sebastien" <s.****@laposte.net> wrote in message So, I'd expect to have a riter() function which would call the
__riter__ special method.
.. . . And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.
If you make a magic method, then the function calling it
needs to be a builtin. They go hand in hand.
Raymond
[Andrew Dalke] However, the example code will work on a string, because list(string)
returns a list, which does have a reverse.
Right. I'll remove the comment.
] Extending slicing minimizes the code overhead but does nothing
] for memory efficiency, beauty, or clarity.
I don't agree with all three of those.
Beauty and clarity are a matter of taste and we differ here.
To some (including me), s[::-1] looks more like line noise
than s.iter_backwards(). Also, I think the word form is
more clear. The situation becomes more acute when the
other indices have values. Compare range[n-1, 0, -1] to
range(1, n-1).iterbackwards(). Whenever negative indicies
are used, it takes more knowledge of the implementation
rules to be able to read,write, understand, and verify the code.
] Should file objects be included? Implementing reverse iteration
] may not be easy though it would be useful on occasion.
I would suggest no - the file may not be seekable.
Agreed.
One thing to bear in mind is that the semantics are slightly
different.
.. . . That is only a very, very minor nit. Feel free to ignore it :)
Agreed. If finalizer call order is important, then the
'while elist: elist.pop()' form is the way to go. If not,
then the for-loop is the way to go.
The former is more terse, but the latter, existing code is
broken down into bite-size chunks and allows verbose
documentation of each step. That isn't possible when
nearly everything is written in one line.
Even the former can be broken down into more lines if
that is what is needed:
result.sort()
result = result[-n]
return [x for score, x in result.iter_backwards()]
Also, a variant implementation may be even more concise,
You must be pretty confident to take on the Timbot's code ;-)
Later in your post you mention a reason against a function
which uses __riter__ magic is that it is "slower than direct
access to the underlying object."
If that's a concern then you should also point out that using
iter_backwards for cases like this is slower than the existing
solution. (It has an extra attr lookup and function call. OTOH,
doesn't the -1, -1 require two calls of the negative unary op
code, or has that been fixed? Good, it has, says dis on
some test code.)
Yes, it's true. All of the speed comments focus on the inner loop.
I'll reword to make that clear. It is definitely an issue. In Py2.2,
there was a generic sequence iterator that used __getitem__ in
iterators for lists and tuples. In Py2.3, we went to a custom
iterator for each object. There was a significant improvement
in performance.
] platform._dist_try_harder()
. . . I think a better reformulation is
verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]
I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.
] random.shuffle() uses for i in xrange(len(x)-1, 0, -1)
This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.
It is an important use case. The replacement code is:
for i in xrange(1,len(x)). iter_backwards()
Whenever the indices have any complexity, the forwards version,
followed by .iter_backwards() is, IMHO, much easier to mentally verify.
There were two intermediate 'reverse' proposals. Here's the four
I know about:
I'll try to add these without making the PEP as long as this thread ;-)
I agree with the conclusion that the last three have unexpected
interactions with mappings. However, iter *also* has unexpected
interactions with mappings.
Yes, but that doesn't make it less of a disadvantage.
The proposed list.iter_backwards() method does not have that
disadvantage.
In your post list a set of problems with the function approaches.
.... I would like to see this list added to the PEP.
Okay. They are added.
I do have'
some qualifiers for the list.
- it doesn't need to be a global function; it could be in,
say, itertools. (I know I originally suggested it as a builtin,
but I'm not saying I'm right ;)
If there is to be a magic method, __riter__, then the access
function needs to be a builtin. They go hand in hand.
BTW, don't underestimate the intensity of resistance to adding
new builtins.
- (see comments above that xrange(5).iter_backwards()
is likely slower than xrange(5, -1, -1) because of the
extra lookup and call)
Reworded it to talk only to the inner loop, "approaches using
__getitem__() are slower using a custom method for
each object."
- the first variant, which only looks for __riter__ and raises
an exception if it doesn't exist, has exactly the same problems
with infinite iterators as calling the new method.
- similarly, it doesn't have additional problems with mappings.
Okay, I've moved the comments around for you.
Also, none of the use cases dealt with infinite sequences so
the lack of appropriate support for them shouldn't be considered
an outright rejection.
The use cases were just the ones I found in the library.
With generators and itertools being new, I expect
infinite iterators to become more common.
Also, I have a preference for creating something that
is as robust as possible. Making a builtin function that
doesn't apply to all objects; behaves badly with mappings;
and crashes with an infinite iterator is not my idea of robust.
I really don't understand the overpowering urge to make this a function
and add a new magic method protocol. Why isn't there a similar rage
against dict.iteritems()?
Perhaps, you'll like it better if the method is named ireverse()
instead of iter_backwards().
BTW, my current belief is that if this PEP leads to code
then it should be:
- a new function, either builtin or in itertools
- which uses __riter__ if present (unless we decide to
use the adapt PEP)
- else uses "for i in itertools.count(): yield[obj[-i]]"
Okay, we simply disagree here. That's fine.
I appreciate the thoughtful comments and careful analysis.
They have helped clarify the pep wording and helped to
define the issues.
Raymond
[Stephen Horne] The question is - is the exercise worthwhile. I'm not sure even though
it's my own idea. I think it is nice in its way, and a lot more
flexible than the 'iter_backward' method proposal, but that is quite
likely a bad thing as it is probably massive overkill for the problem
being solved - and in particular the extra flexibility has costs.
What the hell, though - it was fun to play with for a bit!
Yes, it's a one ton solution to a one ounce problem,
but it was an interesting exercise and diversion.
Raymond Hettinger
[sebastien] So, I'd expect to have a riter() function which would call the
__riter__ special method.
Okay, changed pep to list riter() as an alternative.
And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.
If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.
Raymond
On Thu, 25 Sep 2003, Steve Holden wrote: "Raymond Hettinger" <vz******@verizon.net> wrote ... Please comment on the new PEP for reverse iteration methods.
Looks good. But
[...] It crashes when applied to an infinite iterator.
[...]
Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?
ideally, riter(riter(someInfiniteIterator)) would work, though!
tom
--
I see large and small words on this page, arranged in long rows separated by little dotty characters. Suspect written by little dotty characters, too. -- RonJeffries
Raymond Hettinger: Beauty and clarity are a matter of taste and we differ here.
To some (including me), s[::-1] looks more like line noise
than s.iter_backwards().
Understood. In all this, let me make it clearthat I
have a preference for a function over a method, but won't
call it a wart of the BFDL thinks otherwise. I point these
out because I'm rather detail oriented.
Also, a variant implementation may be even more concise,
You must be pretty confident to take on the Timbot's code ;-)
Well, I did say "may". I wouldn't really want to use it since
inverting the score before the sort is a tricky thing to see and
somewhat unexpected.
(I'm half expecting some enlightening comment about how
negation has subtle interactions with IEEE 754 math -- though
I thought the point of 754 was to make naive approaches like
mine usually correct. :)
verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]
I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.
But that goes against the pretty good motto of "if it ain't broke,
don't touch it." Granted, XP-style tests are supposed to
help out with that, but I don't have a /usr/lib/setup where I
can test this.
] random.shuffle() uses for i in xrange(len(x)-1, 0, -1)
This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.
It is an important use case. The replacement code is:
for i in xrange(1,len(x)). iter_backwards()
Ahh, right. Didn't think about that. You should include that
wording in the PEP.
Whenever the indices have any complexity, the forwards version,
followed by .iter_backwards() is, IMHO, much easier to mentally verify.
I agree. When I do need to go backwards I often forgot to get
both -1s in place. Ie, I'll do (n, 0) instead of (n-1, -1).
Well, don't forgot much and more, but there's some tension in
my mind as I worry if I got it correct or not.
BTW, don't underestimate the intensity of resistance to adding
new builtins.
For me it's two things: remembering what all these new things
do, and attempting to adhere to the 'don't use variables with
the same name as builtins' guiding philosophy. I don't adhere
to the latter wrt the type named 'file'.
Also, I have a preference for creating something that
is as robust as possible. Making a builtin function that
doesn't apply to all objects; behaves badly with mappings;
and crashes with an infinite iterator is not my idea of robust.
The backup which uses __getitem__ and negative offsets
does work for all objects, behaves exactly as badly as the
existing iter, and doesn't crash with an infinite iterator, no?
It's slow, but if you want the performance, make an __riter__.
I really don't understand the overpowering urge to make this a function
and add a new magic method protocol. Why isn't there a similar rage
against dict.iteritems()?
Overpowering?
My preference is for a function because of:
- similaraties to iter, which is also a function
- there are many types of iteration orderings, and not all can be
special cased methods.
- I don't see backwards iteration as all that important
As for iteritems, I was annoyed about it because it meant my
dictionary-like code would become more complicated to
implement, until I came across UserDict.DictMixin, which
lets user-defined code implement just a few methods then
builds the rest from there.
That suggests a UserList.ListMixin might be useful, but it
isn't obvious to me that it would be.
I also note that items is a method so by similarity it makes
sense that iteritems also be a method. Additionally, many
things offer forward iteration, while only dicts offer items.
Should dicts support riteritems, ritervalues?
Okay, we simply disagree here. That's fine.
Yup. I don't think I have anything useful or even interesting
to offer to this dicusssion. I also think that with these changes,
the PEP is pretty complete and solid.
Andrew da***@dalkescientific.com
On Thu, 25 Sep 2003 00:58:45 GMT, "Raymond Hettinger"
<vz******@verizon.net> wrote: Please comment on the new PEP for reverse iteration methods.
One more thought.
Having an iter_backwards method on xrange seems wrong. This suggests
multiple functions/methods...
To generate backward ranges
range_backwards function
xrange_backwards funtion
for i in range_backwards (10) :
...
To iterate an object backwards
either method for appropriate objects, or
iter_backward() function and __iter_backward__ magic method
(I don't mind abbreviating the 'iter', but I think the 'backward'
needs to be kept very obvious and not reduced to an easily missed
'r' prefix).
for i in iter_backwards (seq) :
...
To enumate backwards etc
Add an enumerate_backward function which requires the input to
support __len__.
for i in enumerate_backwards (seq) :
...
For backward slicing
Add a function which translates the slice to a backward equivalent
before applying it...
for i in slice_backwards (seq, start, stop, step) :
...
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
"Raymond Hettinger" <vz******@verizon.net> wrote in message news:<wC*****************@nwrdny01.gnilink.net>... If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.
I don't think so:
We already have this situation for the the copy module:
copy.copy() use the __copy__ magic method and copy.deepcopy() use
__deepcopy__
I've never heard about problems with this behaviour.
In fact it's not the __riter__ magic method which will use riter()
function but the opposite, so an object which define __riter__ doesn't
need to now anything about riter().
Raymond Hettinger wrote: And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.
If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.
I have already seen module copy presented as a counter-example to
your assertion, and I'd like to add module pickle as a second, and
I hope decisive, counter-example. It is, to put it simply, utterly
false that functions corresponding to magic methods "need to be
builtins": it's *perfectly* all right for such functions to live
in standard library modules. Oh, and let's not forget module sets:
the __as_immutable__ and __as_temporarily_immutable__ magic methods,
that are used when making a set of sets, or checking for a set's
membership in another set, can be seen as yet another example (and
in this case there is no wiggling out of it by claiming that the
magicmethod/NON-builtin correspondence is a historical/legacy thing,
as the BDFL approved module sets.py, with just this usage, so very
recently).
I second the motion that function riter, with check for __riter__
and all, should live in module itertools. Reverse iteration is a
RARE need -- far rarer than copying or even pickling -- and there
is no real reason to burden __builtins__ with a function for it.
Alex
I agree. Importing itertools to get at riter()
isn't a significant hardship.
John Roth
"Alex Martelli" <al***@aleax.it> wrote in message
news:8R**********************@news2.tin.it... Raymond Hettinger wrote:
And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.
If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.
I have already seen module copy presented as a counter-example to
your assertion, and I'd like to add module pickle as a second, and
I hope decisive, counter-example. It is, to put it simply, utterly
false that functions corresponding to magic methods "need to be
builtins": it's *perfectly* all right for such functions to live
in standard library modules. Oh, and let's not forget module sets:
the __as_immutable__ and __as_temporarily_immutable__ magic methods,
that are used when making a set of sets, or checking for a set's
membership in another set, can be seen as yet another example (and
in this case there is no wiggling out of it by claiming that the
magicmethod/NON-builtin correspondence is a historical/legacy thing,
as the BDFL approved module sets.py, with just this usage, so very
recently).
I second the motion that function riter, with check for __riter__
and all, should live in module itertools. Reverse iteration is a
RARE need -- far rarer than copying or even pickling -- and there
is no real reason to burden __builtins__ with a function for it.
Alex
Alex Martelli <al***@aleax.it> writes: Raymond Hettinger wrote:
And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.
If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.
I have already seen module copy presented as a counter-example to
your assertion, and I'd like to add module pickle as a second, and
I hope decisive, counter-example. It is, to put it simply, utterly
false that functions corresponding to magic methods "need to be
builtins": it's *perfectly* all right for such functions to live
in standard library modules. Oh, and let's not forget module sets:
the __as_immutable__ and __as_temporarily_immutable__ magic methods,
that are used when making a set of sets, or checking for a set's
membership in another set, can be seen as yet another example (and
in this case there is no wiggling out of it by claiming that the
magicmethod/NON-builtin correspondence is a historical/legacy thing,
as the BDFL approved module sets.py, with just this usage, so very
recently).
I second the motion that function riter, with check for __riter__
and all, should live in module itertools. Reverse iteration is a
RARE need -- far rarer than copying or even pickling -- and there
is no real reason to burden __builtins__ with a function for it.
Is there any concern about the fact that a sequence of direction
changes requires building a new iterator object each time under this
proposal? I was going to implement my own bidirectional iteration
protocol by using a prev() method and use an adaptor object to swap
the meaning of next() and prev(). Python iterators (and GoF
iterators) seem poorly suited as pure sequence position indicators
because you can't reposition them without also accessing elements, so
maybe this is not such a serious issue... but my instincts tell me
that the identity of a bidirectional iterator object could be useful
if we allow the same one to be used in both directions.
FWIW, burdening builtins aside I consider the proposed syntax
for i in xrange(n).iter_backwards():
much uglier than
for i in reverse_view(xrange(n)):
but far superior to
for i in reverse(xrange(n)):
which implies in-place modification of the sequence.
Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Cheers,
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
"David Abrahams" <da**@boost-consulting.com> wrote in message
news:uk***********@boost-consulting.com...
[snip] FWIW, burdening builtins aside I consider the proposed syntax
for i in xrange(n).iter_backwards():
much uglier than
for i in reverse_view(xrange(n)):
but far superior to
for i in reverse(xrange(n)):
which implies in-place modification of the sequence.
How about
from itertools import ireverse
for i in ireverse(xrange(n)):
# suite
ireverse(), like imap(), izip(), etc., suggests that the operation is
iterative, and that no modification of the original sequence will be
performed. Others have suggested riter() (right iteration), in order to form
an association with the iter() builtin. As a matter of taste, I prefer
ireverse().
Sean
In article <WN*********************@news20.bellglobal.com>,
"Sean Ross" <sr***@connectmail.carleton.ca> wrote: How about
from itertools import ireverse
for i in ireverse(xrange(n)):
# suite
ireverse(), like imap(), izip(), etc., suggests that the operation is
iterative, and that no modification of the original sequence will be
performed. Others have suggested riter() (right iteration), in order to form
an association with the iter() builtin. As a matter of taste, I prefer
ireverse().
ireverse, like imap(), izip(), etc., suggests that the operation happens
without the memory overhead of copying the whole sequence into a list
before reversing it. Do you have some plan for how to do that e.g. with
simple generators? Or easy to understand explanation for which things
can be ireversed and which can't?
--
David Eppstein http://www.ics.uci.edu/~eppstein/
Univ. of California, Irvine, School of Information & Computer Science
David Eppstein <ep******@ics.uci.edu> wrote previously:
|ireverse, like imap(), izip(), etc., suggests that the operation happens
|without the memory overhead of copying the whole sequence into a list
|before reversing it. Do you have some plan for how to do that e.g. with
|simple generators? Or easy to understand explanation for which things
|can be ireversed and which can't?
Or even just a DECIDABLE explanation for which things can be 'ireversed'
and which can't[*].
Yours, Lulu...
[*] And if you -can- produce such an explanation, I can almost guarantee
you a Fields Medal, and probably a Nobel Prize also.
--
mertz@ _/_/_/_/_/_/_/ THIS MESSAGE WAS BROUGHT TO YOU BY:_/_/_/_/ v i
gnosis _/_/ Postmodern Enterprises _/_/ s r
..cx _/_/ MAKERS OF CHAOS.... _/_/ i u
_/_/_/_/_/ LOOK FOR IT IN A NEIGHBORHOOD NEAR YOU_/_/_/_/_/ g s
David Eppstein <ep******@ics.uci.edu> wrote previously:
|ireverse, like imap(), izip(), etc., suggests that the operation happens
|without the memory overhead of copying the whole sequence into a list
|before reversing it. Do you have some plan for how to do that e.g. with
|simple generators? Or easy to understand explanation for which things
|can be ireversed and which can't?
Or even just a DECIDABLE explanation for which things can be 'ireversed'
and which can't[*].
Yours, Lulu...
[*] And if you -can- produce such an explanation, I can almost guarantee
you a Fields Medal, and probably a Nobel Prize also.
--
mertz@ _/_/_/_/_/_/_/ THIS MESSAGE WAS BROUGHT TO YOU BY:_/_/_/_/ v i
gnosis _/_/ Postmodern Enterprises _/_/ s r
..cx _/_/ MAKERS OF CHAOS.... _/_/ i u
_/_/_/_/_/ LOOK FOR IT IN A NEIGHBORHOOD NEAR YOU_/_/_/_/_/ g s
David Abrahams wrote:
... Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Sure, but so is denying them, e.g., non-mutating methods such as .index()
and .count(). At least we're _consistently_ arbitrary and capricious!-)
Alex
"David Eppstein" <ep******@ics.uci.edu> wrote in message
news:ep****************************@news.service.u ci.edu...
[snip] ireverse, like imap(), izip(), etc., suggests that the operation happens
without the memory overhead of copying the whole sequence into a list
before reversing it. Do you have some plan for how to do that e.g. with
simple generators? Or easy to understand explanation for which things
can be ireversed and which can't?
Nope. Nor have I intimated I had such, nor that any would be forthcoming.
This is a naming suggestion. I am neither the originator of the proposal nor
a proponent of it, so I see no reason to formulate such a plan nor to give
an explanation of how it should be accomplished. I saw a name, and offered
another. That is all.
P1: "If I make this thing, I propose to call it 'ShomozzleBoB'."
P2: "Really? How about just BoB?"
P3: "Do you have a plan for how to do BoB?"
P2: "Huh?"
In article <dJ*********************@news20.bellglobal.com>,
"Sean Ross" <sr***@connectmail.carleton.ca> wrote: Nope. Nor have I intimated I had such, nor that any would be forthcoming.
This is a naming suggestion. I am neither the originator of the proposal nor
a proponent of it, so I see no reason to formulate such a plan nor to give
an explanation of how it should be accomplished. I saw a name, and offered
another. That is all.
P1: "If I make this thing, I propose to call it 'ShomozzleBoB'."
P2: "Really? How about just BoB?"
P3: "Do you have a plan for how to do BoB?"
P2: "Huh?"
Your proposed name has an implication for the semantics. I was
questioning the feasibility of the implied semantics. I don't think
you're let off the hook by the excuse that "it's only a name".
--
David Eppstein http://www.ics.uci.edu/~eppstein/
Univ. of California, Irvine, School of Information & Computer Science
On Fri, 26 Sep 2003 09:18:18 -0700, David Eppstein
<ep******@ics.uci.edu> wrote: In article <WN*********************@news20.bellglobal.com>,
"Sean Ross" <sr***@connectmail.carleton.ca> wrote:
How about
from itertools import ireverse
for i in ireverse(xrange(n)):
# suite
ireverse(), like imap(), izip(), etc., suggests that the operation is
iterative, and that no modification of the original sequence will be
performed. Others have suggested riter() (right iteration), in order to form
an association with the iter() builtin. As a matter of taste, I prefer
ireverse().
ireverse, like imap(), izip(), etc., suggests that the operation happens
without the memory overhead of copying the whole sequence into a list
before reversing it. Do you have some plan for how to do that e.g. with
simple generators? Or easy to understand explanation for which things
can be ireversed and which can't?
How about this - only support ireverse for sequences (identified using
len, and perhaps with a check for the keys method in case of mapping
types) - not for iterators at all.
Add an xrange_backward function to complement xrange rather than
trying to modify the result of xrange, and where an object can
logically support reverse iteration have a method or property that
returns an alternative generator/iterator.
At its simplest, something like...
class newlist (list) :
def ibackward (self) :
i = len(self)
while i > 0 :
i -= 1
yield i
....is no great burden for most built-in sequence types, and would
define a protocol for other types to follow where appropriate. It
doesn't need a big memory overhead.
Rather than define a separate xrange_backwards, it is worth noticing
that the built-in 'functions' xrange and enumerate actually seem to be
classes. We could add factory methods, set up to effectively give
alternate constructors, to allow syntax such as...
for i in xrange.backward (10) :
...
for i, j in enumerate.backward (seq) :
...
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
Alex Martelli <al***@aleax.it> writes: David Abrahams wrote:
... Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Sure, but so is denying them, e.g., non-mutating methods such as
.index() and .count().
Not IMO. Immutability is a very useful trait.
At least we're _consistently_ arbitrary and capricious!-)
Not in this case.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
"David Eppstein" <ep******@ics.uci.edu> wrote in message
news:ep****************************@news.service.u ci.edu... Your proposed name has an implication for the semantics.
Do the implied semantics of ireverse() differ from those implied by the
names iter_backwards(), reverse_view(), riter(), etc.?
Because, to me, they' re just different words for the same idea.
I was questioning the feasibility of the implied semantics.
Fine. Take it up with the person who proposed the idea.
I don't think you're let off the hook by the excuse that "it's only a
name".
I'm let off the "hook" because I was never on it. The responsibility for
answering your questions re: the feasability of this enterprise do not fall
to me, but to the proponent of the enterprise. It's like someone said they
where going to build a bridge from the mainland to Hawaii, and that they
would paint it green. And I said, "What about painting it red?" And you
asked me, "And how to you propose to build this bridge?". My answer is, "Uh,
why don't you ask the fellow over there who brought up the thing in the
first place. I'm just talking paint colours over here." I think your
questions are mis-directed. And, even if they are not, I don't have your
answers, I'm not going to try to come up with them, and I feel no
responsibility to do so.
I'll put it another way. I'm not interested in the semantics, nor in the
implementation. My interest is in the syntax. If this feature can be added
to the language (and I don't care whether it can or cannot be), then I would
prefer to read code like
for i in ireverse(xrange(n)):
than
for i in xrange(n).iter_backwards(n):
And there ends my superficial concerns in this matter. If my proposed syntax
implies more than the other syntax proposals, then I withdraw the
suggestion. I've no interest in arguing about whether those, or any other,
implications can be met. I'm talking sugar. You're talking substance. And I
don't care. If it can be done, let it be done, and give it pretty name. If
it can't be done, then it doesn't much matter what name you give it.
I hope were done here,
Sean
David Abrahams wrote: Alex Martelli <al***@aleax.it> writes:
David Abrahams wrote:
... Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Sure, but so is denying them, e.g., non-mutating methods such as
.index() and .count().
Not IMO. Immutability is a very useful trait.
Sure, very useful indeed -- and why does YO suggest that add such
NON-mutating methods would damage immutability in the LEAST...? At least we're _consistently_ arbitrary and capricious!-)
Not in this case.
Some amplification would be welcome, because the above comment
on immutability's usefulness is totally obscure to me in context.
Alex
On Fri, 26 Sep 2003 14:51:05 -0400, David Abrahams
<da**@boost-consulting.com> wrote: Alex Martelli <al***@aleax.it> writes:
David Abrahams wrote:
... Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Sure, but so is denying them, e.g., non-mutating methods such as
.index() and .count().
Not IMO. Immutability is a very useful trait.
Yes - and perfectly consistent with having *NON*-mutating methods such
as .index() and .count() ;-)
I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).
Actually, they even seem a little odd in list, to be honest. I'd have
functions, not necessarily even in __builtins__, which work on any
sequence. They just don't seem like everyday operations that should be
built into the object.
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
On Fri, 26 Sep 2003 14:42:28 -0400, "Sean Ross"
<sr***@connectmail.carleton.ca> wrote:
"David Eppstein" <ep******@ics.uci.edu> wrote in message
news:ep****************************@news.service. uci.edu... Your proposed name has an implication for the semantics.
Do the implied semantics of ireverse() differ from those implied by the
names iter_backwards(), reverse_view(), riter(), etc.?
Because, to me, they' re just different words for the same idea.
So far as I can see, it is having the function rather than the method
that creates the issues - that originated with David Abrahams post (at
least tracing back through this thread - it had been suggested
before).
However, your reply was implicitly supporting the use of a function
rather than a method. If 'ireverse' is imported from 'itertools' then
it cannot be a method of the object being reversed.
I think some wires have been slightly crossed - David Eppstein raised
a good point, but should perhaps have aimed it at David Abrahams.
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
On Fri, 2003-09-26 at 11:42, Sean Ross wrote: I was questioning the feasibility of the implied semantics.
Fine. Take it up with the person who proposed the idea.
You proposed it (hear me out). You are missing David's point. In
addition to whatever semantics a general reverse iterator might have, he
is saying that the itertools iterators have an implied additional
semantic of not (necessarily) needing to fully expand an iterable in
memory to operate on it, as a general ireverse() would probably have to
do. So, having ireverse potentially use up VAST quantities of memory,
in order to work, doesn't quite fit into the itertools philosophy.
That is the point he is making, and it is a specific comment on your (I
believe; others have probably proposed it as well) suggestion of an
ireverse() in itertools.
Now, my response to David's point is that currently, cycle() may also
require enough extra storage to remember an entire iterated sequence, so
the itertools philosophy is not a hard rule. Still, ireverse() would
imply some real devilry under the covers...
--
Chad Netzer
"Chad Netzer" <cn*****@sonic.net> wrote in message
news:ma*********************************@python.or g... On Fri, 2003-09-26 at 11:42, Sean Ross wrote:
I was questioning the feasibility of the implied semantics.
Fine. Take it up with the person who proposed the idea.
You proposed it (hear me out). You are missing David's point. In
addition to whatever semantics a general reverse iterator might have, he
is saying that the itertools iterators have an implied additional
semantic of not (necessarily) needing to fully expand an iterable in
memory to operate on it, as a general ireverse() would probably have to
do. So, having ireverse potentially use up VAST quantities of memory,
in order to work, doesn't quite fit into the itertools philosophy.
That is the point he is making, and it is a specific comment on your (I
believe; others have probably proposed it as well) suggestion of an
ireverse() in itertools.
Now, my response to David's point is that currently, cycle() may also
require enough extra storage to remember an entire iterated sequence, so
the itertools philosophy is not a hard rule. Still, ireverse() would
imply some real devilry under the covers...
--
Chad Netzer
Hi.
Thanks for clearing that up.
Yes, "others have ... proposed it as well". I was merely interested in the
appearance of the code, not the underlying implementation. I was not aware
that the name would imply more than the other suggestions simply because it
was housed in itertools. But then that's not really the point, I suppose:
The main point is the "implied additional semantic", regardless of the
function's location. Yes? OK. Fine. I was not concerned with semantics when
I made the suggestion, only sugar, so I couldn't see where David was coming
from. Sorry for the confusion.
Sean
Alex Martelli <al***@aleax.it> writes: David Abrahams wrote:
Alex Martelli <al***@aleax.it> writes:
David Abrahams wrote:
...
Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Sure, but so is denying them, e.g., non-mutating methods such as
.index() and .count().
Not IMO. Immutability is a very useful trait.
Sure, very useful indeed -- and why does YO suggest that add such
NON-mutating methods would damage immutability in the LEAST...?
Sorry, I misread your reply. I didn't know those were missing. At least we're _consistently_ arbitrary and capricious!-)
Not in this case.
Some amplification would be welcome, because the above comment
on immutability's usefulness is totally obscure to me in context.
My mistake; you were right as usual.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
Stephen Horne <$$$$$$$$$$$$$$$$$@$$$$$$$$$$$$$$$$$$$$.co.uk> writes: On Fri, 26 Sep 2003 14:51:05 -0400, David Abrahams
<da**@boost-consulting.com> wrote:
Alex Martelli <al***@aleax.it> writes:
David Abrahams wrote:
...
Also, the idea of denying tuples the ability to reverse iterate seems
arbitrary and capricious.
Sure, but so is denying them, e.g., non-mutating methods such as
.index() and .count().
Not IMO. Immutability is a very useful trait.
Yes - and perfectly consistent with having *NON*-mutating methods such
as .index() and .count() ;-)
Right. Oops, I misread.
I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).
I don't see why not.
Actually, they even seem a little odd in list, to be honest. I'd have
functions, not necessarily even in __builtins__, which work on any
sequence. They just don't seem like everyday operations that should be
built into the object.
It's probably just for optimization purposes.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
"Sean Ross" <sr***@connectmail.carleton.ca> writes: "David Eppstein" <ep******@ics.uci.edu> wrote in message
news:ep****************************@news.service.u ci.edu... Your proposed name has an implication for the semantics.
Do the implied semantics of ireverse() differ from those implied by the
names iter_backwards(), reverse_view(), riter(), etc.?
Because, to me, they' re just different words for the same idea.
Well, the only other place we have an 'i' prefix on a function name,
AFAIK, it means "inplace". I don't mind riter(); it has a nice
resonance with rbegin() in C++.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
[Raymond] It crashes when applied to an infinite iterator.
[Steve Holden] Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?
The PEP proposes adding a method only to a few objects
that don't have degenerate cases: list, str, unicode, xrange
and possibly tuple.
Raymond Hettinger
[Andrew Dalke] verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith("slack-version-")]
I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.
But that goes against the pretty good motto of "if it ain't broke,
don't touch it."
Between major releases, refactoring is fair game. Using newer
constructs like list comprehensions can make this code more
readable and maintainable.
Granted, XP-style tests are supposed to
help out with that, but I don't have a /usr/lib/setup where I
can test this.
Submit a patch, assign to me, and I'll test it.
> random.shuffle() uses for i in xrange(len(x)-1, 0, -1)
This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.
It is an important use case. The replacement code is:
for i in xrange(1,len(x)). iter_backwards()
Ahh, right. Didn't think about that. You should include that
wording in the PEP.
Okay, done!
Whenever the indices have any complexity, the forwards version,
followed by .iter_backwards() is, IMHO, much easier to mentally verify.
I agree. When I do need to go backwards I often forgot to get
both -1s in place. Ie, I'll do (n, 0) instead of (n-1, -1).
Well, don't forgot much and more, but there's some tension in
my mind as I worry if I got it correct or not.
That is a key motivation for the pep!
Also, I have a preference for creating something that
is as robust as possible. Making a builtin function that
doesn't apply to all objects; behaves badly with mappings;
and crashes with an infinite iterator is not my idea of robust.
The backup which uses __getitem__ and negative offsets
does work for all objects, behaves exactly as badly as the
existing iter, and doesn't crash with an infinite iterator, no?
1. Sure, all objects that define __getitem__ in a non-mapping
sense or that don't ignore the index (that was at one time a
common python practice).
2. With infinite iterators, running them to completion causes
the crash. They can be used in other generators, iterators,
or loops that have a break. This is not true with reverse
iterators where the very first call will cause the crash:
itertools.count().next() # this always works
riter(itertools.count()).next() # this always fails
That suggests a UserList.ListMixin might be useful, but it
isn't obvious to me that it would be.
You're the fourth person to suggest it.
If you want to know the issues involved, see my post at:
http://groups.google.com/gr*********...01.gnilink.net
Should dicts support riteritems, ritervalues?
Heck no! I only want reverse iteration for lists, strings, and xrange
where forward and reverse orders make some sense.
I also think that with these changes,
the PEP is pretty complete and solid.
Thanks for the detailed help getting it to that point.
Raymond
In article <dp*****************@nwrdny02.gnilink.net>,
"Raymond Hettinger" <vz******@verizon.net> wrote: [Raymond] It crashes when applied to an infinite iterator.
[Steve Holden] Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?
The PEP proposes adding a method only to a few objects
that don't have degenerate cases: list, str, unicode, xrange
and possibly tuple.
Speaking of which: str, int, etc., used to be functions, but they got
changed to be the initializers of types having those names. Why is
xrange still a function rather than the initializer of the xrange type?
The most visible difference of changing xrange to be the type of the
xrange objects would be more information shown in help(xrange), some of
which could be useful if we are talking about adding more methods to the
xrange objects.
--
David Eppstein http://www.ics.uci.edu/~eppstein/
Univ. of California, Irvine, School of Information & Computer Science
On Sat, 27 Sep 2003 00:08:22 -0700, David Eppstein
<ep******@ics.uci.edu> wrote: Speaking of which: str, int, etc., used to be functions, but they got
changed to be the initializers of types having those names. Why is
xrange still a function rather than the initializer of the xrange type?
"""
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information. xrange
<type 'xrange'>
"""
Looks like a type rather than a function to me, and the help appears
to be very good. Enumerate also seems to be a type with an
initialiser.
Are you using an earlier version?
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
David Abrahams wrote:
... I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).
I don't see why not.
The party line (not saying I necessarily agree, mind you -- just relating
it) is that tuples are meant as semantically heterogeneous containers
(semantically in that the items may happen to be the same type, but
their _meaning_ is disparate: e.g., consider the tuples used in modules
time or stat). So it doesn't really make sense to iterate on them, either;
it just happens to be possible because their items are accessed with []
with progressive naturals, and they have a len(). Associating _names_
to the indices makes more sense and indeed such pseudo-tuples are
starting to be supplied by the standard library (though not a general
mechanism for making your own, yet). Much like you can't iterate in
C++ on a std::pair<>, thus you "shouldn't" be able to iterate on a tuple.
"Frozen" versions of lists and dicts might make more sense -- and lose
only the MUTATING methods. I've seen Ruby experts claim that the
idea of freezing an object looks cool but in practice you end up almost
never using it -- they're speaking from experience, I guess, since in Ruby
they've had the possibility of freezing any object for ages. However, my
intuition (not supported by that much specific experience) makes me
yearn for such a feature...
Alex
Alex Martelli <al***@aleax.it> writes: David Abrahams wrote:
... I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).
I don't see why not.
The party line (not saying I necessarily agree, mind you -- just relating
it) is that tuples are meant as semantically heterogeneous containers
(semantically in that the items may happen to be the same type, but
their _meaning_ is disparate: e.g., consider the tuples used in modules
time or stat). So it doesn't really make sense to iterate on them, either;
it just happens to be possible because their items are accessed with []
with progressive naturals, and they have a len(). Associating _names_
to the indices makes more sense and indeed such pseudo-tuples are
starting to be supplied by the standard library (though not a general
mechanism for making your own, yet). Much like you can't iterate in
C++ on a std::pair<>, thus you "shouldn't" be able to iterate on a
tuple.
Well, (understanding that you don't nececessarily agree with the
above) you can in fact iterate on std::pair<T,T> with the usual C++
iterator protocol, and with a new mixed compile-time/runtime tuple
iterator protocol developed by Doug Gregor, iteration over
heterogeneous tuples is possible too. It certainly is desirable to be
able to do that; it's a real need that has come up in practice.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
David Abrahams wrote:
... Well, (understanding that you don't nececessarily agree with the
above) you can in fact iterate on std::pair<T,T> with the usual C++
iterator protocol,
You mean there's an std::pair::begin etc?! OK, I guess I'm even
rustier on standard C++ than I thought I was -- I could have SWORN
there wasn't. (Std chapter & verse pls? I plan to win bets based
on this tidbit...!-). So I guess the lack of those in gcc is a
breach of the Standard on gcc's part...?
and with a new mixed compile-time/runtime tuple
iterator protocol developed by Doug Gregor, iteration over
heterogeneous tuples is possible too. It certainly is desirable to be
able to do that; it's a real need that has come up in practice.
So, given an arbitrary struct, with fields (generally) of different types,
you can iterate field by field? That's basically what the "heterogeneous
tuple" is supposed to be equivalent to, in the "party line" I was relating
(although the names to go with the fields are only present in a FEW such
tuples, such as those returned by modules time and stat, as of now; more
general tuples still haven't grown the ability to access fields by name,
in Python).
My partial dissent comes from feeling the need for "frozen/immutable
lists" and the fact that tuples are often used as such (including goofy
immutable representations of dicts, e.g. via tuple(thedict.iteritems()),
and the like). If tuples cannot be thought of as immutable lists, then
(I think) we need "immutable/hashable/frozen" lists by other means.
(I need to say "I think" because, as I quoted, Ruby experts claim that
the "anyobject.freeze" feature they do have isn't actually as useful
as it SEEMS it should be -- though I'm not sure I understand why, yet).
Alex
On Sat, 27 Sep 2003 15:22:37 GMT, Alex Martelli <al***@aleax.it>
wrote: David Abrahams wrote:
... Well, (understanding that you don't nececessarily agree with the
above) you can in fact iterate on std::pair<T,T> with the usual C++
iterator protocol,
You mean there's an std::pair::begin etc?! OK, I guess I'm even
rustier on standard C++ than I thought I was -- I could have SWORN
there wasn't. (Std chapter & verse pls? I plan to win bets based
on this tidbit...!-). So I guess the lack of those in gcc is a
breach of the Standard on gcc's part...?
Somehow I think David is mistaken here - I cannot believe that
dereferencing an iterator returns a different datatype depending on
which item it happens to point to at runtime in statically typed C++,
and without that ability to dereference the iterator (1) I cannot see
the point of iterating through a pair, and (2) the 'iterator' would
not be a true iterator as C++ iterators have to comply with one of a
set of standard protocols (forward, bidirectional, random etc) which
all include subscripting.
Of course you can iterate through a container that holds std::pair
objects - you do that every time you iterate through an std::map - but
that isn't the same thing.
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
On Sat, 27 Sep 2003 16:35:10 +0100, Stephen Horne
<$$$$$$$$$$$$$$$$$@$$$$$$$$$$$$$$$$$$$$.co.uk> wrote: set of standard protocols (forward, bidirectional, random etc) which
all include subscripting.
Oops - I've got subscripting on the mind just now. I meant
dereferencing of course.
--
Steve Horne
steve at ninereeds dot fsnet dot co dot uk
Alex Martelli <al***@aleax.it> writes: David Abrahams wrote:
... Well, (understanding that you don't nececessarily agree with the
above) you can in fact iterate on std::pair<T,T> with the usual C++
iterator protocol,
You mean there's an std::pair::begin etc?!
No, that's an entirely different question. I can build an iterator
which iterates over pair<T,T>.
OK, I guess I'm even rustier on standard C++ than I thought I was --
I could have SWORN there wasn't. (Std chapter & verse pls? I plan
to win bets based on this tidbit...!-). So I guess the lack of
those in gcc is a breach of the Standard on gcc's part...?
No, nothing like that.
and with a new mixed compile-time/runtime tuple iterator protocol
developed by Doug Gregor, iteration over heterogeneous tuples is
possible too. It certainly is desirable to be able to do that;
it's a real need that has come up in practice.
So, given an arbitrary struct, with fields (generally) of different
types, you can iterate field by field?
Ah, no. A tuple in C++ is a different thing, and much more like a
Python tuple: http://www.boost.org/libs/tuple
That's basically what the "heterogeneous tuple" is supposed to be
equivalent to, in the "party line" I was relating
Well, that's just nutty. We can make a much better analogy to a
struct in Python with the usual class hacks. But even then, we can
arrange to iterate the attributes.
(although the names to go with the fields are only present in a FEW
such tuples, such as those returned by modules time and stat,
Whaa?? Are they subclassing tuple? Nope, time.struct_time is not
even a tuple. It's just the struct hack with a tuple-like repr()
time.localtime()
(2003, 9, 27, 12, 10, 43, 5, 270, 1) type(time.localtime())
<type 'time.struct_time'> time.struct_time.__bases__
(<type 'object'>,)
as of now; more general tuples still haven't grown the ability to
access fields by name, in Python).
My partial dissent comes from feeling the need for "frozen/immutable
lists" and the fact that tuples are often used as such (including goofy
immutable representations of dicts, e.g. via tuple(thedict.iteritems()),
and the like). If tuples cannot be thought of as immutable lists, then
(I think) we need "immutable/hashable/frozen" lists by other means.
I agree completely.
(I need to say "I think" because, as I quoted, Ruby experts claim that
the "anyobject.freeze" feature they do have isn't actually as useful
as it SEEMS it should be -- though I'm not sure I understand why, yet).
Immutability is extremely useful whenever you have inplace operations
like +=, because it guarantees value stability for other references
to the same object.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
Stephen Horne <$$$$$$$$$$$$$$$$$@$$$$$$$$$$$$$$$$$$$$.co.uk> writes: On Sat, 27 Sep 2003 15:22:37 GMT, Alex Martelli <al***@aleax.it>
wrote:
David Abrahams wrote:
... Well, (understanding that you don't nececessarily agree with the
above) you can in fact iterate on std::pair<T,T> with the usual C++
iterator protocol,
You mean there's an std::pair::begin etc?! OK, I guess I'm even
rustier on standard C++ than I thought I was -- I could have SWORN
there wasn't. (Std chapter & verse pls? I plan to win bets based
on this tidbit...!-). So I guess the lack of those in gcc is a
breach of the Standard on gcc's part...?
Somehow I think David is mistaken here - I cannot believe that
dereferencing an iterator returns a different datatype depending on
which item it happens to point to at runtime in statically typed
C++,
You didn't read carefully enough: I said std::pair<T,T>, not
std::pair<T,U>.
and without that ability to dereference the iterator (1) I cannot see
the point of iterating through a pair, and (2) the 'iterator' would
not be a true iterator as C++ iterators have to comply with one of a
set of standard protocols (forward, bidirectional, random etc) which
all include subscripting.
I'm pretty well familiar with those protocols - I've been working on
the C++ standards committee since 1997 and have written several
related proposals, c.f. http://tinyurl.com/ovpe.
Of course you can iterate through a container that holds std::pair
objects - you do that every time you iterate through an std::map - but
that isn't the same thing.
No, definitely not.
--
Dave Abrahams
Boost Consulting www.boost-consulting.com
"David Abrahams" <da**@boost-consulting.com> wrote in message
news:ul***********@boost-consulting.com... Alex Martelli <al***@aleax.it> writes:
So, given an arbitrary struct, with fields (generally) of different
types, you can iterate field by field?
Ah, no. A tuple in C++ is a different thing, and much more like a
Python tuple: http://www.boost.org/libs/tuple
That's basically what the "heterogeneous tuple" is supposed to be
equivalent to, in the "party line" I was relating
Well, that's just nutty. We can make a much better analogy to a
struct in Python with the usual class hacks. But even then, we can
arrange to iterate the attributes.
(although the names to go with the fields are only present in a FEW
such tuples, such as those returned by modules time and stat,
Whaa?? Are they subclassing tuple? Nope, time.struct_time is not
even a tuple. It's just the struct hack with a tuple-like repr()
time.localtime() (2003, 9, 27, 12, 10, 43, 5, 270, 1) type(time.localtime()) <type 'time.struct_time'> time.struct_time.__bases__
(<type 'object'>,)
Not exactly. It's an object that can be subscripted like a
sequence, so it looks like a tuple for the most common
use cases. I haven't looked at the code, so I don't
know how far they went in putting in the remainder of
tuple behavior, though.
The subscripting and slicing is the key point here, though.
That's what makes it backward compatible. It's not an
extended tuple, and I doubt if it has most of the tuple
methods.
Frankly, I prefer to call it a data object, rather than
a "struct hack." Calling it a data object means that
it might grow other useful behaviors in the future. I'm
not sure about time, but I could appreciate methods
in stat that apply the access and modify dates to another
file in one operation.
as of now; more general tuples still haven't grown the ability to
access fields by name, in Python).
And they most likely won't. It's easy enough to create
a data object that implements the basic part of the sequence
protocol.
John Roth
--
Dave Abrahams
Boost Consulting
www.boost-consulting.com This discussion thread is closed Replies have been disabled for this discussion. Similar topics
72 posts
views
Thread by Raymond Hettinger |
last post: by
|
3 posts
views
Thread by Christoph Becker-Freyseng |
last post: by
|
14 posts
views
Thread by Marcin Ciura |
last post: by
|
8 posts
views
Thread by Micah Elliott |
last post: by
|
2 posts
views
Thread by Bryan Olson |
last post: by
|
18 posts
views
Thread by pocmatos |
last post: by
|
3 posts
views
Thread by Talin |
last post: by
|
4 posts
views
Thread by Tony Lownds |
last post: by
| | | | | | | | | | | | | | | | | | | |
