By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
 440,095 Members | 1,658 Online
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,095 IT Pros & Developers. It's quick & easy.

# a few extensions for the itertools

 P: n/a I wrote a few functions which IMHO are missing in python(s itertools). You can download them here: http://sourceforge.net/project/showf...kage_id=212104 A short description to all the functions: icmp(iterable1, iterable2) -integer Return negative if iterable1 < iterable2, zero if iterable1 == iterable1, positive if iterable1 iterable1. isum(iterable, start=0) -value Returns the sum of the elements of a iterable plus the value of parameter 'start'. When the iterable is empty, returns start. iproduct(iterable, start=0) -value Returns the product of the elements of a iterable times the value of parameter 'start'. When the iterable is empty, returns start. forall(predicate, iterable, default=True) -bool Returns True, when for all elements x in iterable predicate(x) is True. When the iterable is empty, returns default. forany(predicate, iterable, default=False) -bool Returns True, when for any element x in iterable predicate(x) is True. When the iterable is empty, returns default. take(n,iterable) -iterator returns a iterator over the first n elements of the iterator drop(n,iterable) -iterable drops the first n elemetns of iterable and return a iterator over the rest heads(iterable) -iterator over all heads example: for head in heads([1,2,3,4,5,6,7,8,9]): print head output: [] [1] [1, 2] [1, 2, 3] [1, 2, 3, 4] [1, 2, 3, 4, 5] [1, 2, 3, 4, 5, 6] [1, 2, 3, 4, 5, 6, 7] [1, 2, 3, 4, 5, 6, 7, 8] [1, 2, 3, 4, 5, 6, 7, 8, 9] tails(iterable) -iterator over all tails example: for tail in tails([1,2,3,4,5,6,7,8,9]): print tail output: [1, 2, 3, 4, 5, 6, 7, 8, 9] [2, 3, 4, 5, 6, 7, 8, 9] [3, 4, 5, 6, 7, 8, 9] [4, 5, 6, 7, 8, 9] [5, 6, 7, 8, 9] [6, 7, 8, 9] [7, 8, 9] [8, 9] [9] [] fcain(funct,*functs) -function(...,***) fcain(f1,f2,...,fn)(*args,*kwargs) equals f1(f2(...fn(*args,*kwargs))) Nov 19 '06 #1
Share this Question
23 Replies

 P: n/a "Mathias Panzenboeck" I wrote a few functions which IMHO are missing in python(s itertools). You can download them here: http://sourceforge.net/project/showf...kage_id=212104 A short description to all the functions: Just a couple of questions: iproduct(iterable, start=0) -value Returns the product of the elements of a iterable times the value of parameter 'start'. When the iterable is empty, returns start. Wouldn't 1 be a better default value for start? forall(predicate, iterable, default=True) -bool Returns True, when for all elements x in iterable predicate(x) is True. When the iterable is empty, returns default. forany(predicate, iterable, default=False) -bool Returns True, when for any element x in iterable predicate(x) is True. When the iterable is empty, returns default. How are these different from all and any in Python 2.5? -- Paul Nov 19 '06 #2

 P: n/a On Sun, 19 Nov 2006 21:35:24 +0100, Mathias Panzenboeck wrote: I wrote a few functions which IMHO are missing in python(s itertools). You can download them here: http://sourceforge.net/project/showf...kage_id=212104 A short description to all the functions: icmp(iterable1, iterable2) -integer Return negative if iterable1 < iterable2, zero if iterable1 == iterable1, positive if iterable1 iterable1. What does it mean for an iterable to be less than another iterable? That it has fewer items? How do these two iterables compare? iter([1, 2, None, "foo", 3+2j]) def ones(): while 1: yield 1 Which is smaller? isum(iterable, start=0) -value Returns the sum of the elements of a iterable plus the value of parameter 'start'. When the iterable is empty, returns start. You mean just like the built-in sum()? >>sum(xrange(12), 1000) 1066 iproduct(iterable, start=0) -value Returns the product of the elements of a iterable times the value of parameter 'start'. When the iterable is empty, returns start. If I recall, product() was requested about the same time that sum() was introduced, and Guido rejected it as a built-in because it was really only useful for calculating geometric means, and it is easy to do if you need it: def product(it, start=1): # default value of 1 is more sensible than 0 # 1 is the multiplicative identity p = start for x in it: p *= x return p forall(predicate, iterable, default=True) -bool Returns True, when for all elements x in iterable predicate(x) is True. When the iterable is empty, returns default. forany(predicate, iterable, default=False) -bool Returns True, when for any element x in iterable predicate(x) is True. When the iterable is empty, returns default. I vaguely recall plans for all() and any() builtins -- perhaps for Python 2.5? take(n,iterable) -iterator returns a iterator over the first n elements of the iterator Just like itertools.islice(iterable, n). >>list(itertools.islice(xrange(10), 5)) [0, 1, 2, 3, 4] drop(n,iterable) -iterable drops the first n elemetns of iterable and return a iterator over the rest Just like itertools.islice(iterable, n, None) >>list(itertools.islice(xrange(20), 15, None)) [15, 16, 17, 18, 19] (Aside: I think islice would be so much cleaner if it took keyword arguments.) heads(iterable) -iterator over all heads tails(iterable) -iterator over all tails What would you use these for? fcain(funct,*functs) -function(...,***) fcain(f1,f2,...,fn)(*args,*kwargs) equals f1(f2(...fn(*args,*kwargs))) The usual term for this is function composition. -- Steven. Nov 19 '06 #3

 P: n/a Paul McGuire wrote: "Mathias Panzenboeck"

 P: n/a At Sunday 19/11/2006 17:35, Mathias Panzenboeck wrote: >I wrote a few functions which IMHO are missing in python(s itertools).You can download them here:http://sourceforge.net/project/showf...kage_id=212104isum(iterable, start=0) -value Returns the sum of the elements of a iterable plus the value of parameter 'start'. When the iterable is empty, returns start. Isn't the same as the builtin sum? >iproduct(iterable, start=0) -value As others said, start should be 1 >fcain(funct,*functs) -function(...,***) fcain(f1,f2,...,fn)(*args,*kwargs) equals f1(f2(...fn(*args,*kwargs))) I don't understand it, nor even the signature. Perhaps it tries to be "fchain", function composition? But what has it to do with iterables? -- Gabriel Genellina Softlab SRL __________________________________________________ Correo Yahoo! Espacio para todos tus mensajes, antivirus y antispam ˇgratis! ˇAbrí tu cuenta ya! - http://correo.yahoo.com.ar Nov 20 '06 #5

 P: n/a Steven D'Aprano wrote: A short description to all the functions: icmp(iterable1, iterable2) -integer Return negative if iterable1 < iterable2, zero if iterable1 == iterable1, positive if iterable1 iterable1. What does it mean for an iterable to be less than another iterable? That it has fewer items? How do these two iterables compare? iter([1, 2, None, "foo", 3+2j]) def ones(): while 1: yield 1 Which is smaller? Haven't checked the specific implementation, but I would expect it to behave like sequences of the same type, i.e. first compare the first elements of the iterables; if they are equal compare the second elements, and so on, until the first inequality or until the shorter one ends. In your example, the second iterable is smaller. Needless to say, you'd better not compare an infinite iterable with itself ;-) drop(n,iterable) -iterable drops the first n elemetns of iterable and return a iterator over the rest Just like itertools.islice(iterable, n, None) >list(itertools.islice(xrange(20), 15, None)) [15, 16, 17, 18, 19] (Aside: I think islice would be so much cleaner if it took keyword arguments.) How about slice notation ? I just posted in the Cookbook an OO wrapper of itertools that, among other functions, uses slice notation for islice and "+" for chain. Admittedly, my proposal in the py-3k list to make iter() return itertools-enabled iterators was overwhelmingly shot down, but I still like it anyway. FWIW, here's the Cookbook link: http://aspn.activestate.com/ASPN/Coo.../Recipe/498272 George Nov 20 '06 #6

 P: n/a Mathias Panzenboeck

 P: n/a Gabriel Genellina wrote: At Sunday 19/11/2006 17:35, Mathias Panzenboeck wrote: >I wrote a few functions which IMHO are missing in python(s itertools).You can download them here:http://sourceforge.net/project/showf...kage_id=212104 isum(iterable, start=0) -value Returns the sum of the elements of a iterable plus the value of parameter 'start'. When the iterable is empty, returns start. Isn't the same as the builtin sum? No, because the builtin sum want's a list. This can also handle any kind of iterable, so this would work: isum(i**2 for i in xrange(100)) sum would need firs the whole list to be generated: sum([i**2 for i in xrange(100)]) >iproduct(iterable, start=0) -value As others said, start should be 1 Indeed. Can't believe I made that mistake... the mistake is only in the documentation. :) >fcain(funct,*functs) -function(...,***) fcain(f1,f2,...,fn)(*args,*kwargs) equalsf1(f2(...fn(*args,*kwargs))) I don't understand it, nor even the signature. Perhaps it tries to be "fchain", function composition? But what has it to do with iterables? Ups, missed out the 'h'. (Also only in the documentation.) It's like the . operator in haskell: fchain(f,g,h) is the same like lambda *args,**kwargs: f(g(h(*args,**kwargs))) Nov 20 '06 #8

 P: n/a Duncan Booth wrote: Mathias Panzenboeck take(n,iterable) -iterator returns a iterator over the first n elements of the iterator Isn't this just the same as itertools.islice(iterable, n) ? ok, that's true. >drop(n,iterable) -iterable drops the first n elemetns of iterable and return a iterator over the rest and this looks to be the same as itertools.islice(iterable, n, None) same here. Can you give use cases for 'heads' and 'tails'? I'm curious why you would want them. I use them in haskell all the time. But in haskell the lists are all "generators". In haskell you would implement naive string-search like this: import List findIndex (isPrefixOf "bla") (tails "dfvbdbblaesre") Nov 20 '06 #9

 P: n/a Steven D'Aprano wrote: On Sun, 19 Nov 2006 21:35:24 +0100, Mathias Panzenboeck wrote: >I wrote a few functions which IMHO are missing in python(s itertools).You can download them here:http://sourceforge.net/project/showf...kage_id=212104A short description to all the functions:icmp(iterable1, iterable2) -integer Return negative if iterable1 < iterable2, zero if iterable1 == iterable1, positive if iterable1 iterable1. What does it mean for an iterable to be less than another iterable? That it has fewer items? How do these two iterables compare? iter([1, 2, None, "foo", 3+2j]) def ones(): while 1: yield 1 Which is smaller? it's like cmp on lists, but on iterables. [1,2,3] < [1,2,4] [1,2,3] < [1,2,3,0] .... > >isum(iterable, start=0) -value Returns the sum of the elements of a iterable plus the value of parameter 'start'. When the iterable is empty, returns start. You mean just like the built-in sum()? No, because the builtin sum can't handle iterables other than lists. Or dose it? Hmm, maby it dose since any new version and I didn't mention it. >>>sum(xrange(12), 1000) 1066 >iproduct(iterable, start=0) -value Returns the product of the elements of a iterable times the value of parameter 'start'. When the iterable is empty, returns start. If I recall, product() was requested about the same time that sum() was introduced, and Guido rejected it as a built-in because it was really only useful for calculating geometric means, and it is easy to do if you need it: def product(it, start=1): # default value of 1 is more sensible than 0 # 1 is the multiplicative identity p = start for x in it: p *= x return p >forall(predicate, iterable, default=True) -bool Returns True, when for all elements x in iterable predicate(x) is True. When the iterable is empty, returns default. forany(predicate, iterable, default=False) -bool Returns True, when for any element x in iterable predicate(x) is True. When the iterable is empty, returns default. I vaguely recall plans for all() and any() builtins -- perhaps for Python 2.5? all() and any() don't get predicate functions as arguments. > >take(n,iterable) -iterator returns a iterator over the first n elements of the iterator Just like itertools.islice(iterable, n). >>>list(itertools.islice(xrange(10), 5)) [0, 1, 2, 3, 4] ok, ok, ok. I have overseen that. > >drop(n,iterable) -iterable drops the first n elemetns of iterable and return a iterator over the rest Just like itertools.islice(iterable, n, None) >>>list(itertools.islice(xrange(20), 15, None)) [15, 16, 17, 18, 19] (Aside: I think islice would be so much cleaner if it took keyword arguments.) >heads(iterable) -iterator over all headstails(iterable) -iterator over all tails What would you use these for? >fcain(funct,*functs) -function(...,***) fcain(f1,f2,...,fn)(*args,*kwargs) equals f1(f2(...fn(*args,*kwargs))) The usual term for this is function composition. Nov 20 '06 #10

 P: n/a Carl Banks wrote: Paul McGuire wrote: >"Mathias Panzenboeck" >I wrote a few functions which IMHO are missing in python(s itertools).You can download them here:http://sourceforge.net/project/showf...kage_id=212104A short description to all the functions: Just a couple of questions: >>iproduct(iterable, start=0) -valueReturns the product of the elements of a iterabletimes the value of parameter 'start'. When theiterable is empty, returns start. Wouldn't 1 be a better default value for start? I concur; start should default to 1. Bug in the documentation, not in the function. ;) Nov 20 '06 #11

 P: n/a Mathias Panzenboeck wrote: No, because the builtin sum want's a list. the one in Python doesn't: >>def g(): .... for i in range(3): .... print "yield", i .... yield i .... >>sum(g()) yield 0 yield 1 yield 2 3 Nov 20 '06 #12

 P: n/a Mathias Panzenboeck >sum(i**2 for i in xrange(20000000)) 2666666466666670000000L seems to work fine, and judging by the memory usage it pretty obviously doesn't create an intermediate list. Nov 20 '06 #13

 P: n/a Mathias Panzenboeck wrote: No, because the builtin sum can't handle iterables other than lists. Or dose it? Hmm, maby it dose since any new version and I didn't mention it. sum() was added in 2.3, and has always supported arbitrary iterables. Nov 20 '06 #14

 P: n/a On Mon, 20 Nov 2006 09:54:41 +0100, Mathias Panzenboeck wrote: Steven D'Aprano wrote: >On Sun, 19 Nov 2006 21:35:24 +0100, Mathias Panzenboeck wrote: >>I wrote a few functions which IMHO are missing in python(s itertools).You can download them here:http://sourceforge.net/project/showf...kage_id=212104A short description to all the functions:icmp(iterable1, iterable2) -integer Return negative if iterable1 < iterable2, zero if iterable1 == iterable1, positive if iterable1 iterable1. What does it mean for an iterable to be less than another iterable? Thatit has fewer items? How do these two iterables compare?iter([1, 2, None, "foo", 3+2j])def ones(): while 1: yield 1Which is smaller? it's like cmp on lists, but on iterables. [1,2,3] < [1,2,4] [1,2,3] < [1,2,3,0] But that meaningless, as far as I can see. Lists and iterators aren't the same thing. A list is a collection; an iterator is not, but it can be accumulated into a collection. If equality is meaningful for an object, you should be able to test for equality without changing the object. But that isn't true for iterators. Worse, because comparing an iterator consumes items, you can easily get crazy results like the following: >>L = [1, 2, 3, 4, 5]def it(): .... yield 5; yield 0; yield 0; yield 0; yield 0 .... >>it = it() Now, we can compare the first item of L with the first item of it: >>L[0] < it.next() # if True, L < it True So L must be less than it, right? Let's print both objects out in full to check: >>print L [1, 2, 3, 4, 5] >>print list(it) # if L < it, it must be L [0, 0, 0, 0] Oops. I think the GENERAL concept of comparison between iterators is meaningless. A lazy comparison between the items of an iterator and some other iterable may be a useful thing to do, but as a general concept, saying that an iterator compares bigger or smaller or equal to something else doesn't make sense, since the mere fact that you make that comparison will change the iterator. -- Steven. Nov 20 '06 #15

 P: n/a Steven D'Aprano wrote: Worse, because comparing an iterator consumes items, you can easily get crazy results like the following: [snip] However, that doesn't stop the "in" operator: >>a = (x for x in xrange(5))3 in a True >>list(a) [4] I'm not sure if I like it, but at least it's probably a lot more useful than icmp. Carl Banks Nov 20 '06 #16

 P: n/a Mathias Panzenboeck wrote: >>forall(predicate, iterable, default=True) -bool Returns True, when for all elements x in iterable predicate(x) is True. When the iterable is empty, returns default. forany(predicate, iterable, default=False) -bool Returns True, when for any element x in iterable predicate(x) is True. When the iterable is empty, returns default. I vaguely recall plans for all() and any() builtins -- perhaps for Python2.5? all() and any() don't get predicate functions as arguments. all(predicate(x) for x in iterable) any(predicate(x) for x in iterable) -- Roberto Bonvallet Nov 20 '06 #17

 P: n/a Steven D'Aprano wrote: On Mon, 20 Nov 2006 09:54:41 +0100, Mathias Panzenboeck wrote: >Steven D'Aprano wrote: >>On Sun, 19 Nov 2006 21:35:24 +0100, Mathias Panzenboeck wrote:I wrote a few functions which IMHO are missing in python(s itertools).You can download them here:http://sourceforge.net/project/showf...kage_id=212104A short description to all the functions:icmp(iterable1, iterable2) -integer Return negative if iterable1 < iterable2, zero if iterable1 == iterable1, positive if iterable1 iterable1.What does it mean for an iterable to be less than another iterable? Thatit has fewer items? How do these two iterables compare?iter([1, 2, None, "foo", 3+2j])def ones(): while 1: yield 1Which is smaller? it's like cmp on lists, but on iterables.[1,2,3] < [1,2,4][1,2,3] < [1,2,3,0] But that meaningless, as far as I can see. Lists and iterators aren't the same thing. A list is a collection; an iterator is not, but it can be accumulated into a collection. If equality is meaningful for an object, you should be able to test for equality without changing the object. But that isn't true for iterators. Worse, because comparing an iterator consumes items, you can easily get crazy results like the following: >>>L = [1, 2, 3, 4, 5]def it(): ... yield 5; yield 0; yield 0; yield 0; yield 0 ... >>>it = it() Now, we can compare the first item of L with the first item of it: >>>L[0] < it.next() # if True, L < it True So L must be less than it, right? Let's print both objects out in full to check: >>>print L [1, 2, 3, 4, 5] >>>print list(it) # if L < it, it must be L [0, 0, 0, 0] Oops. I think the GENERAL concept of comparison between iterators is meaningless. A lazy comparison between the items of an iterator and some other iterable may be a useful thing to do, but as a general concept, saying that an iterator compares bigger or smaller or equal to something else doesn't make sense, since the mere fact that you make that comparison will change the iterator. I see. Yes, in general you are right, but I implemented it to use it for that (and similar things): icmp(open("foo.txt"),open("bar.txt")) Thats cool, I think. :) panzi Nov 20 '06 #18

 P: n/a Roberto Bonvallet wrote: > all(predicate(x) for x in iterable) any(predicate(x) for x in iterable) Very true, but 2.4 is still very common and forall(perdicate,iterable) is less to write. ;) (And haskell has both, to. all == and, forall == all, ...) Well, but I think you are right. Don't reinvent the wheel, I guess. panzi Nov 20 '06 #19

 P: n/a Duncan Booth wrote: Mathias Panzenboeck No, because the builtin sum want's a list. This can also handle anykind of iterable, so this would work:isum(i**2 for i in xrange(100))sum would need firs the whole list to be generated:sum([i**2 for i in xrange(100)]) Really? >>>sum(i**2 for i in xrange(20000000)) 2666666466666670000000L seems to work fine, and judging by the memory usage it pretty obviously doesn't create an intermediate list. Very strange. I must have made some strange error. I tried that (more than once) and it failed. Don't know why. But now it works here, too. What did I write, when it failed? Nov 20 '06 #20

 P: n/a >>No, because the builtin sum want's a list. This can also >>handle any kind of iterable, so this would work:isum(i**2 for i in xrange(100))sum would need firs the whole list to be generated:sum([i**2 for i in xrange(100)]) Really? >>>>sum(i**2 for i in xrange(20000000)) 2666666466666670000000Lseems to work fine, and judging by the memory usage itpretty obviously doesn't create an intermediate list. Very strange. I must have made some strange error. I tried that (more than once) and it failed. Don't know why. But now it works here, too. What did I write, when it failed? Could you have been trying it on a version of python <2.4? tim@oblique:~\$ python2.3 Python 2.3.5 (#2, Sep 4 2005, 22:01:42) [GCC 3.3.5 (Debian 1:3.3.5-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>sum(i**2 for i in xrange(100)) File "", line 1 sum(i**2 for i in xrange(100)) ^ SyntaxError: invalid syntax tim@oblique:~\$ python2.4 Python 2.4.1 (#2, May 5 2005, 11:32:06) [GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>sum(i**2 for i in xrange(100)) 328350 -tkc Nov 20 '06 #21

 P: n/a Mathias Panzenboeck wrote: It's like the . operator in haskell: fchain(f,g,h) is the same like lambda *args,**kwargs: f(g(h(*args,**kwargs))) I used this once: composedfun = lambda arg: reduce(lambda x, f: f(x), funcs, arg) so I can see the potential need (it is not terribly obvious in short- or long- form). If this belongs anywhere, though, it is functools, not itertools. -Mike Nov 20 '06 #22

 P: n/a Klaas wrote: Mathias Panzenboeck wrote: >It's like the . operator in haskell:fchain(f,g,h) is the same like lambda *args,**kwargs: f(g(h(*args,**kwargs))) I used this once: composedfun = lambda arg: reduce(lambda x, f: f(x), funcs, arg) Nice! Certainly more beautiful than my version. so I can see the potential need (it is not terribly obvious in short- or long- form). If this belongs anywhere, though, it is functools, not itertools. Yes, got in there by accident. -Mike Nov 21 '06 #23

 P: n/a Klaas wrote: Mathias Panzenboeck wrote: It's like the . operator in haskell: fchain(f,g,h) is the same like lambda *args,**kwargs: f(g(h(*args,**kwargs))) I used this once: composedfun = lambda arg: reduce(lambda x, f: f(x), funcs, arg) so I can see the potential need (it is not terribly obvious in short- or long- form). If this belongs anywhere, though, it is functools, not itertools. -Mike Since you are going to exploit the universality of fold [1] here. Maybe we should also take the next step beyond itertools ( i.e. generatortools ) into account? http://aspn.activestate.com/ASPN/Coo.../Recipe/498264 Of course I've nothing against a nice interface for the most convenient usecases just like the proposed isum that could be expressed by greduce. [1] http://www.cs.nott.ac.uk/~gmh/fold.pdf Nov 21 '06 #24

### This discussion thread is closed

Replies have been disabled for this discussion.