Fate of itertools.dropwhile() and itertools.takewhile()

[Raymond Hettinger]

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

* I'm concerned that use cases for the two functions are uncommon and
can obscure code rather than clarify it.

* I originally added them to itertools because they were found in
other functional languages and because it seemed like they would serve
basic building blocks in combination with other itertools allow
construction of a variety of powerful, high-speed iterators. The
latter may have been a false hope -- to date, I've not seen good
recipes that depend on either function.

* If an always true or always false predicate is given, it can be hard
to break-out of the function once it is running.

* Both functions seem simple and basic until you try to explain them
to someone else. Likewise, when reading code containing dropwhile(),
I don't think it is self-evident that dropwhile() may have a lengthy
start-up time.

* Since itertools are meant to be combined together, the whole module
becomes easier to use if there are fewer tools to choose from.

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

Raymond

[Istvan Albert]

On Dec 29, 6:10 pm, Raymond Hettinger <[e-mail]:

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

first off, the itertools module is amazing, thanks for creating it. It
changed the way I think about programming. In fact nowadays I start
all my programs with:

from itertools import *

which may not be the best form, but I got tired of importing every
single function individually or writing out the module name.

Now I never needed the dropwhile() and takewhile() functions, but that
may not mean much. For quite a while I never needed the repeat()
function either. It even looked nonsensical to have an iterator that
simply repeats the same thing over and over. One day I had to solve a
problem that needed repeat() and made me really understand what it was
for and got to marvel at a just how neat the solution was.

i.

[Steven D'Aprano]

On Sat, 29 Dec 2007 15:10:24 -0800, Raymond Hettinger wrote:

* Both functions seem simple and basic until you try to explain them to
someone else.

Oh I don't know about that. The doc strings seem to do an admirable job
to me. Compared to groupby(), the functions are simplicity themselves.

Likewise, when reading code containing dropwhile(), I
don't think it is self-evident that dropwhile() may have a lengthy
start-up time.

*scratches head in confusion*

It isn't? I can understand somebody *under*estimating the start-up time
(perhaps because they overestimate how quickly dropwhile() can iterate
through the items). But surely it is self-evident that a function which
drops items has to drop the items before it can start returning?

* Since itertools are meant to be combined together, the whole module
becomes easier to use if there are fewer tools to choose from.

True, but on the other hand a toolbox with too few tools is harder to use
than one with too many tools.

--
Steven

[bearophileHUGS]

Almost every day I write code that uses itertools, so I find it very
useful, and its functions fast.
Removing useless things and keeping things tidy is often positive. But
I can't tell you what to remove. Here are my usages (every sub-list is
sorted by inverted frequency usage):

I use often or very often:
groupby( iterable[, key])
imap( function, *iterables)
izip( *iterables)
ifilter( predicate, iterable)
islice( iterable, [start,] stop [, step])

I use once in while:
cycle( iterable)
chain( *iterables)
count( [n])
repeat( object[, times])

I have used probably one time or few times:
starmap( function, iterable)
tee( iterable[, n=2])
ifilterfalse( predicate, iterable)

Never used so far:
dropwhile( predicate, iterable)
takewhile( predicate, iterable)

Bye,
bearophile

[Michele Simionato]

On Dec 30, 12:10 am, Raymond Hettinger <[e-mail]:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

I am with Steven D'Aprano when he says that takewhile and dropwhile
are clear enough. On the other hand, in my code
base I have exactly zero occurrences of takewhile and
dropwhile, even if I tend to use the itertools quite
often. That should be telling. If my situations is
common, that means that takewhile and dropwhile are
useless in practice and should be deprecated.
But I will wait for other respondents. It may just be
that I never needed them. I presume you did scans of
large code bases and you did not find occurrences of
takewhile and dropwhile, right?
Michele Simionato

[Marc 'BlackJack' Rintsch]

On Sat, 29 Dec 2007 15:10:24 -0800, Raymond Hettinger wrote:

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

I seem to be in a minority here as I use both functions from time to time.
One "recipe" is extracting blocks from text files that are delimited by a
special start and end line.

def iter_block(lines, start_marker, end_marker):
return takewhile(lambda x: not x.startswith(end_marker),
dropwhile(lambda x: not x.startswith(start_marker),
lines))

Maybe these functions usually don't turn up in code that can be called
"recipes" so often but are useful for themselves.

Ciao,
Marc 'BlackJack' Rintsch

[Istvan Albert]

On Dec 30, 3:29 am, Marc 'BlackJack' Rintsch <[e-mail]:

One "recipe" is extracting blocks from text files that are delimited by a
special start and end line.

Neat solution!

I actually need such functionality every once in a while.

Takewhile + dropwhile to the rescue!

i.

[George Sakkis]

On Dec 30, 4:12 pm, Istvan Albert <[e-mail]:

On Dec 30, 3:29 am, Marc 'BlackJack' Rintsch <[e-mail]:

One "recipe" is extracting blocks from text files that are delimited by a
special start and end line.

Neat solution!

I actually need such functionality every once in a while.

Takewhile + dropwhile to the rescue!

i.

On at least one thread and a recipe for this task (http://
aspn.activestate.com/ASPN/Cookbook/Python/Recipe/521877), the proposed
solutions involved groupby() with an appropriate key function. The
takewhile/dropwhile solution seems shorter and (maybe) easier to read
but perhaps not as flexible and general. Regardless, it's a good
example of takewhile/dropwhile.

George

[Raymond Hettinger]

[bearophile]

Here are my usages (every sub-list is
sorted by inverted frequency usage):

I use often or very often:
groupby( iterable[, key])
imap( function, *iterables)
izip( *iterables)
ifilter( predicate, iterable)
islice( iterable, [start,] stop [, step])

I use once in while:
cycle( iterable)
chain( *iterables)
count( [n])
repeat( object[, times])

I have used probably one time or few times:
starmap( function, iterable)
tee( iterable[, n=2])
ifilterfalse( predicate, iterable)

Never used so far:
dropwhile( predicate, iterable)
takewhile( predicate, iterable)

Thank you for the useful and informative response.
Raymond

[Raymond Hettinger]

[Michele Simionato]

in my code
base I have exactly zero occurrences of takewhile and
dropwhile, even if I tend to use the itertools quite
often. That should be telling.

Thanks for the additional empirical evidence.

I presume you did scans of
large code bases and you did not find occurrences of
takewhile and dropwhile, right?

Yes.
Raymond

[Raymond Hettinger]

[Marc 'BlackJack' Rintsch]

I use both functions from time to time.
One "recipe" is extracting blocks from text files that are delimited by a
special start and end line.

def iter_block(lines, start_marker, end_marker):
* * return takewhile(lambda x: not x.startswith(end_marker),
* * * * * * * * * * *dropwhile(lambda x: not x.startswith(start_marker),
* * * * * * * * * * * * * * * *lines))

Glad to hear this came from real code instead of being contrived for
this discussion. Thanks for the contribution.

Looking at the code fragment, I wondered how that approach compared to
others in terms of being easy to write, self-evidently correct,
absence of awkward constructs, and speed. The lambda expressions are
not as fast as straight C calls or in-lined code, and they also each
require a 'not' to invert the startswith condition. The latter is a
bit problematic in that it is a bit awkward, and it is less self-
evident whether the lines with the markers are included or excluded
from the output (the recipe may in fact be buggy -- the line with the
start marker is included and the line with the end marker is
excluded). Your excellent choice of indentation helps improve the
readability of the nested takewhile/dropwhile calls.

In contrast, the generator version is clearer about whether the start
and end marker lines get included and is easily modified if you want
to change that choice. It is easy to write and more self-evident
about how it handles the end cases. Also, it avoids the expense of
the lambda function calls and the awkwardness of the 'not' to invert
the sense of the test:

def iter_block(lines, start_marker, end_marker):
inblock = False
for line in lines:
if inblock:
if line.startswith(end_marker):
break
yield line
elif line.startswith(start_marker):
yield line
inblock = True

And, of course, for this particular application, an approach based on
regular expressions makes short work of the problem and runs very
fast:

re.search('(^beginmark.*)^endmark', textblock, re.M |
re.S).group(1)
Raymond

[Raymond Hettinger]

FWIW, here is an generator version written without the state flag:

def iter_block(lines, start_marker, end_marker):
lines = iter(lines)
for line in lines:
if line.startswith(start_marker):
yield line
break
for line in lines:
if line.startswith(end_marker):
return
yield line

Raymond

[Paul Hankin]

On Dec 31, 1:25*am, Raymond Hettinger <[e-mail]:

FWIW, here is an generator version written without the state flag:

* * def iter_block(lines, start_marker, end_marker):
* * * * lines = iter(lines)
* * * * for line in lines:
* * * * * * if line.startswith(start_marker):
* * * * * * * * yield line
* * * * * * * * break
* * * * for line in lines:
* * * * * * if line.startswith(end_marker):
* * * * * * * * return
* * * * * * yield line

Here's a (stateful) version that generates all blocks...

import itertools

def iter_blocks(lines, start_marker, end_marker):
inblock = [False]
def line_in_block(line):
inblock[0] = inblock[0] and not line.startswith(end_marker)
inblock[0] = inblock[0] or line.startswith(start_marker)
return inblock[0]
return (block for is_in_block, block in
itertools.groupby(lines, line_in_block) if is_in_block)

If you just want the first block (as the original code did), you can
just take it...

for line in iter_blocks(lines, start_marker, end_marker).next():
... process lines of first block.

I'm not happy about the way the inblock state has to be a 1-element
list to avoid the non-local problem. Is there a nicer way to code it?
Otherwise, I quite like this code (if I do say so myself) as it neatly
separates out the logic of whether you're inside a block or not from
the code that yields blocks and lines. I'd say it was quite readable
if you're familiar with groupby.

And back on topic... I use itertools regularly (and have a functional
background), but have never needed takewhile or dropwhile. I'd be
happy to see them deprecated.

--
Paul Hankin

[Matt Nordhoff]

Raymond Hettinger wrote:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

* I'm concerned that use cases for the two functions are uncommon and
can obscure code rather than clarify it.

* I originally added them to itertools because they were found in
other functional languages and because it seemed like they would serve
basic building blocks in combination with other itertools allow
construction of a variety of powerful, high-speed iterators. The
latter may have been a false hope -- to date, I've not seen good
recipes that depend on either function.

* If an always true or always false predicate is given, it can be hard
to break-out of the function once it is running.

* Both functions seem simple and basic until you try to explain them
to someone else. Likewise, when reading code containing dropwhile(),
I don't think it is self-evident that dropwhile() may have a lengthy
start-up time.

* Since itertools are meant to be combined together, the whole module
becomes easier to use if there are fewer tools to choose from.

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

Raymond

FWIW, Google Code Search shows a few users:

<http://www.google.com/codesearch?q=lang%3Apython+%28drop%7Ctake%29while>

Do any of them make good use of them?
--

[winjer]

On Dec 29 2007, 11:10 pm, Raymond Hettinger <[e-mail]:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

Well I have just this minute used dropwhile in anger, to find the next
suitable filename when writing database dumps using date.count names:

filename = "%02d-%02d-%d" % (now.day, now.month, now.year)
if os.path.exists(filename):
candidates = ("%s.%d" % (filename, x) for x in count(1))
filename = dropwhile(os.path.exists, candidates).next()

Much clearer than the alternatives I think, please keep dropwhile and
takewhile in itertools ;)

Cheers,

Doug.

[Arnaud Delobelle]

On Jan 3, 4:39*pm, "[e-mail]" <[e-mail]:

On Dec 29 2007, 11:10 pm, Raymond Hettinger <[e-mail]:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

Well I have just this minute used dropwhile in anger, to find the next
suitable filename when writing database dumps using date.count names:

* * filename = "%02d-%02d-%d" % (now.day, now.month, now.year)
* * if os.path.exists(filename):
* * * * candidates = ("%s.%d" % (filename, x) for x in count(1))
* * * * filename = dropwhile(os.path.exists, candidates).next()

Much clearer than the alternatives I think, please keep dropwhile and
takewhile in itertools ;)

Wouldn't using ifilterfalse instead of dropwhile produce the same
result?

--
Arnaud

[Paul Rubin]

Raymond Hettinger <py****@rcn.comwrites:

I presume you did scans of
large code bases and you did not find occurrences of
takewhile and dropwhile, right?

Yes.

I think I have used them. I don't remember exactly how. Probably
something that could have been done more generally with groupby.

I remember a clpy thread about a takewhile gotcha, that it consumes an
extra element:

>>from itertools import takewhile as tw
x = range(10)
z = iter(x)
list(tw(lambda i:i<5, z))

[0, 1, 2, 3, 4]

>>z.next()

6

I.e. I had wanted to use takewhile to split a list into the
initial sublist satisfying some condition, and the rest of the
list.

This all by itself is something to at least warn about. I don't
know if it's enough for deprecation.

I've been cooking up a scheme for iterators with lookahead, that I
want to get around to coding and posting. It's a harder thing
to get right than it at first appears.

[Simon Brunning]

On Dec 29, 2007 11:10 PM, Raymond Hettinger <py****@rcn.comwrote:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

Personally, I'd rather you kept them around. I have no FP background,
and I found them easy enough to understand.

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

FWIW, I used them only today: http://tinyurl.com/22q6cb

Not sure if something that ugly counts as a reason for keeping them
around, though!

--
Cheers,
Simon B.
si***@brunningonline.net
http://www.brunningonline.net/simon/blog/
GTalk: simon.brunning | MSN: small_values | Yahoo: smallvalues