Fate of itertools.dropw hile() and itertools.takew hile()

[Raymond Hettinger]

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

* I'm concerned that use cases for the two functions are uncommon and
can obscure code rather than clarify it.

* I originally added them to itertools because they were found in
other functional languages and because it seemed like they would serve
basic building blocks in combination with other itertools allow
construction of a variety of powerful, high-speed iterators. The
latter may have been a false hope -- to date, I've not seen good
recipes that depend on either function.

* If an always true or always false predicate is given, it can be hard
to break-out of the function once it is running.

* Both functions seem simple and basic until you try to explain them
to someone else. Likewise, when reading code containing dropwhile(),
I don't think it is self-evident that dropwhile() may have a lengthy
start-up time.

* Since itertools are meant to be combined together, the whole module
becomes easier to use if there are fewer tools to choose from.

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

Raymond

[Raymond Hettinger]

[Marc 'BlackJack' Rintsch]

I use both functions from time to time.
One "recipe" is extracting blocks from text files that are delimited by a
special start and end line.

def iter_block(line s, start_marker, end_marker):
* * return takewhile(lambd a x: not x.startswith(en d_marker),
* * * * * * * * * * *dropwhile(lamb da x: not x.startswith(st art_marker),
* * * * * * * * * * * * * * * *lines))

Glad to hear this came from real code instead of being contrived for
this discussion. Thanks for the contribution.

Looking at the code fragment, I wondered how that approach compared to
others in terms of being easy to write, self-evidently correct,
absence of awkward constructs, and speed. The lambda expressions are
not as fast as straight C calls or in-lined code, and they also each
require a 'not' to invert the startswith condition. The latter is a
bit problematic in that it is a bit awkward, and it is less self-
evident whether the lines with the markers are included or excluded
from the output (the recipe may in fact be buggy -- the line with the
start marker is included and the line with the end marker is
excluded). Your excellent choice of indentation helps improve the
readability of the nested takewhile/dropwhile calls.

In contrast, the generator version is clearer about whether the start
and end marker lines get included and is easily modified if you want
to change that choice. It is easy to write and more self-evident
about how it handles the end cases. Also, it avoids the expense of
the lambda function calls and the awkwardness of the 'not' to invert
the sense of the test:

def iter_block(line s, start_marker, end_marker):
inblock = False
for line in lines:
if inblock:
if line.startswith (end_marker):
break
yield line
elif line.startswith (start_marker):
yield line
inblock = True

And, of course, for this particular application, an approach based on
regular expressions makes short work of the problem and runs very
fast:

re.search('(^be ginmark.*)^endm ark', textblock, re.M |
re.S).group(1)
Raymond

[Raymond Hettinger]

FWIW, here is an generator version written without the state flag:

def iter_block(line s, start_marker, end_marker):
lines = iter(lines)
for line in lines:
if line.startswith (start_marker):
yield line
break
for line in lines:
if line.startswith (end_marker):
return
yield line

Raymond

[Paul Hankin]

On Dec 31, 1:25*am, Raymond Hettinger <[e-mail] wrote:

FWIW, here is an generator version written without the state flag:

* * def iter_block(line s, start_marker, end_marker):
* * * * lines = iter(lines)
* * * * for line in lines:
* * * * * * if line.startswith (start_marker):
* * * * * * * * yield line
* * * * * * * * break
* * * * for line in lines:
* * * * * * if line.startswith (end_marker):
* * * * * * * * return
* * * * * * yield line

Here's a (stateful) version that generates all blocks...

import itertools

def iter_blocks(lin es, start_marker, end_marker):
inblock = [False]
def line_in_block(l ine):
inblock[0] = inblock[0] and not line.startswith (end_marker)
inblock[0] = inblock[0] or line.startswith (start_marker)
return inblock[0]
return (block for is_in_block, block in
itertools.group by(lines, line_in_block) if is_in_block)

If you just want the first block (as the original code did), you can
just take it...

for line in iter_blocks(lin es, start_marker, end_marker).nex t():
... process lines of first block.

I'm not happy about the way the inblock state has to be a 1-element
list to avoid the non-local problem. Is there a nicer way to code it?
Otherwise, I quite like this code (if I do say so myself) as it neatly
separates out the logic of whether you're inside a block or not from
the code that yields blocks and lines. I'd say it was quite readable
if you're familiar with groupby.

And back on topic... I use itertools regularly (and have a functional
background), but have never needed takewhile or dropwhile. I'd be
happy to see them deprecated.

--
Paul Hankin

[Matt Nordhoff]

Raymond Hettinger wrote:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

* I'm concerned that use cases for the two functions are uncommon and
can obscure code rather than clarify it.

* I originally added them to itertools because they were found in
other functional languages and because it seemed like they would serve
basic building blocks in combination with other itertools allow
construction of a variety of powerful, high-speed iterators. The
latter may have been a false hope -- to date, I've not seen good
recipes that depend on either function.

* If an always true or always false predicate is given, it can be hard
to break-out of the function once it is running.

* Both functions seem simple and basic until you try to explain them
to someone else. Likewise, when reading code containing dropwhile(),
I don't think it is self-evident that dropwhile() may have a lengthy
start-up time.

* Since itertools are meant to be combined together, the whole module
becomes easier to use if there are fewer tools to choose from.

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

Raymond

FWIW, Google Code Search shows a few users:

<http://www.google.com/codesearch?q=la ng%3Apython+%28 drop%7Ctake%29w hile>

Do any of them make good use of them?
--

[winjer]

On Dec 29 2007, 11:10 pm, Raymond Hettinger <[e-mail] wrote:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

Well I have just this minute used dropwhile in anger, to find the next
suitable filename when writing database dumps using date.count names:

filename = "%02d-%02d-%d" % (now.day, now.month, now.year)
if os.path.exists( filename):
candidates = ("%s.%d" % (filename, x) for x in count(1))
filename = dropwhile(os.pa th.exists, candidates).nex t()

Much clearer than the alternatives I think, please keep dropwhile and
takewhile in itertools ;)

Cheers,

Doug.

[Arnaud Delobelle]

On Jan 3, 4:39*pm, "win...@gmail.c om" <win...@gmail.c omwrote:

On Dec 29 2007, 11:10 pm, Raymond Hettinger <[e-mail] wrote:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

Well I have just this minute used dropwhile in anger, to find the next
suitable filename when writing database dumps using date.count names:

* * filename = "%02d-%02d-%d" % (now.day, now.month, now.year)
* * if os.path.exists( filename):
* * * * candidates = ("%s.%d" % (filename, x) for x in count(1))
* * * * filename = dropwhile(os.pa th.exists, candidates).nex t()

Much clearer than the alternatives I think, please keep dropwhile and
takewhile in itertools ;)

Wouldn't using ifilterfalse instead of dropwhile produce the same
result?

--
Arnaud

[Paul Rubin]

Raymond Hettinger <py****@rcn.com writes:

I presume you did scans of
large code bases and you did not find occurrences of
takewhile and dropwhile, right?

Yes.

I think I have used them. I don't remember exactly how. Probably
something that could have been done more generally with groupby.

I remember a clpy thread about a takewhile gotcha, that it consumes an
extra element:

>>from itertools import takewhile as tw
x = range(10)
z = iter(x)
list(tw(lambd a i:i<5, z))

[0, 1, 2, 3, 4]

>>z.next()

6

I.e. I had wanted to use takewhile to split a list into the
initial sublist satisfying some condition, and the rest of the
list.

This all by itself is something to at least warn about. I don't
know if it's enough for deprecation.

I've been cooking up a scheme for iterators with lookahead, that I
want to get around to coding and posting. It's a harder thing
to get right than it at first appears.

[Simon Brunning]

On Dec 29, 2007 11:10 PM, Raymond Hettinger <py****@rcn.com wrote:

I'm considering deprecating these two functions and would like some
feedback from the community or from people who have a background in
functional programming.

Personally, I'd rather you kept them around. I have no FP background,
and I found them easy enough to understand.

These thoughts reflect my own experience with the itertools module.
It may be that your experience with them has been different. Please
let me know what you think.

FWIW, I used them only today: http://tinyurl.com/22q6cb

Not sure if something that ugly counts as a reason for keeping them
around, though!

--
Cheers,
Simon B.
si***@brunningo nline.net
http://www.brunningonline.net/simon/blog/
GTalk: simon.brunning | MSN: small_values | Yahoo: smallvalues