can list comprehensions replace map?

[David Isaac]

Newbie question:

I have been generally open to the proposal that list comprehensions
should replace 'map', but I ran into a need for something like
map(None,x,y)
when len(x)>len(y). I cannot it seems use 'zip' because I'll lose
info from x. How do I do this as a list comprehension? (Or,
more generally, what is the best way to do this without 'map'?)

Thanks,
Alan Isaac

[Andrew Dalke]

Me:

Here's a clever, though not (in my opinion) elegant solution ... This seems a bit more elegant, though the "replace" dictionary is
still a bit of a hack

Here's the direct approach without using itertools. Each list is
iterated over only once. No test against a sequence element is ever
made (either as == or 'is') and the end of the sequence exception
is raised only once per input iterator.

The use of a list for the flag is a bit of a hack. If the list has
1 element then its true, no elements then its false. By doing it this
way I don't need one extra array and one extra indexing/enumeration.

def zipfill(*seqs):
count = len(seqs)
seq_info = [(iter(seq), [1]) for seq in seqs]
while 1:
fields = []
for seq, has_data in seq_info:
if has_data:
try:
fields.append(s eq.next())
except StopIteration:
fields.append(N one)
del has_data[:]
count -= 1
else:
fields.append(N one)
if count:
yield fields
else:
break
Hmm, it should probably yield tuple(fields)

Andrew
da***@dalkescie ntific.com

[Raymond Hettinger]

[Paolino]

Well, for my little experiences use cases in which the lists have different
lengths are rare, but in those cases I don't see the reason of not being
able
to zip to the longest one.What is really strange is that I have to use
map(None,....) for that,instead of another zip-like function which ,at
least
would be intutitive for the average user.Also map(None,...) looks like a
super-hack
and it's not elegant or readable or logic (IMO)

I think zip comes to substitute the tuple.__new__ untolerant
implementation. A dumb like me wuold expect map(tuple,[1,2,3],[2,3,4]) to
work, so pretending map(None,....) would do it is like saying that None
and tuple are near concepts, which is obviously an absurdity.

Yes, map(None, ...) lacks grace and it would be nice if it had never
been done. The more recently implemented zip() does away with these
issues. The original was kept for backwards compatibility. That's
evolution.

My sense for the rest is that your difficulties arise from fighting the
language rather than using it as designed. Most language features are
the result of much deliberation. When design X was chosen over
alternative Y, it is a pretty good cue that X is a more harmonious way
to do things.

Some other languages chose to implement both X and Y. On the plus
side, your intuition likely matches one of the two. On the minus side,
someone else's intuition may not match your own. Also, it leads to
language bloat. More importantly, such a language provides few cues as
to how to select components that work together harmoniously.
Unfortunately, that makes it effortless to mire yourself in deep goo.

My advice is to use the language instead of fighting it. Guido has
marked the trail; don't ignore the signs unless you really know where
you're going.

Raymond
"... and soon you'll feel right as rain." -- from The Matrix

[Christopher Subich]

Andrew Dalke wrote:

Steven Bethard wrote:

Here's one possible solution:

py> import itertools as it
py> def zipfill(*lists) :
... max_len = max(len(lst) for lst in lists)

A limitation to this is the need to iterate over the
lists twice, which might not be possible if one of them
is a file iterator.

Here's a clever, though not (in my opinion) elegant solution

import itertools

def zipfill(*seqs):
count = [len(seqs)]
def _forever(seq):
for item in seq: yield item
count[0] -= 1
while 1: yield None
seqs = [_forever(seq) for seq in seqs]
while 1:
x = [seq.next() for seq in seqs]
if count == [0]:
break
yield x

I like this solution best (note, it doesn't actually use itertools). My
naive solution:
def lzip(*args):
ilist = [iter(a) for a in args]
while 1:
res = []
count = 0
for i in ilist:
try:
g = i.next()
count += 1
except StopIteration: # End of iter
g = None
res.append(g)
if count > 0: # At least one iter wasn't finished
yield tuple(res)
else: # All finished
raise StopIteration

[Andrew Dalke]

Christopher Subich wrote:

My naive solution: ... for i in ilist:
try:
g = i.next()
count += 1
except StopIteration: # End of iter
g = None

...

What I didn't like about this was the extra overhead of all
the StopIteration exceptions. Eg,

zipfill("a", range(1000))

will raise 1000 exceptions (999 for "a" and 1 for the end of the range).

But without doing timing tests I'm not sure which approach is
fastest, and it may depend on the data set.

Since this is code best not widely used, I don't think it's something
anyone should look into either. :)

Andrew
da***@dalkescie ntific.com

[Peter Otten]

Andrew Dalke wrote:

Steven Bethard wrote:

Here's one possible solution:

py> import itertools as it
py> def zipfill(*lists) :
... max_len = max(len(lst) for lst in lists)
A limitation to this is the need to iterate over the
lists twice, which might not be possible if one of them
is a file iterator.

Here's a clever, though not (in my opinion) elegant solution

import itertools

def zipfill(*seqs):
count = [len(seqs)]
def _forever(seq):
for item in seq: yield item
count[0] -= 1
while 1: yield None
seqs = [_forever(seq) for seq in seqs]
while 1:
x = [seq.next() for seq in seqs]
if count == [0]:
break
yield x

This seems a bit more elegant, though the "replace" dictionary is
still a bit of a hack

from itertools import repeat, chain, izip

sentinel = object()
end_of_stream = repeat(sentinel )

def zipfill(*seqs):
replace = {sentinel: None}.get
seqs = [chain(seq, end_of_stream) for seq in seqs]
for term in izip(*seqs):
for element in term:
if element is not sentinel:
break
else:
# All sentinels
break

yield [replace(element , element) for element in term]

Combining your "clever" and your "elegant" approach to something fast
(though I'm not entirely confident it's correct):

def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return
while 1:
yield None
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)

Whether we ran out of active sequences is only tested once per sequence.

Fiddling with itertools is always fun, but feels a bit like reinventing the
wheel in this case. The only excuse being that you might need a lazy
map(None, ...) someday...

Peter

[Andrew Dalke]

Peter Otten wrote:

Combining your "clever" and your "elegant" approach to something fast
(though I'm not entirely confident it's correct):

def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return
while 1:
yield None
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)

Ohh, that's pretty neat passing in 'done' via a mutable default argument.

It took me a bit to even realize why it does work. :)

Could make it one line shorter with

from itertools import chain, izip, repeat
def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return []
return repeat(None)
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)

Go too far on that path and the code starts looking likg

from itertools import chain, izip, repeat
forever, table = repeat(None), {0: []}.get
def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
return table(done[0], forever)
return izip(*[chain(seq, done_iter()) for seq in seqs])

Now add the performance tweak....

def done_iter(done=[len(seqs)], forever=forever , table=table)

Okay, I'm over it. :)

Andrew
da***@dalkescie ntific.com

[Scott David Daniels]

Peter Otten wrote:

def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return
while 1:
yield None
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)

Can I play too? How about:
import itertools

def fillzip(*seqs):
def Nones(countacti ve=[len(seqs)]):
countactive[0] -= 1
while countactive[0]:
yield None
seqs = [itertools.chain (seq, Nones()) for seq in seqs]
return itertools.izip( *seqs)

--Scott David Daniels
Sc***********@A cm.Org

[Peter Otten]

Andrew Dalke wrote:

Peter Otten wrote:

Combining your "clever" and your "elegant" approach to something fast
(though I'm not entirely confident it's correct):

def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return
while 1:
yield None
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)
Ohh, that's pretty neat passing in 'done' via a mutable default argument.

It took me a bit to even realize why it does work. :)

Though I would never have come up with it, were it not for the juxtaposition
of your two variants (I initially disliked the first and tried to improve
on the second), it is an unobvious merger :)
It's a bit fragile, too, as
Could make it one line shorter with from itertools import chain, izip, repeat
def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return []
return repeat(None)
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)
that won't work because done_iter() is now no longer a generator.
In effect you just say

seqs = [chain(seq, repeat(None)) for seq in seqs[:-1]] + [chain(seq[-1],
[])]

I tried

class Done(Exception) :
pass

pad = repeat(None)
def fillzip(*seqs):
def check(active=[len(seqs)]):
active[0] -= 1
if not active[0]:
raise Done
# just to turn check() into a generator
if 0: yield None
seqs = [chain(seq, check(), pad) for seq in seqs]
try
for item in izip(*seqs):
yield item
except Done:
pass

to be able to use the faster repeat() instead of the while loop, and then
stared at it for a while -- in vain -- to eliminate the for item... loop.
If there were a lazy ichain(iter_of_ iters) you could tweak check() to decide
whether a repeat(None) should follow it, but I'd rather not ask Raymond for
that particular addition to the itertools.
Now add the performance tweak....

def done_iter(done=[len(seqs)], forever=forever , table=table)

Okay, I'm over it. :)

Me too. I think. For now...

Peter

[Andrew Dalke]

Me:

Could make it one line shorter with

from itertools import chain, izip, repeat
def fillzip(*seqs):
def done_iter(done=[len(seqs)]):
done[0] -= 1
if not done[0]:
return []
return repeat(None)
seqs = [chain(seq, done_iter()) for seq in seqs]
return izip(*seqs)

Peter Otten: that won't work because done_iter() is now no longer a generator.
In effect you just say

seqs = [chain(seq, repeat(None)) for seq in seqs[:-1]] + [chain(seq[-1],
[])]

It does work - I tested it. The trick is that izip takes iter()
of the terms passed into it. iter([]) -> an empty iterator and
iter(repeat(Non e)) -> the repeat(None) itself.

'Course then the name should be changed.

Andrew
da***@dalkescie ntific.com

[Andrew Dalke]

Scott David Daniels wrote:

Can I play too? How about:

Sweet!
Andrew
da***@dalkescie ntific.com