itertools.izip brokeness

[rurpy]

The code below should be pretty self-explanatory.
I want to read two files in parallel, so that I
can print corresponding lines from each, side by
side. itertools.izip( ) seems the obvious way
to do this.

izip() will stop interating when it reaches the
end of the shortest file. I don't know how to
tell which file was exhausted so I just try printing
them both. The exhausted one will generate a
StopInteration, the other will continue to be
iterable.

The problem is that sometimes, depending on which
file is the shorter, a line ends up missing,
appearing neither in the izip() output, or in
the subsequent direct file iteration. I would
guess that it was in izip's buffer when izip
terminates due to the exception on the other file.

This behavior seems plain out broken, especially
because it is dependent on order of izip's
arguments, and not documented anywhere I saw.
It makes using izip() for iterating files in
parallel essentially useless (unless you are
lucky enough to have files of the same length).

Also, it seems to me that this is likely a problem
with any iterables with different lengths.
I am hoping I am missing something...

#---------------------------------------------------------
# Task: print contents of file1 in column 1, and
# contents of file2 in column two. iterators and
# izip() are the "obvious" way to do it.

from itertools import izip
import cStringIO, pdb

def prt_files (file1, file2):

for line1, line2 in izip (file1, file2):
print line1.rstrip(), "\t", line2.rstrip()

try:
for line1 in file1:
print line1,
except StopIteration: pass

try:
for line2 in file2:
print "\t",line2,
except StopIteration: pass

if __name__ == "__main__":
# Use StringIO to simulate files. Real files
# show the same behavior.
f = cStringIO.Strin gIO

print "Two files with same number of lines work ok."
prt_files (f("abc\nde\nfg h\n"), f("xyz\nwv\nstu \n"))

print "\nFirst file shorter is also ok."
prt_files (f("abc\nde\n") , f("xyz\nwv\nstu \n"))

print "\nSecond file shorter is a problem."
prt_files (f("abc\nde\nfg h\n"), f("xyz\nwv\n" ))
print "What happened to \"fgh\" line that should be in column
1?"

print "\nBut only a problem for one line."
prt_files (f("abc\nde\nfg h\nijk\nlm\n"), f("xyz\nwv\n" ))
print "The line \"fgh\" is still missing, but following\n" \
"line(s) are ok! Looks like izip() ate a line."

[Bengt Richter]

On 9 Jan 2006 08:19:21 GMT, Antoon Pardon <ap*****@forel. vub.ac.be> wrote:

Op 2006-01-05, Bengt Richter schreef <bo**@oz.net> :

On 5 Jan 2006 15:48:26 GMT, Antoon Pardon <ap*****@forel. vub.ac.be> wrote: [...]

But you can fix that (only test is what you see ;-) :
Maybe, but not with this version.
>>> from itertools import repeat, chain, izip
>>> it = iter(lambda z=izip(chain([3,5,8],repeat("Bye")) , chain([11,22],repeat("Bye")) ):z.next(), ("Bye","Bye" ))
>>> for t in it: print t

...
(3, 11)
(5, 22)
(8, 'Bye')

(Feel free to generalize ;-)

The problem with this version is that it will stop if for some reason
each iterable contains a 'Bye' at the same place. Now this may seem
far fetched at first. But consider that if data is collected from

ISTM the job of choosing an appropriate sentinel involves making
that not only far fetched but well-nigh impossible ;-)
experiments certain values may be missing. This can be indicated
by a special "Missing Data" value in an iterable. But this "Missing
Data" value would also be the prime canidate for a fill parameter
when an iterable is exhausted.

ISTM that confuses "missing data" with "end of data stream."
I assumed your choice of terminating sentinel ("Bye") would not have
that problem ;-)

Regards,
Bengt Richter

[Antoon Pardon]

Op 2006-01-10, Bengt Richter schreef <bo**@oz.net> :

On 9 Jan 2006 08:19:21 GMT, Antoon Pardon <ap*****@forel. vub.ac.be> wrote:

Op 2006-01-05, Bengt Richter schreef <bo**@oz.net> :

On 5 Jan 2006 15:48:26 GMT, Antoon Pardon <ap*****@forel. vub.ac.be> wrote: [...] But you can fix that (only test is what you see ;-) :
Maybe, but not with this version.

>>> from itertools import repeat, chain, izip
>>> it = iter(lambda z=izip(chain([3,5,8],repeat("Bye")) , chain([11,22],repeat("Bye")) ):z.next(), ("Bye","Bye" ))
>>> for t in it: print t
...
(3, 11)
(5, 22)
(8, 'Bye')

(Feel free to generalize ;-)

The problem with this version is that it will stop if for some reason
each iterable contains a 'Bye' at the same place. Now this may seem
far fetched at first. But consider that if data is collected from

ISTM the job of choosing an appropriate sentinel involves making
that not only far fetched but well-nigh impossible ;-)

experiments certain values may be missing. This can be indicated
by a special "Missing Data" value in an iterable. But this "Missing
Data" value would also be the prime canidate for a fill parameter
when an iterable is exhausted.

ISTM that confuses "missing data" with "end of data stream."

"end of data stream" implies "missing data". If I'm doing experiments
with a number of materials under a number of tempertures and I want
to compare how copper, iron and lead behaved then when I compare
the results for 400 K and there is no data for lead, I don't care
whether that is because the measurement for 400K was somehow
lost or unsuable or because they stopped the lead measurements at 350K.

It all boils down to no data for lead at 400K, there is no need that
the processing unit differentiates beteen the different reasons for
the missing data. That difference is only usefull for the loop control.
I assumed your choice of terminating sentinel ("Bye") would not have
that problem ;-)

That is true, but what is adequate in one situation doesn't need to
be adequate in general.

--
Antoon Pardon