Re: buffering choking sys.stdin.readlines() ?

[Jean-Paul Calderone]

On Mon, 12 May 2008 08:05:39 -0700 (PDT), cshirky <cs*****@gmail.comwrote:

>Newbie question:

I'm trying to turn a large XML file (~7G compressed) into a YAML file,
and my program seems to be buffering the input.

IOtest.py is just

import sys
for line in sys.stdin.readlines():
print line

but when I run

$ gzcat bigXMLfile.gz | IOtest.py

but it hangs then dies.

file.readlines reads the entire file into a list in memory. You may not
want to do this. You could try, instead, iterating over "sys.stdin",
which should not try to load the entire file into memory.

Jean-Paul