By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
439,993 Members | 1,984 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 439,993 IT Pros & Developers. It's quick & easy.

os.popen and broken pipes

P: n/a

Hi Pythoneers,

I need to process a large number of files which have been packed by the
UNIX compress tool (*.Z files). As I am not aware of a compress
equivalent of the gzip, zipfile or bzip2 modules, I thought I'd use the
uncompress or zcat commands directly to deal with the files:

for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()
This works fine for some files but results in

'write error onstdout: Broken pipe'

emitted by uncompress for others. Using zcat instead of uncompress
changes the wording of the error message but not the result.

I tried to give popen a large bufsize argument but that didn't really
help.

Probably I'm overlooking something obvious here but right now I can't
see it. Any hints?

cu
Philipp

--
Dr. Philipp Pagel Tel. +49-8161-71 2131
Dept. of Genome Oriented Bioinformatics Fax. +49-8161-71 2186
Technical University of Munich
http://mips.gsf.de/staff/pagel
Feb 9 '07 #1
Share this Question
Share on Google+
5 Replies


P: n/a
It could easily be the 2gig file size limitation, how large are the
extracts?

Feb 9 '07 #2

P: n/a
On 2007-02-09, Philipp Pagel <pD*******@gsf.dewrote:
>
Hi Pythoneers,

I need to process a large number of files which have been packed by the
UNIX compress tool (*.Z files). As I am not aware of a compress
equivalent of the gzip, zipfile or bzip2 modules, I thought I'd use the
uncompress or zcat commands directly to deal with the files:

for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()
This works fine for some files but results in

'write error onstdout: Broken pipe'

emitted by uncompress for others. Using zcat instead of uncompress
changes the wording of the error message but not the result.

I tried to give popen a large bufsize argument but that didn't really
help.

Probably I'm overlooking something obvious here but right now I can't
see it. Any hints?
As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.

--
Antoon Pardon
Feb 9 '07 #3

P: n/a
Antoon Pardon <ap*****@forel.vub.ac.bewrote:
On 2007-02-09, Philipp Pagel <pD*******@gsf.dewrote:
for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()

This works fine for some files but results in

'write error onstdout: Broken pipe'
As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.
You are right: some of the files do not fulfill certain
critereia causing so_somehting() to return before the entire file is
processed.

Thanks!

cu
Philipp

--
Dr. Philipp Pagel Tel. +49-8161-71 2131
Dept. of Genome Oriented Bioinformatics Fax. +49-8161-71 2186
Technical University of Munich
http://mips.gsf.de/staff/pagel
Feb 9 '07 #4

P: n/a
Chris <cw****@gmail.comwrote:
It could easily be the 2gig file size limitation, how large are the
extracts?
The files are much smaller than that, so that's not the issue.
Anyway, Antoon pointed me in the right direction.

Thanks for the help
Philipp

--
Dr. Philipp Pagel Tel. +49-8161-71 2131
Dept. of Genome Oriented Bioinformatics Fax. +49-8161-71 2186
Technical University of Munich
http://mips.gsf.de/staff/pagel
Feb 9 '07 #5

P: n/a
In article <eq**********@news.lrz-muenchen.de>,
Philipp Pagel <pD*******@gsf.dewrote:
Antoon Pardon <ap*****@forel.vub.ac.bewrote:
On 2007-02-09, Philipp Pagel <pD*******@gsf.dewrote:
for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()
>
This works fine for some files but results in
>
'write error onstdout: Broken pipe'
As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.

You are right: some of the files do not fulfill certain
critereia causing so_somehting() to return before the entire file is
processed.
Most programming environments don't have this problem, though.

If you like, your program can undo what Python does:

signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for filename in file_list:
...

Then it will work as if you had written it in C, or awk
or whatever.

Donn Cave, do**@u.washington.edu
Feb 9 '07 #6

This discussion thread is closed

Replies have been disabled for this discussion.