473,326 Members | 2,010 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

os.popen and broken pipes


Hi Pythoneers,

I need to process a large number of files which have been packed by the
UNIX compress tool (*.Z files). As I am not aware of a compress
equivalent of the gzip, zipfile or bzip2 modules, I thought I'd use the
uncompress or zcat commands directly to deal with the files:

for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()
This works fine for some files but results in

'write error onstdout: Broken pipe'

emitted by uncompress for others. Using zcat instead of uncompress
changes the wording of the error message but not the result.

I tried to give popen a large bufsize argument but that didn't really
help.

Probably I'm overlooking something obvious here but right now I can't
see it. Any hints?

cu
Philipp

--
Dr. Philipp Pagel Tel. +49-8161-71 2131
Dept. of Genome Oriented Bioinformatics Fax. +49-8161-71 2186
Technical University of Munich
http://mips.gsf.de/staff/pagel
Feb 9 '07 #1
5 5720
It could easily be the 2gig file size limitation, how large are the
extracts?

Feb 9 '07 #2
On 2007-02-09, Philipp Pagel <pD*******@gsf.dewrote:
>
Hi Pythoneers,

I need to process a large number of files which have been packed by the
UNIX compress tool (*.Z files). As I am not aware of a compress
equivalent of the gzip, zipfile or bzip2 modules, I thought I'd use the
uncompress or zcat commands directly to deal with the files:

for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()
This works fine for some files but results in

'write error onstdout: Broken pipe'

emitted by uncompress for others. Using zcat instead of uncompress
changes the wording of the error message but not the result.

I tried to give popen a large bufsize argument but that didn't really
help.

Probably I'm overlooking something obvious here but right now I can't
see it. Any hints?
As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.

--
Antoon Pardon
Feb 9 '07 #3
Antoon Pardon <ap*****@forel.vub.ac.bewrote:
On 2007-02-09, Philipp Pagel <pD*******@gsf.dewrote:
for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()

This works fine for some files but results in

'write error onstdout: Broken pipe'
As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.
You are right: some of the files do not fulfill certain
critereia causing so_somehting() to return before the entire file is
processed.

Thanks!

cu
Philipp

--
Dr. Philipp Pagel Tel. +49-8161-71 2131
Dept. of Genome Oriented Bioinformatics Fax. +49-8161-71 2186
Technical University of Munich
http://mips.gsf.de/staff/pagel
Feb 9 '07 #4
Chris <cw****@gmail.comwrote:
It could easily be the 2gig file size limitation, how large are the
extracts?
The files are much smaller than that, so that's not the issue.
Anyway, Antoon pointed me in the right direction.

Thanks for the help
Philipp

--
Dr. Philipp Pagel Tel. +49-8161-71 2131
Dept. of Genome Oriented Bioinformatics Fax. +49-8161-71 2186
Technical University of Munich
http://mips.gsf.de/staff/pagel
Feb 9 '07 #5
In article <eq**********@news.lrz-muenchen.de>,
Philipp Pagel <pD*******@gsf.dewrote:
Antoon Pardon <ap*****@forel.vub.ac.bewrote:
On 2007-02-09, Philipp Pagel <pD*******@gsf.dewrote:
for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()
>
This works fine for some files but results in
>
'write error onstdout: Broken pipe'
As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.

You are right: some of the files do not fulfill certain
critereia causing so_somehting() to return before the entire file is
processed.
Most programming environments don't have this problem, though.

If you like, your program can undo what Python does:

signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for filename in file_list:
...

Then it will work as if you had written it in C, or awk
or whatever.

Donn Cave, do**@u.washington.edu
Feb 9 '07 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Tim Black | last post by:
In my recent experience, popen os pipes always fail when cwd is a UNC path. Can anyone shed any light on this? Although I've seen lots of UNC path-related problems in this newsgroup, I've not been...
0
by: David H | last post by:
Background. I'm running on WinXP w/ MS Services for Unix installed (to give rsh/rlogin ability), both Python 2.3 and 2.4 version. In linux, I'm running RHEE with python2.3 version. The code...
0
by: Mark Sandler | last post by:
Hi, When i create a process using popen and then if i decide that i am not interested in keeping the process pipes anymore, and I forget the correspond variables . Then, if python tries to destroy...
6
by: Carl J. Van Arsdall | last post by:
I'm not sure the proper way to phrase the question, but let me try. Basically, I'm working with a script where someone wrote: kr = string.strip(os.popen('make kernelrelease').read()) And...
4
by: Kevin Walzer | last post by:
I'm trying to structure a Python script that streams output over a pipe. Here is my code: import os cmd = os.popen('echo foo | sudo -S /usr/sbin/tcpdump -en1') cmd.read() This returns...
13
by: bayer.justin | last post by:
Hi, I am trying to communicate with a subprocess via the subprocess module. Consider the following example: <subprocess.Popen object at 0x729f0> Here hey is immediately print to stdout of...
8
by: dmoore | last post by:
Hi folks, I've seen the following issue come up in multiple posts to this mailing list: I have a python program that spawns a child process with popen or popen2 or popen3 or popen2.popen2...
3
by: Ratko | last post by:
Hi all, I have a python gui app that launches multiple applications using subprocess.Popen class and prints their output in the gui (using PIPEs, threads and wxPython). Everything works great...
25
by: Jeremy Banks | last post by:
Hi. I wondered if anyone knew the rationale behind the naming of the Popen class in the subprocess module. Popen sounds like the a suitable name for a function that created a subprocess, but the...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.