473,699 Members | 2,311 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Capturing stdout incrementally

I have a large set of Python scripts that interface with command line
utilities (primarily Perforce). I am currently capturing ALL the text
output in order to get results and such. I am using the popen functions to
get the stdout, stderr streams.

However, some of the operations take a really long time (copying large files
over the network). If you run Perforce directly (or through os.system,
which doesn't return text output), it shows which files are getting copied,
one at a time. However, if I'm calling it through Python's popen, it
appears to hang while it copies all the files, then suddenly all the text
output appears at once after the operation is done.

Does anyone know a way around this? It is problematic because people think
that the program has hung, when it is really just taking a long time. I
would like the normal stdout to be printed on the screen as it is normally
(but also captured by my Python script simultaneously) .

I am on Windows by the way, so the utilities are printing to the windows
command shell.

thanks for any advice,
MB
Jul 18 '05 #1
5 2134
> I am on Windows by the way, so the utilities are printing to the windows
command shell.


By default, popen in Windows buffers everything coming from the called
app. I believe you can use various Windows system calls in pywin32 in
order to get line buffered output, but I've never done it, so have
little additional advice other than "check out pywin32".

- Josiah
Jul 18 '05 #2
Josiah Carlson <jc******@uci.e du> writes:
I am on Windows by the way, so the utilities are printing to the windows
command shell.


By default, popen in Windows buffers everything coming from the called
app. I believe you can use various Windows system calls in pywin32 in
order to get line buffered output, but I've never done it, so have
little additional advice other than "check out pywin32".


Typically this behavior on Windows (and at least in my experience on
Unix) comes not from popen buffering its input from the called
program, but the called program buffering its output when it isn't a
normal console. I'm not sure if this is default behavior from the C
library, but it's particularly true of cross-platform code that often
uses isatty() to check.

You can even see this behavior with Python itself (e.g., if you pipe a
copy of python to run a script).

In general, the only way to fix this is through the program being
called, and not the program doing the calling. In some cases, the
problem may have a way to disable buffering (for example, the "-u"
command line option with Python), or if it's something you have source
for you can explicitly disable output buffering. For example, Perl
has no command line option, but you can add code to the script to
disable output buffering.

On Unix, a classic way around code for which you have no source is to
run it under expect or some other pty-simulating code rather than a
simple pipe with popen. I'm not sure if there's a good pty/expectish
module that works well under Windows (where simulating what a
"console" is can be tougher).

I've only got an evaluation copy of perforce lying around, but I don't
immediately see any way to control its buffering via command line
options.

-- David
Jul 18 '05 #3
In article <uw***********@ fitlinxx.com>,
David Bolen <db**@fitlinxx. com> wrote:
Josiah Carlson <jc******@uci.e du> writes:
> I am on Windows by the way, so the utilities are printing to the windows
> command shell.

Jul 18 '05 #4
>>>I am on Windows by the way, so the utilities are printing to the windows
command shell.


By default, popen in Windows buffers everything coming from the called
app. I believe you can use various Windows system calls in pywin32 in
order to get line buffered output, but I've never done it, so have
little additional advice other than "check out pywin32".

Typically this behavior on Windows (and at least in my experience on
Unix) comes not from popen buffering its input from the called
program, but the called program buffering its output when it isn't a
normal console. I'm not sure if this is default behavior from the C
library, but it's particularly true of cross-platform code that often
uses isatty() to check.


Your experience in unix has colored your impression of popen on Windows.
The trick with Windows is that pipes going to/from apps are not real
file handles, nor do they support select calls (Windows select comes
from Windows' underlying socket library). If they did, then the Python
2.4 module Popen5 would not be required.

Popen5 is supposed to allow the combination of os.popen and os.system on
all platforms. You get pollable pipes and the signal that the program
ended with. As for how they did it on Windows, I believe they are using
pywin32 or ctypes.

- Josiah
Jul 18 '05 #5
Josiah Carlson <jc******@uci.e du> writes:
Your experience in unix has colored your impression of popen on
Windows. The trick with Windows is that pipes going to/from apps are
not real file handles, nor do they support select calls (Windows
select comes from Windows' underlying socket library). If they did,
then the Python 2.4 module Popen5 would not be required.
Pipes under Windows (at least for the built-in os.popen* calls) are
true OS file handles (in terms of Windows OS system handles), created
via a CreatePipe call which are connected to a child process created
with CreateProcess. You are correct that you can't select on them,
but that's not because they aren't real file handles, but because
Winsock under Windows is the odd man out. Sockets in Winsock aren't
equivalent to other native OS handles (they aren't the "real" file
handles), and select was only written to work with sockets. That's
also why sockets can't directly play in all of the other Windows
synchronization mechanisms (such as WaitFor[Multiple]Object[s]) but
you have to tie a socket to a different OS handle first, and then use
that handle in the sychronization call.
Popen5 is supposed to allow the combination of os.popen and os.system
on all platforms. You get pollable pipes and the signal that the
program ended with. As for how they did it on Windows, I believe they
are using pywin32 or ctypes.


I'm certainly all for additional portability for child process
management - although the internal os.popen* calls under Windows
already give you the exit code of the child process (which does get
some unique values when the process terminates abruptly), just without
any simulated signal bits.

But implementing popen5 under Windows will still run into the same
problem (that of select simply not working for other OS system handles
other than sockets), so I agree that will be a challenge, since
presumably they'll want to return a handle that looks like a Python
file and thus does have to have the underlying OS handle for basic I/O
to work.

Last I saw about the module was in January with a PEP announcement on
python-dev, but the PEP still indicates no Windows support (the
example was built on top of the popen2 module), and the python-dev
discussion led to proposing a start with a pure Python module. I
can't find any code in the current CVS tree related to popen5, so I'm
not sure of the status.

Of course, none of this changes the original question in this thread
in that if the child process is going to select output buffering based
on the "tty" or "console" aspect of the pipe to which its output is
connected, you can't override that from the calling program, but have
to deal with the program being executed (or more properly fake it out
so that the controlling pipe appears more tty/console like). I doubt
any popen* changes will affect that.

-- David

Jul 18 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
8132
by: Mark Wilson CPU | last post by:
This must be easy, but I'm missing something... I want to execute a Perl script, and capture ALL its output into a PHP variable. Here are my 2 files: ------------------------------------- test3.pl ------------------------------------- print "PERL Hello from Perl! (plain print)<br>\n"; print STDERR "PERL This is text sent to STDERR<br>\n"; $output="PERL Some output:<br>\n"; for ($i=0; $i<5; $i++) {
4
2439
by: Avi Kak | last post by:
Is there a Python function in any of the standard-distribution modules that does what the backticks do in Perl? I want to run an external command and I'd like its output to be captured directly in my Python script in the form of a string object. The function os.system() or any of the
1
5383
by: Oz | last post by:
This is long. Bear with me, as I will really go through all the convoluted stuff that shows there is a problem with streams (at least when used to redirect stdout). The basic idea is that my application (VB.NET) will start a process, redirect its stdout and capture that process' output, displaying it in a window. I've written a component for this, and a test application for the component. It allows me to specify a command to execute,...
2
3013
by: sergio | last post by:
i have a huge database that contains large amounts of html that i need to translate to ascii.. i have tried using html2text.py: http://www.aaronsw.com/2002/html2text/ but i could not figure out how to import it and use it as a library without getting errors everywhere..
5
2207
by: Luigi | last post by:
Hi to all! I'd like to execute an external program capturing the stdout/stderr messages at "real-time". I mean that I don't want to wait for the end of the process. If I write a code like this: import os import sys class Runner:
6
2264
by: Ed Leafe | last post by:
I've been approached by a local business that has been advised that they need to start capturing and archiving their instant messaging in order to comply with Sarbanes-Oxley. The company is largely PC, but has a significant number of Macs running OS X, too. Googling around quickly turns up IM Grabber for the PC, which would seem to be just what they need. But there is no equivalent to be found for OS X. So if anyone knows of any such...
3
3366
by: Fuzzyman | last post by:
Hello all, Before I ask the question a couple of notes : * This question is for implementing a script inside the Wing IDE. For some reason using the subprocess module doesn't work so I need a solution that doesn't use this module. * The platform is Windows and I'm happy with a Windoze only solution. :-)
1
1705
by: Falcolas | last post by:
I have a rather strange situation, and I'm not sure my brief experience of Python will let me handle it properly. The situation is this: I have a Java class "X" which I need to call in a Jython script. The output of "X" is sent to stdout using the java call System.out. I need to capture this output, and preform tests on it in the Jython script. My first pass at a design involves two jython scripts. One (we'll call it "Y") whose sole...
4
6002
by: amjadcsu | last post by:
Hi I am trying to execute a command using os.system. this command lists the number of nodes alive in a cluster. I would like to capture the output in list/array in python. IS it possible.?/ Here is my command gstat -a node13 2 ( 0/ 56) OFF node12 2 ( 1/ 63) OFF
0
8685
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8613
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9032
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8908
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
6532
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5869
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4374
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
2344
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2008
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.