473,703 Members | 2,308 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Running long script in the background

Hello,

I am trying to write a python cgi that calls a script over ssh, the
problem is the script takes a very long time to execute so Apache
makes the CGI time out and I never see any output. The script is set
to print a progress report to stdout every 3 seconds but I never see
any output until the child process is killed.

Here's what I have in my python script:

command = "ssh -l root %s /scripts/xen/xen-create-win-vps1.sh %s" %
(host, domuname)
output = os.popen(comman d)
for line in output:
print line.strip()

Here's a copy of the bash script.

http://watters.ws/script.txt

I also tried using os.spawnv to run ssh in the background and nothing
happens.

Does anybody know a way to make output show in real time?

Feb 6 '07 #1
13 5998
Does anybody know a way to make output show in real time?

You can put: #!/usr/bin/python -u
at the top of the script to have unbuffered binary stdout and stderr.

Feb 6 '07 #2
wa*******@gmail .com wrote:
Hello,

I am trying to write a python cgi that calls a script over ssh, the
problem is the script takes a very long time to execute so Apache
makes the CGI time out and I never see any output. The script is set
to print a progress report to stdout every 3 seconds but I never see
any output until the child process is killed.

Here's what I have in my python script:

command = "ssh -l root %s /scripts/xen/xen-create-win-vps1.sh %s" %
(host, domuname)
output = os.popen(comman d)
for line in output:
print line.strip()
try sys.stdout.flus h() after every print.

Or try something like this:
import sys, time

class FlushFile:
def __init__(self, fd):
self.fd = fd
def flush(self):
self.fd.flush()
def write(self, str):
self.fd.write(s tr)
self.fd.flush()

oldstdout = sys.stdout
sys.stdout = FlushFile(sys.s tdout)

for i in range(5):
print "Hello",
time.sleep(0.5)
print

--
Thomas Güttler, http://www.thomas-guettler.de/ http://www.tbz-pariv.de/
E-Mail: guettli (*) thomas-guettler + de
Spam Catcher: ni************* *@thomas-guettler.de

Feb 6 '07 #3
On Feb 6, 8:36 am, "jasonmc" <jasonmccandl.. .@gmail.comwrot e:
Does anybody know a way to make output show in real time?

You can put: #!/usr/bin/python -u
at the top of the script to have unbuffered binary stdout and stderr.

Thanks. I tried that but it still times out waiting for output.

Everything works fine until I call the popen function, then it
freezes. What I want is to print the output in real time, just like
it does when I run it from a shell.

Feb 6 '07 #4
On Feb 6, 10:37 am, "watter...@gmai l.com" <watter...@gmai l.comwrote:
On Feb 6, 8:36 am, "jasonmc" <jasonmccandl.. .@gmail.comwrot e:
Does anybody know a way to make output show in real time?
You can put: #!/usr/bin/python -u
at the top of the script to have unbuffered binary stdout and stderr.

Thanks. I tried that but it still times out waiting for output.

Everything works fine until I call the popen function, then it
freezes. What I want is to print the output in real time, just like
it does when I run it from a shell.

I tried flushing stdout and the same thing happens. As soon as the
os.popen(comman d) line runs it stops there, the next print statement
never even runs.

I've also tried using os.spawnv to make the process run in the
background but then the ssh command never runs.

Feb 6 '07 #5
wa*******@gmail .com wrote:
I tried flushing stdout and the same thing happens. As soon as the
os.popen(comman d) line runs it stops there, the next print statement
never even runs.

I've also tried using os.spawnv to make the process run in the
background but then the ssh command never runs.
Based on what you describe, this isn't a good application for a
single-transaction CGI exchange. The timeouts are not happening at the
level of your CGI script, but rather either at the HTTP server itself or
at the remote client. In either case, fixing it as a one-transaction,
one-script solution is not going to be very feasible.

A more sensible way to do it is to have one logical page (which could be
the same physical page if you want) which accepts job requests, spawns
them off in the background, and offers a link to a second logical page
which sees if the job has completed -- showing the results if it has --
or refreshes periodically if it hasn't yet.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
You could have another fate / You could be in another place
-- Anggun
Feb 6 '07 #6
On Feb 6, 2:02 pm, Dennis Lee Bieber <wlfr...@ix.net com.comwrote:
On 6 Feb 2007 07:37:33 -0800, "watter...@gmai l.com"
<watter...@gmai l.comdeclaimed the following in comp.lang.pytho n:
Everything works fine until I call the popen function, then it
freezes. What I want is to print the output in real time, just like
it does when I run it from a shell.

And you want /this/ in a web page?

I don't think HTTP is designed for that... As I understand it, it
expects to get a complete page back and then the transaction is complete
and forgotten (except for the presence of session cookies). To report
dynamically on a web page tends to either be something like a
timed-redirect (reload) of the same URL with the cookie, and that is a
completely separate transaction starting a new CGI (or equivalent)
process. AJAX techniques may clean up some of this -- by not really
reloading the whole page, instead updating the DOM based upon data
transferred.

Web pages can show output as it's sent. For testing I created a
script on the server that untars a 600 meg volume, I can see each file
name show up in my browser instantly, just like it should. The other
script I'm trying to run won't show anything until the entire process
is complete and it's just a bunch of echo statements in a for loop,
I'm not sure why they behave differently.

Feb 6 '07 #7
wa*******@gmail .com wrote:
Web pages can show output as it's sent. For testing I created a
script on the server that untars a 600 meg volume, I can see each file
name show up in my browser instantly, just like it should. The other
script I'm trying to run won't show anything until the entire process
is complete and it's just a bunch of echo statements in a for loop,
I'm not sure why they behave differently.
In a word: buffering.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
You could have another fate / You could be in another place
-- Anggun
Feb 6 '07 #8
En Tue, 06 Feb 2007 16:44:52 -0300, wa*******@gmail .com
<wa*******@gmai l.comescribió:
On Feb 6, 2:02 pm, Dennis Lee Bieber <wlfr...@ix.net com.comwrote:
>On 6 Feb 2007 07:37:33 -0800, "watter...@gmai l.com"
<watter...@gma il.comdeclaimed the following in comp.lang.pytho n:
Everything works fine until I call the popen function, then it
freezes. What I want is to print the output in real time, just like
it does when I run it from a shell.

And you want /this/ in a web page?

I don't think HTTP is designed for that... As I understand it,
it
expects to get a complete page back and then the transaction is complete
and forgotten (except for the presence of session cookies). To report
If the response does not include a Content-Length header, and has a
Transfer-Encoding: chunked header, then it is sent in chunks (blocks) and
the client is able to process it piece by piece.
See the server docs on how to enable and generate a chunked response. On
Zope 2, by example, it's enough to use response.write( ).
Web pages can show output as it's sent. For testing I created a
script on the server that untars a 600 meg volume, I can see each file
name show up in my browser instantly, just like it should. The other
script I'm trying to run won't show anything until the entire process
is complete and it's just a bunch of echo statements in a for loop,
I'm not sure why they behave differently.
Are you sure the other process is executing? and not buffered? and you're
reading its output line by line?

--
Gabriel Genellina

Feb 6 '07 #9
On Feb 6, 5:26 am, "watter...@gmai l.com" <watter...@gmai l.comwrote:
Hello,

I am trying to write a python cgi that calls a script over ssh, the
problem is the script takes a very long time to execute so Apache
makes the CGI time out and I never see any output. The script is set
to print a progress report to stdout every 3 seconds but I never see
any output until the child process is killed.
<snip>
>
Does anybody know a way to make output show in real time?
Try this:

<code>

# test.py
import os
import sys
import time

def command():
for x in range(5):
print x
sys.stdout.flus h()
time.sleep(1)

def main():
command = 'python -c "import test; test.command()" '
print 'running: %s' % command
output = os.popen(comman d, 'r', 1)
while True:
line = output.readline ()
if line == '':
break
sys.stdout.writ e(line)
sys.stdout.flus h()

if __name__ == '__main__':
main()

</code>

The problem is with using the file-like object returned by popen as an
iterator. It will block until the child process is killed, so just
iterate across it manually.

Pete

Feb 7 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
3320
by: Sticks | last post by:
ok... im not quite sure how to describe my problem. i have a php script that runs through my entire php site and writes the resulting output to html files. this is necessary as the nature of the hosting available to me for this particular page prohibits me from using php/mysql as i would like. my script works simply by using output buffers and an include the relevant section of code is as follows: ob_start();
7
3216
by: Arun | last post by:
Hi, This is a scripting question, but since I am writing the script in python I am posting this question here: I have a python script that runs a simulator (that was written in c++, so I use the system() function to run the simulator) over a list of config files. I want the python script to itself run as well as start the simulations totally in the background (the simulations run for a day or two, so basically I want to start them in...
29
5807
by: pb648174 | last post by:
I have a very long transaction that runs on the same database that other users need to use for existing data. I don't care if they see data from the transaction before it is done and am only using the transaction because I need a way to roll it back if any errors happen during the transaction. Unfortunately all tables affected in the long running transaction are completely locked and nobody else can access any of the affected tables while...
1
6018
by: Anonieko | last post by:
Query: How to display progress bar for long running page Answer: Yet another solution. REFERENCE: http://www.eggheadcafe.com/articles/20050108.asp My only regret is that when click the browser back button, it loads the progress bar again. Any solution to this?
2
4871
by: Dave Hughes | last post by:
Just noticed something rather annoying after upgrading my test box (a Linux server running DB2 UDB v8 for LUW) to fixpak 11 (for reference it was previously on fixpak 7). In the past I've relied heavily on the ability to start multiple long-running SQL scripts in the background and have them all run in parallel. Unfortunately, it seems that somewhere between fixpak 7 and fixpak 11 something has been fixed / broken which prevents more...
14
23160
by: lmttag | last post by:
Hello. We're developing an ASP.NET 2.0 (C#) application and we're trying to AJAX-enable it. We're having problem with a page not showing the page while a long-running process is executing. So, we're looking for a way to display the page with a "please wait..." message while the process is running, and then, when the process is done, update the page with the actual results/page content. We have a page that opens another browser/page...
5
3482
by: This | last post by:
I have a pretty basic emailing script that sends a relatively small number (150) of html emails. The emails are compiled, personalised from a mysql db subscribers list, and sent using mail() - after sending, a small summary html page is sent to the user with number sent, time taken and a simple navigation choice. Up to about 100 emails it all works fine - this takes the server about 27 secs . Any more than that and although the emails are...
3
2424
by: sophie_newbie | last post by:
Hi, I have a cgi script that performs a very long computation that can take several hours to complete. Is there any smart way that I can keep this script running until it is finished (after the user has closed the browser) and email them with the results. The email bit isn't the problem, I just don't know how to keep the code running in the background. I'm sure there is a smart way to do this... Thanks!
0
8669
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9251
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9122
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
9017
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8963
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
5922
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4433
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4687
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3125
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.