Best way to spawn process on back end computer

sophie_newbie

Hi,

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

Thanks.

Oct 16 '08 #1

Subscribe Reply

2091

Simon Brunning

2008/10/16 sophie_newbie <pa**********@g mail.com>:

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

http://edit.kamaelia.org/ might be qworth a look.

--
Cheers,
Simon B.
si***@brunningo nline.net

Oct 16 '08 #2

Robin Becker

sophie_newbie wrote:

Hi,

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

The main problem here is that you'll probably need to detach the job to allow
the current cgi request to return a response to the client.

The implication of that is that the job either has to be anonymous and requires
no further attention or you need to provide some means of making the job
responsive to requests about its status so that a periodic request can be made
by the web page. That implies that the job can be identified and the creation
reponse returns the identity. One of the major problems is that the normal www
user has few privileges and cannot normally write to disk.

I have done this using both external shell scripts to do the main processing and
detaching and or python scripts that know how to detach. It was not terribly
easy or obvious.

Another alternative, as Simon's Kamaelia might indicate, is that you might
consider running a job server to service the cgi script requests on the remote
host. I have also done this as part of a web application. One of the advantages
was that the jobserver can run as any user and thus gets access to whatever the
owner has; additionally by providing a suitable protocol eg XMLRPC you can test
the jobserver without going through the web.
--
Robin Becker

Oct 16 '08 #3

Michael Sparks

sophie_newbie wrote:

Hi,

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

Best is always subjective - what sort of interaction are you after?

Web request to trigger an activity (fire and forget)

Web request to trigger an activity, which produces output suitable for
something else?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want to easily tie
the two things together?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want those two
events to look like part of one operation?

Picking a concrete example...

Or something a bit more like:
* User uploads an image
* Gets converted to a bunch of standard sizes, and placed into a queue
for moderation and then later use?

Or perhaps:
* User uploads a video
* That gets transcoded to a different video format ? (eg video
contribution transcoded to flash video?)

In the latter two cases, this is something I've needed to do, so I created a
simple WSGI filter that dumps the uploaded images/videos into a standard
file location, and then had a script that watches that standard location
for new images & videos, and does those conversions and transcodes.

Strictly speaking to be "clean" about it, you need to dump the file to a
temporary location whilst you're grabbing it from the network and when it's
been written to disk rename it.

I've no idea if this maps to your problem at all. (since I can think of a
few scenarios that match your description).

If it does match, then this code may be handy:
http://code.google.com/p/kamaelia/so...leProcessor.py

If you want to use that code, let me know and I'll package it up. There's an
older version here:
* http://www.kamaelia.org/release/Kama...r-0.1.0.tar.gz

Regards,
Michael.
--
http://www.kamaelia.org/Home

Oct 23 '08 #4

Similar topics

8813

spawn or fork

by: C Gillespie | last post by:

Dear All, I have a function def printHello(): fp = open('file','w') fp.write('hello') fp.close() I would like to call that function using spawn or fork. My questions are:

Python

2120

Tough Spawn Problem

by: googlemike | last post by:

I'm trying to use Python and PyGTK + Glade, along with Bash. I want to make a GUI for the Linux vpnclient command-line tool. (Look for linux-vpnclient.tar.gz on the Internet if you are curious.) Anyway, this vpnclient tool connects to VPN and then stays locked like that while the VPN connection is running. What I was trying to do was either spawn it and then use another command-line tool to check on its status, or shell the process and...

Python

1258

Best Practice for Uploading to SQL Server?

by: The Oracle | last post by:

Hi there, I have an ArrayList containing approximately 20,000 database INSERT command objects. Each of these commands needs to be executed. The commands do not need to be executed in any particular order, nor do they need to be part of a transaction. I'm currently accomplishing this task by iterating through the ArrayList via the foreach construct and Executing each command. This works fine but I'm wondering if there's a way to get...

C# / C Sharp

1619

Best Practice to spawn an Application from a Web Service ?

by: Paul | last post by:

Hi all. Can someone provide some help on the following as there seems to be many different methods of acheiving the same outcome. Basically I am trying to develop a web service which will spawn an exe file to run an import and then return a result back to the web service, this would also need the facility to spawn multiple copies of the application upto a maximum figure..............

ASP.NET

1852

Spawn new browser from server process

by: Teckie03 | last post by:

Hi, does anyone know how to display a seperate browser from server process? My ASP.NET app control (ascx) has Archive button that does its own work , including updating an html page called Index.html. After finishing its work, it should spawn a new browser and display the updated Index.html file. How can I do that? Thank you for any response.

ASP.NET

1525

can web browser spawn specific applicaion process at client side?

by: david | last post by:

It seems that it can not do it due to security. My boss believes that .NET ASP web form application can spawn a local application at the client computer via web browser. Can we do it? How to do it? Is there any alternative way to do it? David

ASP.NET

2685

Spawn/Exec with asterisk in argument

by: jeremyfee | last post by:

The spawn* and exec* functions appear to escape asterisks, I'm guessing all shell characters too, before the spawn/exec'ed process sees them. Is there a way around this? Not sure if this is a bug or a "feature". user$ touch test.txt user$ ls -l * -rw-r--r-- 1 user user 0 Apr 18 18:30 test.txt user$ ls -l \*

Python

4092

Win32::Spawn not executing embedded process

by: somuchh8 | last post by:

Hi, I'm having a lot of trouble with the Win32::Spawn module in perl. Here is my situation, I have a Win32::Spawn call which looks like this: my $success = undef; my $cmdline = EBDTools::os_path("${nh_home}/web/aview/modules/svcrsp-ng/saSync.pl"); $success = Win32::Spawn($^X, "${^X} $cmdline ${optfile}", $pid); if (! $success) { my $lasterr = Win32::GetLastError(); return_configerror("Failed to create commit / sync process...

Perl

2370

portable fork+exec/spawn

by: Brendan Miller | last post by:

I want to spawn a child process based on an external executable that I have the path for. I then want to wait on that executable, and capture it's output. In the os module, fork is only supported on unix, but spawn is only supported on windows. The os.system call is implemented by calling the C system call, which is of course inefficient and has portability gotchas because it calls the underlying system shell (sh for unix, cmd for...

Python

8613

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

9172

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

8908

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

7745

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

4374

Trying to create a lan-to-lan vpn between two differents networks

by: TSSRALBI | last post by:

Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...

Networking - Hardware / Configuration

4626

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

3054

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

2344

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP

2008

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General