473,240 Members | 1,608 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,240 software developers and data experts.

Advice on long running processes

Hello,

I write a lot of CGI scripts, in Python of course. Now I need to
convert some to long-running processes. I'm having trouble finding
resources about the best practices to do that.

I've found a lot of email discussions that say something like, "You
need to educate yourself about the differences when you have long-
running processes" but I've not had a lot of luck with finding things
that explain the differences. I've seen some talk about database
timeouts, for instance, but I'm not sure I understand the problems.
Can anyone here suggest some resources? I'd be happy with web sites,
with buying a book, anything.

I ask here because I write in Python and so if those resources used
Python then that would be super.

Thanks,
Jim

Oct 11 '07 #1
4 2826
co*************@hotmail.com wrote:
Hello,

I write a lot of CGI scripts, in Python of course. Now I need to
convert some to long-running processes. I'm having trouble finding
resources about the best practices to do that.

I've found a lot of email discussions that say something like, "You
need to educate yourself about the differences when you have long-
running processes" but I've not had a lot of luck with finding things
that explain the differences.
The biggest differences between run-and-exit vs. long running processes are
resource management and error recovery. Let's take them one at a time.

Resource management. In a short-lived process, you really don't have to
worry about this at all. Snarf as much memory as you need, open as many
files as you want, and when you exit, the operating system cleans it all up
for you. With a long running process, you have to worry about stuff like
that.

In Python, you're isolate from the low-level details of memory management,
but still need to think about it a bit. Imagine you had code that looked
like this in your main loop:

for request in getNextRequest():
requestList.append (request)
processRequest(request)

requestList is going to keep growing without bounds and eventually will eat
up all available memory in the system and your process will crash.
Everything you store, you also need to delete when you're done with it.

Same with files. In a short-lived process, you can generally open as many
files as you want and never worry about closing them. It unlikely you will
ever run out of file descriptors. In a long running process, that's not
the case. If you open a new file each time you get a request and never
close it, after a few hundred requests (depending on the operating system,
maybe even a few thousand), you'll run out of file descriptors.

The other big thing is error recovery. In a short lived process, if
something fails, you print an error message and exit. In a long running
process, you need to somehow recover from the error and keep going as best
you can. This can be tricky.
Oct 11 '07 #2
Roy Smith wrote:
co*************@hotmail.com wrote:

>>Hello,

I write a lot of CGI scripts, in Python of course. Now I need to
convert some to long-running processes. I'm having trouble finding
resources about the best practices to do that.

I've found a lot of email discussions that say something like, "You
need to educate yourself about the differences when you have long-
running processes" but I've not had a lot of luck with finding things
that explain the differences.


The biggest differences between run-and-exit vs. long running processes are
resource management and error recovery. Let's take them one at a time.

Resource management. In a short-lived process, you really don't have to
worry about this at all. Snarf as much memory as you need, open as many
files as you want, and when you exit, the operating system cleans it all up
for you. With a long running process, you have to worry about stuff like
that.

In Python, you're isolate from the low-level details of memory management,
but still need to think about it a bit. Imagine you had code that looked
like this in your main loop:

for request in getNextRequest():
requestList.append (request)
processRequest(request)

requestList is going to keep growing without bounds and eventually will eat
up all available memory in the system and your process will crash.
Everything you store, you also need to delete when you're done with it.
In particular, it is a good idea to call gc.collect() every now
and then, specially if you are in such a loop. I don't know what is the
gc policy in python, but an application of mine that seemed to eat as much
memory as it was available was reduced to a constant small amount of
memory after I started to call the gc directly.
The other big thing is error recovery. In a short lived process, if
something fails, you print an error message and exit. In a long running
process, you need to somehow recover from the error and keep going as best
you can. This can be tricky.
You should have your main loop inside a try/except, to catch any
exceptions that were not otherwise caught without exiting the application.
Log the exceptions, of course, but in most long running applications
work in a loop like Roy Smith's code above, so if one of them fails it
won't disrupt the others to come.

--
Bruno Barberi Gnecco <brunobg_at_users.sourceforge.net>
The earth is like a tiny grain of sand, only much, much heavier.
Oct 11 '07 #3
co*************@hotmail.com a écrit :
Hello,

I write a lot of CGI scripts, in Python of course. Now I need to
convert some to long-running processes. I'm having trouble finding
resources about the best practices to do that.

I've found a lot of email discussions that say something like, "You
need to educate yourself about the differences when you have long-
running processes" but I've not had a lot of luck with finding things
that explain the differences. I've seen some talk about database
timeouts, for instance, but I'm not sure I understand the problems.
Can anyone here suggest some resources? I'd be happy with web sites,
with buying a book, anything.

I ask here because I write in Python and so if those resources used
Python then that would be super.
As far as I'm concerned, I'd go for one of the available wsgi frameworks.
Oct 11 '07 #4
Thank you to folks for the replies.

Jim

Oct 12 '07 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

13
by: BK | last post by:
Can someone point me to a code sample that illustrates executing long running tasks (asynchronous) from a web application in ASP.NET? I assume that Web Services might come into play at some point,...
29
by: pb648174 | last post by:
I have a very long transaction that runs on the same database that other users need to use for existing data. I don't care if they see data from the transaction before it is done and am only using...
16
by: Justin Lazanowski | last post by:
Cross posting this question on the recommendation of an I have a .NET application that I am developing in C# I am loading information in from a dataset, and then pushing the dataset to a grid,...
3
by: Alex | last post by:
Hi all, I'm looking for some advice on how best to implement storage of access logs into a db/2 8.1.4 database running on a RH 7.2 system. I have 5 (squid) web caches running here that...
8
by: Cider123 | last post by:
I ran into a situation where my Window Service had to process 100,000+ files, when I first noticed I needed to tweak various routines. Everything runs fine, but here's what I ran into: In the...
5
by: Ekempd | last post by:
Hi I need some advice about this situation an how I'm current handling it I hava a ASP.NET solution that in some point start multiples lenghty verification agains diferent databases, my big...
2
by: Steve | last post by:
Hi, I'm trying to design a web server where users can login, initiate long running processes on uploaded files, monitor the progress via a web page and download some results when the process has...
6
by: seb | last post by:
Hi, I am using pygtk for the first times. I am wondering what would be the best "pattern" to interface pygtk with a thread. The thread is collecting informations (over the network for...
1
by: walterbyrd | last post by:
I understand that Python has them, but PHP doesn't. I think that is because mod_php is built into apache, but mod_python is not usually in apache. If mod_python was built into apache, would...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.