Threads vs. Processes on web server

Hi,

I have a question about Python threads vs processes on web servers. I've been writing a cgi script which calls urlopen() at most 30-40 times, and I implemented a separate thread for each urlopen() call so that the total time takes about one second. However, this only works so far for 20 threads or less. Any more threads spawned, and it will give this error:

Exception in thread Thread-2:
Traceback (most recent call last): File
"/usr/lib64/python2.3/threading.py", line 436, in __bootstrap
self.run() File "test.py", line 214, in run current.start() File
"/usr/lib64/python2.3/threading.py", line 410, in start
_start_new_thread(self.__bootstrap, ()) error: can't start new thread

I asked my web hosting service about this, and they said that you can only have "at most 20 concurrent processes" running at the same time. Are threads that are spawned even considered processes? Or is it just because the threads are each taking too much memory?

Furthermore, is there a way to get around this or do some other improvements in my code so I can parse 30-40 urls but keep it down to one or two seconds? Thanks in advance.

Mar 25 '07 #1

Subscribe Post Reply

2369

bartonc

6,596

Expert 4TB

It seems to me that threads are contained in the main python "process" until you start talking about daemon threads:
"

A thread can be flagged as a ``daemon thread''. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set with the setDaemon() method and retrieved with the isDaemon() method.

"

Mar 25 '07 #2

sauce

Thanks bartonc.

I just read somewhere that Python's idea of threads is whatever the OS says threads are and that Linux says threads are lightweight processes. So since I'm only allowed to spawn 20 of these processes, is there anything I can do with my code to still call urlopen() 40 times efficiently? Would Stackless Python help?
My threads looks something like this:

Expand|Select|Wrap|Line Numbers

 
        for i in range(40):

            try:

                current = newThread()

                threadList.append(current)

                current.start()

            except IndexError: pass

        for t in threadlist:

            t.join()

Mar 25 '07 #3

sauce

Hmm...now this is strange.

I tried running the script straight from the terminal (I've never really done this before as all the other times were from the browser as a CGI script), and the terminal output is fine - 46 threads spawned without a problem, output and everything is how I want it to be.

But the script run from firefox hangs on the 23rd-24th thread. How can this be explained? problems with CGI and python?

Mar 25 '07 #4

by: Michael Williams | last post by:

Hi, I am trying to understand the performance implications of running a number of separate ActiveXexe processes as opposed to a single ActiveXexe with multiple threads on a Windows 2000 server....

Visual Basic 4 / 5 / 6

Help needed with logging API and threads

by: Huzefa | last post by:

I am working on a amll project in Java that includes many classes. Each of the classes has a Logger object. I have associated a FileHandler with each of these Logger objects. The file is the same...

Java

fork or threads

by: Andreas Müller | last post by:

i need two loops that run forever. one of it receives data and stores it to a vector. the other one writes the elements of the vector to the disk. this means the vector is a receiver buffer. How...

C / C++

help with concurrency control (threads/processes & signals)

by: Sori Schwimmer | last post by:

Hi, I am working on an application which involves interprocess communication. More to the point, processes should be able to notify other processes about certain situations, so the "notifyees"...

Python

167

What's the connection between objects and threads?

by: darren | last post by:

Hi I have to write a multi-threaded program. I decided to take an OO approach to it. I had the idea to wrap up all of the thread functions in a mix-in class called Threadable. Then when an...

C / C++

Migrating Website to Cloud - Emmanuel Katto

by: emmanuelkatto | last post by:

Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel

General

Looking to do Android software development, any suggestions? Is flutter better?

by: nemocccc | last post by:

hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?

General

Is that possible of reading the .csv file in column wise and the column have different lengths ?

by: Sonnysonu | last post by:

This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...

C / C++

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...

General

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

Online Marketing

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

Windows Server

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

General

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...

Career Advice

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

Microsoft Access / VBA

Threads vs. Processes on web server

Similar topics