473,324 Members | 2,548 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

thread help

Howdy,

Below is a script that I'm using to try and count the number of HTTP
servers within a company's private network. There are 65,536 possible
hosts that may have HTTP servers on them. Any way, I wrote this script
at first w/o threads. It works, but it takes days to run as it probes
one IP at a time... so I thought I'd try to make it threaded so it could
test several dozen IPs at once.

I'm no expert on threading, far from it. Could someone show me how I can
make this work correctly? I want to probe 64 unique IP address for HTTP
servers simultaneously, not the same IP addy 64 times (as I'm doing
now). Any tips would be much appreciated.

Bart

import urllib2, socket, threading, time

class trivialthread(threading.Thread):
def run(self):
socket.setdefaulttimeout(1)

hosts = []
networks = []

# Add the network 192.168.0 possibility.
networks.append("192.168.0.")
n = 0
while n < 255:
n = n + 1
# Generate and add networks 192.168.1-255 to the list of networks.
networks.append("192.168.%s." %(n))

for network in networks:
h = 0
# Add the n.n.n.0 host possibility
hosts.append(network+str(h))
while h < 255:
h = h + 1
# Add hosts 1 - 255 to each network.
hosts.append(network+str(h))

websites = file('websites.txt', 'w')
for ip in hosts:
try:
f = urllib2.urlopen("http://%s" %ip)
f.read()
f.close()
print>> websites, ip
except urllib2.URLError:
print ip
except socket.timeout:
print ip, "Timed Out..................."
except socket.sslerror:
print ip, "SSL Error..................."
websites.close()

if __name__ == '__main__':
threads = []
for x in range(64):
thread = trivialthread()
threads.append(thread)
for thread in threads:
thread.start()
while threading.activeCount() > 0:
print str(threading.activeCount()), "threads running incl. main"
time.sleep(1)
Jul 18 '05 #1
11 1665
In article <ca**********@solaris.cc.vt.edu>,
Bart Nessux <ba*********@hotmail.com> wrote:

I'm no expert on threading, far from it. Could someone show me how I can
make this work correctly? I want to probe 64 unique IP address for HTTP
servers simultaneously, not the same IP addy 64 times (as I'm doing
now). Any tips would be much appreciated.


Create a threading.Thread subclass that takes one IP address and a list
of ports to scan. Start 64 instances of this class, each with a
different IP address.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha
Jul 18 '05 #2
Aahz wrote:
Bart Nessux <ba*********@hotmail.com> wrote:
Could someone show me how I can make this work correctly? I want to probe
64 unique IP address for HTTP servers simultaneously, ...
Create a threading.Thread subclass that takes one IP address and a list
of ports to scan. Start 64 instances of this class, each with a
different IP address.


An alternative is to create a que into which you push IP addresses to
contact, and have each thread read addresses off the queue when they are
free to process them. This has the advantage of decoupling the number
of threads from the number of addresses you want to examine.

-Scott David Daniels
Sc***********@Acm.Org
Jul 18 '05 #3
In article <40********@nntp0.pdx.net>,
Scott David Daniels <Sc***********@Acm.Org> wrote:
Aahz wrote:
Bart Nessux <ba*********@hotmail.com> wrote:

Could someone show me how I can make this work correctly? I want to probe
64 unique IP address for HTTP servers simultaneously, ...


Create a threading.Thread subclass that takes one IP address and a list
of ports to scan. Start 64 instances of this class, each with a
different IP address.


An alternative is to create a que into which you push IP addresses to
contact, and have each thread read addresses off the queue when they are
free to process them. This has the advantage of decoupling the number
of threads from the number of addresses you want to examine.


Absolutely, but that requires a bit more work for someone who isn't
already familiar with threading.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha
Jul 18 '05 #4
Scott David Daniels wrote:
An alternative is to create a que into which you push IP addresses to
contact, and have each thread read addresses off the queue when they are
free to process them. This has the advantage of decoupling the number
of threads from the number of addresses you want to examine.


That is also the general best design pattern for doing threading.
Have a Queue.Queue object where you place work items and have
threads pull items off the queue and execute it. You can use
callbacks or another Queue for placing the results on.

The Queue.Queue class has the nice property that it is very
thread safe, and you can do both blocking and timed waits
on it.

The other problem to deal with that is particular to Python is
how to stop threads when you want to shutdown or cancel actions.
At the very least you can have a 'shutdown' message you place
on the Queue that when any thread reads, it shuts down.

Unfortunately Python doesn't allow interrupting a thread,
so any thread doing something will run to completion. You
can check a variable or something in lines of Python
code, but cannot do anything when in C code. For example
if you do some networking stuff and the C code (eg a DNS
lookup followed by a TCP connect) takes 2 minutes, then
you will have to wait at least that long.

In the simplest case you can just make all your threads be
daemon. Python will shutdown when there are no non-daemon
threads left, so you can just exit your main loop and all
will shutdown. However that means the worker threads just
get abruptly stopped in the middle of what they were
doing.

(IMHO it would be *really* nice if Python provided a way
to interrupt threads).

Roger
Jul 18 '05 #5
Roger Binns wrote:
In the simplest case you can just make all your threads be
daemon. Python will shutdown when there are no non-daemon
threads left, so you can just exit your main loop and all
will shutdown. However that means the worker threads just
get abruptly stopped in the middle of what they were
doing.

(IMHO it would be *really* nice if Python provided a way
to interrupt threads).


Sounds like you can't eat your cake and have it, too. If
you _could_ interrupt threads**, wouldn't that mean "the worker
threads just get abruptly stopped in the middle of what they
were doing"?

-Peter

** There is a way to interrupt threads in Python now, but
it requires an extension routine, or perhaps something with
ctypes. Findable in the archives for this newsgroup/list.
Jul 18 '05 #6
Peter Hansen wrote:
Sounds like you can't eat your cake and have it, too. If
you _could_ interrupt threads**, wouldn't that mean "the worker
threads just get abruptly stopped in the middle of what they
were doing"?


I meant in the same way that can in Java. In that case an
InterruptedException is thrown which the thread can catch
and do whatever it wants with.

As an example at the moment, socket.accept is a blocking
call and if a thread is executing that there is no way
of stopping it.

This would make shutdown and reconfigurations possible.
For example you could interrupt all relevant threads
and they could check a variable to see if they should
shutdown, bind to a different port, abandon the current
work item etc.

Roger
Jul 18 '05 #7
Roger Binns wrote:
Peter Hansen wrote:
Sounds like you can't eat your cake and have it, too. If
you _could_ interrupt threads**, wouldn't that mean "the worker
threads just get abruptly stopped in the middle of what they
were doing"?


I meant in the same way that can in Java. In that case an
InterruptedException is thrown which the thread can catch
and do whatever it wants with.


I didn't think things worked quite that way in Java. For
example, I thought InterruptedException was seen by a thread
only if it had actually been asleep at the time it was sent.

I also didn't know it would actually terminate certain
blocking calls, such as in socket stuff.

Oh well, it's been a while...

-Peter
Jul 18 '05 #8
Scott David Daniels wrote:
Aahz wrote:
Bart Nessux <ba*********@hotmail.com> wrote:
Could someone show me how I can make this work correctly? I want to
probe >>64 unique IP address for HTTP servers simultaneously, ...

Create a threading.Thread subclass that takes one IP address and a list
of ports to scan. Start 64 instances of this class, each with a
different IP address.

An alternative is to create a que into which you push IP addresses to
contact, and have each thread read addresses off the queue when they are
free to process them. This has the advantage of decoupling the number
of threads from the number of addresses you want to examine.

-Scott David Daniels
Sc***********@Acm.Org


I like this idea. I read up on the queue and threading module at
python.org and a few other sites around the Web and came up with this,
however, it doesn't work. I get these errors when it runs:

Exception in thread Thread-149:
Traceback (most recent call last):
File "/usr/lib/python2.3/threading.py", line 434, in __bootstrap
self.run()
File "/usr/lib/python2.3/threading.py", line 414, in run
self.__target(*self.__args, **self.__kwargs)
File "www_reads_threaded_1.py", line 49, in sub_thread_proc
f = urllib2.urlopen(url).read()
File "/usr/lib/python2.3/urllib2.py", line 129, in urlopen
return _opener.open(url, data)
File "/usr/lib/python2.3/urllib2.py", line 324, in open
type_ = req.get_type()
AttributeError: 'NoneType' object has no attribute 'get_type'

The problem I have is this: I know too little about thread programming.
If anyone thinks the code I have below could be made to work for my
tasks (probe 65,000 IPs for HTTP servers using threads to speed things
up), then please *show* me how I might change it in order for it to work.

Thanks again,
Bart

networks = []
hosts = []
urls = []
#socket.setdefaulttimeout(30)
max_threads = 2
http_timeout = 30
start_time = time.time()

# Add the network 192.168.0 possibility.
networks.append("192.168.0.")

# Generate and add networks 192.168.1-255 to the list of networks.
n = 0
while n < 255:
n = n + 1
networks.append("192.168.%s." %(n))

# Generate and add hosts 1-255 to each network
for network in networks:
h = 0
# Add the n.n.n.0 host possibility
hosts.append(network+str(h))
while h < 255:
h = h + 1
hosts.append(network+str(h))

for ip in hosts:
ip = "http://" + ip
urls.append(ip)

urls = dict(zip(urls,urls))
# print urls

# Create a queue of urls to feed the threads
url_queue = Queue.Queue()
for url in urls:
url_queue.put(url)
# print url

def test_HTTP(url_queue):
def sub_thread_proc(url, result):
# try:
f = urllib2.urlopen(url).read()
# except Exception:
# print "Exception"
# else:
result.append(url)
while 1:
try:
url = url_queue.get(0)
except Queue.Empty:
return
result = []
sub_thread = threading.Thread(target=sub_thread_proc,
args=(url,result))
sub_thread.setDaemon(True)
sub_thread.start()
sub_thread.join(http_timeout)
print result

test_HTTP(urls)
Jul 18 '05 #9
Bart Nessux <ba*********@hotmail.com> writes:

The problem I have is this: I know too little about thread programming.
If anyone thinks the code I have below could be made to work for my
tasks (probe 65,000 IPs for HTTP servers using threads to speed things
up), then please *show* me how I might change it in order for it to work.


I haven't been following this thread but if I was doing this I would want to
use asynchronous programming. It would finally force me to get to grips with
twisted.

Eddie
Jul 18 '05 #10
One word Eddie... WOW!

This async stuff is fabulous! It works and it's dead easy for my
application. There is no cpu limit with what I'm doing... only I/O
problems. At your suggestion, I looked at twisted, and then just used
the standard python asyncore module because it looked so darn easy, and
as it turned out, it was.

Thanks a million for the advice. I was looking in the *wrong* direction.

Eddie Corns wrote:
Bart Nessux <ba*********@hotmail.com> writes:
The problem I have is this: I know too little about thread programming.
If anyone thinks the code I have below could be made to work for my
tasks (probe 65,000 IPs for HTTP servers using threads to speed things
up), then please *show* me how I might change it in order for it to work.

I haven't been following this thread but if I was doing this I would want to
use asynchronous programming. It would finally force me to get to grips with
twisted.

Eddie

Jul 18 '05 #11
Peter Hansen wrote:
I didn't think things worked quite that way in Java. For
example, I thought InterruptedException was seen by a thread
only if it had actually been asleep at the time it was sent.

I also didn't know it would actually terminate certain
blocking calls, such as in socket stuff.


http://java.sun.com/j2se/1.4.2/docs/...tml#interrupt()

As is the Java way, they have different types of interrupted
exceptions and sockets etc would need to be InterruptibleChannels.
They also use checked exceptions which makes life a lot harder.

More on Java best practises for thread interruption:

http://java.sun.com/j2se/1.4.2/docs/...precation.html

I would just settle for a nice clean mechanism whereby you can
call Thread.interrupt() and have an InterruptException in that
thread (which ultimately terminates the thread if it isn't
handled anywhere).

Some threads won't be interruptible because they are deep in
extension libraries. Perhaps that can be returned to the
caller of Thread.interrupt().

Roger
Jul 18 '05 #12

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

12
by: serge calderara | last post by:
Dear all, I have a function that I need to run in a thread due to the fact that it can takes long time to execute according to the amount of data to collect. This function is also populating a...
6
by: Tony Proctor | last post by:
Hi everyone We're experiencing some serious anomalies with the scheduling of ASP threads. I'd be interested to hear if anyone knows what algorithm is used (e.g. simple round-robin, or something...
4
by: Leonardo Hyppolito | last post by:
Hello, I am trying to write a multithread program that simulates producers and consumers. My program can have many producers and many consumers (each in a separate thread). It has a storage...
7
by: Ivan | last post by:
Hi I have following problem: I'm creating two threads who are performing some tasks. When one thread finished I would like to restart her again (e.g. new job). Following example demonstrates...
20
by: Bob Day | last post by:
Using VS 2003, VB, MSDE... There are two threads, A & B, that continously run and are started by Sub Main. They instantiationsl of identical code. Thread A handles call activity on telephone...
7
by: Charles Law | last post by:
My first thought was to call WorkerThread.Suspend but the help cautions against this (for good reason) because the caller has no control over where the thread actually stops, and it might have...
1
by: alundi | last post by:
Greetings: I'd like to post this as a new thread to an article in microsoft.public.dotnet.languages.vb originally made by nevin and replied to by Alan Pretre back in December ("ThreadState ==...
6
by: Sergey Poberezovskiy | last post by:
I have the following code in C# that I have trouble converting to VB(2.0): private delegate void openDialog(); private void openWindowsDialog(openDialog open) { Thread thread = new Thread(new...
8
by: Brad Walton | last post by:
Hello. First post, but been doing a bit of reading here. I am working on a project in Java, but decided to switch over to C# after seeing some of the additional features I can get from C#. One of...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.