Multithreaded Telnet sessions

[Richard Bird CCNP, CCDP, MCSE, etc.]

I need some help adding multithreading to an existing python script.
The script was developed to telnet and make changes on a list of cisco
routers.
I figure that by adding multithreading, I'd be able to telnet to
several routers concurrently and speed up the process of making
changes.

Is this feasible using python with its standard telnet lib?
Richard

[Eddie Corns]

li***@myprincesspearls.com (Richard Bird CCNP, CCDP, MCSE, etc.) writes:

I need some help adding multithreading to an existing python script.
The script was developed to telnet and make changes on a list of cisco
routers.
I figure that by adding multithreading, I'd be able to telnet to
several routers concurrently and speed up the process of making
changes. Is this feasible using python with its standard telnet lib?

Yup. Here's the skeleton of code that I use to pull the ARP table from a
bunch of Cisco routers.

---------------------------------------------------------------------------
from threading import Thread, currentThread

# example of how to show which router an error occured on
def warn (msg):
stderr.write ('%s::%s\n' % (currentThread.getName(),msg))

# code to do things to the router
def do_router_things (rtr):
# open connection ...
# do the stuff
# close connection
# all done

# start a thread for each router
threads = []
for rtr in ...:
th = Thread(name=rtr, target=do_router_things, args=(rtr,))
th.start()
threads.append (th)

# wait for all threads to finish
for th in threads:
th.join()
---------------------------------------------------------------------------

Though if the processes were as independent as that I would just use a
separate process for each one (on Unix anyway). If you're collecting data
that needs to be processed then you would probably use a Queue.Queue, pass the
data INTO the Queue from do_router_things and have one more thread that pulls
items FROM the Queue.

HTH, mail me if you'd like more help.

Eddie

[Richard Bird]

Thanks so much for your helpful sample. It proved to be just the nudge
I needed to get the ball rolling.
I've modified the old script to support threading using your outline
and have begun to implement queueing to log information.

I can't get over how much faster it is to push a change to 800+
routers using 20+ threads!

Anything over 20 threads seems to take processor utilization to 100%.
Is there any benefit to starting additional threads once the CPU is at
100% ?

Thanks,
Richard

[Pierre-Frédéric Caillaud]

I can't get over how much faster it is to push a change to 800+
routers using 20+ threads!
yes ? and the more ping time you have, the more you can gain by thread
parallelism.
Anything over 20 threads seems to take processor utilization to 100%.
Is there any benefit to starting additional threads once the CPU is at
100% ?

Probably not. With few threads your app is network-lag-limited. With too
many threads it becomes cpu-limited and might even get slower. I have
noticed that multithreading network code in Python gives up at a rather
low number of threads, in the 20-50 as you noticed, which worries me
because the OS can do a lot better than that. Perhaps is this simply that
python with 1/20 of your CPU is too slow ? If you want to have 100 threads
you'll have to spend a few days coding it in (argh) C++ or (argh) Java.

Or if your brain doesn't explode, you can code it using asynchronous
select and the asynchat module...

Or just be happy that it's 20 times faster than before...

[Eddie Corns]

bi********@gmail.com (Richard Bird) writes:

Anything over 20 threads seems to take processor utilization to 100%.
Is there any benefit to starting additional threads once the CPU is at
100% ?

I think the only correct answer here is "suck it and see". It's going to
depend on what's happening. For instance, if you issue a "write mem" command
then that thread isn't going to be doing anything for a while. So there's no
obvious definitive answer.

Unless there are known limits to how many threads it is practical to use that
others know about? Anyone? My intuition would have suggested that a modern
machine could have easily handled more than 20 telnet sessions. We have less
than 100 Cisco devices but I do have a config save script that I could try
converting to threads, if I get time to convert it I will let you know how
many threads it copes with.

Eddie

[Pierre-Frédéric Caillaud]

I like single-threaded asynchronous network stuff, but programming state
machines is a pain.
Here is a guess at how to do it with pseudo coroutines implemented by
hacking with yield.
If your stuff can be implemented asynchronously (use module asynchat) you
can easily have hundreds of simultaneous connections.

# fast single-threaded queue
class stQueue( object ):
class QueueEmptyError( KeyError ):
pass

def __init__(self):
self._in = 0;
self._q = {}

def put(self, obj):
self._q[self._in] = obj
self._in += 1

def get(self):
if self._q: return self._q.pop( self._in - len(self._q) )
else: raise self.QueueEmptyError

def qsize(self):
return len( self._q )

def __nonzero__(self):
return bool(self._q)

# receiver class, could be a socket to send data
class stReceiver( object ):
def __init__(self,name):
self.name = name
def process( self, data ):
print self.name, ':', data

# stupid coroutines implemented with yield
class asyncSync( object ):
def __init__( self, receiver ):
self.inqueue = stQueue()
self.receiver = receiver

# we expect messages where
# message=0 print "hello"
# message=1 get another message and print it
# message=2 get an integer N, get N messages, print them
# message=3 die
def run(self):
while 1:
while not self.inqueue: # try to get a message
yield None # come back here newt time if no message
msg = self.inqueue.get()

if msg==0:
self.receiver.process( '(0) hello' )
elif msg==1:
while not self.inqueue:
yield None
self.receiver.process( "(1) print a message : %s" % self.inqueue.get()
)
elif msg==2:
while not self.inqueue:
yield None
nmessages = self.inqueue.get()
self.receiver.process( "(2) waiting for %d messages" % nmessages )
while self.inqueue.qsize() < nmessages:
yield None
for i in range(nmessages):
self.receiver.process( "(2) print a message (%d/%d) : %s" %
(i,nmessages,self.inqueue.get()) )
elif msg==3:
break

a = asyncSync( stReceiver( 'thread a' ))
ar = a.run()

ma = iter([ 0, 1, 'yikes', 2, 3, 'i', 'am', 'here', 2, 4, 'i', 'am',
'still', 'here', 0, 3 ])
print ar.next()
a.inqueue.put( ma.next() ) # these lines send a message to the object a
print ar.next() # these trigger some processing in a.run
a.inqueue.put( ma.next() )
print ar.next()
a.inqueue.put( ma.next() ) # these lines are interleaved in random order
just to show it works
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
print ar.next()
print ar.next()
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
print ar.next()
a.inqueue.put( ma.next() )
a.inqueue.put( ma.next() )
print ar.next()
a.inqueue.put( ma.next() )
print "We should crash here"
print ar.next()

[Josiah Carlson]

Richard Bird wrote:

Thanks so much for your helpful sample. It proved to be just the nudge
I needed to get the ball rolling.
I've modified the old script to support threading using your outline
and have begun to implement queueing to log information.

I can't get over how much faster it is to push a change to 800+
routers using 20+ threads!

Anything over 20 threads seems to take processor utilization to 100%.
Is there any benefit to starting additional threads once the CPU is at
100% ?

Sorry I didn't get into this earlier (it's been a while since I browsed
through c.l.py), but I notice that you are doing socket programming with
threads.

At one time I did use threading for sockets (senior undergrad project),
but one thing I discovered is that threading sometimes doesn't scale (it
didn't when it came to the sockets and protocols I was using).

On the other hand, if you can change the way you think, and spend enough
time working with asyncore (and derivatives), you can make asynchronous
sockets work for you*.

Depending on how much time and energy you have to spend on this, you may
want to keep using threads and call it good. Heck, if I didn't need so
many fast connections, I'd have likely stuck with threads.

- Josiah

* Right now I've got some code for a contract that is able to handle 500
sockets (the compile-time limit is 512, and I like to leave a little
breathing room) at 12 megs/second (on a single-processor mobile celeron
1.3 ghz, server and client running on the same machine), using a very
chatty line-based protocol (similar to telnet, most 'lines' are 80 bytes
or less) and my own derivative of asyncore (I may release a generalized
variant, after the contract has ended, if the company says it is alright).