473,224 Members | 1,624 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,224 software developers and data experts.

start a multi-sockets server (a socket/per thread) with different ports but same host

Hi,
I am doing a small project using socket server and thread in python.
This is first time for me to use socket and thread things.
Here is my case. I have 20 socket clients. Each client send a set
of sensor data per second to a socket server. The socket server will
do two things: 1. write data into a file via bsddb; 2. forward the data
to a GUI written in wxpython.
I am thinking the code should work as follow (not sure it is
feasible)
20 threads, each thread takes care of a socket server with a
different port.
I want all socket servers start up and wait for client connection.
In the attached demo code, It stops at the startup of first socket
server somewhere in the following two lines and waits for client call:

lstn.listen(5)
(clnt,ap) = lstn.accept()

Any ideas how to handle these 20 clients? Really appreciate your
suggestions.

Thanks a lot.

Ouyang
import socket
import sys
import threading
class srvr(threading.Thread):
v = ''
vlock = threading.Lock()
id = 0 # next available thread number
def __init__(self,clntsock):
threading.Thread.__init__(self)
self.myid = srvr.id
srvr.id += 1
self.myclntsock = clntsock
def run(self):
while 1:
k = self.myclntsock.recv(1)
if k == '': break
# update v in an atomic manner
srvr.vlock.acquire()
srvr.v += k
srvr.vlock.release()
self.myclntsock.send(srvr.v)
self.myclntsock.close()

#lstn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#port = int(sys.argv[1]) # server port number
#lstn.bind(('', port))
#lstn.listen(5)
nclnt = 20
mythreads = [] # list of all the threads

for i in range(nclnt):
lstn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
lstn.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
lstn.bind(('', 2000+i+1))
lstn.listen(5)
(clnt,ap) = lstn.accept()
s = srvr(clnt)
mythreads.append(s)
s.start()

# shut down the server socket, since it's not needed anymore
#lstn.close()

# wait for all threads to finish
for s in mythreads:
s.join()

print 'the final value of v is', srvr.v

Aug 12 '06 #1
5 4721
Jean-Paul,
Thanks a lot. The code is working. The python twisted is new to me too.
Here are my three more questions:
1. Since the code need to be started in a wxpyhon GUI (either by
clicking a button or up with the GUI), do I have to run the code in a
thread (sorry, I have not tried it yet)?
2. How can I grab the client data in the code? Can you write two lines
for that? I really appreciate that.
3. After I change
self.transport.write(''.join(self.data))
to
self.transport.write(''.join(data))
and scan all the ports with the following code twice (run twice).
First round scanning says "succefully connected". But second round
scanning says "failed". I have to restart your demo code to make it
work.

Ouyang
import sys, threading, socket

class scanner(threading.Thread):
tlist = [] # list of all current scanner threads
maxthreads = int(sys.argv[2]) # max number of threads we're
allowing
evnt = threading.Event() # event to signal OK to create more
threads
lck = threading.Lock() # lock to guard tlist
def __init__(self,tn,host):
threading.Thread.__init__(self)
#self.threadnum = tn # thread ID/port number
self.threadnum = 2000+tn # thread ID/port number
self.host = host # checking ports on this host
def run(self):
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
try:
s.connect((self.host, self.threadnum))
print "%d: successfully connected" % self.threadnum
s.close()
except:
print "%d: connection failed" % self.threadnum
# thread is about to exit; remove from list, and signal OK if we
# had been up against the limit
scanner.lck.acquire()
scanner.tlist.remove(self)
print "%d: now active --" % self.threadnum, scanner.tlist
if len(scanner.tlist) == scanner.maxthreads-1:
scanner.evnt.set()
scanner.evnt.clear()
scanner.lck.release()
def newthread(pn,hst):
scanner.lck.acquire()
sc = scanner(pn,hst)
scanner.tlist.append(sc)
scanner.lck.release()
sc.start()
print "%d: starting check" % pn
print "%d: now active --" % pn, scanner.tlist
newthread = staticmethod(newthread)

def main():
host = sys.argv[1]
#for i in range(1,100):
for i in range(20):
scanner.lck.acquire()
print "%d: attempting check" % i
# check to see if we're at the limit before starting a new thread
if len(scanner.tlist) >= scanner.maxthreads:
# too bad, need to wait until not at thread limit
print "%d: need to wait" % i
scanner.lck.release()
scanner.evnt.wait()
else:
scanner.lck.release()
scanner.newthread(i,host)
for sc in scanner.tlist:
sc.join()

if __name__ == '__main__':
main()


Jean-Paul Calderone 写道:
On 12 Aug 2006 09:00:02 -0700, zxo102 <zx****@gmail.comwrote:
Hi,
I am doing a small project using socket server and thread in python.
This is first time for me to use socket and thread things.
Here is my case. I have 20 socket clients. Each client send a set
of sensor data per second to a socket server. The socket server will
do two things: 1. write data into a file via bsddb; 2. forward the data
to a GUI written in wxpython.
I am thinking the code should work as follow (not sure it is
feasible)
20 threads, each thread takes care of a socket server with a
different port.
I want all socket servers start up and wait for client connection.
In the attached demo code, It stops at the startup of first socket
server somewhere in the following two lines and waits for client call:

Threads aren't the best way to manage the concurrency present in this
application. Instead, consider using non-blocking sockets with an
event notification system. For example, using Twisted, your program
might look something like this:

from twisted.internet import reactor, protocol, defer

class CumulativeEchoProtocol(protocol.Protocol):
def connectionMade(self):
# Stop listening on the port which accepted this connection
self.factory.port.stopListening()

# Set up a list in which to collect the bytes which we receive
self.received = []
def connectionLost(self, reason):
# Notify the main program that this connection has been lost, so
# that it can exit the process when there are no more connections.
self.factory.onConnectionLost.callback(self)

def dataReceived(self, data):
# Accumulate the new data in our list
self.received.append(data)
# And then echo the entire list so far back to the client
self.transport.write(''.join(self.data))

def allConnectionsLost():
# When all connections have been dropped, stop the reactor so the
# process can exit.
reactor.stop()

def main():
# Set up a list to collect Deferreds in. When all of these Deferreds
# have had callback() invoked on them, the reactor will be stopped.
completionDeferreds = []
for i in xrange(20):
# Make a new factory for this port
f = protocol.ServerFactory()

# Make a Deferred for this port's connection-lost event and make
# it available to the protocol by way of the factory.
d = defer.Deferred()
f.onConnectionLost = d
completionDeferreds.append(d)
f.protocol = CumulativeEchoProtocol

# Start listening on a particular port number with this factory
port = reactor.listenTCP(2000 + i + 1, f)

# Make the port object available to the protocol as well, so that
# it can be shut down when a connection is made.
f.port = port

# Create a Deferred which will only be called back when all the other
# Deferreds in this list have been called back.
d = defer.DeferredList(completionDeferreds)

# And tell it to stop the reactor when it fires
d.addCallback(lambda result: allConnectionsLost())

# Start the reactor so things can start happening
reactor.run()

if __name__ == '__main__':
main()

Hope this helps,

Jean-Paul
Aug 12 '06 #2
Jean-Paul,
I just start to learn Twisted. Here is my simple case: I can find
the data sent by clients in dataReceived but I don't know which
client/which port the data is from. After I know where the data comes
from, I can do different things there, for example, write them into
different files via bsddb. I am not sure if it is the correct way to
do it.
def dataReceived(self, data):
# Accumulate the new data in our list
self.received.append(data)
# And then echo the entire list so far back to the client
self.transport.write(''.join(data))

print "============data: ", data
print " which Port? : ", self.factory.port # unforunately it is
an object here.

# if Port == 2001:
# write the data into a file via bsddb
# if Port == 2002:
# write the data into another file via bsddb
# etc .....
Ouyang

Jean-Paul Calderone 写道:
On 12 Aug 2006 10:44:29 -0700, zxo102 <zx****@gmail.comwrote:
Jean-Paul,
Thanks a lot. The code is working. The python twisted is new to me too.
Here are my three more questions:
1. Since the code need to be started in a wxpyhon GUI (either by
clicking a button or up with the GUI), do I have to run the code in a
thread (sorry, I have not tried it yet)?

You can try to use Twisted's wxPython integration support:

http://twistedmatrix.com/projects/co...or.html#auto15

But note the warnings about how well it is likely to work. Using a separate
thread might be the best solution.
2. How can I grab the client data in the code? Can you write two lines
for that? I really appreciate that.

I'm not sure what you mean. The data is available in the `received' attribute
of the protocol instance. Any code which needs to manipulate the data can get
that list and do whatever it likes with it.
3. After I change
self.transport.write(''.join(self.data))
to
self.transport.write(''.join(data))
and scan all the ports with the following code twice (run twice).
First round scanning says "succefully connected". But second round
scanning says "failed". I have to restart your demo code to make it
work.

I intentionally added code which shuts the server off after the first round
of connections is completed, since that seemed to be what your example
program was doing. If you don't want this, just remove the shutdown code.

Jean-Paul
Aug 13 '06 #3
zxo102 wrote:
I am doing a small project using socket server and thread in python.
This is first time for me to use socket and thread things.
Here is my case. I have 20 socket clients. Each client send a set
of sensor data per second to a socket server. The socket server will
do two things: 1. write data into a file via bsddb; 2. forward the data
to a GUI written in wxpython.
I am thinking the code should work as follow (not sure it is
feasible)
20 threads, each thread takes care of a socket server with a
different port.
I want all socket servers start up and wait for client connection.
In the attached demo code, It stops at the startup of first socket
server somewhere in the following two lines and waits for client call:

lstn.listen(5)
(clnt,ap) = lstn.accept()
It will block there, waiting for connection.
Any ideas how to handle these 20 clients? Really appreciate your
suggestions.
One reserved port for each client strikes me as whacked,
as does coding a server to handle exactly 20 of them. Since
you say this is your first socket server, maybe you just
haven't seen the usual techniques.

Normally, one listener socket accepts all the connections.
Each call to accept() returns a new, independent socket for the
connection. You can then start a thread to handle the new
socket. Untested:
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.bind(('', 2000))
listener.listen(5)
while True: # or some should_continue() thing
sock, _ = listener.accept()
thread.start_new_thread(service_function, (sock,))
# Or start threads via class Threading
To update the GUI, you could use the Queue from the Python
library, and call wxPostEvent to tell the GUI go wake up and
check the queue.
--
--Bryan
Aug 13 '06 #4
Bryan,
Thanks for your note. Finally, I have made "one listener socket for
all the connections" work plus Queue-communication between the threads
in wxpython Gui and the threads for socket connections.
Trying to make that twisted example code in this topic for "one
listener socket-all the connections" but failed. That twisted example
only accepts one client connection. I have printed out the Twisted help
file (256 pages). Too much to read.

Ouyang
Bryan Olson 写道:
zxo102 wrote:
I am doing a small project using socket server and thread in python.
This is first time for me to use socket and thread things.
Here is my case. I have 20 socket clients. Each client send a set
of sensor data per second to a socket server. The socket server will
do two things: 1. write data into a file via bsddb; 2. forward the data
to a GUI written in wxpython.
I am thinking the code should work as follow (not sure it is
feasible)
20 threads, each thread takes care of a socket server with a
different port.
I want all socket servers start up and wait for client connection.
In the attached demo code, It stops at the startup of first socket
server somewhere in the following two lines and waits for client call:

lstn.listen(5)
(clnt,ap) = lstn.accept()

It will block there, waiting for connection.
Any ideas how to handle these 20 clients? Really appreciate your
suggestions.

One reserved port for each client strikes me as whacked,
as does coding a server to handle exactly 20 of them. Since
you say this is your first socket server, maybe you just
haven't seen the usual techniques.

Normally, one listener socket accepts all the connections.
Each call to accept() returns a new, independent socket for the
connection. You can then start a thread to handle the new
socket. Untested:
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.bind(('', 2000))
listener.listen(5)
while True: # or some should_continue() thing
sock, _ = listener.accept()
thread.start_new_thread(service_function, (sock,))
# Or start threads via class Threading
To update the GUI, you could use the Queue from the Python
library, and call wxPostEvent to tell the GUI go wake up and
check the queue.
--
--Bryan
Aug 15 '06 #5
"That twisted example only accepts one client connection" if only one
port is available.
zxo102 写道:
Bryan,
Thanks for your note. Finally, I have made "one listener socket for
all the connections" work plus Queue-communication between the threads
in wxpython Gui and the threads for socket connections.
Trying to make that twisted example code in this topic for "one
listener socket-all the connections" but failed. That twisted example
only accepts one client connection. I have printed out the Twisted help
file (256 pages). Too much to read.

Ouyang
Bryan Olson 写道:
zxo102 wrote:
I am doing a small project using socket server and thread in python.
This is first time for me to use socket and thread things.
Here is my case. I have 20 socket clients. Each client send a set
of sensor data per second to a socket server. The socket server will
do two things: 1. write data into a file via bsddb; 2. forward the data
to a GUI written in wxpython.
I am thinking the code should work as follow (not sure it is
feasible)
20 threads, each thread takes care of a socket server with a
different port.
I want all socket servers start up and wait for client connection.
In the attached demo code, It stops at the startup of first socket
server somewhere in the following two lines and waits for client call:
>
lstn.listen(5)
(clnt,ap) = lstn.accept()
It will block there, waiting for connection.
Any ideas how to handle these 20 clients? Really appreciate your
suggestions.
One reserved port for each client strikes me as whacked,
as does coding a server to handle exactly 20 of them. Since
you say this is your first socket server, maybe you just
haven't seen the usual techniques.

Normally, one listener socket accepts all the connections.
Each call to accept() returns a new, independent socket for the
connection. You can then start a thread to handle the new
socket. Untested:
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.bind(('', 2000))
listener.listen(5)
while True: # or some should_continue() thing
sock, _ = listener.accept()
thread.start_new_thread(service_function, (sock,))
# Or start threads via class Threading
To update the GUI, you could use the Queue from the Python
library, and call wxPostEvent to tell the GUI go wake up and
check the queue.


--
--Bryan
Aug 15 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

21
by: Alexander N. Spitzer | last post by:
If I have a machine with 3 virtual IP addresses (192.168.1.), how can I start 3 instances of the same RMI application (each started with different properties/configs), each listening on the port...
8
by: simon place | last post by:
Spent some very frustrating hours recoding to find a way of closing a server socket, i'd not thought it would be any problem, however, after complete failure and as a last resort, i looked at the...
4
by: jas | last post by:
I have a basic client/server socket situation setup....where the server accepts a connection and then waits for commands. On the client side, I create a socket, connect to the server...then I...
2
by: giangiammy | last post by:
hi all, I'd like to implement a server socket in java: something linke the following example. The problem is that the HTML has not the permission to execute instruction serverSocket =...
3
by: O.B. | last post by:
When sharing a Socket between threads, are the socket operations automatically synchronized to support multithreading? For example: Thread1 sets up a server socket to listen and invokes...
0
by: Leo Jay | last post by:
I'd like to read and write the same socket in different threads. one thread is only used to read from the socket, and the other is only used to write to the socket. But I always get a 10022...
0
by: Jean-Paul Calderone | last post by:
On Sat, 23 Aug 2008 02:25:17 +0800, Leo Jay <python.leojay@gmail.comwrote: No - it's just what I said. create_socket creates one socket and passes it to read_socket and write_socket. ...
0
by: ryaneoghan | last post by:
hi, I was just wondering if anyone could help, i am trying to create a client and server socket connection in perl, i want to ask for name and then do a check on the name and ask for a secret word...
0
by: lucy149 | last post by:
hello , i am trying to implement ftp protoco using javal. but when i read the protocol description ,i didnt get how i should have 2 ports does that mean having 2 server sockets , each one with...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: mar23 | last post by:
Here's the situation. I have a form called frmDiceInventory with subform called subfrmDice. The subform's control source is linked to a query called qryDiceInventory. I've been trying to pick up the...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.