473,245 Members | 1,519 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,245 software developers and data experts.

Adding idle timeout capabilities to asyncore

Hi there.
We're talking about an asyncore-based server.
Just for the heck of it I'd like to set a timeout which will
disconnects the clients if they're inactive (i.e., no command or data
transfer in progress) for a long period of time.
I was thinking about putting the timeout value into an attribute:

def settimeout(self, secs):
self.timeout = time.time() + secs

And then have the readable method call time.time() at every loop to
check if time has passed or not:

def readable(self):
if time.time() >= self.timeout:
self.send("Timeout")
self.close()
return 1

My only concern is if it's a good idea calling time.time() so often.
Since A LOT of clients could be connected simultaneously, couldn't it
be a too much resource-intensive operation?
I'd also be curious to know how Twisted implemented this kind of
stuff.
By calling time.time() at every loop?
Thanks in advance.

Oct 22 '07 #1
4 4421
On 22 Ott, 12:28, Giampaolo Rodola' <gne...@gmail.comwrote:
Hi there.
We're talking about an asyncore-based server.
Just for the heck of it I'd like to set a timeout which will
disconnects the clients if they're inactive (i.e., no command or data
transfer in progress) for a long period of time.
I was thinking about putting the timeout value into an attribute:

def settimeout(self, secs):
self.timeout = time.time() + secs

And then have the readable method call time.time() at every loop to
check if time has passed or not:

def readable(self):
if time.time() >= self.timeout:
self.send("Timeout")
self.close()
return 1

My only concern is if it's a good idea calling time.time() so often.
Since A LOT of clients could be connected simultaneously, couldn't it
be a too much resource-intensive operation?
I'd also be curious to know how Twisted implemented this kind of
stuff.
By calling time.time() at every loop?

Thanks in advance.
Calling time.time() is relatively inexpensive in comparison to pure
Python function calls, but indeed, it could be a bottleneck.

I don't know what Twisted does, but what I would do is to add two
variables to each instance of the class you want to add timeouts to;
self.timeout and self.lastdata . self.lastdata would hold the time
for the last time you sent or received data on the socket, and
self.timeout would hold the delay between when you last sent/received
data and when it should be closed due to idle timeout.

Now, since you are smart, you don't need to alter your handle_send()
or handle_receive() methods in your asyncore subclass. Instead, you
rewrite asyncore.poll() to update .lastdata for every socket that is
in either the reader or writer list, which will result in one
time.time() call. Further, to check for timeouts, you only ever need
to check those sockets that are *not* in the readable or writable
lists...

To be precise, add the following block to your own copy of
asyncore.poll just after the 'for fd in e:' block...
#handle timeouts
rw = set(r) + set(w)
now = time.time()
for f in (i for i in rw if i in map):
map[f].lastdata = now
for j in (map[i] for i in map if i not in rw):
if j.timeout+j.lastdata now:
#timeout!
j.handle_close()

You ARE going to need to initialize .timeout and .lastdata members for
every instance, but that shouldn't be so bad (for a socket that
doesn't time out, I would actually suggest a 1 hour or 1 day timeout).

- Josiah

Oct 23 '07 #2
On 23 Ott, 17:34, Josiah Carlson <josiah.carl...@gmail.comwrote:
On 22 Ott, 12:28, Giampaolo Rodola' <gne...@gmail.comwrote:


Hi there.
We're talking about an asyncore-based server.
Just for the heck of it I'd like to set a timeout which will
disconnects the clients if they're inactive (i.e., no command or data
transfer in progress) for a long period of time.
I was thinking about putting the timeout value into an attribute:
def settimeout(self, secs):
self.timeout = time.time() + secs
And then have the readable method call time.time() at every loop to
check if time has passed or not:
def readable(self):
if time.time() >= self.timeout:
self.send("Timeout")
self.close()
return 1
My only concern is if it's a good idea calling time.time() so often.
Since A LOT of clients could be connected simultaneously, couldn't it
be a too much resource-intensive operation?
I'd also be curious to know how Twisted implemented this kind of
stuff.
By calling time.time() at every loop?
Thanks in advance.

Calling time.time() is relatively inexpensive in comparison to pure
Python function calls, but indeed, it could be a bottleneck.

I don't know what Twisted does, but what I would do is to add two
variables to each instance of the class you want to add timeouts to;
self.timeout and self.lastdata . self.lastdata would hold the time
for the last time you sent or received data on the socket, and
self.timeout would hold the delay between when you last sent/received
data and when it should be closed due to idle timeout.

Now, since you are smart, you don't need to alter your handle_send()
or handle_receive() methods in your asyncore subclass. Instead, you
rewrite asyncore.poll() to update .lastdata for every socket that is
in either the reader or writer list, which will result in one
time.time() call. Further, to check for timeouts, you only ever need
to check those sockets that are *not* in the readable or writable
lists...

To be precise, add the following block to your own copy of
asyncore.poll just after the 'for fd in e:' block...

#handle timeouts
rw = set(r) + set(w)
now = time.time()
for f in (i for i in rw if i in map):
map[f].lastdata = now
for j in (map[i] for i in map if i not in rw):
if j.timeout+j.lastdata now:
#timeout!
j.handle_close()

You ARE going to need to initialize .timeout and .lastdata members for
every instance, but that shouldn't be so bad (for a socket that
doesn't time out, I would actually suggest a 1 hour or 1 day timeout).

- Josiah- Nascondi testo tra virgolette -

- Mostra testo tra virgolette -
A really nice hack.
Thank you a lot, Josiah.

Oct 24 '07 #3
On Oct 23, 9:30 am, Jean-Paul Calderone <exar...@divmod.comwrote:
On Tue, 23 Oct 2007 15:34:19 -0000, Josiah Carlson <josiah.carl...@gmail.comwrote:
[snip]
Calling time.time() is relatively inexpensive in comparison to pure
Python function calls, but indeed, it could be a bottleneck.

Did you benchmark this on some system?

There isn't really any "pure Python function calls" that can replace
time.time() in a sensible way, so I made up one that does some random
things (including calling another Python function) and compared it to
time.time:

exarkun@charm:~$ python -m timeit -s '
def g():
return 1. + 2 + 3 * 4 - 5 / 6 + 7 ** 8
def f():
return g()
' 'f()'
1000000 loops, best of 3: 1.39 usec per loop
exarkun@charm:~$ python -m timeit -s 'from time import time as f' 'f()'
100000 loops, best of 3: 4.68 usec per loop
exarkun@charm:~$

Not really useful in any real-world sense, but I still wouldn't
characterize time.time as "relatively inexpensive."

Of course, for any real-world work, one would want to profile the
application to determine if removing calls to time.time() could
make a worthwhile difference.
The particular function call he was talking about was the .readable()
or .writable() in an asyncore.dispatcher subclass. Now, the trick
with any nontrivial asyncore.dispatcher subclass is that you don't
want to signal a socket as writable if you don't have any outgoing
buffer. Similarly, I have seen subclasses that follows a state
machine very strictly (no pipelining, you must finish sending before
you can receive, etc.). In that sense, typically, both
the .readable() and .writable() methods will reference an attribute or
two of the instance.

After running some tests, I notice that it seems like a not
insignificant platform difference. Because on my platform (Windows
and Python 2.3)...
>>setup = '''def g(): return 1. + 2 + 3 * 4 - 5 / 6 + 7 ** 8'''
timeit.Timer('g()', setup).repeat()
[2.0143674536171798, 1.9778226522165654, 1.9959138650798764]
>>setup = '''
.... class foo(object):
.... __slots__ = 'a', 'b'
.... def __init__(self):
.... self.a = 1
.... self.b = 2
.... def foo(self):
.... return self.a and self.b
....
.... inst = foo()
.... '''
>>timeit.Timer('inst.foo()', setup).repeat()
[1.4185994550779242, 1.4186812879064519, 1.4478852871044694]
>>timeit.Timer('f()', 'f = lambda:None').repeat()
[0.58938552412047329, 0.58762154196472238, 0.58324245809257036]
>>timeit.Timer('f()', 'from time import time as f').repeat()
[0.68104482574668168, 0.64085672667252425, 0.6558844364245715]

So to answer your question, even though I hadn't bothered to test it,
my beliefs regarding time.time() being fast in relation to the
particular operation he was expecting to perform, at least according
to my own experience on the platform I use the most (Windows and
Python 2.3) was relatively accurate, though indeed it is slower (but
not much in comparison to what those calls would typically do).

Then again, in the subclasses of asyncore that I write, I don't bother
calling .readable() or .writable() at all, each socket adds and
removes itself from a pair of readable and writable dictionaries
automatically, thus removing perhaps hundreds of function calls for
every loop of asyncore.poll .

What these experiments also tell me is that fetching the time on linux
is slow. But really, since I already wrote code that handles *all* of
the timeout handling with a *single* time.time() call, and that also
generally minimizes all explicit function calls, I'm not sure that
your testing examples were ultimately germane to the conversation (how
would one handle timeouts in asyncore). Or maybe it's just late and
I've had a long day :/ .
In any case, I'm glad that I was able to help Giampaolo.
- Josiah

Oct 25 '07 #4
Giampaolo Rodola' <gn****@gmail.comwrites:
def readable(self):
if time.time() >= self.timeout:
self.send("Timeout")
self.close()
return 1
Don't do it this way. Put all the timeouts into a priority queue
(see the heapq module) so you only need to check the one due to expire
soonest.
Oct 25 '07 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: David M. Wilson | last post by:
Hi peeps, I finally got around recently to doing something serious using Python for serving network clients asynchronously, deciding on asyncore as my starting point. After 2 days or so of...
2
by: Freddie | last post by:
Hi, I've been mangling python-irclib into an asyncore class, so it fits in nicely with the rest of my app. I ran into a problem with asyncore.dispatcher_with_send (Python 2.3.4), though. Not...
0
by: Neil Benn | last post by:
Hello, I was wondering if someone could provide assistance. I'm writing a thin wrapper around asyncore for compatibility with a common Comms API that I use. class...
0
by: Tony Meyer | last post by:
Changes in asyncore from 2.3 to 2.4 mean that asyncore.poll() now passes all the sockets in the map to select.select() to be checked for errors, which is probably a good thing. If an error occurs,...
3
by: Jos | last post by:
Hello. I'm using the asyncore and _chat modules to create a network server. I also have, running in a separate thread(s), a "producer" which needs to "push" data onto the network connection(s)....
1
by: Gordon Smith | last post by:
I'm just getting up to speed on Application Pools in IIS6 and I'm wondering if there would be any good reason to stick with the default value of for the Idle Timeout value for Worker processes...
5
by: shumaker | last post by:
I just read an overview of C# 3.0, and it occured to me that some of these features could make it much easier to write programs that automatically make use of multi core processors. After reading...
5
by: JamesHoward | last post by:
I have a problem with python's asyncore module throwing a bad file descriptor error. The code might be difficult to copy here, but the problem is essentially: The server wants to sever the...
6
by: Andreas Tawn | last post by:
I'm trying to integrate the timeout function from here http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/473878 into a long running automation script and the following code causes IDLE after...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.