asyncore deficiencies

[David M. Wilson]

Hi peeps,

I finally got around recently to doing something serious using Python
for serving network clients asynchronously, deciding on asyncore as my
starting point.

After 2 days or so of fiddling about, I have decided, as many before
me have, that the current asyncore module is the most inflexible,
brain-dead, and useless module to be found on the face of the earth!

It appears that my next stop for well-engineered event-driven
progamming is Twisted, but that is quite a hop from the 500 lines that
is asyncore. I have decided in the meantime to stick with asyncore,
despite it's failings. It has got me thinking though, shouldn't there
be a standard module that does this sort of thing correctly, on a
scale comparable to asyncore? ie. asyncore2

My biggests gripes with asyncore so far have been:

- Exception handling: namely, it's fairly braindead. As an example, if
you are using a private map for loop(), when an Exception is caught it
is dumped to stderr, then the dispatcher causing it is destroyed.
However, the dead dispatcher is not removed from your private map,
resulting in the loop iterating again, and your dead dispatcher
causing an "invalid file descriptor" error.

- API wasted on less than useful features: of the few configurable
parts of asyncore, one of them is the choice between select() and
poll(), which I find to be completely useless, as larger projects
(those that may wish to use poll()) would in my books already have
migrated away from asyncore, and the smaller projects are more
interested in getting code written, rather than worrying about some
low level events strategy. It also makes the API rather tied to
implementation.

- Poor __getattr__ use: this has caused numerous "recursion limit
exceeded" type exceptions, formatted using asyncore's less-than-ideal
so called "compact exception formatter". This results in errors that
are practically uncomprehendable, even as an experienced programmer
you know to look at the top for the true source. The __getattr__ hack
causes behaviour that is far from less than obvious.

- Unclear API: asyncore.loop(timeout = ...) is a parameter passed on
to the underlying poll() or select(). I can see no circumstance for
ever needing to change this. However, I did manage to mistake it for a
"maximum execution time" of the poll function, assuming that after
this time it would return control to the caller. Again, it's unclear.

- Lack of flexibility. I have to use SIGALRM or an external process
connected via a socket to get asyncore to change execution path based
on non-socket events. The only other way of accomplishing this, is to
temporarily empty your map to break out of loop(), then repopulate it,
probably using a couple of global variables while you're at it.

- Unclear interfaces: you must override all methods inherited from a
dispatcher even if you don't use all of them - this makes for verbose
code when it is not necessary.

- Incorrect interfaces: (this is assumed) for a UDP socket,
handle_connect() gets called on first read, despite the fact that UDP
is connectionless, and it doesn't make sense to make an exception of
the first received packet. The idea of taking the strange semantics
detected from select() or poll() and turning them into nice meaningful
names is a good one, however it could be improved.
So far, I'd like to see:

- Less stuffy API: move select() and poll() choice to either an
internal decision based on the number of active dispatchers (bad),
system call availability (better), a module-level configuration value
(ish), or simply use one or the other. Polling objects translate quite
well into how asyncore works internally (but they're not available
everywhere).

- A 'process_once' to allow you to write your own loop()s.

- Non braindead exception handling: if an exception occurs, I want to
see it propagated back up to the parent, as happens everywhere else,
not passed to a handle_error() method or "verbosely ignored".

- Decoupling from socket objects: removal of the __getitem__ magic
which causes screenfuls of abuse for a simple typo at the wrong stage
in execution.

- Possibly combined with the above, a more abstract way of watching
for events from elsewhere? Doing this might ruin the overall
simplicity of the module.
Does anyone have any other ideas? I'm sure there are at least a few I
have forgotten. If there was support for an improved 'asyncore2', I
would very much like to see an improvement on the existing module, not
a silly overcomplication with needless feature adding and
overabstraction (let's not name names here ;).
David.

[Josiah Carlson]

After 2 days or so of fiddling about, I have decided, as many before
me have, that the current asyncore module is the most inflexible,
brain-dead, and useless module to be found on the face of the earth!
A bit of subclassing goes a long ways towards making it easier to handle.

It appears that my next stop for well-engineered event-driven
progamming is Twisted, but that is quite a hop from the 500 lines that
is asyncore. I have decided in the meantime to stick with asyncore,
despite it's failings. It has got me thinking though, shouldn't there
be a standard module that does this sort of thing correctly, on a
scale comparable to asyncore? ie. asyncore2
Indeed. Twisted is a big jump. Those that have gotten used to it seem
to enjoy it quite a bit. It seems to have most every protocol already
implemented, and it seems to interact nicely with various GUI toolkits,
databases, etc. I've stuck with asyncore.

I agree that there should probably be some sort of asyncore2; I have a
handful of boilerplate servers and clients that I use as a base any time
I need something. Optimally, an asyncore2 library would have simple
clients and servers ready to be subclassed and customized as needed.

My biggests gripes with asyncore so far have been:

- Exception handling: namely, it's fairly braindead. As an example, if
you are using a private map for loop(), when an Exception is caught it
is dumped to stderr, then the dispatcher causing it is destroyed.
However, the dead dispatcher is not removed from your private map,
resulting in the loop iterating again, and your dead dispatcher
causing an "invalid file descriptor" error.
I'm curious as to why you need to use a private map. Most uses of
private maps I've seen haven't added any sort of meaningful
functionality that couldn't have been handled by the default map. Yours
could be an exception, but I'm still curious.

- Unclear API: asyncore.loop(timeout = ...) is a parameter passed on
to the underlying poll() or select(). I can see no circumstance for
ever needing to change this. However, I did manage to mistake it for a
"maximum execution time" of the poll function, assuming that after
this time it would return control to the caller. Again, it's unclear.
asyncore.loop loops forever. asyncore.poll does the one-shot deal.
more on this later.
- Lack of flexibility. I have to use SIGALRM or an external process
connected via a socket to get asyncore to change execution path based
on non-socket events. The only other way of accomplishing this, is to
temporarily empty your map to break out of loop(), then repopulate it,
probably using a couple of global variables while you're at it.
Again, you need to use asyncore.poll.
- Unclear interfaces: you must override all methods inherited from a
dispatcher even if you don't use all of them - this makes for verbose
code when it is not necessary.
Agreed. I have a few private classes that I subclass from that take
care of the "unimplemented" warnings.

- Incorrect interfaces: (this is assumed) for a UDP socket,
handle_connect() gets called on first read, despite the fact that UDP
is connectionless, and it doesn't make sense to make an exception of
the first received packet. The idea of taking the strange semantics
detected from select() or poll() and turning them into nice meaningful
names is a good one, however it could be improved.
I've never used asyncore with UDP sockets, but maybe using
"handle_connect = handle_read" in your subclass definition could be
sufficient.

So far, I'd like to see:

- Less stuffy API: move select() and poll() choice to either an
internal decision based on the number of active dispatchers (bad),
system call availability (better), a module-level configuration value
(ish), or simply use one or the other. Polling objects translate quite
well into how asyncore works internally (but they're not available
everywhere).
That is exactly why polling is optional. Select is available on every
platform, but poll is faster on platforms that support it...

- A 'process_once' to allow you to write your own loop()s.
asyncore.poll (or the equivalent poll2 and poll3 if poll is available on
your platform) is what you are looking for. I use it quite often for
things like:

while not QUIT:
asyncore.poll(.01)
#handle any sort of background processing if necessary

It is also quite useful to be able to drop in an asyncore.poll(.001)
inside a GUI application. What I commonly do in wxPython apps is...

#inside __init__
wx.Timer(self, 10001).Start(10, wx.TIMER_CONTINUOUS)
wx.EVT_TIMER(self, 10001, self.PollSocket)

def PollSocket(self, event):
asyncore.poll(.001)
#pull any sort of waiting packets
#from the properly buffered dispatchers

- Non braindead exception handling: if an exception occurs, I want to
see it propagated back up to the parent, as happens everywhere else,
not passed to a handle_error() method or "verbosely ignored".
The trick with doing this is that you end up interrupting other
potentially ready sockets with handling an exception. If you are
feeling fiesty, I believe you can use the following and get the error
propagated back up:

def handle_error(self):
raise

- Decoupling from socket objects: removal of the __getitem__ magic
which causes screenfuls of abuse for a simple typo at the wrong stage
in execution.
Infinite recursion in asyncore? Yikes, I've not hit that bug before.
Care to post some offending code?

- Possibly combined with the above, a more abstract way of watching
for events from elsewhere? Doing this might ruin the overall
simplicity of the module.

Something like...

def handle_read(self):
socket_post_event(self, READ_EVENT)

One could even have a framework for registering even handlers...yikes,
this is starting to sound like wxPython. I don't know if that is a good
idea or not.

[David M. Wilson]

Josiah Carlson <jc******@nospam.uci.edu> wrote...

I agree that there should probably be some sort of asyncore2; I have a
handful of boilerplate servers and clients that I use as a base any time
I need something. Optimally, an asyncore2 library would have simple
clients and servers ready to be subclassed and customized as needed.
I would expect such things to be provided by modules that are
dependant on asyncore, eg. as asynchat is. It's beyond the scope of
what asyncore tries to do.

I'm curious as to why you need to use a private map. Most uses of
private maps I've seen haven't added any sort of meaningful
functionality that couldn't have been handled by the default map. Yours
could be an exception, but I'm still curious.
I don't need a private map, however given that the other option was a
module variable, I naturally chose to provide my own to increase any
required future flexibility. Indeed there is little, if anything to be
gained from using your own map, but the option was there.

Again, you need to use asyncore.poll.
The poll function is not documented in the manual, nor do any
docstrings suggest it is a public interface. Internally asyncore
chooses which of poll, poll2, or poll3 to use inside loop, so calling
these directly also bypasses some of asyncore's OS-independance
functionality.

I've never used asyncore with UDP sockets, but maybe using
"handle_connect = handle_read" in your subclass definition could be
sufficient.
I see this as a workaround, and as such I think the point still
stands. :)

That is exactly why polling is optional. Select is available on every
platform, but poll is faster on platforms that support it...
That is not true at all. If you are feeling investigative, quite a
complete starting point would be <http://www.kegel.com/c10k.html>.
There is lots of information there regarding various OS
implementations async IO performance using poll, epoll, select, aio_*,
etc.

The trick with doing this is that you end up interrupting other
potentially ready sockets with handling an exception. If you are
feeling fiesty, I believe you can use the following and get the error
propagated back up: def handle_error(self):
raise
That's a good point, but I don't have a problem restarting the poll
after handling an exception, a) because it shouldn't happen very
often, b) will probably mean shutting down the service rather than
trying to continue, and c) exceptions are generally expensive special
cases anyway.

If an exception is propogating up from the dispatcher, that means the
dispatcher's method did not handle the exception, which should mean it
gets passed back up the stack as it does with all other code,
eventually ending up with a traceback print if it wasn't handled.

I dislike the handle_error method since it is essentially redundant in
the face of Python's own exception handling mechanism. If a dispatcher
wants it's own function to handle exceptions, it should wrap it's
methods in a try block.

I assume handle_error was provided to allow graceful teardown of a
dispatcher that was erroneous, but it isn't granular enough to be
useful for tearing down complicated objects, given an exception that
could have been generated anywhere in a couple of screenfuls worth of
code, and an object that could be in any state.

- Decoupling from socket objects: removal of the __getitem__ magic
which causes screenfuls of abuse for a simple typo at the wrong stage
in execution.

Infinite recursion in asyncore? Yikes, I've not hit that bug before.
Care to post some offending code?

It was entirely my fault, but here is the example anyway. :)

py> from asyncore import *
py> class D(dispatcher):
... def __init__(self):
... self.foo
... dispatcher.__init__(self)
...
py> D()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in __init__
File "/usr/lib/python2.3/asyncore.py", line 365, in __getattr__
return getattr(self.socket, attr)
File "/usr/lib/python2.3/asyncore.py", line 365, in __getattr__
return getattr(self.socket, attr)
File "/usr/lib/python2.3/asyncore.py", line 365, in __getattr__
return getattr(self.socket, attr)

One could even have a framework for registering even handlers...yikes,
this is starting to sound like wxPython. I don't know if that is a good
idea or not.

NAME asyncore
Basic infrastructure for asynchronous socket service clients and
servers.

I think that answers the question.
Is anyone else interested in this? I have already made small
modifications to the asyncore module to fix a few things, but nothing
mentioned here. I could put together a proposed asyncore2 module if
there was interest. I may do it, out of need, anyway. :)
David.

[Josiah Carlson]

> I would expect such things to be provided by modules that are

dependant on asyncore, eg. as asynchat is. It's beyond the scope of
what asyncore tries to do.
Then what exactly would you like to see in asyncore2?

I don't need a private map, however given that the other option was a
module variable, I naturally chose to provide my own to increase any
required future flexibility. Indeed there is little, if anything to be
gained from using your own map, but the option was there.

You don't even need to offer a variable to the loop construct. By
default, it will use asyncore.map, regardless of where your dispatchers
are defined.

The poll function is not documented in the manual, nor do any
docstrings suggest it is a public interface. Internally asyncore
chooses which of poll, poll2, or poll3 to use inside loop, so calling
these directly also bypasses some of asyncore's OS-independance
functionality.
Seemingly asyncore should do a quick test to see which are available,
and based on some metrics, automatically select the proper function on
module import and call it poll_once(), which would be poll, poll2 or
poll3, based on what the system had available.

It could optionally have a check_again function that should be called
occasionally, which will, on platforms with more than one select/poll
available, rebind poll_once based on which would be faster given the
current size of the socket map.

I've never used asyncore with UDP sockets, but maybe using
"handle_connect = handle_read" in your subclass definition could be
sufficient.

I see this as a workaround, and as such I think the point still
stands. :)

I suppose it would be a relatively minor patch to check what kind of
socket it is on initialization, making "handle_connect = handle_read"
automatic.

That is exactly why polling is optional. Select is available on every
platform, but poll is faster on platforms that support it...

That is not true at all. If you are feeling investigative, quite a
complete starting point would be <http://www.kegel.com/c10k.html>.
There is lots of information there regarding various OS
implementations async IO performance using poll, epoll, select, aio_*,
etc.

Well...select is available on linux, windows, solaris (I've used it on
all three). I would be surprised if it was not available on all of the
BSDs, would anyone know for certain?

That's a good point, but I don't have a problem restarting the poll
after handling an exception, a) because it shouldn't happen very
often, b) will probably mean shutting down the service rather than
trying to continue, and c) exceptions are generally expensive special
cases anyway.
Socket-exceptions usually mean that a connection has died. That kind of
thing can be relatively common. Depending on your application, setting
up a callback for your dispatcher instance to notify any controller (if
necessary) should be relatively easy.

If an exception is propogating up from the dispatcher, that means the
dispatcher's method did not handle the exception, which should mean it
gets passed back up the stack as it does with all other code,
eventually ending up with a traceback print if it wasn't handled.
Unless of course you want your server/client to stay up under all socket
circumstances. Rarely, if ever, do you want your server or client to
crash when exceptions are raised. Certainly such behavior can be useful
when you are initially writing your socket app, but as soon as the thing
becomes live, likely the last thing you want is for it to crash.

I dislike the handle_error method since it is essentially redundant in
the face of Python's own exception handling mechanism. If a dispatcher
wants it's own function to handle exceptions, it should wrap it's
methods in a try block.
I believe they had it call handle_error because then you can write
servers like the following...

#dispatcher subclasses

mydispatcher(listen=('', 9876))
asyncore.loop()

This also allows your various clients and servers to handle exceptions
differently, and for the exception handling to be implemented in the
thing that may be raising the exceptions.

I assume handle_error was provided to allow graceful teardown of a
dispatcher that was erroneous, but it isn't granular enough to be
useful for tearing down complicated objects, given an exception that
could have been generated anywhere in a couple of screenfuls worth of
code, and an object that could be in any state.
If you do some inspection of the exception (check out the traceback
module), you can usually find out where an error occurred, and exactly
what kind of exception it was.

- Decoupling from socket objects: removal of the __getitem__ magic
which causes screenfuls of abuse for a simple typo at the wrong stage
in execution.

Infinite recursion in asyncore? Yikes, I've not hit that bug before.
Care to post some offending code?

It was entirely my fault, but here is the example anyway. :)

With a bit of 'fancy' stack introspection, that infinite recursion error
could be easily fixed. Of course it would end up slowing down attribute
lookups.

I bet there is a quick fix (without introspection), but I'm too tired to
figure it out now, and I have jury duty in 9 hours. Maybe tomorrow evening.

One could even have a framework for registering even handlers...yikes,
this is starting to sound like wxPython. I don't know if that is a good
idea or not.

NAME asyncore
Basic infrastructure for asynchronous socket service clients and
servers.

I think that answers the question.

Good point, it is a bad idea.

Is anyone else interested in this? I have already made small
modifications to the asyncore module to fix a few things, but nothing
mentioned here. I could put together a proposed asyncore2 module if
there was interest. I may do it, out of need, anyway. :)

Feel free to post your changes in some sort of public place (on the web
is probably best). If people want/need it, they'll probably tell you.
With enough support, asyncore could get patched or asyncore2 could be born.

- Josiah

[Alain Pointdexter]

My personal appreciation: asyncore is the most perfect combination of
hard work and incompetence i have ever seen in my life. It is so
anti-pythonic that it should get out of the standard distribution.

Just a personal opinion ...

dw***********@botanicus.net (David M. Wilson) wrote in message news:<99**************************@posting.google. com>...

Josiah Carlson <jc******@nospam.uci.edu> wrote...

I agree that there should probably be some sort of asyncore2; I have a
handful of boilerplate servers and clients that I use as a base any time
I need something. Optimally, an asyncore2 library would have simple
clients and servers ready to be subclassed and customized as needed.

I would expect such things to be provided by modules that are
dependant on asyncore, eg. as asynchat is. It's beyond the scope of
what asyncore tries to do.

I'm curious as to why you need to use a private map. Most uses of
private maps I've seen haven't added any sort of meaningful
functionality that couldn't have been handled by the default map. Yours
could be an exception, but I'm still curious.

I don't need a private map, however given that the other option was a
module variable, I naturally chose to provide my own to increase any
required future flexibility. Indeed there is little, if anything to be
gained from using your own map, but the option was there.

Again, you need to use asyncore.poll.

The poll function is not documented in the manual, nor do any
docstrings suggest it is a public interface. Internally asyncore
chooses which of poll, poll2, or poll3 to use inside loop, so calling
these directly also bypasses some of asyncore's OS-independance
functionality.

I've never used asyncore with UDP sockets, but maybe using
"handle_connect = handle_read" in your subclass definition could be
sufficient.

I see this as a workaround, and as such I think the point still
stands. :)

That is exactly why polling is optional. Select is available on every
platform, but poll is faster on platforms that support it...

That is not true at all. If you are feeling investigative, quite a
complete starting point would be <http://www.kegel.com/c10k.html>.
There is lots of information there regarding various OS
implementations async IO performance using poll, epoll, select, aio_*,
etc.

The trick with doing this is that you end up interrupting other
potentially ready sockets with handling an exception. If you are
feeling fiesty, I believe you can use the following and get the error
propagated back up:

def handle_error(self):
raise

That's a good point, but I don't have a problem restarting the poll
after handling an exception, a) because it shouldn't happen very
often, b) will probably mean shutting down the service rather than
trying to continue, and c) exceptions are generally expensive special
cases anyway.

If an exception is propogating up from the dispatcher, that means the
dispatcher's method did not handle the exception, which should mean it
gets passed back up the stack as it does with all other code,
eventually ending up with a traceback print if it wasn't handled.

I dislike the handle_error method since it is essentially redundant in
the face of Python's own exception handling mechanism. If a dispatcher
wants it's own function to handle exceptions, it should wrap it's
methods in a try block.

I assume handle_error was provided to allow graceful teardown of a
dispatcher that was erroneous, but it isn't granular enough to be
useful for tearing down complicated objects, given an exception that
could have been generated anywhere in a couple of screenfuls worth of
code, and an object that could be in any state.

- Decoupling from socket objects: removal of the __getitem__ magic
which causes screenfuls of abuse for a simple typo at the wrong stage
in execution.

Infinite recursion in asyncore? Yikes, I've not hit that bug before.
Care to post some offending code?

It was entirely my fault, but here is the example anyway. :)

py> from asyncore import *
py> class D(dispatcher):
... def __init__(self):
... self.foo
... dispatcher.__init__(self)
...
py> D()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in __init__
File "/usr/lib/python2.3/asyncore.py", line 365, in __getattr__
return getattr(self.socket, attr)
File "/usr/lib/python2.3/asyncore.py", line 365, in __getattr__
return getattr(self.socket, attr)
File "/usr/lib/python2.3/asyncore.py", line 365, in __getattr__
return getattr(self.socket, attr)

One could even have a framework for registering even handlers...yikes,
this is starting to sound like wxPython. I don't know if that is a good
idea or not.

NAME asyncore
Basic infrastructure for asynchronous socket service clients and
servers.

I think that answers the question.
Is anyone else interested in this? I have already made small
modifications to the asyncore module to fix a few things, but nothing
mentioned here. I could put together a proposed asyncore2 module if
there was interest. I may do it, out of need, anyway. :)
David.

[Josiah Carlson]

> My personal appreciation: asyncore is the most perfect combination of

hard work and incompetence i have ever seen in my life. It is so
anti-pythonic that it should get out of the standard distribution.

Just a personal opinion ...

Funny thing is that asyncore (aka medusa) is one of the highest
performing asyncronous socket libraries for Python that is available.
It also compares nicely (performance-wise) with most anything you could
find in non-interpreted languages.

Honestly, I don't know what your issue is. It is event based, and in
the realm of event driven frameworks, is reasonable.

So what is your problem?

- Josiah