xmlrpc, httplib and SSL

[Roger Binns]

I have just spent several weeks mashing xmlrpc, httplib and SSL (from
M2Crypto) to work together. The current standard library has several
problems:

- Builtin SSL is pretty much useless if you actually care
about security
- Poor HTTP authentication support
- No server side stuff (SSL, HTTP authentication etc)
- Pathological coding to ensure that at most one request is
sent on a connection, rather than reusing an already open
connection (http/1.1 keepalives)
- Pathological coding to ensure that connections are closed
from as many different places as possible
- A broken model for dealing with who owns a connection socket.
It all started when someone added a 'makefile' method that
returned a dup'ed file descriptor since all there are
all those close calls everywhere, so the reference counting
sort of works on UNIX (where dup is a normal operation).
This leads to even more heroic coding to work around the
explicitly coded close's everywhere, and implement yet
another layer of reference counting and encapsulation.
- No real possibility of dealing with things like automatically
reopening connections (eg if you have an HTTP/1.1 connection,
make a request, stay idle for so long the other end closes the
connection, and then try to send a new one)

I now have code that works for me and my project. However it
wouldn't really be appropriate for going back into the standard
library because it spends most of its time having to subvert the
design and implementation of the existing classes.

However I was wondering if anyone was working on fixing the
(IMHO horrible) mess and wants any moral support?

Roger

[Skip Montanaro]

[snip summary of changes]

Roger> I now have code that works for me and my project. However it
Roger> wouldn't really be appropriate for going back into the standard
Roger> library because it spends most of its time having to subvert the
Roger> design and implementation of the existing classes.

Roger> However I was wondering if anyone was working on fixing the (IMHO
Roger> horrible) mess and wants any moral support?

Sounds like you've done most of the work already. Why not post a few
patches to SF for the various affected modules?

Skip

[Roger Binns]

Skip Montanaro wrote:

[snip summary of changes]

Roger> I now have code that works for me and my project. However it
Roger> wouldn't really be appropriate for going back into the standard
Roger> library because it spends most of its time having to subvert the
Roger> design and implementation of the existing classes.

Roger> However I was wondering if anyone was working on fixing the (IMHO
Roger> horrible) mess and wants any moral support?

Sounds like you've done most of the work already. Why not post a few
patches to SF for the various affected modules?

The standard modules need to be redesigned! And M2Crypto would have
to be the standard SSL. And as I state in the first paragraph you quote,
it has to subvert the existing classes. Think of my code as the
result of using chainsaws and band-aids all over the existing code.
The end result works fine for me and my project. But I wouldn't
remotely want to change the existing library to anything like it.

Here were the goals I was trying to meet. The current Python library
does not remotely meet them.

XML-RPC client:

- Connections are made over SSL
- A callback to verify the certificate if there is no
CA or other defined mechanism
- HTTP/1.1 is used, and the connection is reused if
still open
- HTTP authentication is used
- Automatic reopen of connection if above one doesn't
work (eg remote end closed it after timeout)
- Be thread safe (eg one connection per thread - don't
reuse across threads)

XML-RPC server:

- Connections are accepted over SSL
- Callbacks to verify incoming connections, certificates and
credentials of the connections
- Verification of HTTP authentication information
- Connections are kept open/keep-alive (HTTP/1.1)
- Use of a bounded thread pool that handles the connections

Roger

[Etienne Posthumus]

On Mar 27, 2004, at 5:03 AM, Roger Binns wrote:

However I was wondering if anyone was working on fixing the
(IMHO horrible) mess and wants any moral support?

I wanted to do keep the connections on which my clients do XMLRPC calls
open, and after staring at the xmlrpclib.py source for a while, came up
with the class at the bottom of the message. Just thought I would post
it here in the spirit of sharing. You would use it like this:

import httplib, xmlrpclib
s = xmlrpclib.ServerProxy('http://SOMEURL',
transport=PersistTransport())

And then use as normal. When any error occurs, the connection is
closed, seems a bit pessimistic, but I didn't want to do anything more
fancy.

cheers,

Etienne Posthumus
---
http://www.mnemosyne.org/
Cultural Heritage Research
Python, Zope, XML expertise for hire.
Amsterdam, Nederland
----

class PersistTransport(xmlrpclib.Transport):
'''Provides a Transport for the xmlrpclib that uses httplib
supporting persistent connections
Does not close the connection after each request.
'''
connection = None

def request(self, host, handler, request_body, verbose=0):
if not self.connection:
host, extra_headers, x509 = self.get_host_info(host)
self.connection = httplib.HTTPConnection(host)
self.headers = {"User-Agent" : self.user_agent,
"Content-Type" : "text/xml",
"Accept": "text/xml"}
if extra_headers:
for key, item in extra_headers:
self.headers[key] = item

self.headers["Content-Length"] = str(len(request_body))
self.connection.request('POST', handler, request_body,
self.headers)
r = self.connection.getresponse()
if r.status != 200:
self.connection.close()
self.connection = None
raise xmlrpclib.ProtocolError( host + handler, r.status,
r.reason, '' )
data = r.read()
p, u = self.getparser()
p.feed(data)
p.close()
return u.close()

[Roger Binns]

> I wanted to do keep the connections on which my clients do XMLRPC calls

open, and after staring at the xmlrpclib.py source for a while, came up
with the class at the bottom of the message.

Are you sure you are actually getting persistent connections? The
code will auto-close and auto-open the connection even just keeping
the same HTTP connection object.

In order for the connection to not be auto-closed, the remote end
must return a HTTP/1.1 response (see httplib.HTTPResponse._check_close).

If your server end is Python, then it will always close the connection
unless the request was HTTP/1.1 *and* a 'Connection: keep-alive' header
was sent, which the Python client does not do.

The above is all true for Python 2.3. For 2.2 it does HTTP/1.0 IIRC
so you have no hope.

Roger