Wierd M2Crypto bug - phony "peer did not return certificate" error

[John Nagle]

Here's a wierd problem:

I have a little test case for M2Crypto, which just opens up SSL connections to
web servers and reads their certificates. This works fine.

But if I execute

socket.setdefaulttimeout(5.0)

so that the sockets don't wait too long if there's no SSL server, I get
a "peer did not return certificate" exception every time.

Environment:
Windows 2000 SP 4
M2Crypto: M2Crypto-0.17.win32-py2.4.exe
Python: Python 2.4.4 (#71, Oct 18 2006, 08:34:43)
OpenSSL: Win32 OpenSSL v0.9.8d

Notes:
Running M2Crypto does NOT use the version of OpenSSL that comes
with Python. You have to install OpenSSL separately, or
M2Crypto's DLL won't find the OpenSSL DLLS.

There's a good chance that this may be related to:

[ python-Bugs-1098618 ] socket.setdefaulttimeout() breaks smtplib.starttls()

But that should have been fixed in Python 2.4.4, right? Or did that patch
not make it in?

John Nagle

(And no, I can't run Python 2.5, because MySQLdb support doesn't work for
2.5 yet.)

[Heikki Toivonen]

John Nagle wrote:

I have a little test case for M2Crypto, which just opens up SSL
connections to
web servers and reads their certificates. This works fine.

But if I execute

socket.setdefaulttimeout(5.0)

Yup, this is a known problem, this breaks all M2Crypto code that uses
sockets. The Twisted wrapper part still works, as it hands the network
activity to Twisted.

If you can make do without setdefaulttimeout you should be fine.

This is bug https://bugzilla.osafoundation.org/show_bug.cgi?id=2341

The bug has a patch that works on Linux, but it would need Windows and
Mac specific parts before it can be checked in.

--
Heikki Toivonen

[nagle]

That's a problem for me. I need short timeouts; I'm accessing sites
that might or might not have SSL support, and I need to quickly time
out when there's no SSL server.

PyOpenSSL handles timeout correctly, but M2Crypto does not. On the
other hand, M2Crypto actually checks certificates, which PyOpenSSL does
not. So we have two broken implementations.

Python needs a merge here. Read Guido van Rossum's rant on M2Crypto:

http://www.artima.com/weblogs/viewpost.jsp?thread=95863

Both of these packages contain wrappers for OpenSSL, but both wrappers
are incompatible and buggy. M2Crypto also has some Python components.
The PyOpenSSL wrapper, which ships with Python, doesn't expose enough
of the OpenSSL API. M2Crypto exposes more of the API, but doesn't work
as well.

What's probably needed is to implement the additional API functions of
the M2Crypto wrapper in the PyOpenSSL wrapper, so that the M2Crypto
components written in Python could be used with it. Then we'd have one
good implementation instead of two broken ones.

John Nagle
na***@animats.com

(News feed broken, using Google Groups as backup.)

[Heikki Toivonen]

na***@animats.com wrote:

That's a problem for me. I need short timeouts; I'm accessing sites
that might or might not have SSL support, and I need to quickly time
out when there's no SSL server.

You should be able to do short timeouts, just not using the global
setdefaulttimeout. Have you tried Connection.set_socket_read/write_timeout?

Also like I mentioned before, if you use the Twisted wrapper and let
Twisted handle network stuff you should be fine.

Python needs a merge here. Read Guido van Rossum's rant on M2Crypto:

http://www.artima.com/weblogs/viewpost.jsp?thread=95863

That's old news, I believe I have fixed all of the issues mentioned
there already in the 0.16 release.

What's probably needed is to implement the additional API functions of
the M2Crypto wrapper in the PyOpenSSL wrapper, so that the M2Crypto
components written in Python could be used with it. Then we'd have one
good implementation instead of two broken ones.

M2Crypto and pyOpenSSL (and pyOpenSSL-extended, which you might want to
take a look at as well) are implemented pretty differently, so merging
seems unlikely. They are pretty small code wise, though, so it is not an
impossible task.

Personally I think I'd prefer if Python stdlib contained a better SSL
module that did at least all the checks required for safe SSL
connection. (Yeah, yeah, maybe I need to write it myself if nobody else
gets to it;)

--
Heikki Toivonen

[John Nagle]

Heikki Toivonen wrote:

na***@animats.com wrote:

> That's a problem for me. I need short timeouts; I'm accessing sites
that might or might not have SSL support, and I need to quickly time
out when there's no SSL server.

You should be able to do short timeouts, just not using the global
setdefaulttimeout. Have you tried Connection.set_socket_read/write_timeout?

Yes. That does not affect the connect timeout; it's only effective once
the connection has been opened. And adjusting the session timeout
just recreates the blocking/non blocking problem.

Incidentally, "get_socket_read_timeout()" doesn't work. Generates
"EXCEPTION at socket level: unpack str size does not match format",
every time, at least with Python 2.4 on Windows. The lower level
function returns one number as a string, like "7200", and the unpack
function tries to unpack it as "ll", which fails.

Also like I mentioned before, if you use the Twisted wrapper and let
Twisted handle network stuff you should be fine.

That would mean struggling with Twisted and dealing with its bugs.
(For example, has the MySQLdb mess been resolved for Twisted's API?)
This isn't a long-running application; it usually runs as a CGI program.
So Twisted is inappropriate.

M2Crypto and pyOpenSSL (and pyOpenSSL-extended, which you might want to
take a look at as well) are implemented pretty differently, so merging
seems unlikely. They are pretty small code wise, though, so it is not an
impossible task.

That seened good idea, and so I downloaded the source and tried to build
it on a Windows machine to run with Python 2.4. But that build needs
(exactly) Visual Studio 7.1, which I don't have. It also wants a specific
version of OpenSSL, and has a build file which seems to assume a 1998 version
of Visual C++. The last change was in 1995, and it's still at an 0.x version,
so it's effectively abandonware. I have some doubts that it really works
any more. I saw some Python 2.2/2.3 specific code in there.

I could try building on a Linux system, but it's useful to me if Python
works on both Windows and Linux.

>
Personally I think I'd prefer if Python stdlib contained a better SSL
module that did at least all the checks required for safe SSL
connection. (Yeah, yeah, maybe I need to write it myself if nobody else
gets to it;)

What I need is proper SSL operation, certificate chain validation,
useful exception info when a connection fails (including why), and
read acess to the certificate itself in some reasonably sane form.
And I need to time out an SSL connection if it can't connect and
verify within 4 seconds. It's embarassing that this doesn't work,
despite four different wrappers for OpenSSL.

I'm willing to spend a few hundred dollars towards making this happen.

John Nagle

[John Nagle]

OK, I have a one-line fix.

To Connection.py of M2Crypto:

def connect(self, addr):
self.socket.connect(addr)
self.addr = addr
self.socket.settimeout(None) # Back to normal timeout (NEW)
self.setup_ssl()
self.set_connect_state()
ret = self.connect_ssl()
check = getattr(self, 'postConnectionCheck', self.clientPostConnectionCheck)
if check is not None:
if not check(self.get_peer_cert(), self.addr[0]):
raise Checker.SSLVerificationError, 'post connection check failed'
return ret
After the socket is connected, we turn its timeout off.

The effect is that if the caller sets

socket.setdefaulttimeout(timeoutsecs)

before creating the Connection object, that will set the connection
timeout. The socket will be created as non-blocking, but before
any reads or writes are done, we clear the timeout, making it blocking
again. So the connection will time out as requested, but we won't
get errors because the socket is non-blocking and is being used by
code that expects it to block.

Gets usable connection timeouts, and prevents bogus "peer did not
return certificate" errors.

John Nagle
Animats

[Heikki Toivonen]

John Nagle wrote:

def connect(self, addr):
self.socket.connect(addr)
self.addr = addr
self.socket.settimeout(None) # Back to normal timeout (NEW)

I am not sure if this is always appropriate. In fact, doesn't this just
eliminate the timeout completely from this connection?

In your case you could achieve what you want by calling
connection_object.socket.settimeout(4.0) method just before calling
connect, without needing to edit M2Crypto itself.

setdefaulttimeout, socket.settimeout and socket.setblocking are pretty
confusing...

--
Heikki Toivonen

[John Nagle]

Heikki Toivonen wrote:

John Nagle wrote:

>>def connect(self, addr):
self.socket.connect(addr)
self.addr = addr
self.socket.settimeout(None) # Back to normal timeout (NEW)

I am not sure if this is always appropriate. In fact, doesn't this just
eliminate the timeout completely from this connection?

In your case you could achieve what you want by calling
connection_object.socket.settimeout(4.0) method just before calling
connect, without needing to edit M2Crypto itself.

No, this actually works right. It seems that the default timeout
controls the timeout on TCP connect, but puts the socket into
non-blocking mode. So, if as soon as a TCP connection is
opened, but before the SSL handshake starts, the socket timeout
is set to None, then the socket returns to blocking mode before
its first read.

At least on Windows. More later on how this works on Linux.

John Nagle

[John Nagle]