Problem building/using M2Crypto extension module

[Carl Waldbieser]

I tried to adapt the instructions for building the M2Crypto module
(http://sandbox.rulemaker.net/ngps/m2/INSTALL.html) to build a version
compatible with Python2.3, but I've had some mixed results. I actually got
everything to build and install, but when I try to import M2Crypto-- well,
here is a sample session:

import M2Crypto Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\__init__.py", line 7, in ?
import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified. import M2Crypto
dir() ['M2Crypto', '__builtins__', '__doc__', '__name__', 'pywin'] import M2Crypto.Rand Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\Rand.py", line 7, in ?
import m2
File "C:\PYTHON23\Lib\site-packages\M2Crypto\m2.py", line 7, in ?
from __m2crypto import *
ImportError: DLL load failed: The system cannot find the file specified. import M2Crypto.Rand

As you can see, the first time I try to import the module, it fails, but the
next time, it works. The same thing happens when I try to import a
sub-module underneath it.

Has anyone experienced any problems similar to this when building extension
modules? Is there some sort of common trap I might have fallen into?

The following is version info:
I am running Python 2.3.1 on Windows98. I built the extension from source,
using the mingw compiler 3.2.3. I built version 0.9-7c of the OpenSSL
library to use with M2Crypto. This is the first time I used SWIG 1.3.19 and
PExport 0.42h.

This is the first time I have actually tried to build a Python extension on
this PC, so the chances are probably pretty good that I have screwed
something up, but I am not sure how I would go about tracking it down.

If anyone else has successfully built and used this extension for Python2.3
on Windows, I would be glad to hear it, just to know it is possible. If
anyone has any advice or suggestions, I would also be glad to hear that,
too!

Thanks,
Carl Waldbieser

[Ng Pheng Siong]

According to Carl Waldbieser <wa*****@attglobal.net>:

import M2Crypto Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\__init__.py", line 7, in ?
import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified.

Is there a __m2crypto.dll or __m2crypto.pyd (sorry can't remember which)
somewhere?

I haven't tried with Python 2.3 on Windows, but I've done it on FreeBSD. No
change in build procedure.

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.

import M2Crypto

$ python2.3 alltests.py
.................................................. ....................
----------------------------------------------------------------------
Ran 70 tests in 6.756s

OK
On Windows, I use Bloodshed DevC++, which is some version of mingw. Instead
of pydef, I use the free BC++'s impdef, but either should produce
equivalent .def files.

I was going to install Python 2.3 for Windows, then I noticed 2.3.1 and
then I noticed 2.3.2 release candidate, so I decided to wait a little.

Do you have Python 2.2.3, say? Are you able to build on this version?
--
Ng Pheng Siong <ng**@netmemetic.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Carl Waldbieser]

There is a __m2crypto.pyd in my \Python23\Lib\site-packages\M2Crypto
directory. I don't have Python 2.2.3 installed, but I may eventually
re-install it just to see if I can get this working.

Is there some way I can inspect the output files at each step of the build
so as to detect an error in my process? Also, I do not recall using pydef,
and I am not sure what it is. I beilieve that I used a tool called pexport
to make the .def file for python.

If you get around to building it for Python2.3.x, I'd be glad to know how it
went. One thing I got hung up on for a long time was trying to get the
distutils to do the install. If I tried:

python setup.py install
I would get an error because I built the files using mingw and it would say
I needed to build it with Visual C++. Of course, I already had built it for
mingw with:
python setup.py build -cmingw
so I was a bit confused. There was no such flag for the install, as far as
I could tell. After reading through the help for a while, I found a flag
that let me skip the re-build that is implicit in the install. I didn't
think that should have caused a problem, though.

Carl Waldbieser

"Ng Pheng Siong" <ng**@netmemetic.com> wrote in message
news:bl**********@mawar.singnet.com.sg... According to Carl Waldbieser <wa*****@attglobal.net>:

>> import M2Crypto Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\__init__.py", line 7, in ?

import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified.

Is there a __m2crypto.dll or __m2crypto.pyd (sorry can't remember which)
somewhere?

I haven't tried with Python 2.3 on Windows, but I've done it on FreeBSD.

No change in build procedure.

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information. >>> import M2Crypto
>>>

$ python2.3 alltests.py
.................................................. ....................
----------------------------------------------------------------------
Ran 70 tests in 6.756s

OK
On Windows, I use Bloodshed DevC++, which is some version of mingw.

Instead of pydef, I use the free BC++'s impdef, but either should produce
equivalent .def files.

I was going to install Python 2.3 for Windows, then I noticed 2.3.1 and
then I noticed 2.3.2 release candidate, so I decided to wait a little.

Do you have Python 2.2.3, say? Are you able to build on this version?
--
Ng Pheng Siong <ng**@netmemetic.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Ng Pheng Siong]

According to Carl Waldbieser <wa*****@attglobal.net>:

There is a __m2crypto.pyd in my \Python23\Lib\site-packages\M2Crypto
directory.
cd there, import the DLL into the interpreter toplevel:

$ ls -l __m2crypto.so
-rwxr-xr-x 1 ngps ngps 458146 Sep 28 20:59 __m2crypto.so*

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.

import __m2crypto

Does it work?
Is there some way I can inspect the output files at each step of the build
so as to detect an error in my process? Also, I do not recall using pydef,
and I am not sure what it is. I beilieve that I used a tool called pexport
to make the .def file for python.

Sorry, typo, I meant pyexport.

I've just put up 0.12 snapshot #1. There is a SWIG/Makefile.mw, which is
for use with mingw. Take a look and adjust paths (e.g., OpenSSL
include/library files, swig.exe) where necessary. Try it with "make -f
Makefile.mw".
--
Ng Pheng Siong <ng**@netmemetic.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Ng Pheng Siong]

According to Carl Waldbieser <wa*****@attglobal.net>:

If you get around to building it for Python2.3.x, I'd be glad to know how it
went.

Turns out I do have Python 2.3 installed on one of my
rarely-booted-into-Windows computer.

Here's what I get:

C:\ngps\prog\ngps\m2\SWIG>make -f Makefile.mw
c:/pkg/swig/swig.exe -shadow -python _m2crypto.i
gcc -c -DTHREADING -g -Ic:/pkg/py23/include -Ic:/pkg/openssl/include -I. _m2cryp
to_wrap.c
[warnings deleted]
dllwrap --dllname __m2crypto.pyd --driver-name gcc \
--def _m2crypto.def -o __m2crypto.pyd _m2crypto_wrap.o \
-s --entry _DllMain@12 --target=i386-mingw32 c:/pkg/py23/libs/libpyt
hon23.a c:/pkg/openssl/lib/libssl32.a c:/pkg/openssl/lib/libeay32.a
copy _m2crypto.py ..\M2Crypto
1 file(s) copied.
copy __m2crypto.pyd ..\M2Crypto
1 file(s) copied.

C:\ngps\prog\ngps\m2\SWIG>python
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import __m2crypto
C:\ngps\prog\ngps\m2>python setup.py build -cmingw32
running build
running build_py
running build_ext
building '__m2crypto' extension
swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap.c
c:\pkg\swig\swig.exe -python -ISWIG -o SWIG/_m2crypto_wrap.c SWIG/_m2crypto.i
g:\pkg\dev-c++\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\ngps\prog\ngps\m2\SWI
G -Ic:\pkg\openssl/include -Ic:\pkg\py23\include -Ic:\pkg\py23\PC -c SWIG/_m2cry
pto_wrap.c -o build\temp.win32-2.3\Release\swig\_m2crypto_wrap.o -DTHREADING
[warnings deleted]
writing build\temp.win32-2.3\Release\swig\__m2crypto.def
g:\pkg\dev-c++\bin\dllwrap.exe -mno-cygwin -mdll -static --output-lib build\temp
..win32-2.3\Release\swig\lib__m2crypto.a --def build\temp.win32-2.3\Release\swig\
__m2crypto.def -s build\temp.win32-2.3\Release\swig\_m2crypto_wrap.o -Lc:\pkg\op
enssl\lib -Lc:\pkg\py23\libs -Lc:\pkg\py23\PCBuild -lssleay32 -llibeay32 -lpytho
n23 -o build\lib.win32-2.3\M2Crypto\__m2crypto.pyd

C:\ngps\prog\ngps\m2>python setup.py install --skip-build
running install
running install_lib
creating c:\pkg\py23\Lib\site-packages\M2Crypto
copying build\lib.win32-2.3\M2Crypto\ASN1.py -> c:\pkg\py23\Lib\site-packages\M2
Crypto
[blah blah blah]
byte-compiling c:\pkg\py23\Lib\site-packages\M2Crypto\__init__.py to __init__.py
c

C:\>python
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information. import M2Crypto
dir(M2Crypto) ['ASN1', 'BIO', 'DH', 'DSA', 'EVP', 'Err', 'PGP', 'RC4', 'RCS_id', 'RSA', 'Rand'
, 'SSL', 'X509', '__builtins__', '__doc__', '__file__', '__m2crypto', '__name__'
, '__path__', 'decrypt', 'encrypt', 'httpslib', 'm2', 'm2urllib', 'urllib2', 'ut
il'] from M2Crypto import RSA
k = RSA.gen_key(512,65537) .............................++++++++++++
..............++++++++++++

You need mingw versions of libpython23.a, libssl32.a and libeay32.a.

Makefile.mw says:

SSLLIB=c:/pkg/openssl/lib/libssl32.a c:/pkg/openssl/lib/libeay32.a

Whereas, using setup.py, the command line constructed is "-lssleay32
-llibeay32", so you need libssleay32.a and liblibeay32.a. Somewhat
ridiculous looking, yes. ;-). This is because setup.py was written for
VC++; the relevant snippet looks like this:

if os.name == 'nt':
openssl_dir = 'c:\\pkg\\openssl'
include_dirs = [my_inc, openssl_dir + '/include']
library_dirs = [openssl_dir + '\\lib']
libraries = ['ssleay32', 'libeay32']
#libraries = ['ssleay32_bc', 'libeay32_bc']
extra_compile_args = [ "-DTHREADING" ]

So, alternatively, you can modify the "libraries" line to say "['ssl32',
'eay32']" (I think) to use the default library names assigned to OpenSSL
libraries by mingw. (On Unix they're called libssl.so and libcrypto.so.
Duh!)

Notice "['ssleay_bc', 'libeay_bc']". That's for Borland's free compiler
suite. I haven't tested this in a long while. Going forward, it will be
mingw only.

HTH.
--
Ng Pheng Siong <ng**@netmemetic.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Carl Waldbieser]

"Ng Pheng Siong" <ng**@netmemetic.com> wrote in message

cd there, import the DLL into the interpreter toplevel:

$ ls -l __m2crypto.so
-rwxr-xr-x 1 ngps ngps 458146 Sep 28 20:59 __m2crypto.so*

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.

>>> import __m2crypto
>>>

Does it work?

No, it gives me an error each time. Guess there is something wrong with the
..pyd.
I've just put up 0.12 snapshot #1. There is a SWIG/Makefile.mw, which is
for use with mingw. Take a look and adjust paths (e.g., OpenSSL
include/library files, swig.exe) where necessary. Try it with "make -f
Makefile.mw".

That seems to have worked. The .pyd is a little bigger. I copied it into
the site-packages/M2Crypto directory, and I can import M2Crypto now. When I
run the tests, I get:

.................................................. ...............EEEEEE
================================================== ====================
ERROR: test_cipher_mismatch (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
88, in test_cipher_mismatch
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

================================================== ====================
ERROR: test_cipher_ok (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
114, in test_cipher_ok
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

================================================== ====================
ERROR: test_no_such_cipher (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
101, in test_no_such_cipher
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

================================================== ====================
ERROR: test_server_simple (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
54, in test_server_simple
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

================================================== ====================
ERROR: test_tls1_nok (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
65, in test_tls1_nok
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

================================================== ====================
ERROR: test_tls1_ok (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
77, in test_tls1_ok
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

----------------------------------------------------------------------
Ran 70 tests in 2.530s

FAILED (errors=6)

So it looks like I'm not quite out of the woods yet, but these Python errors
at least seem a little more comprehensible to me. I am still not sure what
was going on with the library. Do you have any idea what I did wrong the
first time, or was it just a lucky guess? I would be interested in knowing
so I can try to avoid that kind of mistake in the future.

Thanks for your help!
Carl Waldbieser

[Ng Pheng Siong]

According to Carl Waldbieser <wa*****@attglobal.net>:

That seems to have worked. The .pyd is a little bigger. I copied it into
the site-packages/M2Crypto directory, and I can import M2Crypto now. When I
run the tests, I get:

.................................................. ..............EEEEEE
These are ok; the errors are in test_ssl_win.py, which says:

Win32 version - requires Mark Hammond's Win32 extensions and openssl.exe
on your PATH.

So it looks like I'm not quite out of the woods yet, but these Python errors
at least seem a little more comprehensible to me. I am still not sure what
was going on with the library. Do you have any idea what I did wrong the
first time, or was it just a lucky guess?

I can't tell, really. As I posted in another message, building for Python
2.3 on Windows works for me.

Anyways, looks like you're good to go.

Cheers.
--
Ng Pheng Siong <ng**@netmemetic.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Carl Waldbieser]

Yes, that was the problem-- openssl.exe was not in my PATH. Thanks!

Carl Waldbieser

"Ng Pheng Siong" <ng**@netmemetic.com> wrote in message > These are ok; the
errors are in test_ssl_win.py, which says:

Win32 version - requires Mark Hammond's Win32 extensions and openssl.exe
on your PATH.