A solution to the MSVCRT vs MSVCR71 problem?

[sturlamolden]

This question has been asked many times, and last time I was accused of
spreading FUD. So now I will rather propose a solution.

The reason for the problem is as follows:

The binary installer for Python built by te python.org team is compiled
with Microsoft Visual Studio 2003. It is linked with the C runtime
msvcrt71.dll. The copyright to msvcrt71.dll is owned by Microsoft, and
it is not an integral part of the Windows operating system. This leads
to two distinct problems:

Problem 1: You want to use a particular compiler to build a Python
extension library, but it links with the wrong CRT. Typical examples
are MinGW, Microsoft Visual C++ 6.0 and Microsoft Visual C++ Express.
There is a remedy for MinGW, but not for the other two.

Problem 2: You want to distribute a program created Py2Exe, but has no
license for Visual Studio 2003. You are therefore not allowed to
redistribute msvcrt71.dll. But without this DLL your program will not
work. As a side note: Visual Studio 2003 is out of sale, so if you
don't have it already you may be out of luck.

The solution: modify the IAT of to use the correct binary.

The process is really simple:

Solution to problem 1:

Compile with your compiler or choice, never mind which CRT are used.
Use a 'dumpbin' program, find all references to msvcrt in the binary
DLL file. Create a dll with name "py_crt" that exports these functions
but redirects them to msvcrt71. That is, in the file py_crt.def we put
something like

EXPORTS
malloc=msvcr71. malloc
free=msvcr71.fr ee
etc.

Compile the DLL py_crt.dll and then open your pyd file in binary mode.
Exchange all occurances of the string "msvcrt" (or any other CRT name)
with "py_crt". Now your binary should work just fine. What you need to
make sure is just that the name of the proxy has the same number of
letters as the CRT your compiler linked. So if it is msvcrt81.dll, e.g.
use something like py_crt81.dll instead of py_crt.dll.

Solution to problem 2:

This would be the opposite, involving modifying Python25.dll and all
pyd and dll files gathered by Py2Exe to use msvcrt.dll instead of
msvcr71.dll. One could e.g. create a DLL called py_cr71.dll that
redirects CRT calls to msvcrt.dll or any other CRT of choice. In
py_cr71.def we would then have:

EXPORTS
malloc=msvcrt.m alloc
free=msvcrt.fre e
etc.

And then it is simply a matter of modifying the binaries to load
py_cr71.dll instead of msvcrt71.dll.

What we need to make this work:

1. A dumpbin facility similar to the one that ships with Visual Studio
that can be freely distributed.

2. A python script that parses the output from dumpbin, creates the
proxy and redirects the CRT. (I already have such a script working.)

Finally, a setup script (setup.py) could automate this task for
"problem 1" above.

Another variant of the fake-CRT procedure, which would also work, is to
create a fake msvcrt71.dll that redirects to your CRT of choice, and
replace the msvcrt71.dll that came with Python with that. That is
conceptually simpler, as it involves not binary modifications, but
someone might mistake your msvcrt71.dll for the msvcrt71.dll from
Microsoft. Also antivirus-software might not like attempts to load such
a DLL as it would look like your program is hijacked by a trojan
(although it is not).

What do you think?

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

sturlamolden schrieb:

Problem 2: You want to distribute a program created Py2Exe, but has no
license for Visual Studio 2003. You are therefore not allowed to
redistribute msvcrt71.dll. But without this DLL your program will not
work. As a side note: Visual Studio 2003 is out of sale, so if you
don't have it already you may be out of luck.

I believe this problem doesn't exist. Licensees of Python are permitted
to redistribute mscvr71.dll, as long as they redistribute it in order
to support pythonxy.dll. The EULA says

# You also agree not to permit further distribution of the
# Redistributable s by your end users except you may permit further
# redistribution of the Redistributable s by your distributors to your
# end-user customers if your distributors only distribute the
# Redistributable s in conjunction with, and as part of, the Licensee
# Software, you comply with all other terms of this EULA, and your
# distributors comply with all restrictions of this EULA that are
# applicable to you.

In this text, "you" is the licensee of VS 2003 (i.e. me, redistributing
msvcr71.dll as part of Python 2.5), and the "Redistributabl e" is
msvcr71.dll. The "Licensee Software" is "a software application product
developed by you that adds significant and primary functionality to the
Redistributable s", i.e. python25.dll.

IANAL; this is not legal advise.

Regards,
Martin

[Gabriel Genellina]

At Sunday 21/1/2007 00:07, sturlamolden wrote:

>Solution to problem 1:

Compile with your compiler or choice, never mind which CRT are used.
Use a 'dumpbin' program, find all references to msvcrt in the binary
DLL file. Create a dll with name "py_crt" that exports these functions
but redirects them to msvcrt71. That is, in the file py_crt.def we put
something like

EXPORTS
malloc=msvcr71 .malloc
free=msvcr71.f ree
etc.

Compile the DLL py_crt.dll and then open your pyd file in binary mode.
Exchange all occurances of the string "msvcrt" (or any other CRT name)
with "py_crt". Now your binary should work just fine. What you need to
make sure is just that the name of the proxy has the same number of
letters as the CRT your compiler linked. So if it is msvcrt81.dll, e.g.
use something like py_crt81.dll instead of py_crt.dll.

Solution to problem 2:
[modify external references inside python25.dll to use a different runtime]

This would only work, if runtime dll's were compatibles between them,
and they are not. You can't blindly redirect a call to msvcr71.__xyz
to msvcr80.__xyz and expect that to work magically - it may have a
different number of arguments, or different types, or even may not
exist anymore.
(And what about any symbol exported by ordinal?)
--
Gabriel Genellina
Softlab SRL


_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Gabriel Genellina schrieb:

This would only work, if runtime dll's were compatibles between them,
and they are not. You can't blindly redirect a call to msvcr71.__xyz to
msvcr80.__xyz and expect that to work magically - it may have a
different number of arguments, or different types, or even may not exist
anymore.

Actually, the libraries *are* binary-compatible (on the ABI level). You
just can't mix two libraries in a single program easily.

(And what about any symbol exported by ordinal?)

That doesn't happen for msvcrt, as the import library links by name.

Regards,
Martin

[Gabriel Genellina]

At Sunday 21/1/2007 05:38, Martin v. Löwis wrote:

>Gabriel Genellina schrieb:

This would only work, if runtime dll's were compatibles between them,
and they are not. You can't blindly redirect a call to msvcr71.__xyz to
msvcr80.__xyz and expect that to work magically - it may have a
different number of arguments, or different types, or even may not exist
anymore.

Actually, the libraries *are* binary-compatible (on the ABI level). You
just can't mix two libraries in a single program easily.

That's a good thing - but is this just by accident, or is documentedsomew here?
I remember that I tried something like that in
the past, and failed. (Perhaps earlier versions
where not fully backwards compatible - or I didn't try hard enough thattime).
--
Gabriel Genellina
Softlab SRL


_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[sturlamolden]

Gabriel Genellina wrote:

This would only work, if runtime dll's were compatibles between them,
and they are not.

It is standard C, defined by ANSI and ISO.

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Gabriel Genellina schrieb:

That's a good thing - but is this just by accident, or is documented
somewhere?

It's documented somewhere (although I can't find the documentation
right now - it explains how you can link object files from a static
library compiled with an older compiler version against a new version
of the C library).

I remember that I tried something like that in the past, and failed.
(Perhaps earlier versions where not fully backwards compatible - or I
didn't try hard enough that time).

It's been that way for ages, atleast since they started to support
32-bit code. So if you had problems, they might have had a different
source.

It's a different thing for the C++ libraries, though.

Regards,
Martin

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

sturlamolden schrieb:

>This would only work, if runtime dll's were compatibles between them,
and they are not.

It is standard C, defined by ANSI and ISO.

ANSI and ISO don't define the ABI, though. For example, the definition
of the FILE type might (and does) vary across compilers, even if all
these compilers implement C99.

Regards,
Martin