Python 2.4.2 using msvcrt71.dll on Win and compatibility issues

[Christoph Zwerschke]

The latest Windows distribution of Python 2.4.2 is using the
msvcrt71.dll, while PostgreSQL is using msvcrt.dll.

This lead to the following problem:

I was using PyGreSQL to connect to the PostgreSQL database. There is a
function that prints a query object q simply by writing: "print q". What
happens on the lower levels is that PyObject_Print(q, stdout) is called,
where stdout is the stdout file descriptor of msvcrt71.dll. Then,
PyGreSQL calls the PostgreSQL function PQprint, handing it over that
stdout file descriptor. PQprint in turn calls fprintf of msvcrt.dll,
with the same file descriptor of msvcrt71.dll. This leads to a Windows
exception since the file descriptors of mscvrt.dll and msvcrt71.dll are
obviously not compatible.

I can imagine that other C extensions which are using dlls based on
msvcrt can run into the same problem.

Is there a general strategy to avoid this kind of problems?

-- Christoph

[Martin v. Löwis]

Christoph Zwerschke wrote:

Is there a general strategy to avoid this kind of problems?

In general, the only Microsoft-supported strategy is that you
must use only a single msvcrt in the entire application. So
either recompile PostGres, or recompile Python.

In the specific case, having PQprint print to a string (rather
than to a file) might work. OTOH, it appears there is no "print
to string" code in postgres.

So here is another strategy: flush the stream before going
into postgres, then obtain the fileno() of the stream, and
fdopen it with postgres' iostreams library. That might also
be tricky to implement, but could work.

Regards,
Martin

[Christoph Zwerschke]

Martin v. Löwis wrote:

In general, the only Microsoft-supported strategy is that you
must use only a single msvcrt in the entire application. So
either recompile PostGres, or recompile Python.
Hm, that's really inconvenient (even more so under Windows).
In the specific case, having PQprint print to a string (rather
than to a file) might work. OTOH, it appears there is no "print
to string" code in postgres.
Yes, there is only PQprint which prints to a stream.
So here is another strategy: flush the stream before going
into postgres, then obtain the fileno() of the stream, and
fdopen it with postgres' iostreams library. That might also
be tricky to implement, but could work.

I understand what you mean. But the Postgres dll provides no means to
fdopen a new stream.

-- Christoph

[Martin v. Löwis]

Christoph Zwerschke wrote:

So here is another strategy: flush the stream before going
into postgres, then obtain the fileno() of the stream, and
fdopen it with postgres' iostreams library. That might also
be tricky to implement, but could work.

I understand what you mean. But the Postgres dll provides no means to
fdopen a new stream.

Right. So you must do so in the pygresql module. That means you have
to get hold of msvcrt.dll's (sic) fdopen implementation.

One way to do so would be through linking pygresql to msvcrt.dll,
instead of linking it to msvcr71.dll. That should be only done if
it is certain that there won't be any other resource sharing with
Python (no malloc, no locale, no stdio - see "some" mskb article
for the precise list of restrictions). I would expect that no
such resource sharing happens, but one would have to verify the code
to be certain.

The other option is to use GetProcAddress to obtain the address of
fdopen.

Regards,
Martin

[Christoph Zwerschke]

Martin v. Löwis wrote:

Christoph Zwerschke wrote:

I understand what you mean. But the Postgres dll provides no means to
fdopen a new stream.
Right. So you must do so in the pygresql module. That means you have
to get hold of msvcrt.dll's (sic) fdopen implementation.

One way to do so would be through linking pygresql to msvcrt.dll,
instead of linking it to msvcr71.dll.

I think this would only shift the problem. Because then I would have to
convert the msvcr71 stream I get from Python to a msvcrt stream. Using
fileno() (of msvcrt) to get the file descriptor will probably not work.
The other option is to use GetProcAddress to obtain the address of
fdopen.

That would be an option. However, I would need to know the handle of the
PostgreSQL dll module. Is there an easy way to get it (without having to
hard-code the explicit name of the dll)? In any way, it seems I have to
insert a lot of platform-dependent, ugly code.

In my case, it is probably easier to simply mimic the Postgres PQprint()
function provided by the dll in PyGreSQL. It is not such a complicated
thing. Plus it is only called with fixed parameters, so I don't need the
complete functionality. PQprint() is considered obsolete anyway and may
vanish one day.

But thanks a lot for your help. Sometimes you don't know whether you're
doing something terribly wrong or missing a very easy solution if you
don't discuss with others.

-- Christoph

[Martin v. Löwis]

Christoph Zwerschke wrote:

I think this would only shift the problem. Because then I would have to
convert the msvcr71 stream I get from Python to a msvcrt stream. Using
fileno() (of msvcrt) to get the file descriptor will probably not work.
It actually would:

#define _fileno(_stream) ((_stream)->_file)

This definition is the same in all CRT version, plus the _file member
is at the same offset in all CRT versions. Microsoft apparently wants
to support linkage of object files build with one CRT version against
a different CRT version.

It *is* hacky, of course.
But thanks a lot for your help. Sometimes you don't know whether you're
doing something terribly wrong or missing a very easy solution if you
don't discuss with others.

Right. I come more and more to the conclusion that you shouldn't really
be using the CRT on Win32.

Anyway, I just proposed to have Python 2.5 link against msvcrt.dll
on python-dev, and got mixed responses.

Regards,
Martin

[Ross Ridge]

Martin v. Löwis wrote:

In general, the only Microsoft-supported strategy is that you
must use only a single msvcrt in the entire application. So
either recompile PostGres, or recompile Python.

If you want a compiled version of Python that already uses
MSVCRT then you try using pyMingGW:

http://jove.prohosting.com/iwave/ipython/pyMinGW.html

[Christoph Zwerschke]

Martin v. Löwis wrote:

Christoph Zwerschke wrote:

I think this would only shift the problem. Because then I would have to
convert the msvcr71 stream I get from Python to a msvcrt stream. Using
fileno() (of msvcrt) to get the file descriptor will probably not work.

It actually would:

#define _fileno(_stream) ((_stream)->_file)

This definition is the same in all CRT version, plus the _file member
is at the same offset in all CRT versions. Microsoft apparently wants
to support linkage of object files build with one CRT version against
a different CRT version.

It *is* hacky, of course.

I see. So this approach could actually work. Thanks for the hint.

-- Christoph