New versions breaking extensions, etc.

[Jive]

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing? Why won't extensions compiled
to run with 2.3 also work with 2.4? Why does it matter whether a component
was compiled with VC++ 6.0 or 7.1? I've been using MS stuff for 6 years.
Before I started using Python, I don't remember ever having a program tell
me I had to use an OLDER version of something to make the program work.
Newer, maybe, but older, never. I had to revert to Python 2.3 because I
have applications that will not run under Python 2.4.

Although I have been a full-time software developer since the late 70's, I
don't know much about Microsoft stuff. (Amazing trick that, eh?) But I've
been reading some things on the MS web pages that say dot-NET and all that
should NOT break existing stuff.
http://msdn.microsoft.com/library/en..._libraries.asp I
don't pretend to understand it all, but certainly one would not expect MS to
screw things up too badly on purpose. The value of MS Windows lies in the
applications that it can run. That's part of the genius of the Evil Genius.
He understood when others didn't that when Joe Blow in his basement develops
a program to run under MS Windows, Joe Blow is making money for the E.G.,
and it doesn't cost the E.G. a penny.

In my own small domain, I cannot even consider coming out with new releases
that are not drop-in replacements for what went before. Customers won't go
for it. It's an absolute deal-breaker.

What puzzles me is that, apparently, the incompatibility of Python 2.4 and
extensions built for 2.3 did not come as a nasty surprise. I get the
impression that it was expected. Can someone explain that to me?

I hope this doesn't sound like I'm complaining. I think Python is great.
And the price can't be beat!

[Cameron Laird]

In article <11************ **********@f14g 2000cwb.googleg roups.com>,
John Machin <sj******@lexic on.net> wrote:

Jive wrote:

"Martin v. Löwis" <ma****@v.loewi s.de> wrote in message
news:41******** ******@v.loewis .de...

> OTOH, people who only have VC6 just need to buy VS.NET 2003,
> which is still available.

I don't even know how to do that! :-) What's the difference between

VC++

.net Standard and Visual Studio .net Pro? (Besides $370?) Is the

former

C++ only, but with the IDE, and the later the whole shebang with

SourceSafe,

VBASIC, and all that?

OH NO! I've gone seriously off-topic. Please don't call the Spanish
Inquisiton. Allow me to re-phrase the question: What do I need to

build

(on-topic) Python extensions?

Short answer to Jive's question: (1) free non-MS C compiler (either
MinGW or Borland) (2) inner calm.

I really can't understand what all the screaming and yelling is about.
Windows Python is built using an MS compiler. Those extension
developers who can't/won't buy the MS compiler use either the free
MinGW compiler or the free Borland 5.5 compiler (or both!). Yes, you

[Cameron Laird]

In article <ma************ *************** ***********@pyt hon.org>,
Nick Coghlan <nc******@email .com> wrote:

Jive wrote:

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing?

The problem is the Python C/API. At the moment, it exposes things
directly (like
data structures) that may change size between major version releases. The other
issue is that the interpreter and the extensions may be linked to different
versions of the Microsoft runtime.

This is a solvable problem, but it would require:
1. Major additions to the C/API - a "Python Major Version Agnostic" API that
hides all data structures behind opaque pointers, never passes file descriptors
between the interpreter and the extension and never gets Python to
delete memory
allocated by the extension (or vice-versa)
2. Rewriting extensions to use the new API functions instead of the current
major version specific ones.

Such an API is likely to be both harder to work with and slower, simply due to
the limitations of C.

To this date, no-one has cared enough about the problem to put in the effort
required to make the C API version agnostic. Given that the API almost always

[Cameron Laird]

In article <Ru************ *********@news. easynews.com>,
Jive <so*****@micros oft.com> wrote:

[Jive]

"Cameron Laird" <cl****@lairds. us> wrote in message
news:32******** ****@lairds.us. ..

Tcl went through this epiphany a few years, and has largely,
though not exclusively, committed to use of the once-more-
redirected API called Stubs <URL: http://wiki.tcl.tk/stubs >.
Some Tcl-ers are orgasmic about the wonders of Stubs, some
ignore it, a few don't like it, and most, perhaps, aren't
even aware of its existence. Performance doesn't appear to
be much of an issue, though.

It uses function pointer tables (like COM), but in the "other direction"
apparently.

Bit by bit, I am remembering now just how stupifyingly brain-dead the MS DLL
scheme is.
Back in '87 or so, I wrote a better dynamic linker (for a Unix) in a couple
of days.

[Cameron Laird]

In article <1e************ **********@news .easynews.com>,
Jive <so*****@micros oft.com> wrote:

[Jive]

"Martin v. Löwis" <ma****@v.loewi s.de> wrote in message
news:41******** **************@ news.freenet.de ...

It's not hard-coded in the linker, but hard-coded in the import library.
So if you link with msvcrt.lib (which might not be the precise name
of the import library - I cannot look up the precise name right now),
msvcrt.lib will in turn refer to msvcr71.dll.

But it makes no difference, no? The problem is that both Python.exe and the
extensions are *compiled* to link with a *particular* crt. (How "dyanamic"
is that?) We could probably kluge around the problem if it were not for the
fact that one crt might perversely define some struct, (FILE, for example),
with difference layouts in different crt DLL's. Right? The header files
contain the poison.

So it would seem that as far as the crt goes, we are at the mercy of the
micro soft ones. They could introduce an incompatible crt at any time.

By the way, I've googled around and found others outside the Python
community who have run into the same problem. In some cases, they have
opted just to stay with VC 6.0. One said, "At least the 6.0 compiler is 2.5
times as fast." Groan.

[Robin Becker]

Cameron Laird wrote:
......

.
Part of the trick is that it demands deep understanding
to detect the antisynergies that arise from the interac-
tions of the DLL, registry, and filesystem schemes. I
know it was only this year that I realized the whole
installation-requires-reboot absurdity is a consequence
of DLL (mis-)design.

That's judgmental of me. What I'm saying is *I* wouldn't
do an OS that way. It's certain, though, that I'll never
extract as much consumer surplus as Microsoft has, so my
evaluations should count for little.

what seems strange is that although the OS hasn't changed we need a
whole new framework just because the compiler underwent a version change.

That would not happen in a *nix style OS because the framework is the
OS. In the M$ world the $ value of causing reams of people to get the
new compiler outways any sense of what would be a decent approach to OS
design etc.

It's sad that people who are otherwise sensible about opensource seem to
be a bit silly about the poisoned apples. There was no rational reason
for me to upgrade to VC 7.x, but now I'm forced to by my preferred language.
--
Robin Becker

[Martin v. Löwis]

Jive wrote:

But it makes no difference, no? The problem is that both Python.exe and the
extensions are *compiled* to link with a *particular* crt.
No, that is not (really) the case. They are compiled to link with
msvcrt.lib, which could, at link time, then become msvcrt.dll,
msvcrt40.dll, or msvcr71.dll.
(How "dyanamic" is that?)
The actual code of the DLL is not provided until run-time - only
at run-time, the precise implementation of the CRT routines is
available - and it indeed may vary from system to system, as different
versions of msvcrt40.dll might be installed (or of python24.dll, for
that matter).
We could probably kluge around the problem if it were not for the
fact that one crt might perversely define some struct, (FILE, for example),
with difference layouts in different crt DLL's. Right?
Wrong. struct FILE binds us the the MS CRT (so the Borland or glibc CRTs
are incompatible), but struct FILE has stayed the same for more than 10
years now, so you can, in principle, mix object files compiled with
different compilers. One would need to check the header files to
determine they are really compatible, but that appears to be the intent,
and Microsoft does not dare to change it (except perhaps for the _ctype
thing, which I don't fully understand).

What really hurts is that the CRT has a number of global variables: the
global FILE objects (FILE 0 to FILE 63, I believe), the global heap,
and the global atexit handlers. If you have multiple CRTs, you get
multiple copies of these globals, and the different routines in the
different CRTs each operate on their own copy of the global variable.
So it would seem that as far as the crt goes, we are at the mercy of the
micro soft ones. They could introduce an incompatible crt at any time.
They could, but they won't. I believe they have long given up changing
or improving the CRT, since that would make things worse in most cases.
The only things they still do is to add new features; the existing ABI
apparently is never touched (again, perhaps except for a _ctype change
I don't understand).
By the way, I've googled around and found others outside the Python
community who have run into the same problem. In some cases, they have
opted just to stay with VC 6.0. One said, "At least the 6.0 compiler is 2.5
times as fast." Groan.

This is what Python did for 2.3. Already at that time, user pressure
was quite high to move to a more recent compiler; with 2.4, it was
clear that we do need some of the new features (in particular,
ongoing commercial availability, and IPv6 support in winsock.h).

Regards,
Martin

[Martin v. Löwis]

Cameron Laird wrote:

Part of the trick is that it demands deep understanding
to detect the antisynergies that arise from the interac-
tions of the DLL, registry, and filesystem schemes. I
know it was only this year that I realized the whole
installation-requires-reboot absurdity is a consequence
of DLL (mis-)design.

Can you elaborate? To me, that problem only originates from
the OS lack of support for deleting open files. If you could
delete a shared libary that is still in use (as you can on
Unix), the put the new version of the DLL in the place, it
would all work fine: new processes would use the new DLL.
Existing processes continue to use the old DLL, which would
be only deleted if the last process using it terminates.
It might be sensible to restart a few long-running processes
so that they also use the new DLL, but apart from that,
installation does need to imply reboot even on a system
that uses DLLs.

Regards,
Martin

[Jive]

"Robin Becker" <ro***@SPAMREMO VEjessikat.fsne t.co.uk> wrote in message
news:41******** ******@jessikat .fsnet.co.uk...

Cameron Laird wrote:
There was no rational reason
for me to upgrade to VC 7.x, but now I'm forced to by my preferred language. --
Robin Becker

That's the way I feel about it too. Actually, I'm getting pushed toward
Microsoft from other directions, also for no good reason -- and for a whole
lot of computers, not just for the one on my desk.

Open letter to Bill Gates:

Dear Bill,

Okay. You win again. It was silly of me to think Python would help me
escape your clutches. Look at how good I am being now. I am typing this
into an Outlook Express window, using a Microsoft operating system that you
will no doubt find a way to make me replace very soon. You beat me fair and
square. Well, square, anyway. Congratulations . I give up.

Jive