New versions breaking extensions, etc.

[Jive]

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing? Why won't extensions compiled
to run with 2.3 also work with 2.4? Why does it matter whether a component
was compiled with VC++ 6.0 or 7.1? I've been using MS stuff for 6 years.
Before I started using Python, I don't remember ever having a program tell
me I had to use an OLDER version of something to make the program work.
Newer, maybe, but older, never. I had to revert to Python 2.3 because I
have applications that will not run under Python 2.4.

Although I have been a full-time software developer since the late 70's, I
don't know much about Microsoft stuff. (Amazing trick that, eh?) But I've
been reading some things on the MS web pages that say dot-NET and all that
should NOT break existing stuff.
http://msdn.microsoft.com/library/en..._libraries.asp I
don't pretend to understand it all, but certainly one would not expect MS to
screw things up too badly on purpose. The value of MS Windows lies in the
applications that it can run. That's part of the genius of the Evil Genius.
He understood when others didn't that when Joe Blow in his basement develops
a program to run under MS Windows, Joe Blow is making money for the E.G.,
and it doesn't cost the E.G. a penny.

In my own small domain, I cannot even consider coming out with new releases
that are not drop-in replacements for what went before. Customers won't go
for it. It's an absolute deal-breaker.

What puzzles me is that, apparently, the incompatibility of Python 2.4 and
extensions built for 2.3 did not come as a nasty surprise. I get the
impression that it was expected. Can someone explain that to me?

I hope this doesn't sound like I'm complaining. I think Python is great.
And the price can't be beat!

[Robert]

VS7 is a really a vastly different beastie than VS6.
On 12/10/04 9:31 PM, in article YZ************* *******@news.ea synews.com,
"Jive" <so*****@micros oft.com> wrote:

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing? Why won't extensions compiled
to run with 2.3 also work with 2.4? Why does it matter whether a component
was compiled with VC++ 6.0 or 7.1? I've been using MS stuff for 6 years.
Before I started using Python, I don't remember ever having a program tell
me I had to use an OLDER version of something to make the program work.
Newer, maybe, but older, never. I had to revert to Python 2.3 because I
have applications that will not run under Python 2.4.

Although I have been a full-time software developer since the late 70's, I
don't know much about Microsoft stuff. (Amazing trick that, eh?) But I've
been reading some things on the MS web pages that say dot-NET and all that
should NOT break existing stuff.
http://msdn.microsoft.com/library/en....time_librarie
s.asp I
don't pretend to understand it all, but certainly one would not expect MS to
screw things up too badly on purpose. The value of MS Windows lies in the
applications that it can run. That's part of the genius of the Evil Genius.
He understood when others didn't that when Joe Blow in his basement develops
a program to run under MS Windows, Joe Blow is making money for the E.G.,
and it doesn't cost the E.G. a penny.

In my own small domain, I cannot even consider coming out with new releases
that are not drop-in replacements for what went before. Customers won't go
for it. It's an absolute deal-breaker.

What puzzles me is that, apparently, the incompatibility of Python 2.4 and
extensions built for 2.3 did not come as a nasty surprise. I get the
impression that it was expected. Can someone explain that to me?

I hope this doesn't sound like I'm complaining. I think Python is great.
And the price can't be beat!

[News M Claveau /Hamster-P]

Hi !

But, if Python is as much sensitive to the passage of an external software,
version 6 (obsolete) with a version 7 (obsolete also), it is worrying.

Michel Claveau

[Nick Coghlan]

Jive wrote:

Can someone explain to me why Python 2.4 on MS Windows has these backward
compatibility problems? What am I missing?

The problem is the Python C/API. At the moment, it exposes things directly (like
data structures) that may change size between major version releases. The other
issue is that the interpreter and the extensions may be linked to different
versions of the Microsoft runtime.

This is a solvable problem, but it would require:
1. Major additions to the C/API - a "Python Major Version Agnostic" API that
hides all data structures behind opaque pointers, never passes file descriptors
between the interpreter and the extension and never gets Python to delete memory
allocated by the extension (or vice-versa)
2. Rewriting extensions to use the new API functions instead of the current
major version specific ones.

Such an API is likely to be both harder to work with and slower, simply due to
the limitations of C.

To this date, no-one has cared enough about the problem to put in the effort
required to make the C API version agnostic. Given that the API almost always
remains *source* compatible, within a month or two of a new Python release, the
extension developers have generally tweaked their build environments to also
produce binaries for the new release.

In the current case (Python 2.4), the transition from msvcrt to msvcrt71, and
the poor quality of the free development tools offered by Microsoft has
exacerbated the problem on Windows. The Python developers are looking at
improving the documentation and support for building extensions with Windows
compilers other than Visual Studio .Net 2003 (mainly MinGW and the free MS tools).

Although key extension developers could always try asking the PSF to buy them a
Visual Studio license ;)

Cheers,
Nick.

--
Nick Coghlan | nc******@email. com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Jive]

"Nick Coghlan" <nc******@iinet .net.au> wrote in message
news:ma******** *************** *************** @python.org...

Jive wrote:

Can someone explain to me why Python 2.4 on MS Windows has these backward compatibility problems? What am I missing?
The problem is the Python C/API. At the moment, it exposes things directly

(like data structures) that may change size between major version releases.
Okay. That could be fixed.
The other
issue is that the interpreter and the extensions may be linked to different versions of the Microsoft runtime.
Doesn't Microsoft have an answer for that? There are (at last count) nine
skillion ActiveX
components in the wild. Surely Microsoft didn't blast them into oblivion
with dot-net -- did it?

This is a solvable problem, but it would require:
1. Major additions to the C/API - a "Python Major Version Agnostic" API that hides all data structures behind opaque pointers, never passes file descriptors between the interpreter and the extension and never gets Python to delete memory allocated by the extension (or vice-versa)
I remember having trouble with that last bit once. But I seem to recall
there was an easy
answer. Something to do with the main ap and the DLL using the same heap or
some
such thing.
2. Rewriting extensions to use the new API functions instead of the current major version specific ones.
That's a one-time thing, for those who choose to do it. Certainly the old
API
would still be available.

Such an API is likely to be both harder to work with and slower, simply due to the limitations of C.

To this date, no-one has cared enough about the problem to put in the effort required to make the C API version agnostic. Given that the API almost always remains *source* compatible, within a month or two of a new Python release, the extension developers have generally tweaked their build environments to also produce binaries for the new release.
It would be a Good Thing to put a stop to that requirement, no?

In the current case (Python 2.4), the transition from msvcrt to msvcrt71, and the poor quality of the free development tools offered by Microsoft has
exacerbated the problem on Windows.
No doubt. I don't understand what the deal is with msvcr71 and all that,
but I have
figured out that it's a dirty rotten shame. If you google for msvcr71, you
will
get pages and pages of links to plaintive cries for help. We are not alone.
The Python developers are looking at
improving the documentation and support for building extensions with Windows compilers other than Visual Studio .Net 2003 (mainly MinGW and the free MS tools).

That would be good. But is using VC++ 6.0 right out? As it stands, I can
no longer
build extensions that will run with a standard dot-net Python 2.4 release.
Or at least I don't
know how. And if I build 2.4 under VC++ 6.0 and distribute that around the
company, it's not
clear to me if we can then use new third party extensions. Some
documentation
clearing it all up would be very helpful. Maybe it exists somewhere
already. This is
a Microsoft problem, not a Python problem.

Like I said, dirty rotten shame.

Thinking it over, I'm wondering why the Python crowd went with dot-NET in
the first place.
Surely the world would be a better, happier place without MS jerking
everyone around.
We work our fingerprints to the bone writing code to make their stinking
little OS do something
useful, and what have they ever given us in return? -- Besides the
aquaducts?

Jive

[Jive]

P.s. Does anyone know how to make Outlook Express leave my damned line-ends
alone? If I want line-ends. I know where to find the ENTER key.

[JanC]

Jive schreef:

P.s. Does anyone know how to make Outlook Express leave my damned
line-ends alone? If I want line-ends. I know where to find the ENTER
key.

Google for "oe-quotefix", but the best solution is to use a proper
newsreader. ;-)
--
JanC

"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9

[Mike Meyer]

"Jive" <so*****@micros oft.com> writes:

"Nick Coghlan" <nc******@iinet .net.au> wrote in message
news:ma******** *************** *************** @python.org...

To this date, no-one has cared enough about the problem to put in the

effort

required to make the C API version agnostic. Given that the API almost

always

remains *source* compatible, within a month or two of a new Python

release, the

extension developers have generally tweaked their build environments to

also

produce binaries for the new release.

It would be a Good Thing to put a stop to that requirement, no?

Actually, there's a problem on Unix that may not exist on
Windows. Python is installed in <PREFIX>/lib/python<version>/. This
lets us have multiple versions of Python installed at the same time,
which is a good thing.

Now, installed packages and extensions go in
<PREFIX>/lib/python<version>/site-packages. This means that you can't
use *any* of the previously installed packages in the new
version, even if they were pure python. Since I use TMDA, my mail
stopped working when I installed python 2.4.

The real solution is a database of installed packages (which may well
be needed for the apt-get/CPAN functionality that people want) so that
a single python script can fetch and install all the installed
packages.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Martin v. Löwis]

Jive wrote:

Why won't extensions compiled to run with 2.3 also work with 2.4?

I believe nobody has answered *this* question, yet:

Python extensions built for 2.3 link with python23.dll, Python
extensions build for 2.4 link with python24.dll.

pythonxy.dll has global variables, e.g. the pointers to True,
False, None, or the dict type object. If you have two copies
of the Python runtime, you get two copies of each local object.
This cannot work: we really need to rely on having only one
dict type, or else "is" comparisons with the dict type fail.

Regards,
Martin

[Martin v. Löwis]

Jive wrote:

The other
issue is that the interpreter and the extensions may be linked to
different versions of the Microsoft runtime.

Doesn't Microsoft have an answer for that?

Microsoft's answer to this question is: don't do that. Never
mix different versions of the CRT in a single application.
There are (at last count) nine skillion ActiveX
components in the wild. Surely Microsoft didn't blast them into oblivion
with dot-net -- did it?
No, it didn't - but so what? The point is that ActiveX controls normally
don't use the CRT in dangerous ways, since they are constrained to use
Co* function calls only.
I remember having trouble with that last bit once. But I seem to recall
there was an easy
answer. Something to do with the main ap and the DLL using the same heap or
some
such thing.
Something like that, for sure.

[require that extensions be recompiled] It would be a Good Thing to put a stop to that requirement, no?
No. The maintenance cost of such a new API would be significantly
higher, and Python would lose some of its attractiveness for C
developers.
No doubt. I don't understand what the deal is with msvcr71 and all that,
You probably either need to learn what the deal is, or else you have to
trust others that there is no better way.
That would be good. But is using VC++ 6.0 right out? As it stands, I can
no longer build extensions that will run with a standard dot-net Python
2.4 release. Or at least I don't know how.
The most easy way to do this is to use VC7.1 (VS.NET 2003). That will
work out of the box (you need to box first, of course).
Thinking it over, I'm wondering why the Python crowd went with dot-NET in
the first place.
We did not go to .NET. Python 2.4 has nothing to do with .NET. It just
happens that the C compiler we use ships as part of a product that has
..NET in its name - the C compiler itself is Microsoft Visual C 7.1,
aka Microsoft C 13.10.

That compiler is now used because it has a number of improvements over
Microsoft C 12.00, and because VC6 is no longer commercially available.
Many developers have only VS .NET 2003 available, so if Python would
continue to be built with VC6, those people could not build
extensions for it - they could not even buy the compiler they needed,
anymore. OTOH, people who only have VC6 just need to buy VS.NET 2003,
which is still available. Or else they find out what alternative
compiler arrangements can be made.
Surely the world would be a better, happier place without MS jerking
everyone around.

It would surely be better if the operating system shipped with a
compiler, so anybody could rebuild things, and you wouldn't need
binary distributions in the first place :-)

Regards,
Martin