bytecode non-backcompatibility

[Maurice LING]

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure). It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

Cheers
Maurice

[Robert Kern]

Maurice LING wrote:

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure).
Then this has nothing to do with bytecode incompatibility. Only
pure-Python modules get compiled to bytecode. You mean binary
compatibility of the C API.

If you're going to have significant improvements in the core
interpreter, you have to break binary compatibility from time to time.
It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

This problem is mitigated somewhat by the fact that the devs keep binary
compatibility within a major revision series (e.g. 2.3.0, 2.3.1, 2.3.2,
....).

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[Fuzzyman]

Maurice LING wrote:

Hi,

I've been using Python for about 2 years now, for my honours project and now my postgrad project. I must say that I am loving it more and more now. From my knowledge, Python bytecodes are not back-compatible. I must say that my technical background isn't strong enough but is there any good reason for not being back-compatible in bytecodes?

It *shouldn't* be a problem for pure python modules. The interpreter
will recognise that the bytecode has the wrong 'magic number' and
recompile.

Seeing as a new major version of python will probably be installed in a
new directory at the very least you will have to copy the modules
across. If they have install files (or a setup.py) wouldn't it be
better to use that *anyway* ?
My problem is not about pure python modules or libraries but the problem is with 3rd party libraries with C bindings (not python pure). It means that with every upgrade of python, I have to reinstall all my 3rd party libraries which can be quite a bit of work...

It is a nuisance. It's more of a nuisance when third part modules with
'c' components are compiled for the new version of python (a windows
specific problem).

I did find that the last upgrade forced me to evaluate which extensions
I actually used. The answer was 'a few less than I thought'. It became
a good opportunity to spring clean my 'site-packages' folder.

Best Regards,

Fuzzy
http://www.voidspace.org.uk/python
I do hope this problem will be sorted out some day.

Cheers
Maurice

[Terry Reedy]

I think the OP is confusing three different CPython implementation
features -- bytecodes, marshal format, and C API, all of which change
pretty slowly -- and a MS Windows/C (dis)feature.

CPython bytecodes only concern code written in Python but are mostly not a
concern because recompilation is automatic when needed. The exception is
code that directly mucks around with bytecodes, especially the intentially
semi-undocumented numerical codes.

Code objects also include marshaled object values. The code object for
'haha = 987654321' must contain representations for the constants 'haha'
and '987654321' as well as the bytecode for the assignment. So even if
bytecodes remain the same, marshal can change (as it did for 2.4, I
believe, and as it will for floats in 2.5 to fix a bug) and trigger auto
recompiles.

C extensions interact with the interpreter via function calls that
constitute the C API. As much as possible, the developers consciously
avoid changes that break old code or even old binaries. Guido has claimed
that for Linux, there are extension binaries compiled for 2.0 that still
work with 2.4.

However, for MS Windows/C, there is a 'feature' with respect to DLLs that
requires recompilation of extensions to work with a new version of the
Python DLL, even if the C API is unchanged (or so people who understand
this have said). So the OPs complaint about having to get and install new
extension binaries for each Python version might well be directed to MS.

This does not, of course, negate the idea that it would be nice if the
update process were somehow make easier. Indeed, the more-batteries
included distributions from ActiveState and Enthought specifically aim at
this.

Terry J. Reedy

[Maurice LING]

> It is a nuisance. It's more of a nuisance when third part modules with

'c' components are compiled for the new version of python (a windows
specific problem).

I did find that the last upgrade forced me to evaluate which extensions
I actually used. The answer was 'a few less than I thought'. It became
a good opportunity to spring clean my 'site-packages' folder.

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?

I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the
trouble if I do a "fink selfupdate" and "fink update-all" and finds that
my Python is replaced by a newer version, say Python 2.4. Then all my
3rd party libraries in /sw/lib/python2.3/site-packages are un-usable...
And I will have to re-download and re-install all the libraries again.

Yes, I do agree that it is a nice time to clean out "site-packages" but
imagine the work of a system admin who has to re-install 50 packages...
I do believe that we can use some help here, if possible... Perhaps
someone can enlighten us on the technicalities of "so" files and/or C
bindings in Python...

Now I understand that Python bytecodes are only dealing with pure python
source codes. However, the same question lies, how can it (set of
bytecodes) be made stable, like Java bytecodes, which are pretty stable?
Perhaps a better question will be, what techniques Java VM designers
use that enables Java class files (and JAR files) to be stable and
usable across versions that is lacking in Python?

Thanks.

Cheers
Maurice

[Dennis Lee Bieber]

On Tue, 26 Apr 2005 14:47:21 +1000, Maurice LING <ma*********@acm.org>
declaimed the following in comp.lang.python:

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?
Linux equivalent of a Windows DLL -- "shared object" file I
think... a binary object file that is dynamically loaded/linked at run
time. The application binding just knows the name/offset into the object
for each function call.
I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the
trouble if I do a "fink selfupdate" and "fink update-all" and finds that
my Python is replaced by a newer version, say Python 2.4. Then all my
3rd party libraries in /sw/lib/python2.3/site-packages are un-usable...
And I will have to re-download and re-install all the libraries again.
If Fink stuffs a Python 2.4 in your /sw/lib/python2.3/ directory
-- blame Fink! (whatever Fink is...) A Python 2.4 install would be in a
..../python2.4/... directory, and your 2.3 should be untouched. The
/search path/ might have been changed to find 2.4 first, and /that/
might be a problem.

-- ================================================== ============ <
wl*****@ix.netcom.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
================================================== ============ <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.netcom.com/> <

[Mike Meyer]

Maurice LING <ma*********@acm.org> writes:

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?
Shared object files. They're executable code that can be linked into a
program at runtime. Like DLL's on Windows.
I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine
the trouble if I do a "fink selfupdate" and "fink update-all" and
finds that my Python is replaced by a newer version, say Python
2.4. Then all my 3rd party libraries in
/sw/lib/python2.3/site-packages are un-usable... And I will have to
re-download and re-install all the libraries again.
Yeah, it's pain. It would be nice if setup.py kept a list of installed
packages that you could extract before updating, so you would know
what needed to be updated.

Last time I updated, I used the systems package handler. I asked it
for the names of all the Python packages that were installed and saved
that to a file. Then I installed the new version of Python. Then I
edited the package list to turn it into a series of package update
commands, and ran that file.
Now I understand that Python bytecodes are only dealing with pure
python source codes. However, the same question lies, how can it (set
of bytecodes) be made stable, like Java bytecodes, which are pretty
stable? Perhaps a better question will be, what techniques Java VM
designers use that enables Java class files (and JAR files) to be
stable and usable across versions that is lacking in Python?

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Maurice LING]

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

My arguments to the developers will be:

1. Python had gone from a purely scripting language to a general purpose
programming language.

2. The current compilation scheme (compiling to bytecode as and when it
is needed) works well for scripting purposes but is less desirable in
commercial settings. Less distribution happens when it is used purely
for scripting purposes, such as system maintenance or tuning.

3. Using Python in commercial settings will usually require distribution
of resulting software and it is may or may not be desirable to
distribute source codes as well. Unless the application is frozen,
distributing source code is a must.

4. One advantage that Java platform has is that it does not require the
release of source files and still promotes platform-independence.

5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.

6. Unstable bytecodes also makes research work into Python, such as,
just-in-time compilation and just-in-time specialization unfavourable as
they may only be applicable to a specific version of Python. There is
much less chance of getting a project grant than if the same project is
applied for Java (stable bytecodes).

What other point are there?

I may be chopped by saying this but by having a stable set of bytecodes,
we may lose a means of optimization. But we may gain more from filling
the need for commerical distribution of applications writing in Python
and ease of upgrading...

At current stage, every time a new version of Python is installed in my
server or system, I have to test and ensure the needed libraries are
there... which may be a horror in corporate settings. Couple that with
messy dependencies of the libraries to be installed. I remembered the
time I was trying to install Eric 3, the dependencies makes me want to
give up... I have to install Qt, then pyQt, then something else, then
Eric3. Imagine you need 10 of those libraries... which may happen... I
am not yet masochistic enough to take pleasures in this...

I also hope some python developers are reading this thread as well......
Call this a desperate plea from some of us......

Cheers
Maurice

[Steve Holden]

Maurice LING wrote:

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

[... arguments about bytecode stability ...]
What other point are there?

I may be chopped by saying this but by having a stable set of bytecodes,
we may lose a means of optimization. But we may gain more from filling
the need for commerical distribution of applications writing in Python
and ease of upgrading...

At current stage, every time a new version of Python is installed in my
server or system, I have to test and ensure the needed libraries are
there... which may be a horror in corporate settings. Couple that with
messy dependencies of the libraries to be installed. I remembered the
time I was trying to install Eric 3, the dependencies makes me want to
give up... I have to install Qt, then pyQt, then something else, then
Eric3. Imagine you need 10 of those libraries... which may happen... I
am not yet masochistic enough to take pleasures in this...

I also hope some python developers are reading this thread as well......
Call this a desperate plea from some of us......

Cheers
Maurice

There's enough attention to backward compatibility that the chance of
code breakage is quite small for pure-Python modules. Given that case,
all we really need for them is a suitable way of moving them forward to
an updated distribution (though, of course, the module authors may have
brought out new versions that use more advanced language features, there
is no real compulsion to migrate to those if you seek stability).

I don't believe that the major problem is the changes to the bytecodes -
extension modules (those written in C and/or C++) don't generally use
the bytecodes anyway. The real problem is the API to the interpreter,
which again the developers retain the right to change between minor (but
not micro) versions.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/

[Mike Meyer]

Maurice LING <ma*********@acm.org> writes:

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

My arguments to the developers will be:

1. Python had gone from a purely scripting language to a general
purpose programming language.

I think that's irrelevant. It just means that there are more
applications around that have to be dealt with when you update.
2. The current compilation scheme (compiling to bytecode as and when
it is needed) works well for scripting purposes but is less desirable
in commercial settings. Less distribution happens when it is used
purely for scripting purposes, such as system maintenance or tuning.
The solution with the current situation depends on what you mean by
"commercial settings".
3. Using Python in commercial settings will usually require
distribution of resulting software and it is may or may not be
desirable to distribute source codes as well. Unless the application
is frozen, distributing source code is a must.
People have recommended that you distribute the Python interpreter
with commercial applications. This makes installing them much easier
for the end user. It also means the byte code and the interpreter will
always match.
4. One advantage that Java platform has is that it does not require
the release of source files and still promotes platform-independence.
Java is only "platform-independent" to the degree that the jvm and
libraries you need for your application are already available. A
quick google finds applications that require jvm versions - so I
suspect that you can't always run Java code built for a new version of
the jvm on older jvms. So you have to make sure the target platform
has a recent enough implementation of the jvm installed. You may have
problems if you try using a different groups jvm as well. I haven't
looked into that in a number of years.

So you can't just ship java bytecodes to someone expecting they'll be
able to run it out of the box. They may well need to update their java
environment, or install a second one (I recall one time having four
Java products that took in total three different jvms to run. Bleah.)

This isn't really different from Python. In both cases, the end user
has to have a suitable version of the bytecode interpreter and
libraries installed. Java is a little better in that they provide
backwards compatability.
5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.
Unstable bytecodes have nothing to do with these problems.

The current CPython installation process puts the python command and
the libraries for different versions in different directories. This
allows you to have multiple versions installed so you can keep old
scripts working with the old version until you've had time to test
them. It also makes life much easier on the developers, as they can
have a development version installed on the machine at the same time
as they have a production version without breaking the old scripts. It
also means you have to reinstall all your modules on the new
installation - which is what makes the update process tedious for me.

Now, this could be mitigated by having Python libraries installed in
the same location for all versions. You could fix all the bytecode
files by running compileall.py as a script. Sometimes, a module won't
work properly on a new version of Python, and will have to be
updated. You'll have to find those by trial and error and fix them as
you find them. You'll also have to recompile all the C modules, which
will break running scripts on the old interpreter.

Under the currewnt system, old scripts that are run by the new
interpreter will break if all the modules they need aren't installed
yet. Once they're installed, the scripts should work.

If you have scripts that use C modules that aren't installed in the
standard Python search path, they may well break on the new
interpreter until you recompile them. Having both versions of the
interpreter available makes this tolerable.

The cure I proposed seems worse than disease. If you've got a better
solution, I'm sure we'd be interested in hearing it.
6. Unstable bytecodes also makes research work into Python, such as,
just-in-time compilation and just-in-time specialization unfavourable
as they may only be applicable to a specific version of Python. There
is much less chance of getting a project grant than if the same
project is applied for Java (stable bytecodes).
This doesn't seem to have stopped the Psyco project from turning out a
jit compiler. I can't speak to grant applications, though. If you're
really interested, you could apply to the PSF for a grant.
I may be chopped by saying this but by having a stable set of
bytecodes, we may lose a means of optimization. But we may gain more
from filling the need for commerical distribution of applications
writing in Python and ease of upgrading...
The commercial distribution needs can be met without going to stable
bytecodes. The ease of upgrading needs aren't met by going to stable
bytecodes.
At current stage, every time a new version of Python is installed in
my server or system, I have to test and ensure the needed libraries
are there... which may be a horror in corporate settings. Couple that
with messy dependencies of the libraries to be installed. I remembered
the time I was trying to install Eric 3, the dependencies makes me
want to give up... I have to install Qt, then pyQt, then something
else, then Eric3. Imagine you need 10 of those libraries... which may
happen... I am not yet masochistic enough to take pleasures in this...

Fixing the byte-code problem won't help with the dependency
problem. If anything, it'll make it worse, because you'll have old
libraries that were installed with previous versions of Python to
contend with. If those are acceptable, that's all well and good. But
for complex packages like eric - which depends on sip and qt and a
number of other things - the latest versions tend to rely on having
up-to-date libraries. So you install eric, and watch it fail because
some library is out of date. Update that library and repeat. The
easiest way to deal with this is to find all the libraries it needs,
and just update them all.

The dependencies problem is actually pretty easy to solve. In fact, in
many Python environments, it's already solved. On my system, if I
install Eric3 using the provided installation package, the
dependencies will be picked up automatically. Not very helpfull if
you're not on such a system, I know. The long-term solution is for
PyPI to grow to include this functionality.

Thank you,
<mike

--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Maurice LING]

>

2. The current compilation scheme (compiling to bytecode as and when
it is needed) works well for scripting purposes but is less desirable
in commercial settings. Less distribution happens when it is used
purely for scripting purposes, such as system maintenance or tuning.

The solution with the current situation depends on what you mean by
"commercial settings".

Commerically, I will mean distributing the program without the codes.
The best way under the current method is to freeze the application. In
Java, you can just distribute the JAR files or CLASS files. I can
imagine there will be some level of touchiness when source codes are
distributed...

3. Using Python in commercial settings will usually require
distribution of resulting software and it is may or may not be
desirable to distribute source codes as well. Unless the application
is frozen, distributing source code is a must.

People have recommended that you distribute the Python interpreter
with commercial applications. This makes installing them much easier
for the end user. It also means the byte code and the interpreter will
always match.

4. One advantage that Java platform has is that it does not require
the release of source files and still promotes platform-independence.

Java is only "platform-independent" to the degree that the jvm and
libraries you need for your application are already available. A
quick google finds applications that require jvm versions - so I
suspect that you can't always run Java code built for a new version of
the jvm on older jvms. So you have to make sure the target platform
has a recent enough implementation of the jvm installed. You may have
problems if you try using a different groups jvm as well. I haven't
looked into that in a number of years.

So you can't just ship java bytecodes to someone expecting they'll be
able to run it out of the box. They may well need to update their java
environment, or install a second one (I recall one time having four
Java products that took in total three different jvms to run. Bleah.)

This isn't really different from Python. In both cases, the end user
has to have a suitable version of the bytecode interpreter and
libraries installed. Java is a little better in that they provide
backwards compatability.

I think backward compatibility is good enough. A Java JAR file compiled
under for JVM 1.2 can still run in JVM 1.4. I don't think anyone will or
can expect forward compatibility, that will require godly powers...

According to Steve Holden's reply, Python C API is only backward
compatible in minor versions but not major versions. The issue is, why
not major versions as well unless there's something exceedingly prohibitive?

5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.

Unstable bytecodes have nothing to do with these problems.

The current CPython installation process puts the python command and
the libraries for different versions in different directories. This
allows you to have multiple versions installed so you can keep old
scripts working with the old version until you've had time to test
them. It also makes life much easier on the developers, as they can
have a development version installed on the machine at the same time
as they have a production version without breaking the old scripts. It
also means you have to reinstall all your modules on the new
installation - which is what makes the update process tedious for me.

Now, this could be mitigated by having Python libraries installed in
the same location for all versions. You could fix all the bytecode
files by running compileall.py as a script. Sometimes, a module won't
work properly on a new version of Python, and will have to be
updated. You'll have to find those by trial and error and fix them as
you find them. You'll also have to recompile all the C modules, which
will break running scripts on the old interpreter.

Under the currewnt system, old scripts that are run by the new
interpreter will break if all the modules they need aren't installed
yet. Once they're installed, the scripts should work.

If you have scripts that use C modules that aren't installed in the
standard Python search path, they may well break on the new
interpreter until you recompile them. Having both versions of the
interpreter available makes this tolerable.

The cure I proposed seems worse than disease. If you've got a better
solution, I'm sure we'd be interested in hearing it.

Perhaps this is another call for Python version of CPAN (CPyAN or PYAN).
It can be modelled after Fink or Darwinports.

If I remembered correctly, Fink uses apt-get and curl.

What can be done in PYAN is to encourage all 3rd party library
developers to centralize their libraries in it, which I think all will
gladly respond. All that is needed to be deposited in PYAN is a
description file, like the simplest setup.py file. All that is needed in
this description file is
1. where to download the source codes (e.g. CVS)?
2. version number?
3. standard stuffs (optional) like authors, descriptions, copyright?

Python can then have a built-in mechanism to read the description file
and download the source codes and do the standard "sudo python setup.py
install" to install the library into site-package. The main thing this
mechamisn does is to maintain what had been installed in site-package
through it and what versions. So when newer versions of Python is
installed, there can be a script to take the old site-package log,
download the same set of libraries and install them for the new version
of Python.

Darwinports uses a very simple means to achieve this. All the
description files are in a CVS repository. The actual source codes for
the libraries may or may not reside in the CVS repository. Darwinports
system installation is little more than checking out the entire
repository of description files. When user wants to install a library,
it then downloads the library's source codes and performs the standard
"configure, make, make install" operations. Of course, the root user
will have to do a update to update to the latest version of description
files.

I think something like this can be set up for Python quite easily. I
recall some time back, there's a long discussion about setting up
Python's version of CPAN but I can't recall the contents of the
discussions.

Fixing the byte-code problem won't help with the dependency
problem. If anything, it'll make it worse, because you'll have old
libraries that were installed with previous versions of Python to
contend with. If those are acceptable, that's all well and good. But
for complex packages like eric - which depends on sip and qt and a
number of other things - the latest versions tend to rely on having
up-to-date libraries. So you install eric, and watch it fail because
some library is out of date. Update that library and repeat. The
easiest way to deal with this is to find all the libraries it needs,
and just update them all.

The dependencies problem is actually pretty easy to solve. In fact, in
many Python environments, it's already solved. On my system, if I
install Eric3 using the provided installation package, the
dependencies will be picked up automatically. Not very helpfull if
you're not on such a system, I know. The long-term solution is for
PyPI to grow to include this functionality.

I must say that I do not quite understand your system. Please enlighten me.

Thanks and cheers
Maurice

[Terry Reedy]

"Maurice LING" <ma*********@acm.org> wrote in message
news:d4**********@domitilla.aioe.org...

Now I understand that Python bytecodes are only dealing with pure python
source codes.

Then stop blaming (machine-independent) CPython 'bytecodes' for any
problems you have with compiled-to-machine-language C extensions, which
have nothing to do with bytecodes. Bytecodes also have nothing directly to
do with source-code compatibility.

[Terry Reedy]

"Maurice LING" <ma*********@acm.org> wrote in message
news:d4**********@domitilla.aioe.org...

I think backward compatibility is good enough. A Java JAR file compiled
under for JVM 1.2 can still run in JVM 1.4. I don't think anyone will or
can expect forward compatibility, that will require godly powers...

One difference between Java and Python is this: Java bytecodes are, as I
understand it, part of the Java language definition. CPython bytecodes are
intentionally not part of the language at all. Except maybe fore PyPy,
other implementations do not use them. Jython translates Python source to
Java bytecodes. Pyrex translates augmented Python source to C, to be
compiled to native machine code. Ironman translates, I presume, to .NET
common language. PyParrot (don't know if it has an official name)
translates to Parrot bytecodes. Viper translated to OCamel.

If you want an implementation with frozen bytecodes, you are free to make
one ;-)

Terry J. Reedy

[Maurice LING]

Terry Reedy wrote:

"Maurice LING" <ma*********@acm.org> wrote in message
news:d4**********@domitilla.aioe.org...

Now I understand that Python bytecodes are only dealing with pure python
source codes.

Then stop blaming (machine-independent) CPython 'bytecodes' for any
problems you have with compiled-to-machine-language C extensions, which
have nothing to do with bytecodes. Bytecodes also have nothing directly to
do with source-code compatibility.

technicalities are wrong but situation remains unchanged.

maurice

[Maurice LING]

One difference between Java and Python is this: Java bytecodes are, as I
understand it, part of the Java language definition. CPython bytecodes are
intentionally not part of the language at all. Except maybe fore PyPy,
other implementations do not use them. Jython translates Python source to
Java bytecodes. Pyrex translates augmented Python source to C, to be
compiled to native machine code. Ironman translates, I presume, to .NET
common language. PyParrot (don't know if it has an official name)
translates to Parrot bytecodes. Viper translated to OCamel.

If you want an implementation with frozen bytecodes, you are free to make
one ;-)

Terry J. Reedy

So there are currently 7 implementations or variations of the Python
language and you are suggesting I make another one? If I am to do that,
I will use CPython 2.4.1 and call it a implementation that maintains the
current set of bytecodes (newer versions can have more bytecodes and
deprecated bytecodes are still left there for backward compatibility)
and C API interface. Till now I still do not know what is so exceedingly
prohibitive to do that? Why should I create such a direct fork?
Personally I do not have the resources to maintain this fork.

The Pacman (binary executable) that I played on a 286 a decade ago still
works well on my brother's pentium III system, so from the point of a
user, it is backward compatible. The same can't be said for .pyc files
or .so files (I may be wrong with .so files here).

What I do have resources (time and energy) for is to work with the
maintainers of PyPI to implement the package maintenance system I've
described......

maurice

[Terry Reedy]

"Maurice LING" <ma*********@acm.org> wrote in message
news:d4**********@domitilla.aioe.org...

So there are currently 7 implementations or variations of the Python
language and you are suggesting I make another one?
Perhaps you missed the winkie ;-)
The Pacman (binary executable) that I played on a 286 a decade ago still
works well on my brother's pentium III system, so from the point of a
user, it is backward compatible.
How well does it work on a P4 WinXP system? If it does, you're lucky. I
have games that will not run in spite of the compatibility settings being
set. And so I will keep a Win95 machine and a Win98 machine for as long as
they run.
The same can't be said for .pyc files
The same *can* be said for some decade-old .py files. Certainly, most
everything written for 2.0 onward still works. The same will also be true
for .pyc files as long as you run them with their corresponding binary and
as long as the binary stills run.

However, .pyc files are private, temporary caches created by a CPython
interpreter for its own use and tied to its current implementation
technology. They are not intended to be a public distribution form and
indeed cannot be a means to run Python programs on other interpreters.
Guido has stated that he want the freedom to change or even replace parts
of the internal implementation as he sees fit. (2.5 will probably get a new
compiler, with the difference mostly transparent to users.)

One of the principles of OOP is separation of interface from
implementation. User of a class should only use the public interface and
not depend on private implementation. One reason is so implementation can
be changed even radically as long as interface is kept constant. The
Python language is interface. CPython bytecodes are implementation.
What I do have resources (time and energy) for is to work with the
maintainers of PyPI to implement the package maintenance system I've
described......

Good. I agree that we need more along that line.

Terry J. Reedy

[Maurice LING]

> The same *can* be said for some decade-old .py files. Certainly, most

everything written for 2.0 onward still works. The same will also be true
for .pyc files as long as you run them with their corresponding binary and
as long as the binary stills run.

However, .pyc files are private, temporary caches created by a CPython
interpreter for its own use and tied to its current implementation
technology. They are not intended to be a public distribution form and
indeed cannot be a means to run Python programs on other interpreters.
Guido has stated that he want the freedom to change or even replace parts
of the internal implementation as he sees fit. (2.5 will probably get a new
compiler, with the difference mostly transparent to users.)

One of the principles of OOP is separation of interface from
implementation. User of a class should only use the public interface and
not depend on private implementation. One reason is so implementation can
be changed even radically as long as interface is kept constant. The
Python language is interface. CPython bytecodes are implementation.

From a technical perspective, I can accept that .pyc files are private
and temporary. To a certain extend, it does helps development cycle.
Every time I amend my source codes, I just run it without having to
consider or needing to re-compile the source files.

The idea of having to release the program or library as source files
does ring alarms in many executives in corporate world. Less freezing
the modules (which renders it platform dependent) or using Jython (which
is not possible when C extensions are involved), there is no middle
grounds in CPython for distribution without source codes.

Every now and then, there will be new threads in this list about people
asking the means and possibilities of releasing a module/program without
having to release the source code (there's another new thread on it
today, "Can .py be compiled?")...

What I do have resources (time and energy) for is to work with the
maintainers of PyPI to implement the package maintenance system I've
described......

Good. I agree that we need more along that line.

I've posted my idea on catelog-sig mailing list. Hope it get picked up...

Cheers
Maurice

[Mike Meyer]

Maurice LING <ma*********@acm.org> writes:

5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules. Unstable bytecodes have nothing to do with these problems. The
current CPython installation process puts the python command and
the libraries for different versions in different directories. This
allows you to have multiple versions installed so you can keep old
scripts working with the old version until you've had time to test
them. It also makes life much easier on the developers, as they can
have a development version installed on the machine at the same time
as they have a production version without breaking the old scripts. It
also means you have to reinstall all your modules on the new
installation - which is what makes the update process tedious for me.
Now, this could be mitigated by having Python libraries installed in
the same location for all versions. You could fix all the bytecode
files by running compileall.py as a script. Sometimes, a module won't
work properly on a new version of Python, and will have to be
updated. You'll have to find those by trial and error and fix them as
you find them. You'll also have to recompile all the C modules, which
will break running scripts on the old interpreter.
Under the currewnt system, old scripts that are run by the new
interpreter will break if all the modules they need aren't installed
yet. Once they're installed, the scripts should work.
If you have scripts that use C modules that aren't installed in the
standard Python search path, they may well break on the new
interpreter until you recompile them. Having both versions of the
interpreter available makes this tolerable.
The cure I proposed seems worse than disease. If you've got a better
solution, I'm sure we'd be interested in hearing it.

Perhaps this is another call for Python version of CPAN (CPyAN or
PYAN). It can be modelled after Fink or Darwinports.

If I remembered correctly, Fink uses apt-get and curl.

What can be done in PYAN is to encourage all 3rd party library
developers to centralize their libraries in it, which I think all will
gladly respond. All that is needed to be deposited in PYAN is a
description file, like the simplest setup.py file. All that is needed
in this description file is
1. where to download the source codes (e.g. CVS)?
2. version number?
3. standard stuffs (optional) like authors, descriptions, copyright?

To copy all of what CPAN does, you need to know what other modules it
depends on, so you can automatically download those as well.
Python can then have a built-in mechanism to read the description file
and download the source codes and do the standard "sudo python
setup.py install" to install the library into site-package.
I don't like this - it would make Python depend on sudo being
available. I'd rather it not do that, and let each systems
administrator issue the command according to *their* security policy.
I think something like this can be set up for Python quite easily. I
recall some time back, there's a long discussion about setting up
Python's version of CPAN but I can't recall the contents of the
discussions.

It's not clear it's easy. It's *very* clear it won't happen until
someone steps up to do it. The latter is a problem.

The dependencies problem is actually pretty easy to solve. In fact,
in
many Python environments, it's already solved. On my system, if I
install Eric3 using the provided installation package, the
dependencies will be picked up automatically. Not very helpfull if
you're not on such a system, I know. The long-term solution is for
PyPI to grow to include this functionality.

I must say that I do not quite understand your system. Please enlighten me.

FreeBSD has a packaging system (like fink on the Mac) called either
"Ports" (if you build from source) or "Packages" (if you install
pre-compiled binaries). I use Ports, so I'm going to talk about
them. But Packages are built from Ports, and all the information in
the port winds up in the package, so you get the same functionality
with a slightly different command set.

Basically, each port has a list of all the other ports it depends
on. When you install a port, the system will check to see if each of
the dependent ports is installed - usually by checking if a file it
should install exists - and if it doesn't, will install the dependent
port. The dependencies of that port are recursively checked. So if I
were to install the Eric3 port, it would note that I need the
qscintilla, py-sip and py-qt (ports of python modules are all named
"py-<something>"). py-sip and py-qt are already installed, so it would
skip thos. qscintilla isn't, so that would get installed. qscintilla
doesn't have any dependencies, so that would stop there. After
qscintilla is installed, eric3 would be installed.

All of this is recorded in the Packages database, and I can query that
to find out what python modules are installed, which module installed
a specific file, etc. I can also use that to cleanly remove the
module, which the python setup.py doesn't do for me.

I'm not familiar with fink, but I suspect it offers much the same
functionality. rpm's and apt-get offer that functionality for many
Linux systems. Solaris has it's own package system as well.

Basically, for common Unix platforms, the dependency problems you ran
into with Eric3 should be solved by the platforms package system,
assuming that Eric3 is available as a package.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Mike Meyer]

Maurice LING <ma*********@acm.org> writes:

The idea of having to release the program or library as source files
does ring alarms in many executives in corporate world. Less freezing
the modules (which renders it platform dependent) or using Jython
(which is not possible when C extensions are involved), there is no
middle grounds in CPython for distribution without source codes.

You're right - and byte codes *as they exist now* aren't an acceptable
middle ground, either. The problem isn't that the interpreter might
change (that can be defeated by distributing the interpreter with the
bytecode files), but that the byte code can be decompiled to get your
Python source back. See <URL:
http://www.crazy-compilers.com/decompyle/ > for an example.

Selling byte codes to someone who's worried about shipping source is
selling them snake oil. I think it's unprofessional, at best.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Maurice LING]

>>What can be done in PYAN is to encourage all 3rd party library

developers to centralize their libraries in it, which I think all will
gladly respond. All that is needed to be deposited in PYAN is a
description file, like the simplest setup.py file. All that is needed
in this description file is
1. where to download the source codes (e.g. CVS)?
2. version number?
3. standard stuffs (optional) like authors, descriptions, copyright?

To copy all of what CPAN does, you need to know what other modules it
depends on, so you can automatically download those as well.

Yes, dependencies are necessary to be declared. The maintainers of each
package should list the dependencies as well to make it work.

Python can then have a built-in mechanism to read the description file
and download the source codes and do the standard "sudo python
setup.py install" to install the library into site-package.

I don't like this - it would make Python depend on sudo being
available. I'd rather it not do that, and let each systems
administrator issue the command according to *their* security policy.

If you are installing packages into your home directory, then sudo is
not needed. But if you are installing it for everybody's use, then it is
necessary. Fink runs using superuser privileges.

I think something like this can be set up for Python quite easily. I
recall some time back, there's a long discussion about setting up
Python's version of CPAN but I can't recall the contents of the
discussions.

It's not clear it's easy. It's *very* clear it won't happen until
someone steps up to do it. The latter is a problem.

I've posted my method in catelog-sig list. And is willing to offer some
time on it... if that helps... But to get it working, I think it will
need 5-10x more time than I can put into it...

Cheers
maurice

[Maurice LING]

Mike Meyer wrote:

Maurice LING <ma*********@acm.org> writes:

The idea of having to release the program or library as source files
does ring alarms in many executives in corporate world. Less freezing
the modules (which renders it platform dependent) or using Jython
(which is not possible when C extensions are involved), there is no
middle grounds in CPython for distribution without source codes.

You're right - and byte codes *as they exist now* aren't an acceptable
middle ground, either. The problem isn't that the interpreter might
change (that can be defeated by distributing the interpreter with the
bytecode files), but that the byte code can be decompiled to get your
Python source back. See <URL:
http://www.crazy-compilers.com/decompyle/ > for an example.

I know that bytecodes can be decompiled back to python source. Similarly
Java class files can be decompiled back to Java source files quite
easily. IANAL, however, I do think that reverse engineering can be seen
as a purposeful act in the eyes of law. I remembered that EU Council
Directive 9/250/ECC does explicits the allowable scope of decompilation
(for interoperability, preserving expression of ideas, achieving
performance of objectives).

For companies, I think it is touchy to ask permission to release the
source codes even though they may be alright with releasing the same
Java JAR file into the community.

We do know that regardless of whatever methods of compilation there may
be, if some genius is willing to spend sufficient time to crack a
program. Even if he has to read the binary codes, it will still break.
But that is a wilfull act rather than just looking at the source codes
when it is distributed with it. It's just like picking up your money
from the ground if you had dropped it as compared to pickpocketing you
as compared to hold you at knife point and robbing you...... Same end
result but it is viewed differently in the eyes of law and morals.


Selling byte codes to someone who's worried about shipping source is
selling them snake oil. I think it's unprofessional, at best.

<mike

[Martin v. Löwis]

Maurice LING wrote:

Now I understand that Python bytecodes are only dealing with pure python
source codes. However, the same question lies, how can it (set of
bytecodes) be made stable, like Java bytecodes, which are pretty stable?
Perhaps a better question will be, what techniques Java VM designers
use that enables Java class files (and JAR files) to be stable and
usable across versions that is lacking in Python?

The Java class file format is not any more stable than the Python .pyc
format - I think every major release of the JVM has changed details
in the class file format.

The difference is that the newer JVMs accept old class files, by
means of separate loader code for each class file version that was
ever released (and very possibly also support for class file versions
that were never released).

The same approach would be possible in Python, but nobody has
contributed code to do so. It is unlikely that future Python
versions will provide such compatibility with the current byte
code format, either, unless somebody steps forward and volunteers
to maintain that compatibility.

Maintaining this backwards compatibiltiy is a tedious and boring
task, given the many much easier alternatives, so that volunteers are
unlikely to jump in. Sun manages to provide the compatibility by
paying somebody to actually maintain it.

Regards,
Martin

[Martin v. Löwis]

Maurice LING wrote:

technicalities are wrong but situation remains unchanged.

For C modules, it is very likely that new versions of Python
will continue to break the ABI, by changing the layout of
structures.

The most straight-forward way to deal with it as a sysadmin
or user is to install multiple versions of Python on a single
machine. If Fink considers python2.4 as a replacement for
python2.3, then this is a flaw in Fink. In Debian, there is
a python package, which currently depends on python2.3.
Sometime in the future, it will depend on python2.4. Users
which update will then get python2.4, however, python2.3
will remain installed and usable, with all the extension
modules that were installed for it.

Regards,
Martin

[Terry Reedy]

"Maurice LING" <ma*********@acm.org> wrote in message
news:d4**********@domitilla.aioe.org...

From a technical perspective, I can accept that .pyc files are private
and temporary. To a certain extend, it does helps development cycle.
Every time I amend my source codes, I just run it without having to
consider or needing to re-compile the source files. From a user perspective, source code is the run-time program. The idea of having to release the program or library as source files does
ring alarms in many executives in corporate world.

I understand that people use Python while resisting its purpose and design.
But I also understand that it is *their* responsibilty to hide their code,
which may possibly mean not using Python, or which may mean developing
proprietary methods to translate to something designed to *not* be
readable. For all we know, some of the developers have been paid to do
exactly that -- and not talk about it.

Python is *designed* for human readability. That is one of its big
features! The same also seems somewhat true for CPython's bytecodes,
especially when disassembled with the dis module that comes with the
interpreter. You even get all the object names included in the code
object.

Terry J. Reedy

[Maurice LING]

Martin v. Löwis wrote:

Maurice LING wrote:

technicalities are wrong but situation remains unchanged.

For C modules, it is very likely that new versions of Python
will continue to break the ABI, by changing the layout of
structures.

The most straight-forward way to deal with it as a sysadmin
or user is to install multiple versions of Python on a single
machine. If Fink considers python2.4 as a replacement for
python2.3, then this is a flaw in Fink. In Debian, there is
a python package, which currently depends on python2.3.
Sometime in the future, it will depend on python2.4. Users
which update will then get python2.4, however, python2.3
will remain installed and usable, with all the extension
modules that were installed for it.

Regards,
Martin

Fink does not consider python2.4 to be a replacement for python2.3. In
fact, you can install python2.2, 2.3 and 2.4 in the same machine with
Fink. It will maintain 3 sets of libraries as /sw/lib/python2.2,
/sw/lib/python2.3 and /sw/lib/python2.4. The chore is that when say Fink
installs python2.4, all the libraries in /sw/lib/python2.3/site-packages
have to be re-installed into /sw/lib/python2.4/site-packages, one by
one. There is no simple way of doing that... which makes any system
admin needing to re-install 50 3rd party libraries into
/sw/lib/python2.4/site-packages a big task, as well as satisfying the
necessary dependencies.

So if C extension API (or whatever that is really called) is stable, the
system admin can just copy all of /sw/lib/python2.3/site-packages into
/sw/lib/python2.4/site-packages and it should work. From what you've
said, it seems that this isn't possible. So my alternative solution is
that PyPI have a mechanism to maintain what had been installed in the
site-package directory and to download the libraries and install into
the new site-package directory...

What do you think?

Cheers
maurice

[Terry Reedy]

"Maurice LING" <ma*********@acm.org> wrote in message
news:d4**********@domitilla.aioe.org...

So my alternative solution is that PyPI have a mechanism to maintain what
had been installed in the site-package directory and to download the
libraries and install into the new site-package directory... What do you think?

I doubt anyone disputes that upgrades are more hassle than we would like.
My main point was that freezing CPython's technology is not the solution.
Any other upgrade helper that you can contribute will be welcome.

Terry J. Reedy

[Martin v. Löwis]

Maurice LING wrote:

So if C extension API (or whatever that is really called) is stable, the
system admin can just copy all of /sw/lib/python2.3/site-packages into
/sw/lib/python2.4/site-packages and it should work.
It would be counter-productive to make it stable, as this would render
certain desirable changes impossible. Backwards compatibility on C API
(source) level is a goal, binary compatibility within a 2.x release
is a goal, binary compatibility across 2.x releases is not.

[automatically install a set of packages] What do you think?

That would certainly be possible. Contributions are welcome.

Regards,
Martin

[Maurice LING]

Terry Reedy wrote:

I doubt anyone disputes that upgrades are more hassle than we would like.
My main point was that freezing CPython's technology is not the solution.
Any other upgrade helper that you can contribute will be welcome.

Terry J. Reedy

So there is no way of releasing a close-source application written in
Python?

Forget about reverse engineering or decompiling, things like EU Council
Directive, Berne Convention , and legislation of individual countries
are to be used and argued for handling what decompilation is allowed or
not allowed. It does make a difference between looking at the codes
when it is given and performing an act of decompilation to look at the
codes. As mentioned in one of my postings, it may be analogous to the
difference between you dropping some cash on the ground and I take it,
and I performing the act of pickpocketing and take your money.

I am not advocating close source in any sense and I believe the choice
of open or close source distribution is individual's and corporate's rights.

Cheers
Maurice

[Maurice LING]

Martin v. Löwis wrote:

[automatically install a set of packages]

What do you think?

That would certainly be possible. Contributions are welcome.

Regards,
Martin

I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...

Maurice

[Martin v. Löwis]

Maurice LING wrote:

I doubt anyone disputes that upgrades are more hassle than we would
like. My main point was that freezing CPython's technology is not the
solution. Any other upgrade helper that you can contribute will be
welcome.

So there is no way of releasing a close-source application written in
Python?

I fail to see the relationship between the statement and your question.

Again, whether or not the C API is frozen has nothing to do with
whether or not the bytecode format is frozen. So any implication
from not freezing the C API to not being able to ship byte-code
only is invalid.

As for only shipping byte code: sure, that is certainly possible,
and people do that all the time.

Regards,
Martin

[Martin v. Löwis]

Maurice LING wrote:

I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...

Posting questions is not the only way to find answers. The source code
of PyPI is available: sf.net/projects/pypi.

Regards,
Martin

[Maurice LING]

Martin v. Löwis wrote:

Maurice LING wrote:

I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...

Posting questions is not the only way to find answers. The source code
of PyPI is available: sf.net/projects/pypi.

Regards,
Martin

I've browsed the source codes of PyPI in sf.net, nothing pops up as very
useful to me... I reckoned that sitting in here or hoping for replies
from catelog-sig isn't going to help much.

On that, I've prototyped a very simple proof-of-concept of what I have
in mind, regarding a Fink-like tool for Python's 3rd party libraries
management. Please bear in mind that this is a proof-of-concept
prototype, really bare bones. It can be found in 'centipyde' module in
sf.net/projects/ib-dwb. In case the CVS hadn't updated by the time
someone reads this, the directory layout is this:

.../centipyde
.../centipyde/centipyde.py
.../centipyde/pgkinfo
.../centipyde/pgkinfo/ply15.info
ply15.info contains the following text (pretty much modelled against Fink):

package=ply15
maintainer=.
dependencies=.
downloadurl=http://systems.cs.uchicago.edu/ply/ply-1.5.tar.gz
prebuildscript=tar zxvf ply-1.5.tar.gz
sourcedir=ply-1.5
buildscript=python setup.py build
installscript=sudo python setup.py install
centipyde.py is the following:
================================================== ===

"""
Author: Maurice H.T. Ling <ma*********@acm.org>
Copyright (c) 2005 Maurice H.T. Ling
Date created : 28th April 2005
"""

PKGINFOPATH = 'pkginfo'
INSTALL_LOG = 'install.log'

import os
import string
import sys

def install_package(package_name):
f = open(os.getcwd() + os.sep + PKGINFOPATH + os.sep + package_name
+ '.info', 'r')
install_info = {}
for line in f.readlines():
line = string.split(line, '=')
if line[1][-1] == os.linesep:
install_info[line[0]] = string.strip(line[1][:-1])
else: install_info[line[0]] = string.strip(line[1])
f.close()
print "Package Installation Information: " + str(install_info)

os.system('curl -O ' + str(install_info['downloadurl']))

preinstall = []
preinstall = string.split(install_info['prebuildscript'], ';')
for cmd in preinstall: os.system(cmd)

cwd = os.getcwd()
print cwd
os.chdir(os.path.join(os.getcwd(), install_info['sourcedir']))
print os.getcwd()

buildscript = []
buildscript = string.split(install_info['buildscript'], ';')
for cmd in buildscript: os.system(cmd)

installscript = []
installscript = string.split(install_info['installscript'], ';')
for cmd in installscript: os.system(cmd)
if sys.argv[1] == 'install':
install_package(sys.argv[2])

================================================== ===

When I run "python centipyde.py install ply15", PLY1.5 gets downloaded
from David Beazley's website, uncompressed and installed into the
site-package.

All comments and code donations are welcomed.

Cheers
Maurice

[Mike Meyer]

Maurice LING <ma*********@acm.org> writes:

Python can then have a built-in mechanism to read the description file
and download the source codes and do the standard "sudo python
setup.py install" to install the library into site-package.

I don't like this - it would make Python depend on sudo being
available. I'd rather it not do that, and let each systems
administrator issue the command according to *their* security policy.

If you are installing packages into your home directory, then sudo is
not needed. But if you are installing it for everybody's use, then it
is necessary. Fink runs using superuser privileges.

No, sudo isn't necessary. It isn't provided by
default for all Unix installations, so Python would have to add a
dependency on it, which would be a bad thing.

sudo is sufficient. Other means are also sufficient. It would be wrong
for Python to assume a specific Unix security model (i.e. - "sudo")
for installations.

<mike

--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Mike Meyer]

Maurice LING <ma*********@acm.org> writes:

So if C extension API (or whatever that is really called) is stable,
the system admin can just copy all of /sw/lib/python2.3/site-packages
into /sw/lib/python2.4/site-packages and it should work. From what
you've said, it seems that this isn't possible.
Correct. This isn't possible. It's not clear it's *desirable*,
either. For one thing, code can fail to port across versions, so
things will break at random (instead of all at once, I admit). For
another, I dislike the idea of a module sitting unupdate for long
periods of time (decades, say). Updating Python provides a good excuse
to update all the python modules, etc.

Now, it would be nice if Python provided an easy way to do the
update. The FreeBSD packaging system happens to provide a way to
automate this process, but it's still not exactly easy.
So my alternative
solution is that PyPI have a mechanism to maintain what had been
installed in the site-package directory and to download the libraries
and install into the new site-package directory...

PyPI is the wrong place to maintain a record of installed
software. Distutils is the software that does the installing (ok, on
most modules, anyway), and is the software that should record what
packages are installed.

Neither distutils nor PyPI has a "download" functionality. For that
matter, PyPI doesn't have a hard requirement on listing where a module
comes from. I'd say distutils is the software that should provide it,
so I can say:

python setup.py install --depends

and get all the dependencies. But that's not at all clear. PyPI will
have to cooperate by providing URLs to packages, instead of to pages.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.