bytecode non-backcompatibili ty

[Maurice LING]

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure). It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

Cheers
Maurice

[Robert Kern]

Maurice LING wrote:

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure).
Then this has nothing to do with bytecode incompatibility . Only
pure-Python modules get compiled to bytecode. You mean binary
compatibility of the C API.

If you're going to have significant improvements in the core
interpreter, you have to break binary compatibility from time to time.
It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

This problem is mitigated somewhat by the fact that the devs keep binary
compatibility within a major revision series (e.g. 2.3.0, 2.3.1, 2.3.2,
....).

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[Fuzzyman]

Maurice LING wrote:

Hi,

I've been using Python for about 2 years now, for my honours project and now my postgrad project. I must say that I am loving it more and more now. From my knowledge, Python bytecodes are not back-compatible. I must say that my technical background isn't strong enough but is there any good reason for not being back-compatible in bytecodes?

It *shouldn't* be a problem for pure python modules. The interpreter
will recognise that the bytecode has the wrong 'magic number' and
recompile.

Seeing as a new major version of python will probably be installed in a
new directory at the very least you will have to copy the modules
across. If they have install files (or a setup.py) wouldn't it be
better to use that *anyway* ?
My problem is not about pure python modules or libraries but the problem is with 3rd party libraries with C bindings (not python pure). It means that with every upgrade of python, I have to reinstall all my 3rd party libraries which can be quite a bit of work...

It is a nuisance. It's more of a nuisance when third part modules with
'c' components are compiled for the new version of python (a windows
specific problem).

I did find that the last upgrade forced me to evaluate which extensions
I actually used. The answer was 'a few less than I thought'. It became
a good opportunity to spring clean my 'site-packages' folder.

Best Regards,

Fuzzy
http://www.voidspace.org.uk/python
I do hope this problem will be sorted out some day.

Cheers
Maurice

[Terry Reedy]

I think the OP is confusing three different CPython implementation
features -- bytecodes, marshal format, and C API, all of which change
pretty slowly -- and a MS Windows/C (dis)feature.

CPython bytecodes only concern code written in Python but are mostly not a
concern because recompilation is automatic when needed. The exception is
code that directly mucks around with bytecodes, especially the intentially
semi-undocumented numerical codes.

Code objects also include marshaled object values. The code object for
'haha = 987654321' must contain representations for the constants 'haha'
and '987654321' as well as the bytecode for the assignment. So even if
bytecodes remain the same, marshal can change (as it did for 2.4, I
believe, and as it will for floats in 2.5 to fix a bug) and trigger auto
recompiles.

C extensions interact with the interpreter via function calls that
constitute the C API. As much as possible, the developers consciously
avoid changes that break old code or even old binaries. Guido has claimed
that for Linux, there are extension binaries compiled for 2.0 that still
work with 2.4.

However, for MS Windows/C, there is a 'feature' with respect to DLLs that
requires recompilation of extensions to work with a new version of the
Python DLL, even if the C API is unchanged (or so people who understand
this have said). So the OPs complaint about having to get and install new
extension binaries for each Python version might well be directed to MS.

This does not, of course, negate the idea that it would be nice if the
update process were somehow make easier. Indeed, the more-batteries
included distributions from ActiveState and Enthought specifically aim at
this.

Terry J. Reedy

[Maurice LING]

> It is a nuisance. It's more of a nuisance when third part modules with

'c' components are compiled for the new version of python (a windows
specific problem).

I did find that the last upgrade forced me to evaluate which extensions
I actually used. The answer was 'a few less than I thought'. It became
a good opportunity to spring clean my 'site-packages' folder.

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?

I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the
trouble if I do a "fink selfupdate" and "fink update-all" and finds that
my Python is replaced by a newer version, say Python 2.4. Then all my
3rd party libraries in /sw/lib/python2.3/site-packages are un-usable...
And I will have to re-download and re-install all the libraries again.

Yes, I do agree that it is a nice time to clean out "site-packages" but
imagine the work of a system admin who has to re-install 50 packages...
I do believe that we can use some help here, if possible... Perhaps
someone can enlighten us on the technicalities of "so" files and/or C
bindings in Python...

Now I understand that Python bytecodes are only dealing with pure python
source codes. However, the same question lies, how can it (set of
bytecodes) be made stable, like Java bytecodes, which are pretty stable?
Perhaps a better question will be, what techniques Java VM designers
use that enables Java class files (and JAR files) to be stable and
usable across versions that is lacking in Python?

Thanks.

Cheers
Maurice

[Dennis Lee Bieber]

On Tue, 26 Apr 2005 14:47:21 +1000, Maurice LING <ma*********@ac m.org>
declaimed the following in comp.lang.pytho n:

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?
Linux equivalent of a Windows DLL -- "shared object" file I
think... a binary object file that is dynamically loaded/linked at run
time. The application binding just knows the name/offset into the object
for each function call.
I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine the
trouble if I do a "fink selfupdate" and "fink update-all" and finds that
my Python is replaced by a newer version, say Python 2.4. Then all my
3rd party libraries in /sw/lib/python2.3/site-packages are un-usable...
And I will have to re-download and re-install all the libraries again.
If Fink stuffs a Python 2.4 in your /sw/lib/python2.3/ directory
-- blame Fink! (whatever Fink is...) A Python 2.4 install would be in a
..../python2.4/... directory, and your 2.3 should be untouched. The
/search path/ might have been changed to find 2.4 first, and /that/
might be a problem.

-- =============== =============== =============== =============== == <
wl*****@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
wu******@dm.net | Bestiaria Support Staff <
=============== =============== =============== =============== == <
Home Page: <http://www.dm.net/~wulfraed/> <
Overflow Page: <http://wlfraed.home.ne tcom.com/> <

[Mike Meyer]

Maurice LING <ma*********@ac m.org> writes:

I find this part of the story a nuisance, C components in 3rd party
modules... What are the C components compiled into? What are actually
"so" files?
Shared object files. They're executable code that can be linked into a
program at runtime. Like DLL's on Windows.
I am using Fink to maintain my Python installation and Fink maintains
the 3rd party libraries in /sw/lib/python2.3/site-packages. Imagine
the trouble if I do a "fink selfupdate" and "fink update-all" and
finds that my Python is replaced by a newer version, say Python
2.4. Then all my 3rd party libraries in
/sw/lib/python2.3/site-packages are un-usable... And I will have to
re-download and re-install all the libraries again.
Yeah, it's pain. It would be nice if setup.py kept a list of installed
packages that you could extract before updating, so you would know
what needed to be updated.

Last time I updated, I used the systems package handler. I asked it
for the names of all the Python packages that were installed and saved
that to a file. Then I installed the new version of Python. Then I
edited the package list to turn it into a series of package update
commands, and ran that file.
Now I understand that Python bytecodes are only dealing with pure
python source codes. However, the same question lies, how can it (set
of bytecodes) be made stable, like Java bytecodes, which are pretty
stable? Perhaps a better question will be, what techniques Java VM
designers use that enables Java class files (and JAR files) to be
stable and usable across versions that is lacking in Python?

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Maurice LING]

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

My arguments to the developers will be:

1. Python had gone from a purely scripting language to a general purpose
programming language.

2. The current compilation scheme (compiling to bytecode as and when it
is needed) works well for scripting purposes but is less desirable in
commercial settings. Less distribution happens when it is used purely
for scripting purposes, such as system maintenance or tuning.

3. Using Python in commercial settings will usually require distribution
of resulting software and it is may or may not be desirable to
distribute source codes as well. Unless the application is frozen,
distributing source code is a must.

4. One advantage that Java platform has is that it does not require the
release of source files and still promotes platform-independence.

5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.

6. Unstable bytecodes also makes research work into Python, such as,
just-in-time compilation and just-in-time specialization unfavourable as
they may only be applicable to a specific version of Python. There is
much less chance of getting a project grant than if the same project is
applied for Java (stable bytecodes).

What other point are there?

I may be chopped by saying this but by having a stable set of bytecodes,
we may lose a means of optimization. But we may gain more from filling
the need for commerical distribution of applications writing in Python
and ease of upgrading...

At current stage, every time a new version of Python is installed in my
server or system, I have to test and ensure the needed libraries are
there... which may be a horror in corporate settings. Couple that with
messy dependencies of the libraries to be installed. I remembered the
time I was trying to install Eric 3, the dependencies makes me want to
give up... I have to install Qt, then pyQt, then something else, then
Eric3. Imagine you need 10 of those libraries... which may happen... I
am not yet masochistic enough to take pleasures in this...

I also hope some python developers are reading this thread as well......
Call this a desperate plea from some of us......

Cheers
Maurice

[Steve Holden]

Maurice LING wrote:

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

[... arguments about bytecode stability ...]
What other point are there?

I may be chopped by saying this but by having a stable set of bytecodes,
we may lose a means of optimization. But we may gain more from filling
the need for commerical distribution of applications writing in Python
and ease of upgrading...

At current stage, every time a new version of Python is installed in my
server or system, I have to test and ensure the needed libraries are
there... which may be a horror in corporate settings. Couple that with
messy dependencies of the libraries to be installed. I remembered the
time I was trying to install Eric 3, the dependencies makes me want to
give up... I have to install Qt, then pyQt, then something else, then
Eric3. Imagine you need 10 of those libraries... which may happen... I
am not yet masochistic enough to take pleasures in this...

I also hope some python developers are reading this thread as well......
Call this a desperate plea from some of us......

Cheers
Maurice

There's enough attention to backward compatibility that the chance of
code breakage is quite small for pure-Python modules. Given that case,
all we really need for them is a suitable way of moving them forward to
an updated distribution (though, of course, the module authors may have
brought out new versions that use more advanced language features, there
is no real compulsion to migrate to those if you seek stability).

I don't believe that the major problem is the changes to the bytecodes -
extension modules (those written in C and/or C++) don't generally use
the bytecodes anyway. The real problem is the API to the interpreter,
which again the developers retain the right to change between minor (but
not micro) versions.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/

[Mike Meyer]

Maurice LING <ma*********@ac m.org> writes:

All you have to do is convince the developers to declare the set of
bytecodes fixed, and they'd be stable. I don't think that will be
easy, as it removes one of the methods used to improve the performance
of Python. Since rebuilding the bytecode files is trivial (just invoke
compileall.py in the library as a script), this isn't really a
problem. The only thing that breaks are modules that muck about inside
the bytecode. I don't know how much bytecode improvements have sped
things up, but I do know that if I ever used a module that mucked
around with byte codes, I wasn't aware of it - so this is a tradeoff
I'm more than happy to make.

Thanks Mike,

My arguments to the developers will be:

1. Python had gone from a purely scripting language to a general
purpose programming language.

I think that's irrelevant. It just means that there are more
applications around that have to be dealt with when you update.
2. The current compilation scheme (compiling to bytecode as and when
it is needed) works well for scripting purposes but is less desirable
in commercial settings. Less distribution happens when it is used
purely for scripting purposes, such as system maintenance or tuning.
The solution with the current situation depends on what you mean by
"commercial settings".
3. Using Python in commercial settings will usually require
distribution of resulting software and it is may or may not be
desirable to distribute source codes as well. Unless the application
is frozen, distributing source code is a must.
People have recommended that you distribute the Python interpreter
with commercial applications. This makes installing them much easier
for the end user. It also means the byte code and the interpreter will
always match.
4. One advantage that Java platform has is that it does not require
the release of source files and still promotes platform-independence.
Java is only "platform-independent" to the degree that the jvm and
libraries you need for your application are already available. A
quick google finds applications that require jvm versions - so I
suspect that you can't always run Java code built for a new version of
the jvm on older jvms. So you have to make sure the target platform
has a recent enough implementation of the jvm installed. You may have
problems if you try using a different groups jvm as well. I haven't
looked into that in a number of years.

So you can't just ship java bytecodes to someone expecting they'll be
able to run it out of the box. They may well need to update their java
environment, or install a second one (I recall one time having four
Java products that took in total three different jvms to run. Bleah.)

This isn't really different from Python. In both cases, the end user
has to have a suitable version of the bytecode interpreter and
libraries installed. Java is a little better in that they provide
backwards compatability.
5. Unstable bytecodes makes updating to a newer version of Python very
tedious and risk breaking old scripts, if they uses C modules.
Unstable bytecodes have nothing to do with these problems.

The current CPython installation process puts the python command and
the libraries for different versions in different directories. This
allows you to have multiple versions installed so you can keep old
scripts working with the old version until you've had time to test
them. It also makes life much easier on the developers, as they can
have a development version installed on the machine at the same time
as they have a production version without breaking the old scripts. It
also means you have to reinstall all your modules on the new
installation - which is what makes the update process tedious for me.

Now, this could be mitigated by having Python libraries installed in
the same location for all versions. You could fix all the bytecode
files by running compileall.py as a script. Sometimes, a module won't
work properly on a new version of Python, and will have to be
updated. You'll have to find those by trial and error and fix them as
you find them. You'll also have to recompile all the C modules, which
will break running scripts on the old interpreter.

Under the currewnt system, old scripts that are run by the new
interpreter will break if all the modules they need aren't installed
yet. Once they're installed, the scripts should work.

If you have scripts that use C modules that aren't installed in the
standard Python search path, they may well break on the new
interpreter until you recompile them. Having both versions of the
interpreter available makes this tolerable.

The cure I proposed seems worse than disease. If you've got a better
solution, I'm sure we'd be interested in hearing it.
6. Unstable bytecodes also makes research work into Python, such as,
just-in-time compilation and just-in-time specialization unfavourable
as they may only be applicable to a specific version of Python. There
is much less chance of getting a project grant than if the same
project is applied for Java (stable bytecodes).
This doesn't seem to have stopped the Psyco project from turning out a
jit compiler. I can't speak to grant applications, though. If you're
really interested, you could apply to the PSF for a grant.
I may be chopped by saying this but by having a stable set of
bytecodes, we may lose a means of optimization. But we may gain more
from filling the need for commerical distribution of applications
writing in Python and ease of upgrading...
The commercial distribution needs can be met without going to stable
bytecodes. The ease of upgrading needs aren't met by going to stable
bytecodes.
At current stage, every time a new version of Python is installed in
my server or system, I have to test and ensure the needed libraries
are there... which may be a horror in corporate settings. Couple that
with messy dependencies of the libraries to be installed. I remembered
the time I was trying to install Eric 3, the dependencies makes me
want to give up... I have to install Qt, then pyQt, then something
else, then Eric3. Imagine you need 10 of those libraries... which may
happen... I am not yet masochistic enough to take pleasures in this...

Fixing the byte-code problem won't help with the dependency
problem. If anything, it'll make it worse, because you'll have old
libraries that were installed with previous versions of Python to
contend with. If those are acceptable, that's all well and good. But
for complex packages like eric - which depends on sip and qt and a
number of other things - the latest versions tend to rely on having
up-to-date libraries. So you install eric, and watch it fail because
some library is out of date. Update that library and repeat. The
easiest way to deal with this is to find all the libraries it needs,
and just update them all.

The dependencies problem is actually pretty easy to solve. In fact, in
many Python environments, it's already solved. On my system, if I
install Eric3 using the provided installation package, the
dependencies will be picked up automatically. Not very helpfull if
you're not on such a system, I know. The long-term solution is for
PyPI to grow to include this functionality.

Thank you,
<mike

--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.