bytecode non-backcompatibili ty

[Maurice LING]

Hi,

I've been using Python for about 2 years now, for my honours project and
now my postgrad project. I must say that I am loving it more and more
now. From my knowledge, Python bytecodes are not back-compatible. I must
say that my technical background isn't strong enough but is there any
good reason for not being back-compatible in bytecodes?

My problem is not about pure python modules or libraries but the problem
is with 3rd party libraries with C bindings (not python pure). It means
that with every upgrade of python, I have to reinstall all my 3rd party
libraries which can be quite a bit of work...

I do hope this problem will be sorted out some day.

Cheers
Maurice

[Maurice LING]

Mike Meyer wrote:

Maurice LING <ma*********@ac m.org> writes:

The idea of having to release the program or library as source files
does ring alarms in many executives in corporate world. Less freezing
the modules (which renders it platform dependent) or using Jython
(which is not possible when C extensions are involved), there is no
middle grounds in CPython for distribution without source codes.

You're right - and byte codes *as they exist now* aren't an acceptable
middle ground, either. The problem isn't that the interpreter might
change (that can be defeated by distributing the interpreter with the
bytecode files), but that the byte code can be decompiled to get your
Python source back. See <URL:
http://www.crazy-compilers.com/decompyle/ > for an example.

I know that bytecodes can be decompiled back to python source. Similarly
Java class files can be decompiled back to Java source files quite
easily. IANAL, however, I do think that reverse engineering can be seen
as a purposeful act in the eyes of law. I remembered that EU Council
Directive 9/250/ECC does explicits the allowable scope of decompilation
(for interoperabilit y, preserving expression of ideas, achieving
performance of objectives).

For companies, I think it is touchy to ask permission to release the
source codes even though they may be alright with releasing the same
Java JAR file into the community.

We do know that regardless of whatever methods of compilation there may
be, if some genius is willing to spend sufficient time to crack a
program. Even if he has to read the binary codes, it will still break.
But that is a wilfull act rather than just looking at the source codes
when it is distributed with it. It's just like picking up your money
from the ground if you had dropped it as compared to pickpocketing you
as compared to hold you at knife point and robbing you...... Same end
result but it is viewed differently in the eyes of law and morals.


Selling byte codes to someone who's worried about shipping source is
selling them snake oil. I think it's unprofessional, at best.

<mike

[Martin v. Löwis]

Maurice LING wrote:

Now I understand that Python bytecodes are only dealing with pure python
source codes. However, the same question lies, how can it (set of
bytecodes) be made stable, like Java bytecodes, which are pretty stable?
Perhaps a better question will be, what techniques Java VM designers
use that enables Java class files (and JAR files) to be stable and
usable across versions that is lacking in Python?

The Java class file format is not any more stable than the Python .pyc
format - I think every major release of the JVM has changed details
in the class file format.

The difference is that the newer JVMs accept old class files, by
means of separate loader code for each class file version that was
ever released (and very possibly also support for class file versions
that were never released).

The same approach would be possible in Python, but nobody has
contributed code to do so. It is unlikely that future Python
versions will provide such compatibility with the current byte
code format, either, unless somebody steps forward and volunteers
to maintain that compatibility.

Maintaining this backwards compatibiltiy is a tedious and boring
task, given the many much easier alternatives, so that volunteers are
unlikely to jump in. Sun manages to provide the compatibility by
paying somebody to actually maintain it.

Regards,
Martin

[Martin v. Löwis]

Maurice LING wrote:

technicalities are wrong but situation remains unchanged.

For C modules, it is very likely that new versions of Python
will continue to break the ABI, by changing the layout of
structures.

The most straight-forward way to deal with it as a sysadmin
or user is to install multiple versions of Python on a single
machine. If Fink considers python2.4 as a replacement for
python2.3, then this is a flaw in Fink. In Debian, there is
a python package, which currently depends on python2.3.
Sometime in the future, it will depend on python2.4. Users
which update will then get python2.4, however, python2.3
will remain installed and usable, with all the extension
modules that were installed for it.

Regards,
Martin

[Terry Reedy]

"Maurice LING" <ma*********@ac m.org> wrote in message
news:d4******** **@domitilla.ai oe.org...

From a technical perspective, I can accept that .pyc files are private
and temporary. To a certain extend, it does helps development cycle.
Every time I amend my source codes, I just run it without having to
consider or needing to re-compile the source files. From a user perspective, source code is the run-time program. The idea of having to release the program or library as source files does
ring alarms in many executives in corporate world.

I understand that people use Python while resisting its purpose and design.
But I also understand that it is *their* responsibilty to hide their code,
which may possibly mean not using Python, or which may mean developing
proprietary methods to translate to something designed to *not* be
readable. For all we know, some of the developers have been paid to do
exactly that -- and not talk about it.

Python is *designed* for human readability. That is one of its big
features! The same also seems somewhat true for CPython's bytecodes,
especially when disassembled with the dis module that comes with the
interpreter. You even get all the object names included in the code
object.

Terry J. Reedy

[Maurice LING]

Martin v. Löwis wrote:

Maurice LING wrote:

technicalitie s are wrong but situation remains unchanged.

For C modules, it is very likely that new versions of Python
will continue to break the ABI, by changing the layout of
structures.

The most straight-forward way to deal with it as a sysadmin
or user is to install multiple versions of Python on a single
machine. If Fink considers python2.4 as a replacement for
python2.3, then this is a flaw in Fink. In Debian, there is
a python package, which currently depends on python2.3.
Sometime in the future, it will depend on python2.4. Users
which update will then get python2.4, however, python2.3
will remain installed and usable, with all the extension
modules that were installed for it.

Regards,
Martin

Fink does not consider python2.4 to be a replacement for python2.3. In
fact, you can install python2.2, 2.3 and 2.4 in the same machine with
Fink. It will maintain 3 sets of libraries as /sw/lib/python2.2,
/sw/lib/python2.3 and /sw/lib/python2.4. The chore is that when say Fink
installs python2.4, all the libraries in /sw/lib/python2.3/site-packages
have to be re-installed into /sw/lib/python2.4/site-packages, one by
one. There is no simple way of doing that... which makes any system
admin needing to re-install 50 3rd party libraries into
/sw/lib/python2.4/site-packages a big task, as well as satisfying the
necessary dependencies.

So if C extension API (or whatever that is really called) is stable, the
system admin can just copy all of /sw/lib/python2.3/site-packages into
/sw/lib/python2.4/site-packages and it should work. From what you've
said, it seems that this isn't possible. So my alternative solution is
that PyPI have a mechanism to maintain what had been installed in the
site-package directory and to download the libraries and install into
the new site-package directory...

What do you think?

Cheers
maurice

[Terry Reedy]

"Maurice LING" <ma*********@ac m.org> wrote in message
news:d4******** **@domitilla.ai oe.org...

So my alternative solution is that PyPI have a mechanism to maintain what
had been installed in the site-package directory and to download the
libraries and install into the new site-package directory... What do you think?

I doubt anyone disputes that upgrades are more hassle than we would like.
My main point was that freezing CPython's technology is not the solution.
Any other upgrade helper that you can contribute will be welcome.

Terry J. Reedy

[Martin v. Löwis]

Maurice LING wrote:

So if C extension API (or whatever that is really called) is stable, the
system admin can just copy all of /sw/lib/python2.3/site-packages into
/sw/lib/python2.4/site-packages and it should work.
It would be counter-productive to make it stable, as this would render
certain desirable changes impossible. Backwards compatibility on C API
(source) level is a goal, binary compatibility within a 2.x release
is a goal, binary compatibility across 2.x releases is not.

[automatically install a set of packages] What do you think?

That would certainly be possible. Contributions are welcome.

Regards,
Martin

[Maurice LING]

Terry Reedy wrote:

I doubt anyone disputes that upgrades are more hassle than we would like.
My main point was that freezing CPython's technology is not the solution.
Any other upgrade helper that you can contribute will be welcome.

Terry J. Reedy

So there is no way of releasing a close-source application written in
Python?

Forget about reverse engineering or decompiling, things like EU Council
Directive, Berne Convention , and legislation of individual countries
are to be used and argued for handling what decompilation is allowed or
not allowed. It does make a difference between looking at the codes
when it is given and performing an act of decompilation to look at the
codes. As mentioned in one of my postings, it may be analogous to the
difference between you dropping some cash on the ground and I take it,
and I performing the act of pickpocketing and take your money.

I am not advocating close source in any sense and I believe the choice
of open or close source distribution is individual's and corporate's rights.

Cheers
Maurice

[Maurice LING]

Martin v. Löwis wrote:

[automatically install a set of packages]

What do you think?

That would certainly be possible. Contributions are welcome.

Regards,
Martin

I've emailed to catelog-sig mailing list and is still waiting to hear
something. Currently, I have no idea of the structure of PyPI. I hope I
can hear from them soon and generate some starting points...

Maurice

[Martin v. Löwis]

Maurice LING wrote:

I doubt anyone disputes that upgrades are more hassle than we would
like. My main point was that freezing CPython's technology is not the
solution. Any other upgrade helper that you can contribute will be
welcome.

So there is no way of releasing a close-source application written in
Python?

I fail to see the relationship between the statement and your question.

Again, whether or not the C API is frozen has nothing to do with
whether or not the bytecode format is frozen. So any implication
from not freezing the C API to not being able to ship byte-code
only is invalid.

As for only shipping byte code: sure, that is certainly possible,
and people do that all the time.

Regards,
Martin