Python compilers?

[Svein Ove Aas]

Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

Come to think of it, is anyone working on a sexpr-enabled version of
Python, or anything similar? I really miss my macros whenever I try to
use it...

[Ville Vainio]

>>>>> "Andrew" == Andrew MacIntyre <an*****@bullseye.apana.org.au> writes:

There is also GCJ as part of the GCC, which can compile both .class
and .java files. Its libraries aren't complete yet, but I'm sure it's
only a matter of time.

Andrew> Hmmm... anyone tried GCJ on Jython?

Native code will not help much if the created native code is of type:

arg=lookup(object1, "fooarg")
f = lookup(object2,"foomethod")
call(f,arg)

For the performance that is expected of native code we need direct
dispatching with the addresses of the functions known at the compile
time, or via direct indexing of linear virtual tables.

(I'm speaking of static compilation here - what I said may not apply
to psyco)

--
Ville Vainio http://tinyurl.com/2prnb

[Heather Coppersmith]

On Sat, 22 May 2004 05:54:24 GMT,
Carl Banks <im*****@aerojockey.invalid> wrote:

... There's two questions remaining for me: 1. These claims that Lisp code can approach 50 percent the speed
of C, is that with or without the optional type declarations?
That would be *with* type declarations.
2. If you don't use the declarations, does compiling Lisp help?
If it does (and nothing I've read indicated that is doesn't),
it definitely casts some doubt on the claim that compiling
Python wouldn't help. That's kind of been my point all
along.
In _Common Lisp The Language, Second Edition_, by Guy Steele
(CLTL2), on page 686, in section 25.1.3 ("Compilation
Environment") there is a lengthy list of assumptions the compiler
makes, including:

o ... within a named function, a recursive call to a function
of the same name refers to the same function [barring
notinline declarations] ...

o ... a call within the file being compiled to a named
function that is defined in that file refers to that
function [barring notinline declarations] ...

o ... the signature (or "interface contract") of all built-in
Common Lisp functions will not change ...

o ... the signature (or "interface contract") of functions
with ftype information will not change.

o ... any type definition made with defstruct or deftype in
the compile-type environment will retain the same definition
in the run-time environment. It may also assume that a
class defined in the compile-time environment will be
defined in the same run-time environment in such a way as to
have the same superclasses and metaclass ...

o ... if type declarations are present in the compile-time
environment, the corresponding variables and functions
present in the run-time environment will actually be of
those types ...

So compiled Lisp is a less dynamic than Python, and the built-in
functionality is large and guaranteed to match its textbook
definitions, which gives Lisp Compilers more than a fighting
chance. I'm pretty sure I've seen that summary on c.l.p before.
I think (and I'm wrapping this up, cause I think I made my
point) compiling Python could help, even without type
declarations, but probably not as much as in Lisp. It could
still make inferences or educated guesses, like Lisp compilers
do; just maybe not as often.

I agreem, with emphasis on "could" and some hesitation on how much
guessing I want my compiler to do.

Regards,
Heather

--
Heather Coppersmith
That's not right; that's not even wrong. -- Wolfgang Pauli

[Terry Reedy]

"Heather Coppersmith" <[e-mail]> wrote in message
news:m2************@unique.phony.fqdn...

In _Common Lisp The Language, Second Edition_, by Guy Steele
(CLTL2), on page 686, in section 25.1.3 ("Compilation
Environment") there is a lengthy list of assumptions the compiler
makes, including:

o ... within a named function, a recursive call to a function
of the same name refers to the same function [barring
notinline declarations] ...

o ... a call within the file being compiled to a named
function that is defined in that file refers to that
function [barring notinline declarations] ...

o ... the signature (or "interface contract") of all built-in
Common Lisp functions will not change ...
Interesting. PyCode can be sped up by making the same assumptions.
Richard Hettinger's recent recipe implements these assumptions. (It was
rejected as part of the standard lib for being too implementation specific,
but remains available in the archives and Python Cookbook site.)

[snip less applicable to Python today stuff]
So compiled Lisp is a less dynamic than Python,

The balance between flexibity and speed continues to be debated by the
developers.

Terry J. Reedy

[Günter Jantzen]

"Terry Reedy" <tj*****@udel.edu> schrieb im Newsbeitrag
news:ma*************************************@pytho n.org...

So compiled Lisp is a less dynamic than Python,

The balance between flexibity and speed continues to be debated by the
developers.

And should be debated by the users, too.

Not all of Pythons dynamics is really necessary to write good programms.
Writing good software needs creativity and discipline. We like to work with
Python, because it allows us creativitity. Sometimes to enforce a grain of
discipline in Python would not be bad. There have been many discussions
about programming idioms and design patterns in the last years. They give
not so high level languages the ability to simulate high level features.
They could also help to give a very high level language more solid ground.
And they normalize the way to handle common classes of problems

"""There should be one-- and preferably only one --obvious way to do it.
"""
We could try to search patterns, idioms, antipatterns which could establish
a better way to use the Python language:
-- to avoid unnecessary dynamics
-- to avoid unnecessary features
-- to establish the 'obvious way to do it'
-- without loosing the power and expressiveness of python

"""In the face of ambiguity, refuse the temptation to guess.
"""
Possible sideeffect:.- it could be easier for mechanical tools (Pyrex,
Psycho, PyPy, Starkiller) to make Python faster

Not all people will agree about a useful set of patterns. But this is not
necessary now.
Patterns and idioms do not change a language immediately. They do not fall
from heaven, it is necessary to play around and to gain some experience.

I think patterns define a subset of a language and some constraints on
programms
Maybe this constrained subset (or subsets if there are controversal ideas)
can be formalized and be checked by 'pylintplus' or 'pythoncheckerX' (do not
google).
Or there would be a configurable universal 'Monsterchecker'. In the first
line of our sources we expose a checkpolicy and the checker will follow this
policy.
If patterns can not be formalized so easy, they can be established by
convention.

I do not know the result of this process ...
Now some ideas what could be done in the near future

1) Integration of Pyrex into Python.
[strong variant]
Syntactically this languages are close. Maybe they could be changed so, that
every valid Pyrex source is also a valid Python source with -hopefully- the
same semantics. Special Pyrex tokens would be allowed, but meaningless when
used in the Python context.
Benefit: more homogenous environment. Little changes in the program could be
made without a C-Compilation step.
Maybe the support of interfaces would be necessary in Python, but this is
not a bad idea.
Then rewrite some parts of the standard library in Pyrex

[weak variant]
Rewrite some parts of the standard library in Pyrex

2) Find a set of constraints which make it easier for Psyco (now) and
Starkiller (later) to do their job

3) Implement Monsterchecker modes for 1 and 2

Regards
Guenter

[Andrew MacIntyre]

On Sat, 22 May 2004, Ville Vainio wrote:

>> "Andrew" == Andrew MacIntyre <an*****@bullseye.apana.org.au> writes: >> There is also GCJ as part of the GCC, which can compile both .class
>> and .java files. Its libraries aren't complete yet, but I'm sure it's
>> only a matter of time.

Andrew> Hmmm... anyone tried GCJ on Jython?

Native code will not help much if the created native code is of type:

arg=lookup(object1, "fooarg")
f = lookup(object2,"foomethod")
call(f,arg)

For the performance that is expected of native code we need direct
dispatching with the addresses of the functions known at the compile
time, or via direct indexing of linear virtual tables.

(I'm speaking of static compilation here - what I said may not apply
to psyco)

I was asking more out of interest in the potential for creating
distributable binaries (which is usually one of the desires of people
looking for compilers), than performance specifically. Of course it
would be nice if a GCJ compiled Jython app could at least match the
performance of CPython for the same app.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: an*****@bullseye.apana.org.au (pref) | Snail: PO Box 370
an*****@pcug.org.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Konstantin Veretennicov]

"Terry Reedy" <tj*****@udel.edu> wrote in message news:<ma*************************************@pyth on.org>...

Interesting. PyCode can be sped up by making the same assumptions.
Richard Hettinger's recent recipe implements these assumptions. (It was
rejected as part of the standard lib for being too implementation specific,
but remains available in the archives and Python Cookbook site.)

I believe that's /Raymond/ Hettinger.

Anyway, the recipe is a great stuff:
http://aspn.activestate.com/ASPN/Coo.../Recipe/277940
- kv

[Jacek Generowicz]

Heiko Wundram <he*****@ceosg.de> writes:

Am Dienstag, 18. Mai 2004 13:41 schrieb Jacek Generowicz:

Native compilers for other languages just as dynamic as Python
exist. These compilers manage to achieve very significant speed
increases[*].
In Python this isn't true. Python, instead of LISP, is "completely" dynamic,

In what way is Lisp less dynamic then Python? If what follows is the
basis of your argument, then think again?
meaning that it's pretty impossible to do type-inference for each function
that is called (even checking types isn't possible). E.g. how do you expect
type-inference to work with the pickle module? string -> something/Error
would be the best description what pickle does. For the function which calls
pickle, do you want to create versions for each possible output of Pickle?
Which outputs of Pickle are possible?

Which outputs of the standard Common Lisp function 'read'[*] are possible?
[*] http://www.lisp.org/HyperSpec/Body/chap-23.html

[Jacek Generowicz]

Paul Rubin <http://ph****@NOSPAM.invalid> writes:

Carl Banks <im*****@aerojockey.invalid> writes:

I don't follow you. In what way is Python dynamic that Lisp isn't?

class foo: ... def bar(self, x):
... return x*x
... a = foo()
a.bar(3) 9 a.bar = lambda x: x*x*x
a.bar(3) 27

I'm afraid you'll have to try harder.
[1]> (defclass foo () ())
#<STANDARD-CLASS FOO>
[2]> (defmethod bar ((foo foo) x)
(* x x))
#<STANDARD-METHOD (#<STANDARD-CLASS FOO> #<BUILT-IN-CLASS T>)>
[3]> (defparameter a (make-instance 'foo))
A
[4]> (bar a 3)
9
[5]> (defmethod bar ((self (eql a)) x)
(* x x x))
#<STANDARD-METHOD ((EQL #<FOO #x203185F9>) #<BUILT-IN-CLASS T>)>
[6]> (bar a 3)
27

[Jacek Generowicz]

Paul Rubin <http://ph****@NOSPAM.invalid> writes:

Carl Banks <im*****@aerojockey.invalid> writes:

The example above kills any attempt to turn a.bar() into a static
procedure call.

Of course it does--but it's one method. A compiler, if it's good,
would only make the optization on methods named "bar", and it could
probably pare the number of possible classes it could happen to down
to only a few.

How could it possibly know? The reassignment of a.bar could happen
anytime, anywhere in the code. Maybe even in an eval.

I mean you could have a Turing nightmare on your hands, with all kinds
of crazy setattrs and execs and stuff, in both Python and Lisp, and
then there's not much a compiler could do but emit totally general
code. I assume Lisp compilers do this sometimes.

Lisp compilers might have to do that sometimes, but Python compilers
would have to do it ALL the time. Psyco took one way out, basically
generating code at runtime and caching it for specific operand types,
but the result is considerable code expansion compared to precompilation.

I see how seasoned Lispers can get pretty tired of these sorts of
arguments. Please bear in mind that Lisp has been the playground for a
lot of very clever people whose aim was to solve difficult problems
efficiently ... and they've been at it for about 3 decades before
Python was even thought of. Instead of claiming that Python is
something revolutionarily new in the area of dynamicity (it isn't) and
that compiling it is impossible or futile (it isn't) have a look at
what the Lispers learned in their study of the subject over the four
decades that they have been at it. (You should only do this, of
course, if you are interested in how Python might benefit from being
compiled; if you don't care, then at least don't hurl unfounded
"opinions" about how it is impossible because Python is so amazingly
dynamic.)

Of course certain things are easier to compile to efficient machine
code than others. How does the existence of the problem spots take
away from the usefulness of compiling the easier parts? It's possible
to have significant gains. Bigger gains come with harder work. Bigger
gains can be had in some areas than in others. Certain areas are more
likely to be speed critical than others to a larger proportion of
users (number crunching, for instance). Different compiler
implementors make different choices regarding which areas of the
language they target with hard compiler optimizations.

Imagine that I would like to be able to write effecient numeric code
in pure Python, for example ... I really don't care that any instance
methods that I add will not be dispatched within a single cycle, do I?