By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
457,976 Members | 1,162 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 457,976 IT Pros & Developers. It's quick & easy.

pyvm -- faster python

P: n/a
Hi.

pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/
Facts about pyvm:
- It's FAST. According to the "cooked-bench" benchmark suite it finishes
in 55% of the time python takes;)
- It's SMALL. Currently the source code is under 15k lines with the
builtin modules.
- It's new. Uses no code from CPython.
- It's incomplete. Not even near the stability and quality of python.
It needs A LOT of work before it can be compared to CPython.
Moreover, at the time it lacks many things like closures, long numbers
new style classes, etc.
- It's incompatible with CPython. Not all programs run.
- The C API is incompatible. You can't run C modules (a thin wrapper to
make pyvm appear as libpython *could* be possible but not a goal
AFAIC)
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!

Cheers,

Stelios
Jul 19 '05 #1
Share this Question
Share on Google+
53 Replies


P: n/a

Stelios Xanthakis wrote:
Hi.

pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/
Facts about pyvm:
- It's FAST. According to the "cooked-bench" benchmark suite it finishes in 55% of the time python takes;)
- It's SMALL. Currently the source code is under 15k lines with the
builtin modules.
- It's new. Uses no code from CPython.
- It's incomplete. Not even near the stability and quality of python.
It needs A LOT of work before it can be compared to CPython.
Moreover, at the time it lacks many things like closures, long numbers new style classes, etc.
- It's incompatible with CPython. Not all programs run.
- The C API is incompatible. You can't run C modules (a thin wrapper to make pyvm appear as libpython *could* be possible but not a goal
AFAIC)
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!

Cheers,

Stelios


Hi Stelios,

could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter? There is AFAIK no such ambitious
project that has ever survived. The last one I remember died shortly
after it was born:

http://www.python.org/pycon/papers/pymite/

This is sad because it is still challenging to offer a tiny interpreter
of a dynamic language for glue code/RAD on tiny hardware. A lot of
effort was spent to provide Java for microcontrollers especially for
SmartCards. I think a lot of people would show interest in Your project
if it gets somehow focussed and does not seem to be redundant.

Ciao,
Kay

Jul 19 '05 #2

P: n/a
I've seen the benchmarks, they look quite interesting.

This project is probably a LOT of work; maybe people can tell us about
such efforts *before* doing so much work, so we can discuss it, and
avoid wasting time.

Maybe you can explain us why it is so fast, and/or maybe you can work
with the other developers to improve the speed of the normal CPython,
this can require equal or less work for you, and it can produce more
long-lasting results/consequences for your work.

Bye,
Bearophile

Jul 19 '05 #3

P: n/a
Kay Schluehr wrote:
Stelios Xanthakis wrote:
pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/


could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter? There is AFAIK no such ambitious
project that has ever survived. ..


If you check the URL more closely, you'll notice that this is a
university site.

It seem likely this being done at least partly as a school project,
which is certainly motivation enough for such a thing, *even* if it
doesn't survive (since it will pay for itself in learning).

-Peter
Jul 19 '05 #4

P: n/a
> could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter?


I'd also be curious to know if the performance gains would remain
once it gets fleshed out with things like closures, long numbers,
new style classes and a C library compatibility shim.

Roger
Jul 19 '05 #5

P: n/a
Stelios Xanthakis <sx****@ceid.upatras.gr> writes:
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!


Are you going to release the source? If not, it's a lot less interesting.
Jul 19 '05 #6

P: n/a
djw
Paul Rubin wrote:
Stelios Xanthakis <sx****@ceid.upatras.gr> writes:
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Are you going to release the source? If not, it's a lot less interesting.


From the website:

"...the source will be released when it becomes more complete and stable."
Jul 19 '05 #7

P: n/a
djw <dw****@vcd.hp.com> writes:
Paul Rubin wrote:
Stelios Xanthakis <sx****@ceid.upatras.gr> writes:
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Are you going to release the source? If not, it's a lot less interesting.


From the website:

"...the source will be released when it becomes more complete and stable."


If he releases the source now, it'll probably become more complete and
stable sooner...

Remember, release early, release often!

Dan
Jul 19 '05 #8

P: n/a
Kay Schluehr wrote:

could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter? There is AFAIK no such ambitious
project that has ever survived. The last one I remember died shortly
after it was born:


The motivation is that I just needed some bytecode engine. I deciced
to go with python bytecode, because well, IMO it's the best!
Eventually it turned out what it is.

I'm afraid this may end up dead before unborn too.
So it depends what people want. If nobody cares, I just learned python
the hard way and the project does to the abandonware cemetary.
Stelios

Jul 19 '05 #9

P: n/a
be************@lycos.com wrote:
This project is probably a LOT of work; maybe people can tell us about
such efforts *before* doing so much work, so we can discuss it, and
avoid wasting time.

It is a lot of work indeed.
Usually, when people announce "we shall create X", it doesn't happen.
And you don't know if you can do it at all *before*.

Maybe you can explain us why it is so fast, and/or maybe you can work
with the other developers to improve the speed of the normal CPython,
this can require equal or less work for you, and it can produce more
long-lasting results/consequences for your work.


The reason is that it's built from the scratch.
Guido would disagree with that, see py-dev thread:

http://www.mail-archive.com/py******.../msg01225.html
There *are* alternative ways to do some things in the vm and Joel
is simply wrong:)
Could this be .... Python 3000!!!? :)
Stelios
Jul 19 '05 #10

P: n/a
Roger Binns wrote:
could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter?

I'd also be curious to know if the performance gains would remain
once it gets fleshed out with things like closures, long numbers,
new style classes and a C library compatibility shim.


I guess it will. There are several places with /*XXX:speed up*/
and a couple more ideas to gain exrta speed. On the other hand
pyvm does not check for Bad Internal Calls and other checks Cpython
does to be OK in all cases (for example if __cmp__ modifies a list
while it's being sorted, in pyvm you *can* get a crash).

There is no plan to implement closures; unless there is a way to
turn closures to default arguments with bytecode hacking. But yes,
closures are interesting, especially in generator expressions and
lambdas.

I am not very interested on C compatibility. Once the source is
open people can implement it if they want but I'd rather go with
an advanced ctypes module which will make loading dynamic libraries
a matter of python code.
Stelios

Jul 19 '05 #11

P: n/a
[Stelios Xanthakis]
I'm afraid this may end up dead before unborn too. So it depends what
people want. If nobody cares, [...]


People might not care so much about what could be done about your
project, unless you give them proper and complete means for evaluating
the state of affairs. Your project is very likely to die if you keep
your sources closed, and yourself loose interest in the project.

Opening your sources is no guarantee either that the community will
adopt your project. But this might give your project a better chance.

--
François Pinard http://pinard.progiciels-bpi.ca
Jul 19 '05 #12

P: n/a

"Stelios Xanthakis" <sx****@ceid.upatras.gr> wrote in message
news:42**************@ceid.upatras.gr...
Maybe you can explain us why it is so fast, and/or maybe you can work
with the other developers to improve the speed of the normal CPython,
this can require equal or less work for you, and it can produce more
long-lasting results/consequences for your work.

The reason is that it's built from the scratch.
Guido would disagree with that, see py-dev thread:


Guido, like me, believes in being correct before being fast.
http://www.mail-archive.com/py******.../msg01225.html
There *are* alternative ways to do some things in the vm and Joel
is simply wrong:)


The smiley doesn't negate your silliness. Joel explicitly talked about
complex, mature, debugged, in-use systems, with actual or potential
competitive replacements, not laboratory development toys and prototypes.
Three years between releases *is* a long time. And the Python codebase
*does* embody a lot of hard-won knowledge, hardly available elsewhere, that
should not be tossed. A particular example is how to get certain things to
work across numerous platforms in the face of ambuiguity and 'not defined'
behavior in the C standard and implementations.

Its possible that some future reference Python will be based on a new,
built-from-scratch core developed *in parallel* with current Python. But
that will only happen after it has been tested on a hundred different
platforms with a hundred different applications and packages (as happens,
more or less, with each new CPython release).

Terry J. Reedy

Jul 19 '05 #13

P: n/a
> I am not very interested on C compatibility.

That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon. What ultimately killed it was the problem of having
a decent library. You don't have to make the C library
compatibility totally high performance, but having any form of
it there will make adoption by others easier.

Roger
Jul 19 '05 #14

P: n/a
"Roger Binns" <ro****@rogerbinns.com> writes:
That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon.
I think it would be enough to retarget SWIG.
What ultimately killed it was the problem of having a decent
library.


Nah, I think people just lost interest. Otherwise, why isn't Jython
dead?
Jul 19 '05 #15

P: n/a
Paul Rubin wrote:
"Roger Binns" <ro****@rogerbinns.com> writes:
That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon.

I think it would be enough to retarget SWIG.

What ultimately killed it was the problem of having a decent
library.

Nah, I think people just lost interest. Otherwise, why isn't Jython
dead?


I thought Jython had full access to the entire library of standard Java,
plus at least the Python parts of the Python standard library...

-Peter
Jul 19 '05 #16

P: n/a
"Paul Rubin" <http://ph****@NOSPAM.invalid> wrote in message news:7x************@ruckus.brouhaha.com...
"Roger Binns" <ro****@rogerbinns.com> writes:
That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon.


I think it would be enough to retarget SWIG.


That is also a lot of work, and a lot of toolkits that use Swig
for Python bindings are very strongly tied to the Swig-Python
internals. For example I doubt there are many where you could
tell Swig to output Java and it would just work. (I know none
of mine would work, nor would wxPython).
What ultimately killed it was the problem of having a decent
library.


Nah, I think people just lost interest. Otherwise, why isn't Jython
dead?


Err, you proved my point! Prothon was fine at the VM level. The
author couldn't figure out how to get a decent sized "standard"
library, and ultimately ended up abandoning his VM for .Net since
that gets you a large standard library. Jython also gets a large
standard library from the Java ones.

Roger
Jul 19 '05 #17

P: n/a
"Roger Binns" <ro****@rogerbinns.com> writes:
Err, you proved my point! Prothon was fine at the VM level. The
author couldn't figure out how to get a decent sized "standard"
library, and ultimately ended up abandoning his VM for .Net since
that gets you a large standard library. Jython also gets a large
standard library from the Java ones.


What kind of stuff is in the existing Python C library that couldn't
be reimplemented or retargeted pretty easily? Most of it is either
wrappers for standard C functions (system calls), stuff that shouldn't
be in C in the first place (the Python compiler and interpreter), etc.

I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.
Jul 19 '05 #18

P: n/a

Paul Rubin wrote:
"Roger Binns" <ro****@rogerbinns.com> writes:
Err, you proved my point! Prothon was fine at the VM level. The
author couldn't figure out how to get a decent sized "standard"
library, and ultimately ended up abandoning his VM for .Net since
that gets you a large standard library. Jython also gets a large
standard library from the Java ones.
What kind of stuff is in the existing Python C library that couldn't
be reimplemented or retargeted pretty easily? Most of it is either
wrappers for standard C functions (system calls), stuff that

shouldn't be in C in the first place (the Python compiler and interpreter), etc.

Delete the "standard" and You still obtain huge librarys for .Net, Java
and Python. I also regret that Prothon starved in infancy but it might
be exeggerated to demand that each language designer or one of his
apostels should manage a huge community that enjoys doing redundant
stuff like writing Tk-bindings, regexp-engines and all that.
I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.


PyPy is a beast on it's own and provides interesting new ideas to
Python as well as runtime design in general. But I'm not quite shure
that it will stay in the boundarys of the current Python community
process and will subordinate to Guidos will. As a Marxist would say:
who controls the runtime also controls the language. Sooner or later
Python will go the LISP way of having a standardized "Common-Python" (
std_objectspace) and a number of dialects and DSLs running in their own
derived object spaces. Maybe Python 3000 is an illusion and will fade
away like a Fata Morgana the closer we seem come.

Kay

Jul 19 '05 #19

P: n/a

"Paul Rubin" <http://ph****@NOSPAM.invalid> wrote in message news:7x************@ruckus.brouhaha.com...
"Roger Binns" <ro****@rogerbinns.com> writes:
What kind of stuff is in the existing Python C library that couldn't
be reimplemented or retargeted pretty easily? Most of it is either
wrappers for standard C functions (system calls), stuff that shouldn't
be in C in the first place (the Python compiler and interpreter), etc.
Some examples are gui toolkits (eg wxPython), SSL (eg M2Crypto, pyopenssl)
and database (pysqlite, APSW). These aren't in the shipped with Python
library but are widely used.
I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.


You don't have to stay backwards compatible. It is best to provide
some sort of way of using the old extensions even if it is suboptimal
(eg some sort of mapping shim). The extensions can later be improved,
but if there is no simple way of recompiling and running then it will
take a long time for all the extensions to get ported.

I already get burnt out on the matrix of CPython versions and different
platforms. Adding another interpretter would make life even harder
for extension authors.

Roger
Jul 19 '05 #20

P: n/a
"Kay Schluehr" <ka**********@gmx.net> writes:
Delete the "standard" and You still obtain huge librarys for .Net, Java
and Python. I also regret that Prothon starved in infancy but it might
be exeggerated to demand that each language designer or one of his
apostels should manage a huge community that enjoys doing redundant
stuff like writing Tk-bindings, regexp-engines and all that.
Maybe there needs to be some kind of standard FFI (foreign function
interface) like Lisp implementations have.
Sooner or later Python will go the LISP way of having a standardized
"Common-Python" ( std_objectspace) and a number of dialects and DSLs
running in their own derived object spaces. Maybe Python 3000 is an
illusion and will fade away like a Fata Morgana the closer we seem come.


Right now Python feels about like Maclisp in the 1970's must have felt
(that was before my time so I can't know for certain). Lots of
hackerly excitement, lots of cruft. It needs to ascend to the next
level. PyPy looks like the best vehicle for that so far. See

http://catb.org/~esr/jargon/html/M/MFTL.html

for the canonical remark about languages that can't be used to
implement their own compilers. Python is fun and useful, but it
really isn't mature until PyPy is released for production use.
Jul 19 '05 #21

P: n/a
"Roger Binns" <ro****@rogerbinns.com> writes:
Some examples are gui toolkits (eg wxPython), SSL (eg M2Crypto, pyopenssl)
and database (pysqlite, APSW). These aren't in the shipped with Python
library but are widely used.
M2Crypto is a straightforward SWIG wrapper around OpenSSL, I thought.
I don't know about wxPython or pysqlite. It seems to me that some
kind of SQL client should be part of the stdlib. But why isn't a SWIG
wrapper enough? The CPython stdlib has for possibly good reasons
avoided SWIG but a new implementation doesn't need to.
You don't have to stay backwards compatible. It is best to provide
some sort of way of using the old extensions even if it is suboptimal
(eg some sort of mapping shim).
Yeah, there's something to be said for that, if it doesn't cause too
much pain.
I already get burnt out on the matrix of CPython versions and different
platforms. Adding another interpretter would make life even harder
for extension authors.


There's already Jython and Python.net is on its way. This is
something Lisp users have dealt with for decades and they've developed
workable machinery for it. Maybe Python can adopt some of those
methods.
Jul 19 '05 #22

P: n/a
Roger Binns wrote:
I am not very interested on C compatibility.

That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon. What ultimately killed it was the problem of having
a decent library. You don't have to make the C library
compatibility totally high performance, but having any form of
it there will make adoption by others easier.


There are two kinds of C modules: those that do have a knowledge
of the C API (like sre, tkinter, etc) and those that are just C/C++
libraries which are simply wrapped as modules. For the latter there
are two solutions besides adding a wrapper which makes pyvm appear
as libpython:
- an advanced ctypes module which will make dlopening libraries
and wrapping their symbols behind python functions, a matter of
python code. I'm considering this approach to provide things
like 'raw_input'.
- hacking SWIG. Shouldn't be too hard and will instantly give
us access to wx, qt, etc.

The thing is that the C API of pyvm is IMHO superior and much more fun.
You can wrap the entire sockets module in a couple of hours and also
enjoy it. I wish I could clone myself to port the entire std library
to pyvm -- so much fun it is:)
thanks,

Stelios
Jul 19 '05 #23

P: n/a
Paul Rubin wrote:

I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.


I didn't know much about PyPy. It seems that pyvm is *exactly* what
pypy needs to boost its performance. Does pypy has the vm in python
as well? Does pypy have a compiler that produces 2.4 bytecodes?

I think the next step in speeding up python (not that it's slow, I'm
in the group of people that don't think Python is slow), is the AST
compiler. An AST compiler for CPython is in the works AFAIK, but it
would be more powerful in written in python. It would be also easy
to have 'alternative' compilers that, for example are not whitespace
sensitive, or the code looks like perl :), but they all produce python
bytecode.
Stelios
Jul 19 '05 #24

P: n/a
In article <m1************@home.rogerbinns.com>,
Roger Binns <ro****@rogerbinns.com> wrote:
could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter?


I'd also be curious to know if the performance gains would remain
once it gets fleshed out with things like closures, long numbers,
new style classes and a C library compatibility shim.

Roger


And Unicode. And ...
Jul 19 '05 #25

P: n/a
Stelios Xanthakis wrote:
There are two kinds of C modules: those that do have a knowledge
of the C API (like sre, tkinter, etc) and those that are just C/C++
libraries which are simply wrapped as modules. For the latter there
are two solutions besides adding a wrapper which makes pyvm appear
as libpython:
- an advanced ctypes module which will make dlopening libraries
and wrapping their symbols behind python functions, a matter of
python code. I'm considering this approach to provide things
like 'raw_input'.
- hacking SWIG. Shouldn't be too hard and will instantly give
us access to wx, qt, etc.


No, writing a pyvm modules for SWIG won't give you compatibility with
most existing SWIG wrappers. Such wrappers are very, very rarely "pure
SWIG". Almost all nontrivial wrappers include ad hoc typemaps that use
the Python C API.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

Jul 19 '05 #26

P: n/a
Stelios Xanthakis <sx****@ceid.upatras.gr> writes:
I didn't know much about PyPy. It seems that pyvm is *exactly* what
pypy needs to boost its performance. Does pypy has the vm in python
as well? Does pypy have a compiler that produces 2.4 bytecodes?


Pypy makes native machine code, not bytecode.
Jul 19 '05 #27

P: n/a
Stelios Xanthakis <sx****@ceid.upatras.gr> writes:
- hacking SWIG. Shouldn't be too hard and will instantly give
us access to wx, qt, etc.


You can't assume that because some package is a C/C++ library wrapped
for Python that it uses SWIG. pyqt, for example, doesn't use SWIG at
all. It uses SIP, which is considerably more complicated than SWIG.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 19 '05 #28

P: n/a
Paul Rubin wrote:
PyPy looks like the best vehicle for that so far. See

http://catb.org/~esr/jargon/html/M/MFTL.html

for the canonical remark about languages that can't be used to
implement their own compilers.


Which makes it clear that the remark is only intended
to apply to *compiled* languages.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Jul 19 '05 #29

P: n/a
Greg Ewing <gr**@cosc.canterbury.ac.nz> writes:
PyPy looks like the best vehicle for that so far. See
http://catb.org/~esr/jargon/html/M/MFTL.html
for the canonical remark about languages that can't be used to
implement their own compilers.


Which makes it clear that the remark is only intended to apply to
*compiled* languages.


Yes, there are several Python compilers already, (Psyco, Jython followed
up by your favorite JIT compiler, IronPython (similarly), Pyrex (not
quite the same input language but I think it should be counted), etc.
It's true that CPython doesn't have a compiler and that's a serious
deficiency. A lot of Python language features don't play that well
with compilation, and that's often unnecessary. So I hope the baseline
implementation changes to a compiled one before the language evolves
too much more.
Jul 19 '05 #30

P: n/a

Paul Rubin wrote:
"Kay Schluehr" <ka**********@gmx.net> writes:
Delete the "standard" and You still obtain huge librarys for .Net, Java and Python. I also regret that Prothon starved in infancy but it might be exeggerated to demand that each language designer or one of his
apostels should manage a huge community that enjoys doing redundant
stuff like writing Tk-bindings, regexp-engines and all that.
Maybe there needs to be some kind of standard FFI (foreign function
interface) like Lisp implementations have.
Sooner or later Python will go the LISP way of having a standardized "Common-Python" ( std_objectspace) and a number of dialects and DSLs running in their own derived object spaces. Maybe Python 3000 is an
illusion and will fade away like a Fata Morgana the closer we seem

come.
Right now Python feels about like Maclisp in the 1970's must have felt (that was before my time so I can't know for certain). Lots of
hackerly excitement, lots of cruft. It needs to ascend to the next
level. PyPy looks like the best vehicle for that so far. See

http://catb.org/~esr/jargon/html/M/MFTL.html

for the canonical remark about languages that can't be used to
implement their own compilers. Python is fun and useful, but it
really isn't mature until PyPy is released for production use.


Yes. What we are seeking for and this may be the meaning of Armins
intentiously provocative statement about the speed of running HLLs is a
successor of the C-language and not just another VM interpreter that is
written in C and limits all efforts to extend it in a flexible and
OO-manner. Python is just the most promising dynamic OO-language to
follow this target.

Ciao,
Kay

Jul 19 '05 #31

P: n/a
On 11 May 2005 19:48:42 -0700, Paul Rubin <http://ph****@NOSPAM.invalid> wrote:
Stelios Xanthakis <sx****@ceid.upatras.gr> writes:
I didn't know much about PyPy. It seems that pyvm is *exactly* what
pypy needs to boost its performance. Does pypy has the vm in python
as well? Does pypy have a compiler that produces 2.4 bytecodes?


Pypy makes native machine code, not bytecode.

I thought they hoped to experiment with many targets, which would presumably
include generating code for different CPUs and VMs in various representations,
not necessarily just machine code via low level C or ASM, but I haven't checked
current status re that.

Regards,
Bengt Richter
Jul 19 '05 #32

P: n/a
Kay Schluehr wrote:

Yes. What we are seeking for and this may be the meaning of Armins
intentiously provocative statement about the speed of running HLLs is a
successor of the C-language and not just another VM interpreter that is
written in C and limits all efforts to extend it in a flexible and
OO-manner. Python is just the most promising dynamic OO-language to
follow this target.


Bytecode engine is the best method for dynamic code execution
("exec", eval, etc). A low level OOP language would be very suitable
for a python VM.

pyvm has that. A big part of it is written in "lightweight C++" [1].
That makes it less portable as the lwc preprocessor is using GNU-C
extensions. However, it's the same extensions also used by the linux
kernel and AFAIK the intel compiler supports them too.

So probably the bigger "competitor" of pyvm is boost-python.
And that's one reason the release of the source is stalled until it
gets better.
Stelios

[1] http://students.ceid.upatras.gr/~sxanth/lwc/
Jul 19 '05 #33

P: n/a
- hacking SWIG. Shouldn't be too hard and will instantly give
us access to wx, qt, etc.


Mike> You can't assume that because some package is a C/C++ library
Mike> wrapped for Python that it uses SWIG. pyqt, for example, doesn't
Mike> use SWIG at all. It uses SIP, which is considerably more
Mike> complicated than SWIG.

PyGTK uses its own lisp-y thing as the wrapper IDL. Actually, it's specific
to GTK and is AFAIK is used to wrap GTK by languages besides Python.

Skip

Jul 19 '05 #34

P: n/a
[Paul Rubin]
It's true that CPython doesn't have a compiler and that's
a serious deficiency.


Hi, Paul. I did not closely follow all of the thread, so maybe my
remark below, only repeats what others might have said and I missed?

Deep down, why or how not having a [traditional, to-native-code]
compiler is a deficiency for CPython? We already know that such a beast
would not increase speed so significantly, while using much more memory.

It is true that a standard traditional compiler for CPython would allow
one would be to check the box:

[x] has a compiler

in the fashion of the day language information sheet, and for some
readers, not having that box checked is a deficiency in itself. :-)

So far, it seems that the only way to get speed is to attach static
type information to some variables. Some compilation avenues do it
through information added either in Python source code or in extraneous
declarative files, other approaches do it by delaying compilation
until such information is discovered at run-time. The former taints
the purity of real CPython as the only source. The later often shows
spectacular speed gain, but not always, and may bloat size unboudedly.

--
François Pinard http://pinard.progiciels-bpi.ca
Jul 19 '05 #35

P: n/a
Paul Rubin wrote:
Yes, there are several Python compilers already ... It's true that CPython doesn't have a compiler and that's a serious
deficiency. A lot of Python language features don't play that well
with compilation, and that's often unnecessary. So I hope the baseline
implementation changes to a compiled one before the language evolves
too much more.


Years ago, presented at one of the Python conferences, was a
program to generate C code from the byte code. It would still
make calls to the Python run-time library (just as C does to
its run-time library).

The presenter did some optimizations, like not decref at the
end of one instruction when the next immediately does an incref
to it. The conclusion I recall was that it wasn't faster -
at best a few percent - and there was a big memory hit because
of all the duplicated code. One thought was that the cache miss
caused some of the performance problems.

Does that count as a compiler?

Andrew
da***@dalkescientific.com

Jul 19 '05 #36

P: n/a
Stelios Xanthakis wrote:
Kay Schluehr wrote:

Yes. What we are seeking for and this may be the meaning of Armins
intentiously provocative statement about the speed of running HLLs is a successor of the C-language and not just another VM interpreter that is written in C and limits all efforts to extend it in a flexible and
OO-manner. Python is just the most promising dynamic OO-language to
follow this target.

Bytecode engine is the best method for dynamic code execution
("exec", eval, etc). A low level OOP language would be very suitable
for a python VM.


Why this? eval() consumes a string, produces a code object and executes
it. Wether the code-object is bytecode or a chunk of machine code makes
a difference in the runtime but does not alter the high level
behavioural description of eval(). In either way the compile() function
behind eval is a JIT.
pyvm has that. A big part of it is written in "lightweight C++" [1].
That makes it less portable as the lwc preprocessor is using GNU-C
extensions.
Hmmm... I'm not amazingly happy that it is not ANSI-C++. To be honest I
don't want to learn lw-C++ and the peculiaritys of the translator and
debuging through translated code. This may be an interesting study on
it's own but it is just not me who is interested in it. Using an
experimental language to write an experimental VM is clearly out of my
motivation ( only as a remark for the quest of contribution not as a
general criticism ).
However, it's the same extensions also used by the linux
kernel and AFAIK the intel compiler supports them too.

So probably the bigger "competitor" of pyvm is boost-python.


How could boost-Python be a "competitor"? Isn't it just an ANSI
C++-binding that relates heavily on templates that are not even
supported by lw-C++?

Regards,
Kay

Jul 19 '05 #37

P: n/a
François Pinard <pi****@iro.umontreal.ca> writes:
Deep down, why or how not having a [traditional, to-native-code]
compiler is a deficiency for CPython? We already know that such a beast
would not increase speed so significantly, while using much more memory.
I'd say the opposite. The 4x speedup from Psyco is quite significant.
The speedup would be even greater if the language itself were more
compiler-friendly.
So far, it seems that the only way to get speed is to attach static
type information to some variables.
No of course not, there's quite a bit of overhead in interpretation in
general. Plus, having to do a dictionary lookup for 'bar' in every
call like foo.bar() adds overhead and I don't think I'd call fixing
that as similar to adding static type info.
Some compilation avenues do it through information added either in
Python source code or in extraneous declarative files, other
approaches do it by delaying compilation until such information is
discovered at run-time.
Also, lots of times one can do type inference at compile time.
The former taints the purity of real CPython as the only source.


I don't understand this. What purity? Why is real CPython the
only source? There are dozens of C compilers and none of them is
the "only source". Why should Python be different?
Jul 19 '05 #38

P: n/a
Andrew Dalke <da***@dalkescientific.com> writes:
Years ago, presented at one of the Python conferences, was a program
to generate C code from the byte code.... The conclusion I recall
was that it wasn't faster - at best a few percent - and there was a
big memory hit because of all the duplicated code. One thought was
that the cache miss caused some of the performance problems. Does
that count as a compiler?


I would say it counts as a compiler and that other languages have
used a similar compilation approach and gotten much better speedups.
For example, Java JIT compilers. The DEC Scheme-to-C translator
and Kyoto Common Lisp also produced C output from their compilers
and got really significant speedups. Part of the problem may be
with the actual Python language. One of the reasons for wanting
real compilation as a high priority, is that the presence of a
compiler will affect future evolution of the language, in a way
that makes it more conducive to compilation.

Despite the shrieks of the "Python is not Lisp!" crowd, Python
semantics and Lisp semantics aren't THAT different, and yet compiled
Lisp implementations com completely beat the pants off of interpreted
Python in terms of performance. I don't think Python can ever beat
carefully coded C for running speed, but it can and should aim for
parity with compiled Lisp.
Jul 19 '05 #39

P: n/a
> I don't think Python can ever beat
carefully coded C for running speed, but it can and should aim for
parity with compiled Lisp.


But common lisp compilers often beat C compilers in speed for similar
tasks
of moderate complexity. In particular, CMUCL beats GCC in numerical
computations.

David

Jul 19 '05 #40

P: n/a
[Paul Rubin]
François Pinard <pi****@iro.umontreal.ca> writes:
Deep down, why or how not having a [traditional, to-native-code]
compiler is a deficiency for CPython? We already know that such a
beast would not increase speed so significantly, while using much
more memory.
I'd say the opposite. The 4x speedup from Psyco is quite significant.
The speedup would be even greater if the language itself were more
compiler-friendly.
Psyco is not a traditional compiler. Compilation occurs at run-time, as
typing information is not available without observation.
So far, it seems that the only way to get speed is to attach static
type information to some variables. No of course not, there's quite a bit of overhead in interpretation in
general. Plus, having to do a dictionary lookup for 'bar' in every
call like foo.bar() adds overhead and I don't think I'd call fixing
that as similar to adding static type info.
The dictionary lookup does not occur for local variables. Changes are
planned in Python so lookups may be avoided in a few cases of global
variables. But until (and despite!) such changes occur, compilation
does not buy you much, here. This has been much studied and debated.
Some compilation avenues do it through information added either in
Python source code or in extraneous declarative files, other
approaches do it by delaying compilation until such information is
discovered at run-time. Also, lots of times one can do type inference at compile time.
Yes, particularily in simple or small programs. I vaguely remember
that the unborn Viper (or Vyper?) was promising compile time type
inference. It seems that so far, in many years, no one really succeeded
in demonstrating a definite advantage in this direction, at least, not
enough for the community to follow in.
The former taints the purity of real CPython as the only source.

I don't understand this. What purity? Why is real CPython the
only source? There are dozens of C compilers and none of them is
the "only source". Why should Python be different?


Because it is. There are not dozens of Python compilers. A few, maybe.

Pyrex gives me a lot of speed, given I taint my Python sources with
cdefs, ctypes, etc. Oh, I'm absolutely happy with Pyrex when I starve
for speed, but I know and you know that we are not writing pure Python
then. And if, for not touching sources, one supplements them with
separate declarative files, my Python code may look pure, but I have to
maintain those files as well, and Python is not my only source anymore.

So, to get back to the origin of our argument, I'm still not tempted
to think that Python not having a compiler makes it deficient, because
having a compiler (at least in the traditional acceptation of the term)
would not buy it much enough.

--
François Pinard http://pinard.progiciels-bpi.ca
Jul 19 '05 #41

P: n/a
Paul Rubin wrote:
Despite the shrieks of the "Python is not Lisp!" crowd, Python
semantics and Lisp semantics aren't THAT different, and yet compiled
Lisp implementations com completely beat the pants off of interpreted
Python in terms of performance.


I know little about Lisp compilation, so I could be mistaken, but I was
under the impression that one of the differences between Python & Lisp
is directly relevant to compilation issues.

Python, as a procedural language, makes extensive use of globals &
mutable variables. Not only can the contents of a variable change, but
that change can non-explicitly affect a function in a "remote" part of
the program, hence the requirement for the Global Interpreter Lock.
Tracking how changes propagate in Python is non-trivial, as evidenced by
the difficulty of replacing the GIL with a less "obtrusive" alternative.

IIUC, in Lisp, as a functional language, "all politics is local."
Global-like variables are much rarer, and mutability is severely
limited. In this case it is much easier to track how a piece of code
alters the program's state - the propagation of those changes are
handled explicitly, and once you have a reference to a piece of data,
you don't have to worry about some other function changing the value on
you. As such, it's a lot easier to write an optimizing compiler - you
can place much greater limits on what is known at compile time.

It's been quite a while since I've looked at Lisp/functional languages,
though, so I could be misunderstanding what I remember.
Jul 19 '05 #42

P: n/a
Rocco Moretti <ro**********@hotpop.com> writes:
Python, as a procedural language, makes extensive use of globals &
mutable variables.... IIUC, in Lisp, as a functional language, "all
politics is local." Global-like variables are much rarer, and
mutability is severely limited.


Some people write Lisp code in a functional style, but not everyone.
Lisp provides mutable objects as much as Python does--maybe more,
for example, Lisp strings are mutable. Overall I'd say there's not
much difference between Lisp and Python in this regard. Lisp lets
the programmer supply static type declarations that the compiler
can use, but those are optional. When you use them, you get faster
output code.
Jul 19 '05 #43

P: n/a
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
Andrew Dalke <da***@dalkescientific.com> writes:
Years ago, presented at one of the Python conferences, was a program
to generate C code from the byte code.... The conclusion I recall
was that it wasn't faster - at best a few percent - and there was a
big memory hit because of all the duplicated code. One thought was
that the cache miss caused some of the performance problems. Does
that count as a compiler?
I would say it counts as a compiler and that other languages have
used a similar compilation approach and gotten much better speedups.
For example, Java JIT compilers. The DEC Scheme-to-C translator
and Kyoto Common Lisp also produced C output from their compilers
and got really significant speedups. Part of the problem may be
with the actual Python language.


The DEC Scheme->C compiler (yes, it was an honest to gods compiler -
it just generated C instead of machine code) doesn't sound very
similar to generating C code from byte code. The latter sounds like a
straightforward translation of byte code to C. The Scheme->C
compiler, on the other hand, used all the optimizations known to LISP
compiler writers. It would, with the right settings, produce code that
was comparable to hand-coded C. So unless the Python compiler that
produced the byte code in the first place does all those optimizations
(which I don't know), you're not going to get results that compare
with the Scheme->C system.
Despite the shrieks of the "Python is not Lisp!" crowd, Python
semantics and Lisp semantics aren't THAT different, and yet compiled
Lisp implementations com completely beat the pants off of interpreted
Python in terms of performance. I don't think Python can ever beat
carefully coded C for running speed, but it can and should aim for
parity with compiled Lisp.


There are people out there who claim that compiling carefully coded
LISP can compete with compiling carefully coded C. They use such
systems in real-world, time-critical applications. The thing is, to
get that performance, they had to go back and tell the compiler what
the types of all the variables are, and disable all the runtime type
checking.

So if this is a real goal, you now have some idea of what to look
forward to in Python's future.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 19 '05 #44

P: n/a
Hi,

Kay Schluehr wrote:

Why this? eval() consumes a string, produces a code object and executes
it. Wether the code-object is bytecode or a chunk of machine code makes
a difference in the runtime but does not alter the high level
behavioural description of eval(). In either way the compile() function
behind eval is a JIT.


It makes a difference because
1) You depend on a compiler. Especially if you want to optimize the
machine code you depend on a big program (no tcc for example)
2) Optimizing machine code, needs a lot of code/time.
3) What about debugging info? Dwarf?
4) What if there's a new architecture?

The bytecode compiler is *much* faster to produce the bytecode assembly.
Also, for the other part of the thread, I think that bytecode may
be in fact faster than machine code JIT. Here is a theory:

Suppose that for each algorithm there is the "ideal implementation"
which executes at the speed limit where it can't be done any faster.
For small algorithms the speed limit may be known but for more
complex programs, it's just a theoretical limit. Now for such very
big programs bytecode has the advantage that it achieves very good
code re-use; everything is those 400kB of the libpython core and that
does not increase with the size of the program. In this case, bytecode
is the driver that operates the buldozer (the buldozer being C). So the
theory is that for very complex programs a bytecode+core-library is
closer to the ideal implementation than a JIT which produces/compiles
megabytes of machine code.

Evidence for that may be that all those JIT efforts don't get any
great speed ups (psyco is different as it exposes static-ness).

Of course, if we care about speed we'll use Pyrex to convert some
heavy routines to C. For example if we need FFT, it's madness to
do it in the HLL. Make it part of the buldozer.

Stelios
Jul 19 '05 #45

P: n/a
Stelios Xanthakis wrote:
Kay Schluehr wrote:

Yes. What we are seeking for and this may be the meaning of Armins
intentiously provocative statement about the speed of running HLLs is a
successor of the C-language and not just another VM interpreter that is
written in C and limits all efforts to extend it in a flexible and
OO-manner. Python is just the most promising dynamic OO-language to
follow this target.

Bytecode engine is the best method for dynamic code execution
("exec", eval, etc). A low level OOP language would be very suitable
for a python VM.

pyvm has that. A big part of it is written in "lightweight C++" [1].


Realy ? I have downloaded the lwc distribution and checked it out.
It was a surprise that none of the examples are working.
I'm using SuSE 9.0 with gcc 3.3.1 ...

Is there a working version of lwc ???

Regards

Armin

That makes it less portable as the lwc preprocessor is using GNU-C
extensions. However, it's the same extensions also used by the linux
kernel and AFAIK the intel compiler supports them too.

So probably the bigger "competitor" of pyvm is boost-python.
And that's one reason the release of the source is stalled until it
gets better.
Stelios

[1] http://students.ceid.upatras.gr/~sxanth/lwc/

Jul 19 '05 #46

P: n/a
Armin Steinhoff wrote:

pyvm has that. A big part of it is written in "lightweight C++" [1].

Realy ? I have downloaded the lwc distribution and checked it out.
It was a surprise that none of the examples are working.
I'm using SuSE 9.0 with gcc 3.3.1 ...


:(
Is there a working version of lwc ???


pyvm is written in lwc-2.0 which is not yet released because
nobody's using it.
Stelios
Jul 19 '05 #47

P: n/a
Stelios Xanthakis wrote:

Also, for the other part of the thread, I think that bytecode may
be in fact faster than machine code JIT.


Forgot to add: It depends of course on how good is the bytecode.
IMO Python's bytecode is pretty good for its purpose which is
executing a dynamic language with dynamic types and namespaces.

Also, inside pyvm some things are done with internal bytecode
objects. One such thing is "list(generator)". It has been
proved that doing this in bytecode is faster than C.

Stelios
Jul 19 '05 #48

P: n/a
Stelios Xanthakis wrote:
Armin Steinhoff wrote:

pyvm has that. A big part of it is written in "lightweight C++" [1].
Realy ? I have downloaded the lwc distribution and checked it out.
It was a surprise that none of the examples are working.
I'm using SuSE 9.0 with gcc 3.3.1 ...


:(
Is there a working version of lwc ???


pyvm is written in lwc-2.0 which is not yet released because
nobody's using it.


As you mentioned it ... lwc-2.0 is used for pyvm. So it is used :)

Do you have an idea when lwc-2.0 will be releast ?

Everyone who are interested in pyvm will need it ...
-- Armin



Stelios

Jul 19 '05 #49

P: n/a
Armin Steinhoff wrote:
Is there a working version of lwc ???


pyvm is written in lwc-2.0 which is not yet released because
nobody's using it.

As you mentioned it ... lwc-2.0 is used for pyvm. So it is used :)

Do you have an idea when lwc-2.0 will be releast ?

Everyone who are interested in pyvm will need it ...


It will be included together with pyvm. Normally if you want to just
compile pyvm you do not need lwc, it's a preprocessor that generates C
from C++ and the C for pyvm will be pre-generated. You'd need lwc for
modifying/hacking pyvm.
Stelios

Jul 19 '05 #50

53 Replies

This discussion thread is closed

Replies have been disabled for this discussion.