pyvm -- faster python

[Stelios Xanthakis]

Hi.

pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/
Facts about pyvm:
- It's FAST. According to the "cooked-bench" benchmark suite it finishes
in 55% of the time python takes;)
- It's SMALL. Currently the source code is under 15k lines with the
builtin modules.
- It's new. Uses no code from CPython.
- It's incomplete. Not even near the stability and quality of python.
It needs A LOT of work before it can be compared to CPython.
Moreover, at the time it lacks many things like closures, long numbers
new style classes, etc.
- It's incompatible with CPython. Not all programs run.
- The C API is incompatible. You can't run C modules (a thin wrapper to
make pyvm appear as libpython *could* be possible but not a goal
AFAIC)
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!

Cheers,

Stelios

[Terry Reedy]

"Stelios Xanthakis" <sx****@ceid.upatras.gr> wrote in message
news:42**************@ceid.upatras.gr...

Also, for the other part of the thread, I think that bytecode may
be in fact faster than machine code JIT. Here is a theory: Suppose that for each algorithm there is the "ideal implementation"
which executes at the speed limit where it can't be done any faster. ....Now for such very
big programs bytecode has the advantage that it achieves very good
code re-use; everything is those 400kB of the libpython core and that
does not increase with the size of the program. In this case, bytecode
is the driver that operates the buldozer (the buldozer being C).
This, of course, is exactly what CPython does. Bytecodes index into a
switch table that jumps to interface code that calls C routines, most of
which have been optimized, on and off, for about 15 years.
So the
theory is that for very complex programs a bytecode+core-library is
closer to the ideal implementation than a JIT which produces/compiles
megabytes of machine code.

Evidence for that may be that all those JIT efforts don't get any
great speed ups (psyco is different as it exposes static-ness).

Of course, if we care about speed we'll use Pyrex to convert some
heavy routines to C. For example if we need FFT, it's madness to
do it in the HLL. Make it part of the buldozer.

And of course, it already is (replacing 'bulldozer' with 'set of
construction equipment'), under the name NumPy or Numarray. And both of
these can take advantage of an Atlas implementation of the BLAS, when
available, that is tuned to a specific platform with the intention of being
the 'ideal implementation' of the Basic Linear Algebra Subroutines for that
platform.

So I congratulate you on conceptually re-inventing Python + C extensions.
But I haven't understood yet what conceptual variations you have or plan to
make on the current implementation.

Current CPyVM simulates a stack machine. This has the virtual of
simplicity and clarity even for someone like me who has never seriously
written assembler. People have made proposals to switch to simulating a
(more complex) register machine, with a new set of bytecodes, that
theoretically would be faster. Do you have any opinion on this issue? You
seem to like the current byte code set while wanting to run faster.

Terry J. Reedy

[Susan A. Smith]

I have read about parrot. How is that progressing?

Stelios Xanthakis wrote:

Hi.

pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/
Facts about pyvm:
- It's FAST. According to the "cooked-bench" benchmark suite it finishes
in 55% of the time python takes;)
- It's SMALL. Currently the source code is under 15k lines with the
builtin modules.
- It's new. Uses no code from CPython.
- It's incomplete. Not even near the stability and quality of python.
It needs A LOT of work before it can be compared to CPython.
Moreover, at the time it lacks many things like closures, long numbers
new style classes, etc.
- It's incompatible with CPython. Not all programs run.
- The C API is incompatible. You can't run C modules (a thin wrapper to
make pyvm appear as libpython *could* be possible but not a goal
AFAIC)
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!

Cheers,

Stelios

[Michael Sparks]

Stelios Xanthakis wrote:
....

- It's incompatible with CPython. Not all programs run. .... - The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!
Whatever the merits of a system like this, a closed system with bugs
(read: incompatibility with the standard python) will be considered to
"not work" unfortunately - unless people can fix the (or get fixed)
bugs they encounter.

Releasing as closed source guarantees people will come back to you to
get the code fixed, or abandon using your code in favour of CPython,
Jython or IronPython if they want speed now. (Or PyPy if they want
speed later :-)

Releasing open source means that people *may* fix their own bugs, or
abandon the code.

In your release notes you state: WHERE IS THE SOURCE?:
The source code of pyvm is not yet released. Whether it will and
'when' depends on the interest of the community. Right now I cannot
afford the maintainance costs (fix the source, remove/insert comments,
write docs, fix known but harmless bugs, process bug reports, etc).
I'm not making any money from pyvm, but at least I'd like to avoid
paying for it too!

Whilst that's fair enough - it's your code, your decision - it might be
worth considering that if these are your only objections consider this:
* Interest from the community is likely to be low unless you release
the source. It will be an interesting curio, but no more than that.
* Releasing as open source does NOT imply you have to support (or
market) the code - it's simply releasing.
* If the code isn't stable, bear in mind that the existing python
test suite can largely be used to test your VM and improve it - if
you have something that works and you intend at *some* point to
release the code as open source the sooner you do so, the faster
your project *may* mature.

At the end of the day though, it's your code, you choose what to do with
it. Personally I find your project curious, and if you had fun creating
the project (or its useful in some other way), then it strikes me as a
positive thing (your release URL implies you're a student!).

Best Regards,
Michael.
--
Mi************@rd.bbc.co.uk
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

[Stelios Xanthakis]

Hi Michael

[...]

Releasing open source means that people *may* fix their own bugs, or
abandon the code.

[...]

I agree with all the points made.

Moreover let me add that "code is one expression of a set of good
ideas", and ideas want to be free! ;)

I've decided to release the source code of pyvm as soon as it's ready.

Right now *it doesn't* make much sense to give the source because it is
still at an early development stage. Even if I did and people sent
patches they wouldn't apply because I still make big changes to the
architecture of it. I'd like to keep it in this status where I can
modify the structure of the program until it becomes really
developer-friendly. And IMO it doesn't make sense to release incomplete
open source projects: either give something that's good and people can
happily hack, or don't do it at all. Giving out the source of an
unstable project will most likely harm it (see CherryOS incidents).

The bottomline is that I estimate that pyvm will be ready within
the summer.
Thanks,

Stelios