Squisher -- a lightweight, self-contained alternative to eggs?

[Adam Atlas]

I wrote this little program called Squisher that takes a ZIP file
containing Python modules and generates a totally self-contained .pyc
file that imports a specified module therein. (Conveniently, Python's
bytecode parser ignores anything after an end marker, and the
zipimport mechanism skips any non-ZIP data at the beginning of a
file!) For example, say you have a directory called 'foo', which
contains an __init__.py, which contains "print 'hello'; x = 123", and
a thing.py, which contains "y = 456". If you make a ZIP archive of
this and run it through Squisher, you'll get a single .pyc file which
can be imported by any Python installation anywhere just like any
other module, without requiring users to install any supporting
mechanisms (like setuptools), and giving all the flexibility and
stability of zipimport. In other words, you can do this simply by
dropping the foo.pyc file into a directory:

>>import foo

hello

>>foo.x

123

>>from foo.thing import y
y

456

Of course, this is a stupid and useless example, but you can do it
with any package you could use as an egg, yet without requiring people
to install setuptools (or any other external glue) to use it. I've
tested it with several large packages, including SQLAlchemy.

Right now I'm just testing and polishing up the code... in the
meantime, any comments?

[Bruno Desthuilliers]

Adam Atlas a écrit :
(snip)

If you make a ZIP archive of
this and run it through Squisher, you'll get a single .pyc file which
can be imported by any Python installation anywhere just like any
other module, without requiring users to install any supporting
mechanisms (like setuptools),

setuptools offers *much* more than a convenient way to package source code.

[Adam Atlas]

This could be easily made into a distutils extension (which was my
intention all along, though that's not implemented yet). That's not
the point. This is not intended as a "way to package source code".
It's analogous to bdist, not sdist. The convenience gain is for the
users, not (primarily) the developers, of the package.

To use an Egg, you have to either put it in your sys.path manually or
install setuptools and use it to install the Egg in some global scope.
The advantage of Squisher is that you can take a whole package, squish
it into one .pyc file, and distribute that. Then, anyone who downloads
it can get a simple module file that can be used anywhere a usual .py
file can, without editing sys.path or installing it somewhere.
(Actually, this could be combined with eggs, since they're just ZIP
files. You could use the same file as an egg or as a squished module
simply by changing its extension.)

....Or am I missing something obvious about setuptools that already
does this and making a fool of myself? :)

[Adam Atlas]

Ah... heh, sorry, I misread your message as "a much more convenient
way" rather than "much more than a convenient way". Anyway, I
understand that, and I do indeed find setuptools useful and use it on
a regular basis.

But my other points still stand. This would be a moot point if
setuptools were part of the standard library, but it's not, and I
don't see why people should have to bother installing it if they
simply want to try a small package. Look at Beautiful Soup, for
example. It's distributed as a single .py file, and that's great. With
most modules, all I want to do is download them and plop them into my
project directory. You can always copy it into site-packages if you
want to access it globally, and you can always unzip it if you need to
see the source.

So I *will* retract my statement that this could be an "alternative"
to eggs -- ideally, it would be an addition, since it doesn't break
compatibility at all. You could download an egg and rename it to .pyc,
and import it like any other module, and at any point later, you could
rename it back to .egg and use setuptools to install it if you wish.

[Stef Mientki]

Adam Atlas wrote:

Ah... heh, sorry, I misread your message as "a much more convenient
way" rather than "much more than a convenient way". Anyway, I
understand that, and I do indeed find setuptools useful and use it on
a regular basis.

But my other points still stand. This would be a moot point if
setuptools were part of the standard library, but it's not, and I
don't see why people should have to bother installing it if they
simply want to try a small package. Look at Beautiful Soup, for
example. It's distributed as a single .py file, and that's great. With
most modules, all I want to do is download them and plop them into my
project directory. You can always copy it into site-packages if you
want to access it globally, and you can always unzip it if you need to
see the source.

So I *will* retract my statement that this could be an "alternative"
to eggs -- ideally, it would be an addition, since it doesn't break
compatibility at all. You could download an egg and rename it to .pyc,
and import it like any other module, and at any point later, you could
rename it back to .egg and use setuptools to install it if you wish.

Good point to make these things much easier!

But possibly I'm the only Windows user here,
as I still find these talks all very difficult to understand,
and I really don't understand why all this complexity is necessary,
setuptools ? eggs ? zips ? pythonpaths ?

As a normal Windows user,
I'm used to run an install file,
and hit just 1 button.
As a normal Windows programmer,
I'm used to create a simple Inno-setup file,
and my users can behave as a simplistic and happy Windows user.

But I guess the needed complexity is all thanks to NIX ;-)

I think I never would have started with Python,
if I didn't bounced into the Enthought-edition.

--
cheers,
Stef Mientki
http://pic.flappie.nl

[MonkeeSage]

Adam,

Sounds like a nice idea to me. Pretty ingenious use of the zip/
bytecode headers and all too. Post a message when you release it
please.

Regards,
Jordan

[Bruno Desthuilliers]

Adam Atlas a écrit :

Ah... heh, sorry, I misread your message as "a much more convenient
way" rather than "much more than a convenient way".

!-)

(snip)

But my other points still stand.

Indeed.

[Bruno Desthuilliers]

Stef Mientki a écrit :

Adam Atlas wrote:

>Ah... heh, sorry, I misread your message as "a much more convenient
way" rather than "much more than a convenient way". Anyway, I
understand that, and I do indeed find setuptools useful and use it on
a regular basis.

But my other points still stand. This would be a moot point if
setuptools were part of the standard library, but it's not, and I
don't see why people should have to bother installing it if they
simply want to try a small package. Look at Beautiful Soup, for
example. It's distributed as a single .py file, and that's great. With
most modules, all I want to do is download them and plop them into my
project directory. You can always copy it into site-packages if you
want to access it globally, and you can always unzip it if you need to
see the source.

So I *will* retract my statement that this could be an "alternative"
to eggs -- ideally, it would be an addition, since it doesn't break
compatibility at all. You could download an egg and rename it to .pyc,
and import it like any other module, and at any point later, you could
rename it back to .egg and use setuptools to install it if you wish.

Good point to make these things much easier!

But possibly I'm the only Windows user here,

I don't think so.

as I still find these talks all very difficult to understand,
and I really don't understand why all this complexity is necessary,
setuptools ? eggs ? zips ? pythonpaths ?

As a normal Windows user,
I'm used to run an install file,
and hit just 1 button.

As a normal *n*x user, I'm used to run a single command line. Don't even
have to hit a button !-)

But I guess the needed complexity is all thanks to NIX ;-)

FUD.

[Paul Boddie]

Stef Mientki wrote:

>
As a normal Windows user,
I'm used to run an install file,
and hit just 1 button.
As a normal Windows programmer,
I'm used to create a simple Inno-setup file,
and my users can behave as a simplistic and happy Windows user.

But I guess the needed complexity is all thanks to NIX ;-)

"Au contraire, as they say in England." - Steve X. Citement

Most UNIX-based (or at least GNU/Linux or BSD-based) Python users are
likely to have Python provided by their vendor together with a
"constellation" of packages. As long as the software they need is
packaged by their distribution, it's a matter of selecting the package
concerned; other packages upon which the desired software depends will
be installed automatically. You don't usually need to click through
some wizard ("next", "I agree", "next", "next", "next", "next",
"next", "finish"), either.

Paul

[MonkeeSage]

Stef,

What Adam is talking about has nothing to do with windows or *nix.
He's talking about packing one or more .py files into a single
archive, which can be imported just like the regular .py files. This
means you can distribute a whole bunch of module files/dirs as a
single .pyc file. It just makes it easier to have a single file to
distribute rather than a whole bunch, or a zip that one has to unzip
into a certain directory, &c. So you can have myscript.py +
myscriptfiles.pyc vs. myscript.py + somedepdir/ + someotherdepdir/ +
somedepfile.py...&c.

If you still don't get it, you don't need it.

Regards,
Jordan

[Adam Atlas]

Okay, here's the prototype...
<http://adamatlas.org/2007/03/Squisher-0.1.py>
It's meant to be run as a command line program (pass it a directory or
a zip file as an argument). By default it will save the new file to
the argument's base name plus '.pyc'. You can override this with -o.

Obviously it's very much a work in progress... try it out and let me
know the results.

[Peter Wang]

On Mar 5, 12:31 am, "Adam Atlas" <[e-mail]:

Right now I'm just testing and polishing up the code... in the
meantime, any comments?

How does this work with compiled extension modules?

[Adam Atlas]

Doesn't seem to work. I guess zipimport doesn't support that by
default... but if I remember correctly, Setuptools adds that. Maybe
I'll take a look at how it does it (I think by extracting the .so to /
tmp?) and see how easy it would be to integrate it here.

[Adam Atlas]

I updated it.
http://adamatlas.org/2007/03/Squisher-0.2.py

New Things:
- It supports C extensions within squished packages.
- It supports including squished packages within other squished
packages. (That is, you can have a package that includes a .pyc
generated by this, and turn that whole package into one .pyc.)
- If you import it after you import setuptools in a setup.py, it will
override a bit of setuptools so that bdist_egg will result in an egg
that can also be used as a squished package (if you rename it to
something ending in .pyc).
- It puts a helpful docstring at the top of generated .pyc files,
mainly for the purpose of being easy to see if you open the file in a
text editor. (It tells you what can be done with the file -- name it
something .pyc, or unzip it, or, if applicable, use it as an egg.)

[Bruno Desthuilliers]

Adam Atlas a écrit :

Doesn't seem to work. I guess zipimport doesn't support that by
default... but if I remember correctly, Setuptools adds that. Maybe
I'll take a look at how it does it (I think by extracting the .so to /
tmp?)

or to another known location, IIRC.

and see how easy it would be to integrate it here.