By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,617 Members | 1,708 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,617 IT Pros & Developers. It's quick & easy.

crossplatform py2exe - would it be useful?

P: n/a
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.

Since py2exe is a distutils extension, and since C compilers are
commonly available on most platforms except Windows, it would be fairly
easy to let py2exe generate a C source file installing this import hook,
and let distutils' C compiler build an executable file from this.

Would this be useful, or would I be wasting my time, since McMillan
installer, cx_freeze, and tools/freeze already do it?

At the end of this post you'll find excerpts from the readme file, which
is currently the only documentation available.

Thomas

The readme file:

A new and improved py2exe for Python 2.3
========================================

Uses the zipimport mechanism, so it requires Python 2.3 or later. The
zipimport mechanism is able to handle the early imports of the
warnings and also the encodings module which is done by Python.

Creates a single directory, which must be deployed completely.

(Most of this is based on ideas of Mark Hammond:) Can create any
number of console and gui executables in this directory, plus
optionally a windows service exe, plus optionally an exe and dll com
server. The com servers can expose one or more com object classes.

All pure Python files are contained in a single zip archive, which is
shared by all the executables. The zip archive may also be used by
programs embedding Python. Since extension modules cannot be imported
from zipfiles, a simple pure Python loader is included in the zipfile
which loads the extension from the file system (without requiring that
the directory is in sys.path).

It would be nice if the executables could be run with only a single
sys.path entry containing the absolute filename of the zipfile, but it
seems for dll com servers the executable's directory is also
needed. The absolute filenames are constructed at runtime from the
directory containing the executable, and the zipfile name specified at
build time.

The way has changed how build targets are specified in the setup
script. py2exe installs it own Distribution subclass, which enables
additional keyword arguments to the setup function:

console = [...] # list of scripts to convert into console executables
windows = [...] # list of scripts to convert into gui executables
com_servers = [...] # list of fully qualified class names to build into the exe com server
service = [...] # list of fully qualified class names to build into a service executable
zipfile = "xxx.zip" # filename of the zipfile containing the pure Python modules

All of the above arguments are optional. The zipfile name defaults to
'library.zip'.
Jul 18 '05 #1
Share this Question
Share on Google+
20 Replies


P: n/a
Hi Thomas;

I've tried Freeze before, but I'm hooked on py2exe because it's so
SIMPLE and flexible, and handles all the weird cases automatically...
For those of us that aren't C gurus, anything that makes the process
easier is welcome.

To clarify your proposal, would you simply build the executable using
the same setup.py script on other platforms too, or would you have to
do more manual steps?

Thanks for the great work so far... py2exe ROCKS! q:]

Kevin.
Thomas Heller <th*****@python.net> wrote in message news:<pt**********@python.net>...
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.

Since py2exe is a distutils extension, and since C compilers are
commonly available on most platforms except Windows, it would be fairly
easy to let py2exe generate a C source file installing this import hook,
and let distutils' C compiler build an executable file from this.

Would this be useful, or would I be wasting my time, since McMillan
installer, cx_freeze, and tools/freeze already do it?

At the end of this post you'll find excerpts from the readme file, which
is currently the only documentation available.

Thomas

The readme file:

A new and improved py2exe for Python 2.3
========================================

Uses the zipimport mechanism, so it requires Python 2.3 or later. The
zipimport mechanism is able to handle the early imports of the
warnings and also the encodings module which is done by Python.

Creates a single directory, which must be deployed completely.

(Most of this is based on ideas of Mark Hammond:) Can create any
number of console and gui executables in this directory, plus
optionally a windows service exe, plus optionally an exe and dll com
server. The com servers can expose one or more com object classes.

All pure Python files are contained in a single zip archive, which is
shared by all the executables. The zip archive may also be used by
programs embedding Python. Since extension modules cannot be imported
from zipfiles, a simple pure Python loader is included in the zipfile
which loads the extension from the file system (without requiring that
the directory is in sys.path).

It would be nice if the executables could be run with only a single
sys.path entry containing the absolute filename of the zipfile, but it
seems for dll com servers the executable's directory is also
needed. The absolute filenames are constructed at runtime from the
directory containing the executable, and the zipfile name specified at
build time.

The way has changed how build targets are specified in the setup
script. py2exe installs it own Distribution subclass, which enables
additional keyword arguments to the setup function:

console = [...] # list of scripts to convert into console executables
windows = [...] # list of scripts to convert into gui executables
com_servers = [...] # list of fully qualified class names to build into the exe com server
service = [...] # list of fully qualified class names to build into a service executable
zipfile = "xxx.zip" # filename of the zipfile containing the pure Python modules

All of the above arguments are optional. The zipfile name defaults to
'library.zip'.

Jul 18 '05 #2

P: n/a
Thomas Heller wrote:
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.

Since py2exe is a distutils extension, and since C compilers are
commonly available on most platforms except Windows, it would be fairly
easy to let py2exe generate a C source file installing this import hook,
and let distutils' C compiler build an executable file from this.

Would this be useful, or would I be wasting my time, since McMillan
installer, cx_freeze, and tools/freeze already do it?


I think it would be a WONDERFUL idea: py2exe is the simplest to use
of all the tools you mention, and it would save a lot of ink^H^H^H pixels,
each time I point people to it, to be able to omit the blurb about "if
you're interested in deploying to Windows platform only, then"...:-).
Alex

Jul 18 '05 #3

P: n/a
On Wed, Aug 06, 2003 at 08:36:20PM +0200, Thomas Heller wrote:
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.


Now that zipimport is part of Python the code required for bootstrapping
a py2exe runtime is just:

myscript -c "import sys; sys.path.insert(0, sys.executable); import foo"

This reduces the difference between the custom interpreter supplied with
py2exe and the standard interpreter to just a few lines of C.

The obvious question is - why not go all the way and put this little
hook into the standard Python distribution? This way py2exe could be a
platform-independent pure Python application. In fact, py2exe wouldn't
actually be necessary because anyone could create a zip file manually and
append it to the executable but it's more convenient to have a tool that
automates the process and finds the required dependencies.

Oren

Jul 18 '05 #4

P: n/a
Oren Tirosh wrote:
On Wed, Aug 06, 2003 at 08:36:20PM +0200, Thomas Heller wrote:
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.


Now that zipimport is part of Python the code required for bootstrapping
a py2exe runtime is just:

myscript -c "import sys; sys.path.insert(0, sys.executable); import foo"

This reduces the difference between the custom interpreter supplied with
py2exe and the standard interpreter to just a few lines of C.

The obvious question is - why not go all the way and put this little
hook into the standard Python distribution? This way py2exe could be a
platform-independent pure Python application. In fact, py2exe wouldn't
actually be necessary because anyone could create a zip file manually and
append it to the executable but it's more convenient to have a tool that
automates the process and finds the required dependencies.


Sounds like a good idea to me, if a sensible name is chosen for the
"main module" (I propose 'main':-). Take it to Python-Dev...? I even
wonder if it's unobtrusive enough to be considered (as a "bugfix"...:-)
for 2.3.1, rather than having to get into 2.4 and thus wait a LONG time...
Alex
Jul 18 '05 #5

P: n/a
Oren Tirosh <or*******@hishome.net> writes:
On Wed, Aug 06, 2003 at 08:36:20PM +0200, Thomas Heller wrote:
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.
Now that zipimport is part of Python the code required for bootstrapping
a py2exe runtime is just:

myscript -c "import sys; sys.path.insert(0, sys.executable); import foo"


Yes, something like this is also what I am thinking now. And 'myscript'
is just a copy of the standard interpreter. Although the sys.path entry
must be present *before* Py_Initialize is called.
This reduces the difference between the custom interpreter supplied with
py2exe and the standard interpreter to just a few lines of C.

The obvious question is - why not go all the way and put this little
hook into the standard Python distribution? This way py2exe could be a
platform-independent pure Python application. In fact, py2exe wouldn't
actually be necessary because anyone could create a zip file manually and
append it to the executable but it's more convenient to have a tool that
automates the process and finds the required dependencies.


Yes, and modulefinder is now in the standard library.

OTOH, py2exe does a little bit more: It has a mechanism to supply
modules to include which modulefinder doesn't find, exclude modules
which are unneeded although found, can detect whether Tkinter is used
and copy it, scan (on Windows) extensions for dlls they need (wxPython
needs the wxWindows dll, for example), handle hidden imports from C code
in Python itself and extensions, and so on.

And it works around the fact that extension modules cannot be loaded
from zipfiles, it creates a pure Python loader included in the zip for
them.

Having said that, I would have nothing against py2exe included in the
standard distribution, and the hooks in place.

Thanks,

Thomas
Jul 18 '05 #6

P: n/a
Alex Martelli <al***@aleax.it> writes:
Oren Tirosh wrote:
On Wed, Aug 06, 2003 at 08:36:20PM +0200, Thomas Heller wrote:
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.
Now that zipimport is part of Python the code required for bootstrapping
a py2exe runtime is just:

myscript -c "import sys; sys.path.insert(0, sys.executable); import foo"

This reduces the difference between the custom interpreter supplied with
py2exe and the standard interpreter to just a few lines of C.

The obvious question is - why not go all the way and put this little
hook into the standard Python distribution? This way py2exe could be a
platform-independent pure Python application. In fact, py2exe wouldn't
actually be necessary because anyone could create a zip file manually and
append it to the executable but it's more convenient to have a tool that
automates the process and finds the required dependencies.


Sounds like a good idea to me, if a sensible name is chosen for the
"main module" (I propose 'main':-).


My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?
Take it to Python-Dev...? I even wonder if it's unobtrusive enough to
be considered (as a "bugfix"...:-) for 2.3.1, rather than having to
get into 2.4 and thus wait a LONG time...
Alex


Thomas
Jul 18 '05 #7

P: n/a
Thomas Heller wrote:
...
myscript -c "import sys; sys.path.insert(0, sys.executable); import foo"
... Sounds like a good idea to me, if a sensible name is chosen for the
"main module" (I propose 'main':-).


My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?


Well, most main scripts ARE coded with the "if __name__=='__main__':"
convention, after all, so an "import __main__" can be seen as a way
to just piggyback on that existing convention rather than inventing a
new one in addition. So, I concede it's better than "import main".
Alex

Jul 18 '05 #8

P: n/a
Alex Martelli <al***@aleax.it> writes:
Thomas Heller wrote:
...
myscript -c "import sys; sys.path.insert(0, sys.executable); import foo" ... Sounds like a good idea to me, if a sensible name is chosen for the
"main module" (I propose 'main':-).


My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?


Well, most main scripts ARE coded with the "if __name__=='__main__':"
convention, after all, so an "import __main__" can be seen as a way
to just piggyback on that existing convention rather than inventing a
new one in addition. So, I concede it's better than "import main".


How would the hook be triggered? The zipimporter code would probably add
argv[0] to sys.path, and then try to 'import __main__' or something like
this. The problem is that a standalone executable python would have to
disable the standard Python command line flags and environment
variables, but they are parse *before* Py_Initialize() is called.

And I hope that the options set by the command line flags and env vars
should now come from the __main__ script itself.

Thomas
Jul 18 '05 #9

P: n/a
Thomas Heller wrote:
Alex Martelli <al***@aleax.it> writes:
Thomas Heller wrote:
...
> myscript -c "import sys; sys.path.insert(0, sys.executable); import
> foo" ...
Sounds like a good idea to me, if a sensible name is chosen for the
"main module" (I propose 'main':-).

My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?


Well, most main scripts ARE coded with the "if __name__=='__main__':"
convention, after all, so an "import __main__" can be seen as a way
to just piggyback on that existing convention rather than inventing a
new one in addition. So, I concede it's better than "import main".


How would the hook be triggered? The zipimporter code would probably add
argv[0] to sys.path, and then try to 'import __main__' or something like
this. The problem is that a standalone executable python would have to
disable the standard Python command line flags and environment
variables, but they are parse *before* Py_Initialize() is called.


Ah, yes, good point. So, the executable needs to know whether to do
the usual commandline and environment processing, or not, _before_
calling Py_Inizialize. One approach might be to trigger this based
on the executable's own *name* -- do the full commandline and environment
processing if and only if the executable's name starts with (case-
insensitive, probably, to be safe...) the six letters 'python', but
not otherwise. There are, no doubt, other alternative ways, too, but
this one seems dirt-simple and practically sufficient.

And I hope that the options set by the command line flags and env vars
should now come from the __main__ script itself.


I'm not sure I understand what you mean. Anyway, I do see that if
my 'foobar.exe' is a python.exe + appended zipfile, then running
'foobar -i' should just put '-i' in sys.argv[1], and NOT gobble it up
to mean "enter interactive mode", for example.
Alex

Jul 18 '05 #10

P: n/a
Alex Martelli <al***@aleax.it> writes:
Thomas Heller wrote:
Alex Martelli <al***@aleax.it> writes:
Thomas Heller wrote:
...
>> myscript -c "import sys; sys.path.insert(0, sys.executable); import
>> foo"
...
> Sounds like a good idea to me, if a sensible name is chosen for the
> "main module" (I propose 'main':-).

My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?

Well, most main scripts ARE coded with the "if __name__=='__main__':"
convention, after all, so an "import __main__" can be seen as a way
to just piggyback on that existing convention rather than inventing a
new one in addition. So, I concede it's better than "import main".

How would the hook be triggered? The zipimporter code would probably add
argv[0] to sys.path, and then try to 'import __main__' or something like
this. The problem is that a standalone executable python would have to
disable the standard Python command line flags and environment
variables, but they are parse *before* Py_Initialize() is called.


Ah, yes, good point. So, the executable needs to know whether to do
the usual commandline and environment processing, or not, _before_
calling Py_Inizialize.


Exactly. And it may even be useful to do specail command line
processing, an PY2EXEVERBOSE flag might be useful.
One approach might be to trigger this based
on the executable's own *name* -- do the full commandline and environment
processing if and only if the executable's name starts with (case-
insensitive, probably, to be safe...) the six letters 'python', but
not otherwise. There are, no doubt, other alternative ways, too, but
this one seems dirt-simple and practically sufficient.


On windows, where the interpreter is in a dll, providing a custom
equivalent to python.exe (as py2exe currently does) is pretty simple.
On systems where the interpreter is staically linked, there's no other
choice than to recompile and relink the whole pythonm if I understand
correctly.
And I hope that the options set by the command line flags and env vars
should now come from the __main__ script itself.


I'm not sure I understand what you mean. Anyway, I do see that if
my 'foobar.exe' is a python.exe + appended zipfile, then running
'foobar -i' should just put '-i' in sys.argv[1], and NOT gobble it up
to mean "enter interactive mode", for example.


You understood. Yes, the command line flags must be passed into
sys.argv. But I still want to set the optimize flag and the unbuffered
flag at *build* time. I'm quite sure all this cannot be encoded into the
filename.

Right now, py2exe embeds a struct containing a magic value plus these
flags into the exe, just before the zip-archive, but all this unpacking
has to be done from C code (because these flags are not writable from
Python code), so all this has to be part of the hook.

Thomas

PS: Since py2exe, even on Linux, doesn't really create a single file
executable, there are always some shared libs needed, maybe the first
step would be to create a directory containing the interpreter
executable with an appended ziparchive, the shared libs needed (zlib.so,
maybe more), and a bash script containing something like this (you'll
probably see how rusty my *nix skills are nowadays):

#!/bin/sh
exec python_with_zip -c "<<EOF
the script itself
"
EOF
What are the disadvantages of a shell-script against an (elf) executable?
Jul 18 '05 #11

P: n/a
On Thu, Aug 07, 2003 at 02:46:24PM +0200, Thomas Heller wrote:
Alex Martelli <al***@aleax.it> writes:
Oren Tirosh wrote:
On Wed, Aug 06, 2003 at 08:36:20PM +0200, Thomas Heller wrote:
I'm currently working on a new version of py2exe, which will require
Python 2.3 and later, because it uses the zipimport mechanism.

Now that zipimport is part of Python the code required for bootstrapping
a py2exe runtime is just:

myscript -c "import sys; sys.path.insert(0, sys.executable); import foo"

This reduces the difference between the custom interpreter supplied with
py2exe and the standard interpreter to just a few lines of C.

The obvious question is - why not go all the way and put this little
hook into the standard Python distribution? This way py2exe could be a
platform-independent pure Python application. In fact, py2exe wouldn't
actually be necessary because anyone could create a zip file manually and
append it to the executable but it's more convenient to have a tool that
automates the process and finds the required dependencies.


Sounds like a good idea to me, if a sensible name is chosen for the
"main module" (I propose 'main':-).


My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?


You can't import __main__ - you'll get the one already in sys.modules.
The code for the main script needs to be executed in __main__'s dict.

Guido might not like it if the interpreter always attempted to open
its executable image file to check for an appended zip. It could cause
problems on some obscure environments. A possible alternative would be
to have a configuration area inside the executable that can be modified
by an external program (e.g. py2exe). The program would search for a
signature string and modify the section after it. The configuration
area can be as simple as a string that overrides the command line
arguments.

Oren

Jul 18 '05 #12

P: n/a
Oren Tirosh wrote:
...
> Sounds like a good idea to me, if a sensible name is chosen for the
> "main module" (I propose 'main':-).
My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?


You can't import __main__ - you'll get the one already in sys.modules.


If there's a __main__ in the zip, we could remove (even assuming it's
already there) the yet-empty sys.modules['__main__'].
The code for the main script needs to be executed in __main__'s dict.

Guido might not like it if the interpreter always attempted to open
its executable image file to check for an appended zip. It could cause
So, I reiterate an idea I've already expressed: key on the executable
file's name. If it's at least six characters long and the first six
characters are (case-insensitive) 'p', 'y', 't', 'h', 'o', 'n' in this
order, forget the whole thing and proceed like now; in other words,
use whatever new tricks we insert *if and only if* the executable file's
name does NOT start with (case-insensitive) 'python'.
problems on some obscure environments. A possible alternative would be
to have a configuration area inside the executable that can be modified
by an external program (e.g. py2exe). The program would search for a
signature string and modify the section after it. The configuration
area can be as simple as a string that overrides the command line
arguments.


I suspect "obscure environments" may make it hard for py2exe to find
the needed signature and get at the 'configuration area' (depending on
how executable files are stored when seen as stream of bytes). Still,
such an area would also be useful for other purposes, as you mention
(e.g., supplying the -O switch "at compile time", and the like). So,
perhaps, we could simply test the executable's name FIRST, and if the
name starts with "python" just do nothing, otherwise look at the
configuration area (string) and so on. On any "obscure environment"
where the set of tricks doesn't work, one would simply have to avoid
renaming or copying the python interpreter to weird names, and otherwise
would be just about as well or badly off as today.
Alex

Jul 18 '05 #13

P: n/a
Thomas Heller wrote:
...
Ah, yes, good point. So, the executable needs to know whether to do
the usual commandline and environment processing, or not, _before_
calling Py_Inizialize.
Exactly. And it may even be useful to do specail command line
processing, an PY2EXEVERBOSE flag might be useful.


I guess it might, yes.

One approach might be to trigger this based
on the executable's own *name* -- do the full commandline and environment
processing if and only if the executable's name starts with (case-
insensitive, probably, to be safe...) the six letters 'python', but
not otherwise. There are, no doubt, other alternative ways, too, but
this one seems dirt-simple and practically sufficient.


On windows, where the interpreter is in a dll, providing a custom
equivalent to python.exe (as py2exe currently does) is pretty simple.
On systems where the interpreter is staically linked, there's no other
choice than to recompile and relink the whole pythonm if I understand
correctly.


Python 2.3 now supports building a .so (or whatever) on many systems,
needing just a switch on ./configure. If one chooses to go for a
static build anyway, sure, one will have to link statically. But
no recompile and relink would be needed (unless 'pythonm' is something
I don't know about rather than just a typo for 'python'...?). E.g.,
on Linux:

$ cat main.py
print "Hello world"
$ python -c 'import main'
Hello world
$ zip main main.pyc
adding: main.pyc (deflated 31%)
$ cp /usr/local/bin/python2.3 ./myapp
$ cat main.zip >>myapp
$ ./myapp -c 'import sys; sys.path.insert(0,"myapp"); import main'
Hello world
$

So, if I could just insert that "-c string" into myapp in some
way -- and have myapp, which is just a copy of the python 2.3
interpreter, execute it in some way instead of the arguments
[leaving the arguments to use for sys.argv] -- I'd be in clover,
quite indifferently as to whether myapp is statically OR
dynamically linked. Admittedly at this level it doesn't work
with __main__, as Oren suggested ('import __main__' is a noop
unless one deletes sys.modules['__main__'], and if one does it's
"ImportError: Cannot re-init internal module __main__"). Which
is why I've reverted to 'main' without underscores;-).

Anyway, the zipfile would of course be prepared in much more
sophisticated (but presumably platform independent?) ways, and
the suitable string (including e.g. a -O or whatever) could be
inserted (by a py2exe tool or the like) into a suitable config
area in the executable, as Oren suggested.
And I hope that the options set by the command line flags and env vars
should now come from the __main__ script itself.


I'm not sure I understand what you mean. Anyway, I do see that if
my 'foobar.exe' is a python.exe + appended zipfile, then running
'foobar -i' should just put '-i' in sys.argv[1], and NOT gobble it up
to mean "enter interactive mode", for example.


You understood. Yes, the command line flags must be passed into
sys.argv. But I still want to set the optimize flag and the unbuffered
flag at *build* time. I'm quite sure all this cannot be encoded into the
filename.


Right. But it COULD easily be encoded in a "configuration area".
Right now, py2exe embeds a struct containing a magic value plus these
flags into the exe, just before the zip-archive, but all this unpacking
has to be done from C code (because these flags are not writable from
Python code), so all this has to be part of the hook.
Oh yes, it surely would need to be part of the hook.

Thomas

PS: Since py2exe, even on Linux, doesn't really create a single file
executable, there are always some shared libs needed, maybe the first
step would be to create a directory containing the interpreter
executable with an appended ziparchive, the shared libs needed (zlib.so,
Perhaps this step could be left to a separate tool (basically we're talking
about a self-unpackaging zipfile, it seems to me).
maybe more), and a bash script containing something like this (you'll
probably see how rusty my *nix skills are nowadays):

#!/bin/sh
exec python_with_zip -c "<<EOF
the script itself
"
EOF
What are the disadvantages of a shell-script against an (elf) executable?


For example, a shell script cannot be set-userid (it wouldn't be secure).
Alex

Jul 18 '05 #14

P: n/a
Alex Martelli <al***@aleax.it> writes:
Oren Tirosh wrote:
...
> Sounds like a good idea to me, if a sensible name is chosen for the
> "main module" (I propose 'main':-).

My choice would have been __main__ :-) Is it really the correct way to
'import __main__' instead of 'running' it?
You can't import __main__ - you'll get the one already in sys.modules.


If there's a __main__ in the zip, we could remove (even assuming it's
already there) the yet-empty sys.modules['__main__'].
The code for the main script needs to be executed in __main__'s dict.


Then we name the boot module __boot__ and import this from the zip.
This could then execute the script in the __main__ module's namespace.
So, I reiterate an idea I've already expressed: key on the executable
file's name. If it's at least six characters long and the first six
characters are (case-insensitive) 'p', 'y', 't', 'h', 'o', 'n' in this
order, forget the whole thing and proceed like now; in other words,
use whatever new tricks we insert *if and only if* the executable file's
name does NOT start with (case-insensitive) 'python'.
problems on some obscure environments. A possible alternative would be
to have a configuration area inside the executable that can be modified
by an external program (e.g. py2exe). The program would search for a
signature string and modify the section after it. The configuration
area can be as simple as a string that overrides the command line
arguments.
I suspect "obscure environments" may make it hard for py2exe to find
the needed signature and get at the 'configuration area' (depending on
how executable files are stored when seen as stream of bytes). Still,
such an area would also be useful for other purposes, as you mention
(e.g., supplying the -O switch "at compile time", and the like). So,
perhaps, we could simply test the executable's name FIRST, and if the
name starts with "python" just do nothing, otherwise look at the
configuration area (string) and so on.


Sounds much like the way py2exe already works now. It locates the
appended zip-file by searching the exefile from the end, then finds the
beginning of the zipfile, and looks for a magic number there, which is
used to verify that the next n bytes before this position is a C
structure containing the required flags.

I don't like the idea to scan the executable for a magic signature
without further hints where this should be.
On any "obscure environment"
where the set of tricks doesn't work, one would simply have to avoid
renaming or copying the python interpreter to weird names, and otherwise
would be just about as well or badly off as today.


From reading the McMillan installer sources some time ago, I have the
impression that on some obscure platforms it's not possible to append
the structure and the zipfile to the executable, and on other obscure
platforms (or maybe runtime environments, maybe a cgi executable started
from apache) it may be difficult to the pathname if the exefile.

But, all in all, it sounds like a plan. Although I have the impression
that it may be difficult to convince the python-dev crowd to include
this in 2.3.1. (Is anyone of them reading this thread?)

Thomas
Jul 18 '05 #15

P: n/a
Thomas Heller wrote:
...
But, all in all, it sounds like a plan. Although I have the impression
that it may be difficult to convince the python-dev crowd to include
this in 2.3.1. (Is anyone of them reading this thread?)


Sure -- me, for one;-). Seriously, _let's_ take it to python-dev and
see which way the wind blows -- then we may pep &c accordingly...
Alex

Jul 18 '05 #16

P: n/a
Bengt Richter wrote:
...
Is there a big difference for you between e.g.,

wget <some url>/py2exefiedapp.exe
py2exefiedapp

and

wget <some url>/py2exefiedapp.uff
uffunwrap --launch exefiedapp.uff


Yes. Specifically, in the first case (on a Unix-like system) I could
interpose a suitable set-userid setting change such as:

sudo chmod u+s py2exefiedapp

while in the second case I couldn't.

Furthermore, my cousin, who has installed no extras at all compared
to what comes with his operating system (and runs an operating system
without 'apt-get', 'urpmi', or similar 'download-on-demand' functionality)
would still be able to take full advantage of the first approach w/o
having to previously install ANY other piece of software; to take
advantage of the second approach, he would have to first download and
install 'uffunwrap', and he just ain't gonna do that.

That's two strikes against the ".uff" approach and in favour of the
'.exe' one. I can see potential advantages for the '.uff', too, in
widely different scenarios; but these issues indicate to me that it
just can't replace the '.exe'. Therefore, I would suggest you pursue
the .uff as a third-party alternative -- while, on the other hand,
"makers of .exe's" have long been available as third-party alternatives,
and the thrilling aspect of this latest round of ideas is that we seem
to be very close to being able to integrate them in the Python standard
distribution, with a resulting potential for an interesting boost to
Python's popularity. It makes a psychological difference, quite a big
one, whether some functionality is integrated in a standard distribution
or has to be separately downloaded and installed as a third-party add-on.

"Ability to build directly executable files" would make a big 'selling'
point if it were in Python's standard distribution, while "ability to
wrap files into an archive which still needs a separate utility to
unwrap and run", useful as it may be, just doesn't have the same level
of raw appeal to typical punters currently wondering about Python.
Alex

Jul 18 '05 #17

P: n/a
On Sat, 09 Aug 2003 17:11:57 GMT, Alex Martelli <al***@aleax.it> wrote:
Bengt Richter wrote:
...
Is there a big difference for you between e.g.,

wget <some url>/py2exefiedapp.exe
py2exefiedapp

and

wget <some url>/py2exefiedapp.uff
uffunwrap --launch exefiedapp.uff
Yes. Specifically, in the first case (on a Unix-like system) I could
interpose a suitable set-userid setting change such as:

sudo chmod u+s py2exefiedapp

while in the second case I couldn't.


If you know enough to use sudo, I think you would find it trivial to
use uffunwrap without auto-launch (i.e., just like an installer or
untarrer or unzipper etc.) and do your sudo chmod on whatever you liked.
I don't think this is a areal argument against uffunwrap. In fact, by
including a cmd/bash/sh/whatever file and launching that instead of
the main app, you could have a prompted automated sequence where you might
just have to remember the password when prompted (and maybe confirm that, yes,
you really mean it ;-)

I think there is good reason to minimize the number of times one gives control
to .exe's one hasn't very well authenticated. Imagine if all zip files came
in the form of .exe's! Wouldn't that make you nervous? Much better IMO to have
a single trusted tool that deals with them all as safely as possible.

Furthermore, my cousin, who has installed no extras at all compared
to what comes with his operating system (and runs an operating system
without 'apt-get', 'urpmi', or similar 'download-on-demand' functionality)
would still be able to take full advantage of the first approach w/o
having to previously install ANY other piece of software; to take
advantage of the second approach, he would have to first download and
install 'uffunwrap', and he just ain't gonna do that. For a one-time thing, I think he *should*, and you should twist his arm,
because I think it would result in a safer modus operandi. If the plan for
Python is to generate .exe's per se as self-executing distribution containers,
(other than the major wise installer distribution files, which have md5's posted)
then I really think some considerable thought ought to be expended on making
that safe. I have no problem checking md5's, but your cousin might.

I would envision uffunwrap to be a small executable that can unwrap,launch,
check integrity, and potentially do an automatic autenticity check (though
featuritis can cause growth). The packing partner program would be in Python,
say uff.py, and be able to package a uff file automatically based on a uff header
template edited to specify the requisite file sources on lines inserted following each
normal header line (which specifies an included file and where it goes in the unwrapping
context). The normal header lines also have text/binary flags and either actual sizes
and dates and optional digests, or placeholders for them to be created from specified sources.
That way an option to revalidate a new packing with only one file changed can be supported,
and only the header line for the new source should be updated.

I like short names, so maybe uffunwrap should just be uff.exe and the packer uff.py
(which also can have command line options for unpacking and various fancy stuff,
since it is python.

But if the heart it set on a single .exe for the user experience of having an
apparent single executable to run, then as I mentioned previously, it's
not that hard, and as you mentioned, there are various .exe builder out there.

Indeed, I believe winzip can generate an auto-extracting, auto-launching exe that
it can also recognize as a zip archive. Mabe even old pkzip could so something similar.

A single exe could be built on tar, tgz, zip or most any archive format as a multi-file
container embedded as a single binary resource.

I got interested in factoring out the container. The information itself is not that different
from a zip or tar file, but I have long been bugged by the IMO severely kludgy way file content
types are represented and/or inferred through magic and/or extension hints etc. etc.

IMO there ought to be a way of associating file type and other metadata with file data
other than through file names and/or extensions or multi/nefarious magic. So I thought,
what about introducing a single open-ended magic prefix (postfix or indirection can work too)
to data that could do the job. So I got to thinking, maybe a utf-8 header having some general
structure that can identify what you'd like to say about the data itself, as opposed the the
particular file system container that happens to contain it.

My original thought was to have a metadata prefix for single files, and just tweak a file
system implementation to keep this data in the first n 512-byte blocks of what would
ordinarily be data space, but add an offset into the file definition, so that the header
could be skipped transparently for seek and open etc., and look like an ordinary file,
but allow some kind of access to the metadata, maybe by opening with an 'm' mode to include
the metadata prefix as part of the apparent file. Or maybe to exclude it, so naive opens
will see the metadata. To have international content description I thought utf-8 would work.
Then it was a matter of choosing a standard format an minimal content for the header. Then
I got interested in something else ;-)

One reason for a universal text header is that then any file can be opened in a text editor
and you should at least see the header. Or just do head -20 some.uff to have a peek.

Well, this latest thread came up, and I thought to expand the idea to a segmented file.
with a header field for every segment. rfc2822 seemed like a possible format for the header,
other than it's supposed to be ascii. I wanted a universal format. So I'm debating utf-8 or -16,
and settled on 8 for now, because it's more readable if you see it raw.

I thought I could fairly easily implement packing methodology and at first I thought to use
the data as embedded/appended .exe resource, but had second thoughts about .exe's. Anyway,
it could obviously work as a microinstaller tool as well as a launcher. So what does it have
that wise or winzip etc don't have?

++ Potential for really simple and small open source code, both python and C.
+ support for unicode descriptions etc. (On windows it wouldn't be so hard to send the header
to the clipboard for insertion into an editor that can show unicode, e.g., notepad.)
+ Potential to detect current console encoding and output header accordingly for
localized interactive viewing w/o editor (instead of just assuming latin-1 and printing ?'s)
+ probable pretty good portability for a lot of the unwrapper.
++ platform independence of the .uff format (since endianness, packing, encoding, whatever can
all be specified in the utf-8 header, and the rest is binary with specified endianness overall,
and segment-wise also describable.

To get fast unpacking, I'd probably specify align=512 in the X-UFF-Packing: field.
To get really fast unpacking, I might spawn separate threads to copy segments to files
in parallel, but that's a future optimization. YM would vary with OS, controllers, etc.

Yet another self-unpacking archive is not rocket science or that interesting, but the idea
of a universal, file-system-independent, self-describing data format, seems to me the important
part. It would be like a universal bar code system for data, and would mean you could do
away with file extension associations, and you could see the orignal name for the data in some
native language, no matter how many times it had been contained in variously named and dated files
-- which are really only container names, not data names (except by unreliable dual name usage).
When you make new data, or modify existing data, that's when the data descriptions should change.
The file system names in use will only be temporary locator info, and are really separate
semantically.

Yada, yada ...

It wouldn't be that hard to do a single-exe version that can carry the data appended. (Though I'm
not 100% sure all .exe formats permit that any more, so it might be a matter of getting some
template pieces and faking what the linker does to include a binary resource officially
within an exe. But you still need un unpacking function to put python.exe and theapp.py
and theextension.dll and config.txt etc. into separate files, and maybe setting environment
and path, before kicking off python.exe.

You can also buy a copy of winzip to create self-extracting-and-launching launching zip files,
I think.

UFF is different. For one thing it is platform independent as a container format.
Of course executable binary contents destined for different platforms will be
different (and BTW the opportunity exists to package several versions selectable
at startup, even if just localization strings for a given app). But the basic
content is binary or text, and text is stored in the uff file with \n EOLs, unless
it is some special encoding, in which case it should be flagged as binary. When it
is unwrapped, text destination file is opened with 'w' not 'wb', so it becomes
what is normal for the platform. Of course you can ship ascii as binary too.

My current concept for packing (as opposed to unwrapping) a uff file, is to
drive it using a copy of the header as a template, and just e.g., add in
a source: specification after each line that needs a file to pack into the whole, e.g.,
...
X-UFF-Pkt: 3: t 456 ./myConfig.txt ...
source: build2/cfg.dat
...
where relative paths are taken re a prefix specified elsewhere, maybe a packing command line option.
Anyway, it becomes a simple and I think flexible framework for lots of possibilites. Since you
can include whatever you want and launch anything you want from the included -- or from an
assumed user environment, since it's like having an internal command line. Speaking of which,
you could possibly prefix e.g. #! uffunwrap -x and make the resulting .uff executable.
BTW, does sudo chmod u+s get the setuid effect passed on to the interpreting executable?

That's two strikes against the ".uff" approach and in favour of the
'.exe' one. I can see potential advantages for the '.uff', too, in
widely different scenarios; but these issues indicate to me that it
just can't replace the '.exe'. Therefore, I would suggest you pursue
the .uff as a third-party alternative -- while, on the other hand,
"makers of .exe's" have long been available as third-party alternatives,
and the thrilling aspect of this latest round of ideas is that we seem
to be very close to being able to integrate them in the Python standard
distribution, with a resulting potential for an interesting boost to
Python's popularity. It makes a psychological difference, quite a big
one, whether some functionality is integrated in a standard distribution
or has to be separately downloaded and installed as a third-party add-on. Agreed. But I don't see why uff.exe and uff.py couldn't be standard. Since
downloading and running .exe's with a big python payload is attractive to
some, why wouldn't downloading a 50k or 100k uff.exe be attractive? ;-)
"Ability to build directly executable files" would make a big 'selling'
point if it were in Python's standard distribution, while "ability to
wrap files into an archive which still needs a separate utility to
unwrap and run", useful as it may be, just doesn't have the same level
of raw appeal to typical punters currently wondering about Python.

Not that hard to take a copy of uff.exe and append the payload and have your
directly executable file. I can do it, but I'm not sure it's a good idea.
It could be fine for official python stuff, just like the windows installer
..exes are fine (but I only say that because I trust the Timbot ;-)
And I can check the md5's on those.

But in general, I don't see that the executable buys me much except worry.
The final executable is prepared by some few actions in any case. I'd rather
be having a data-driven tool I trust do it than something I don't wholly trust
maybe do it on the fly. Plus if the content really is multiple files, re-executing
the original exe may mean loading the whole thing, even if it notices that it
doesn't need to repeat its initial disgorging of content.

I just don't like the work of first making sure it's really the .exe I intended to get.
If I have a separate trusted tool that makes checking and looking easy, I prefer it.
It's why I prefer zipped or tgz files to gee-whiz Installshield-prepared or any other
installation .exes.

You can never trust those buggers to ask you politely whether you would like
their latest and greatest to override current file associations (or which), or
replace ctl3d.dll with something newer and supposedly better, etc. (Unless,
of course, you know the timbot put it together ;-)

Regards,
Bengt Richter
Jul 18 '05 #18

P: n/a
On Sat, 09 Aug 2003 19:59:42 +0100, Marc Wilson <ma**@cleopatra.co.uk> wrote:
In comp.lang.python, Alex Martelli <al***@aleax.it> (Alex Martelli) wrote
in <xX*********************@news2.tin.it>::

|That's two strikes against the ".uff" approach and in favour of the
|'.exe' one. I can see potential advantages for the '.uff', too, in
|widely different scenarios; but these issues indicate to me that it
|just can't replace the '.exe'. Therefore, I would suggest you pursue
|the .uff as a third-party alternative -- while, on the other hand,
|"makers of .exe's" have long been available as third-party alternatives,
|and the thrilling aspect of this latest round of ideas is that we seem
|to be very close to being able to integrate them in the Python standard
|distribution, with a resulting potential for an interesting boost to
|Python's popularity. It makes a psychological difference, quite a big
|one, whether some functionality is integrated in a standard distribution
|or has to be separately downloaded and installed as a third-party add-on.
|
|"Ability to build directly executable files" would make a big 'selling'
|point if it were in Python's standard distribution, while "ability to
|wrap files into an archive which still needs a separate utility to
|unwrap and run", useful as it may be, just doesn't have the same level
|of raw appeal to typical punters currently wondering about Python.

For instance, I have written a couple of little progs in Python to solve
problems for a client. I need to make them into executables, because, while
he's willing to install an .exe I've written, he's less sure about one
written by someone who has no contract with him, and no comeback if it goes
taters.

You make the contract. If you use open source, he'll come back to you, and you'll
be able to fix it, or find someone here who can. Python itself is written
"by someone who has no contract with him" (presumably). How do you square that?

How does he feel about installing via something you've made with Installshield
(commercial installer .exe packager)? He will certainly have no easy way to deal
with any problem with that other than call you, and you will have no easy way
to check on what happened, because you can't just look at the code, or ask someone
here who knows about the code, because it isn't openly available (AFAIK; for a price
most commercial things are possible).

uff.exe is (well, might be if I don't use up my time budget here ;-) just a tool either
just to install or to install and automatically execute something in a way you decided,
and having the possibility that automatically executing means getting some files in place
and maybe doing some preliminary things you don't really want embedded in your app just
because you have a "single-exe" requirement.

If it's open source and all available through python.org, where's the problem
with that? ISTM, he's better protected, and has all kinds of options.

Regards,
Bengt Richter
Jul 18 '05 #19

P: n/a
On Fri, Aug 08, 2003 at 05:31:35PM +0200, Thomas Heller wrote:
....
Sounds much like the way py2exe already works now. It locates the
appended zip-file by searching the exefile from the end, then finds the
beginning of the zipfile, and looks for a magic number there, which is
used to verify that the next n bytes before this position is a C
structure containing the required flags.

I don't like the idea to scan the executable for a magic signature
without further hints where this should be.


I wasn't suggesting scanning the executable at runtime. When py2exe
reads the python executablein order to create a copy with an appended
zip archive it might as well scan for a signature and modify the
customization area that follows it. When this customized interpreter
loads the customization values will already be in memory in the form of
an initialized variable.

If the interpreter detects that the customization area has been modified
it will interpret it as command-line arguments instead of argv. This
enables complete control of things like compiler flags, optimization
flag, ignoring environment, etc. A short "-c" string can be used for
bootstrapping the appended zip. The call to PySys_SetArgv would use the
real argv, if course.

Oren

Jul 18 '05 #20

P: n/a
In comp.lang.python, bo**@oz.net (Bengt Richter) (Bengt Richter) wrote in
<bh**********@216.39.172.122>::

|>For instance, I have written a couple of little progs in Python to solve
|>problems for a client. I need to make them into executables, because, while
|>he's willing to install an .exe I've written, he's less sure about one
|>written by someone who has no contract with him, and no comeback if it goes
|>taters.
|You make the contract. If you use open source, he'll come back to you, and you'll
|be able to fix it, or find someone here who can. Python itself is written
|"by someone who has no contract with him" (presumably). How do you square that?
|
|How does he feel about installing via something you've made with Installshield
|(commercial installer .exe packager)? He will certainly have no easy way to deal
|with any problem with that other than call you, and you will have no easy way
|to check on what happened, because you can't just look at the code, or ask someone
|here who knows about the code, because it isn't openly available (AFAIK; for a price
|most commercial things are possible).
Hey, I didn't say it was a rational position. :)
--
Marc Wilson

Cleopatra Consultants Limited - IT Consultants
2 The Grange, Cricklade Street, Old Town, Swindon SN1 3HG
Tel: (44/0) 70-500-15051 Fax: (44/0) 870 164-0054
Mail: in**@cleopatra.co.uk Web: http://www.cleopatra.co.uk
__________________________________________________ _______________
Try MailTraq at https://my.mailtraq.com/register.asp?code=cleopatra
Jul 18 '05 #21

This discussion thread is closed

Replies have been disabled for this discussion.