would be nice: import from archive

Dan Perl <dp***@rogers.com> wrote:

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?
Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.
I am quite new to python so I should ask first whether there is already
something like that, although I did a search already. Or maybe such a
feature has already been discussed somewhere? If this is an original idea,
how can I propose it for future releases?

It's there already, and has been for over a year now;-).
Alex

Jul 18 '05 #6

Anthony Baxter

On Fri, 27 Aug 2004 17:15:49 GMT, Dan Perl <dp***@rogers.com> wrote:

My bad. Right after posting the message I found the 'Index of Python
Enhancement Proposals (PEPs)'
and there is PEP 273, 'Import Modules from Zip Archives', submitted by
James C. Ahlstrom. Thank you, Jim! And there are even two implementations
already.

Note that the PEP is not up-to-date. zipimport "just works" in Python
2.3 and Python 2.4. In the following example, we use the -v flag to
show where imports are coming from.

bonanza% cat hello.py
def hello():
print 'hello world'
bonanza% zip hello.zip hello.py
adding: hello.py (deflated 8%)
bonanza% rm hello.py
bonanza% python2.4 -v
Python 2.4a2 (#3, Aug 24 2004, 01:25:51)
[GCC 3.4.0 20040613 (Red Hat Linux 3.4.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
... lots and lots of lines showing default imports snipped ...

import hello Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named hello import sys
sys.path.append('hello.zip')
import hello # zipimport: found 1 names in hello.zip
dlopen("/usr/local/lib/python2.4/lib-dynload/zlib.so", 2);
import zlib # dynamically loaded from
/usr/local/lib/python2.4/lib-dynload/zlib.so
# zipimport: zlib available
import hello # loaded from Zip hello.zip/hello.py hello.hello()

hello world

At least on this box (linux) there's even a zip file on the default sys.path!

Anthony

Jul 18 '05 #7

al*****@yahoo.com (Alex Martelli) writes:

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.

Jar files are -signed- zip files. Is there some reason to not do that
for Python?

Jul 18 '05 #8

Dan Perl

I am using 2.3 and I tried importing from a zip file in a script of my own
and, yes, it works.

I never used the -v flag before but it's great for debugging some problems.
Thanks for mentioning it!

Dan

"Anthony Baxter" <an***********@gmail.com> wrote in message
news:ma**************************************@pyth on.org...

Note that the PEP is not up-to-date. zipimport "just works" in Python
2.3 and Python 2.4. In the following example, we use the -v flag to
show where imports are coming from.

Jul 18 '05 #9

Dan Perl

----- Original Message -----
From: "Alex Martelli" <al*****@yahoo.com>
Newsgroups: comp.lang.python
Sent: Friday, August 27, 2004 1:37 PM
Subject: Re: would be nice: import from archive

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.

[...]
It's there already, and has been for over a year now;-).

Alex

Alright, alright! And given the fact that it was added only in 2.3, that
explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
2nd ed. ;-)

Dan

Jul 18 '05 #10

Paul Rubin <http://ph****@NOSPAM.invalid> wrote:

al*****@yahoo.com (Alex Martelli) writes:
Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.

Jar files are -signed- zip files. Is there some reason to not do that
for Python?

I know of no reason to forbid support for such "signing", no. If you
want to offer a patch to zipimport to let it support whatever signing,
encryption, or other devilry appeals to you, this is definitely the
right moment if you hope to see it happen in Python 2.4.

If you mean patching zipimport to _forbid_ importing from any zipfile
whatsoever, including a plain vanilla one, I think it's too late for
THAT for Python 2.4 -- not sure what backwards incompatibilities that
might cause, but it's definitely not the kind of thing you can spring on
the world in a release that's already fast moving towards its
hopefully-last alpha release. (Adding functionality is one thing,
breaking compatibility with something that a previous release allowed is
quite another...).
Alex

Jul 18 '05 #11

Dan Perl <dp***@rogers.com> wrote:
...

Alright, alright! And given the fact that it was added only in 2.3, that
explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
2nd ed. ;-)

If somebody submits a good recipe about it, I'll be overjoyed to add it
(not sure what chapter -- 'files'? 'system administration'? 'programs
about programs'? -- ah well, I'll find a spot!-).

In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
breath!-), I'll write it up just like I will for all the new delights
since 2.2 -- but for the Cookbook I'm supposed to use recipes that
people do submit to the online cookbook site... OK, I and my co-editors
_do_ perform a lot of editing and merging, and occasionally do add a
recipe or three, but mostly it IS the book you all have written, with
over a hundred authors covering the subjects THEY think matter...
Alex

Jul 18 '05 #12

al*****@yahoo.com (Alex Martelli) writes:

I know of no reason to forbid support for such "signing", no. If you
want to offer a patch to zipimport to let it support whatever signing,
encryption, or other devilry appeals to you, this is definitely the
right moment if you hope to see it happen in Python 2.4.
I think the simplest is to just have zipimport understand jar files
and their signatures. There's enough supporting infrastructure needed
that getting it in 2.4 is probably asking a bit much, though.
If you mean patching zipimport to _forbid_ importing from any zipfile
whatsoever, including a plain vanilla one, I think it's too late for
THAT for Python 2.4

If signing is supported, then there has to be a way to reject imports
whose signatures don't verify. It could be a runtime option or
something, I guess. How does zipimport work anyway? I don't see it
in the library doc index for 2.3.

Jul 18 '05 #13

al*****@yahoo.com (Alex Martelli) writes:

Dan Perl <dp***@rogers.com> wrote:
...
Alright, alright! And given the fact that it was added only in 2.3, that
explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
2nd ed. ;-)

If somebody submits a good recipe about it, I'll be overjoyed to add it
(not sure what chapter -- 'files'? 'system administration'? 'programs
about programs'? -- ah well, I'll find a spot!-).

In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
breath!-), I'll write it up just like I will for all the new delights
since 2.2 -- but for the Cookbook I'm supposed to use recipes that
people do submit to the online cookbook site... OK, I and my co-editors
_do_ perform a lot of editing and merging, and occasionally do add a
recipe or three, but mostly it IS the book you all have written, with
over a hundred authors covering the subjects THEY think matter...

It would be great to have one example with more than one file.

From the discussion I got curious and tested it here and -- since
Python's so efficient I wasn't surprised that -- it worked.

$ cat test.py
def test():
print "Test from file 1"

$ cat test2.py
def test():
print "Test from file 2"

import sys
sys.path.append('test.zip')
import test
import test2
test.test() Test from file 1 test2.test() Test from file 2

I also noticed that there was no '.pyc' created for that import, as is
usually done for uncompressed modules.

Be seeing you,
--
Godoy. <go***@ieee.org>

Jul 18 '05 #14

Dan Perl

I thought about submitting a recipe but I couldn't think of a way to use it
in a good code 'snippet'. I am using the zipimport feature now to save
several configuration files together in a zip file (so I can have many
configurations saved in a convenient format, better than a new directory for
each configuration). That would be more of an example for using multiple
configurations but, anyway, it doesn't make for a short, well-contained
example. Sorry, Alex.

BTW, there are two books on Python that I keep on my Safari bookshelf:
'Python Cookbook' and 'Learning Python'. So my jab was made will all the
respect I can muster. I am really looking forward to the second edition,
recipe on zipimport or not.

Dan

"Alex Martelli" <al*****@yahoo.com> wrote in message
news:1gj7a9r.g7ec5a1g89rj0N%al*****@yahoo.com...

Dan Perl <dp***@rogers.com> wrote:
...
Alright, alright! And given the fact that it was added only in 2.3, that explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the 2nd ed. ;-)

If somebody submits a good recipe about it, I'll be overjoyed to add it
(not sure what chapter -- 'files'? 'system administration'? 'programs
about programs'? -- ah well, I'll find a spot!-).

In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
breath!-), I'll write it up just like I will for all the new delights
since 2.2 -- but for the Cookbook I'm supposed to use recipes that
people do submit to the online cookbook site... OK, I and my co-editors
_do_ perform a lot of editing and merging, and occasionally do add a
recipe or three, but mostly it IS the book you all have written, with
over a hundred authors covering the subjects THEY think matter...
Alex

Jul 18 '05 #15

Paul Rubin <http://ph****@NOSPAM.invalid> wrote:

al*****@yahoo.com (Alex Martelli) writes:
I know of no reason to forbid support for such "signing", no. If you
want to offer a patch to zipimport to let it support whatever signing,
encryption, or other devilry appeals to you, this is definitely the
right moment if you hope to see it happen in Python 2.4.

I think the simplest is to just have zipimport understand jar files
and their signatures. There's enough supporting infrastructure needed
that getting it in 2.4 is probably asking a bit much, though.
If you mean patching zipimport to _forbid_ importing from any zipfile
whatsoever, including a plain vanilla one, I think it's too late for
THAT for Python 2.4

If signing is supported, then there has to be a way to reject imports
whose signatures don't verify. It could be a runtime option or
something, I guess. How does zipimport work anyway? I don't see it
in the library doc index for 2.3.

While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking
at the Python source distribution, files:
dist/src/Modules/zipimport.c
dist/src/Lib/test/test_zipimport.py

The excu^H^H^H^H reason for the documentation scarcity, you can read at
the end of the docstring for zipimport...:
"""
It is usually not needed to use the zipimport module explicitly; it is
used by the builtin import mechanism for sys.path items that are paths
to Zip archives.
"""
Alex

Jul 18 '05 #16

Dan Perl <dp***@rogers.com> wrote:

I thought about submitting a recipe but I couldn't think of a way to use it
in a good code 'snippet'. I am using the zipimport feature now to save
several configuration files together in a zip file (so I can have many
configurations saved in a convenient format, better than a new directory for
each configuration). That would be more of an example for using multiple
configurations but, anyway, it doesn't make for a short, well-contained
example. Sorry, Alex.
Alas! Your analysis seems spot-on and is pretty close to why I haven't
done a zipimport recipe myself -- it's a very useful feature but it's
not easy to think of a way to show it off in a recipe that's just how
recipes should be -- short, self-contained, readable. We can still hope
that somebody else does think of something...

BTW, there are two books on Python that I keep on my Safari bookshelf:
'Python Cookbook' and 'Learning Python'. So my jab was made will all the
respect I can muster. I am really looking forward to the second edition,
recipe on zipimport or not.

I did take your friendly jab as just that, friendly and pleasant, and
answered in just the same vein. (I do hope the reason the Nutshell
isn't on your Safari bookshelf is that you have the paper copy always at
hand, right?-)
Alex

Jul 18 '05 #17

al*****@yahoo.com (Alex Martelli) writes:

While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking

OK. I'll look at the PEP. I think adding signing is hairy enough
that it should have its own round of discussion; should I see about
editing the PEP to add something about signing?

Jul 18 '05 #18

Paul Rubin <http://ph****@NOSPAM.invalid> wrote:

al*****@yahoo.com (Alex Martelli) writes:
While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking

OK. I'll look at the PEP. I think adding signing is hairy enough
that it should have its own round of discussion; should I see about
editing the PEP to add something about signing?

I think that would be an excellent idea. If it was just about allowing
import from signed zipfiles it might not be needed, but how best to let
the user optionally DIS-allow imports from UN-signed files does appear
to be something requiring a little debate. An environment variable
would have the advantage of letting the disallowing work even for the
early imports that Python does before application code gets control, but
some people dislike relying on environment variables particularly for
security-related configuration tasks. Would it make sense to rely on a
naming convention instead? I.e. foo.zip would be unsigned but bar.jar
would have to be signed or else no go. This would have the advantage of
allowing substantial granularity in controlling this.

Side question, does module zipfile already have the code to allow
reading such signed files? Otherwise a first, uncontentious step, in
parallel with the PEP, might be a patch to add that ability to zipfile.
Alex

Jul 18 '05 #19

al*****@yahoo.com (Alex Martelli) writes:

Would it make sense to rely on a naming convention instead?
I.e. foo.zip would be unsigned but bar.jar would have to be signed
or else no go. This would have the advantage of allowing
substantial granularity in controlling this.
I think this is reasonable, except what does the import statement look
like? Do you say something like "import frob from bar.jar"?
Side question, does module zipfile already have the code to allow
reading such signed files?

I think jar files are just zip files containing an extra file (called
"manifest") that has signatures in it. So you can import from a jar
as if it were a zip.

Jul 18 '05 #20

Just

In article <m3************@g2ctech.com>, Jorge Godoy <go***@ieee.org>
wrote:

It would be great to have one example with more than one file.

From the discussion I got curious and tested it here and -- since
Python's so efficient I wasn't surprised that -- it worked.

$ cat test.py
def test():
print "Test from file 1"

$ cat test2.py
def test():
print "Test from file 2"

import sys
sys.path.append('test.zip')
import test
import test2
test.test() Test from file 1 test2.test() Test from file 2

I also noticed that there was no '.pyc' created for that import, as is
usually done for uncompressed modules.

The zipimport module will never write to the zip archive, so for most
efficient imports, you have to store .pyc data in there yourself.
zipimport is mostly meant as a repackaging tool, and typical zip files
only contain .pyc files.

Just

Jul 18 '05 #21

Paul Rubin <http://ph****@NOSPAM.invalid> writes:

Side question, does module zipfile already have the code to allow
reading such signed files?

I think jar files are just zip files containing an extra file (called
"manifest") that has signatures in it. So you can import from a jar
as if it were a zip.

But to add to that, if module zipfile is going to eventually expect
jar files to be signed, the first patch needed is that if it doesn't
have code to actually check the signatures, it should refuse to load
jar files.

I guess I better check into what Java does about this. It's been a
while since I've used Java, but I seem to remember that signing is not
mandatory.

Jul 18 '05 #22

Paul Rubin <http://ph****@NOSPAM.invalid> wrote:

Paul Rubin <http://ph****@NOSPAM.invalid> writes:
Side question, does module zipfile already have the code to allow
reading such signed files?
I think jar files are just zip files containing an extra file (called
"manifest") that has signatures in it. So you can import from a jar
as if it were a zip.

But to add to that, if module zipfile is going to eventually expect
jar files to be signed, the first patch needed is that if it doesn't
have code to actually check the signatures, it should refuse to load
jar files.

Presumably that would be an optional argument on the ZipFile constructor
specifying what to do about signatures -- defaulting to 'ignore' for
backwards compatibility, I guess, but possibly 'strict' or 'optional' or
something.

I guess I better check into what Java does about this. It's been a
while since I've used Java, but I seem to remember that signing is not
mandatory.

OK, but it might make for a nice optional feature anyway.
Alex

Jul 18 '05 #23

Paul Rubin <http://ph****@NOSPAM.invalid> wrote:

al*****@yahoo.com (Alex Martelli) writes:
Would it make sense to rely on a naming convention instead?
I.e. foo.zip would be unsigned but bar.jar would have to be signed
or else no go. This would have the advantage of allowing
substantial granularity in controlling this.

I think this is reasonable, except what does the import statement look
like? Do you say something like "import frob from bar.jar"?

No, you say, as always:

import frob

Importing looks at each item on sys.path, and each item can be:

1. a directory X -- then import looks for X/frob.py or a subdirectory
X/frob/ containing an __init__.py (or in either case .pyc or .pyo)

2. a zipfile X.zip -- then import looks inside (unsigned) file X.zip for
a frob.py, frob.pyc, etc

3. [only novelty...] a signed zipfile X.jar -- then import verifies the
signature then if valid proceed as in 2

Side question, does module zipfile already have the code to allow
reading such signed files?

I think jar files are just zip files containing an extra file (called
"manifest") that has signatures in it. So you can import from a jar
as if it were a zip.

But it might be nice to check signatures automatically if reading such
files is a common task.
Alex

Jul 18 '05 #24

al*****@yahoo.com (Alex Martelli) writes:

I guess I better check into what Java does about this. It's been a
while since I've used Java, but I seem to remember that signing is not
mandatory.

OK, but it might make for a nice optional feature anyway.

Well, in Java, jars are the only thing you can import from, and you
need to be able to import from unsigned ones or else you have to sign
them all the time. If we use a naming convention, then if you want an
unsigned archive you can just name it .zip instead of .jar. But
still, I better check. I do remember that in Javascript (which also
used jar files under the Netscape browsers of the day), you could load
code from unsigned jars and the code could do normal operations. But
code that did "dangerous" operations wouldn't run unless it was loaded
from a signed jar file.

Jul 18 '05 #25

Benjamin Niemann

Alex Martelli wrote:

Paul Rubin <http://ph****@NOSPAM.invalid> wrote:

al*****@yahoo.com (Alex Martelli) writes:
While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking
OK. I'll look at the PEP. I think adding signing is hairy enough
that it should have its own round of discussion; should I see about
editing the PEP to add something about signing?

I think that would be an excellent idea. If it was just about allowing
import from signed zipfiles it might not be needed, but how best to let
the user optionally DIS-allow imports from UN-signed files does appear
to be something requiring a little debate. An environment variable
would have the advantage of letting the disallowing work even for the
early imports that Python does before application code gets control, but
some people dislike relying on environment variables particularly for
security-related configuration tasks. Would it make sense to rely on a
naming convention instead? I.e. foo.zip would be unsigned but bar.jar
would have to be signed or else no go. This would have the advantage of
allowing substantial granularity in controlling this.

Isn't the purpose of signatures that the importing program can trust the
module? If it's implemented as you suggest, an attacker could just
inject path to an unsigned module into PYTHONPATH to fool a program. How
about something like

require_signature('mymodule')
import mymodule

or

import mymodule
verify_module(mymodule)

Another question is, where to place (require|verify)_signature() (that
could also take a CA key (or list of) as optional argument to only allow
modules signed by this CA). It must not be imported from an untrusted
module.
The whole signing thing probably make only sense, if python and it's
stdlib can be trusted (=signed).

Or am I missing other useful applications of signed archives?

Side question, does module zipfile already have the code to allow
reading such signed files? Otherwise a first, uncontentious step, in
parallel with the PEP, might be a patch to add that ability to zipfile.
Alex

Jul 18 '05 #26

Benjamin Niemann <pi**@odahoda.de> writes:

import mymodule
verify_module(mymodule)
This is no good. The import runs any code in the module, so the sig
has to verify BEFORE the module loads.
Another question is, where to place (require|verify)_signature() (that
could also take a CA key (or list of) as optional argument to only
allow modules signed by this CA). It must not be imported from an
untrusted module.

Correct, that's the messy infrastructure I mentioned. My basic idea is
"do whatever Java does".

Jul 18 '05 #27

Benjamin Niemann <pi**@odahoda.de> wrote:
...

Isn't the purpose of signatures that the importing program can trust the
module? If it's implemented as you suggest, an attacker could just
inject path to an unsigned module into PYTHONPATH to fool a program. How
If the attacker is able to alter sys.path then it does not matter
whether zipfiles are even considered -- the attacker could simply
position a .pyc file early on the path.
about something like

require_signature('mymodule')
import mymodule
This could be made to work, but only if _every_ module was so checked
before importing it; otherwise, even just one unchecked module could
easily subvert __import__ or other aspects of the import hook mechanism.

So, if you're considering this approach, it makes more sense to switch
on module checking globally in an early phase of Python's startup
(because Python starts importing modules pretty early indeed). New
conventions will also be needed for signature of .py, .pyc, .pyo, and
..so (or other binary DLLoid files containing Python extensions).

It appears to me that this is a project of orders of magnitude more work
than the original idea, which didn't assume the attacker could freely
alter sys.path, and protected only against altered or replaced zipfiles
specifically -- presumably files that have been legitimately placed on
the path by authorized agents.

or

import mymodule
verify_module(mymodule)
Too late:'import mymodule' runs code in mymodule, shutting the barn door
after mymodule has tramped all over your system is little use.

Another question is, where to place (require|verify)_signature() (that
could also take a CA key (or list of) as optional argument to only allow
modules signed by this CA). It must not be imported from an untrusted
module.
The whole signing thing probably make only sense, if python and it's
stdlib can be trusted (=signed).
The stdlib Python (.pyc) parts could be moved into a .jar (signed) just
as easily as into a .zip (unsigned). The EXE and DLL's involved may be
quite a problem, though, since for those you're in the hands of the
operating system -- what could Python itself possibly do to stop an
altered Python.Exe from running?!

Or am I missing other useful applications of signed archives?

Remote distribution of code. My program's startup checks if an updated
version of foobar.jar purports to be available, and if so downloads it
and places it where the previous version used to be. Admittedly, in
this case, checking once and for all right after the download might work
better than checking after each and every import. (Not sure why but
this reminds me of the old 'end to end approach' issue;-).

Unfortunately, Python currently doesn't have a working 'sandbox'
mechanism where code might run in a resricted way if it hadn't passed
all needed checks. This lack, among other things, may certainly lessen
the usefulness of checks performed at (or, rather, just before) import
time.
Alex

Jul 18 '05 #28

Benjamin Niemann

Alex Martelli wrote:

Benjamin Niemann <pi**@odahoda.de> wrote:
...
Isn't the purpose of signatures that the importing program can trust the
module? If it's implemented as you suggest, an attacker could just
inject path to an unsigned module into PYTHONPATH to fool a program. How

If the attacker is able to alter sys.path then it does not matter
whether zipfiles are even considered -- the attacker could simply
position a .pyc file early on the path.

about something like

require_signature('mymodule')
import mymodule

This could be made to work, but only if _every_ module was so checked
before importing it; otherwise, even just one unchecked module could
easily subvert __import__ or other aspects of the import hook mechanism.

So, if you're considering this approach, it makes more sense to switch
on module checking globally in an early phase of Python's startup
(because Python starts importing modules pretty early indeed). New
conventions will also be needed for signature of .py, .pyc, .pyo, and
.so (or other binary DLLoid files containing Python extensions).

It appears to me that this is a project of orders of magnitude more work
than the original idea, which didn't assume the attacker could freely
alter sys.path

Mmmm, seems I missed this point...
, and protected only against altered or replaced zipfiles
specifically -- presumably files that have been legitimately placed on
the path by authorized agents.

or

import mymodule
verify_module(mymodule)

Too late:'import mymodule' runs code in mymodule, shutting the barn door
after mymodule has tramped all over your system is little use.

correct

Another question is, where to place (require|verify)_signature() (that
could also take a CA key (or list of) as optional argument to only allow
modules signed by this CA). It must not be imported from an untrusted
module.
The whole signing thing probably make only sense, if python and it's
stdlib can be trusted (=signed).

The stdlib Python (.pyc) parts could be moved into a .jar (signed) just
as easily as into a .zip (unsigned). The EXE and DLL's involved may be
quite a problem, though, since for those you're in the hands of the
operating system -- what could Python itself possibly do to stop an
altered Python.Exe from running?!

A use-case that came to my mind was a suid program that wants to verify
that everything that it imports is what it expects, specifically not to
allow the non-root user to inject any malicious replacement modules.
Doing this with module signatures is probably not the easiest way...

Or am I missing other useful applications of signed archives?

Remote distribution of code. My program's startup checks if an updated
version of foobar.jar purports to be available, and if so downloads it
and places it where the previous version used to be. Admittedly, in
this case, checking once and for all right after the download might work
better than checking after each and every import. (Not sure why but
this reminds me of the old 'end to end approach' issue;-).

Yes, this could be handled by a generic 'file-signature-verification'
mechanism. No need to delay verification until import.

Unfortunately, Python currently doesn't have a working 'sandbox'
mechanism where code might run in a resricted way if it hadn't passed
all needed checks. This lack, among other things, may certainly lessen
the usefulness of checks performed at (or, rather, just before) import
time. Yep. Python treats us as adults. But adults often have to deal with
children and the environment should give adults some kind of authority
and places where the kiddies can play without causing damage... ;) This
is currently not the case for Python.

Alex

Jul 18 '05 #29

Terry Reedy

"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7x************@ruckus.brouhaha.com...

Benjamin Niemann <pi**@odahoda.de> writes:
import mymodule
verify_module(mymodule)

This is no good. The import runs any code in the module, so the sig
has to verify BEFORE the module loads.

'import x' is syntactic sugar for 'x = __import__('x')'. I do not see it
as necessary that sugar for the common case need cover every possible case.
So, how about giving __import__ had an optional param 'signed' defaulted to
False, to allow signed =True or signed = CA?

Terry J. Reedy

Jul 18 '05 #30

"Terry Reedy" <tj*****@udel.edu> writes:

'import x' is syntactic sugar for 'x = __import__('x')'. I do not see it
as necessary that sugar for the common case need cover every possible case.
So, how about giving __import__ had an optional param 'signed' defaulted to
False, to allow signed =True or signed = CA?

Man, that __import__ thing is ugly. I think it's better to extend the
syntax, e.g.
import x(a,b) => __import__('x', {'a':None, 'b':None})
import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})

so you could say
import x(signed)
or
import x(signed, certfile='mycerts.pem')

or whatever.

Jul 18 '05 #31

Terry Reedy

"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7x************@ruckus.brouhaha.com...

"Terry Reedy" <tj*****@udel.edu> writes:
'import x' is syntactic sugar for 'x = __import__('x')'. I do not see
it
as necessary that sugar for the common case need cover every possible
case.
So, how about giving __import__ had an optional param 'signed' defaulted
to
False, to allow signed =True or signed = CA?
Man, that __import__ thing is ugly.

Yes... but importing from signed zips is sufficiently rare and esoteric
that I would not see surface ugliness that accompanies using current syntax
as the most important consideration.

I think it's better to extend the syntax, e.g.
import x(a,b) => __import__('x', {'a':None, 'b':None})
import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})

Identifier(args) is currently a call of identifier with args and
overloading that syntax to mean somthing similar but different is, to me,
even uglier in a different sort of way.

Terry J. Reedy

Jul 18 '05 #32

"Terry Reedy" <tj*****@udel.edu> writes:

I think it's better to extend the syntax, e.g.
import x(a,b) => __import__('x', {'a':None, 'b':None})
import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})

Identifier(args) is currently a call of identifier with args and
overloading that syntax to mean somthing similar but different is, to me,
even uglier in a different sort of way.

Ok, use brackets instead: import x[a,b].
Then it's just a matter of overloading the index operator on __import__.

Jul 18 '05 #33

Peter Otten

Paul Rubin wrote:

"Terry Reedy" <tj*****@udel.edu> writes:
> I think it's better to extend the syntax, e.g.
> import x(a,b) => __import__('x', {'a':None, 'b':None})
> import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})

Identifier(args) is currently a call of identifier with args and
overloading that syntax to mean somthing similar but different is, to me,
even uglier in a different sort of way.

Ok, use brackets instead: import x[a,b].
Then it's just a matter of overloading the index operator on __import__.

@signed
@certfile('mycerts.pem')
import x

anybody?
Peter

Jul 18 '05 #34

Just <ju**@xs4all.nl> writes:

The zipimport module will never write to the zip archive, so for most
efficient imports, you have to store .pyc data in there yourself.
zipimport is mostly meant as a repackaging tool, and typical zip files
only contain .pyc files.

They aren't created even outside of the zip archive, this is what I
meant ;-)

--
Godoy. <go***@ieee.org>

Jul 18 '05 #35

Just

In article <m3************@g2ctech.com>, Jorge Godoy <go***@ieee.org>
wrote:

Just <ju**@xs4all.nl> writes:
The zipimport module will never write to the zip archive, so for most
efficient imports, you have to store .pyc data in there yourself.
zipimport is mostly meant as a repackaging tool, and typical zip files
only contain .pyc files.

They aren't created even outside of the zip archive, this is what I
meant ;-)

But since .pyc's are always generated in the same directory as the .py
files, where else would you expect them to be generated?

Just

Jul 18 '05 #36

Just <ju**@xs4all.nl> writes:

But since .pyc's are always generated in the same directory as the .py
files, where else would you expect them to be generated?

At the directory where the zip archive is stored.

--
Godoy. <go***@ieee.org>

Jul 18 '05 #37

Just

In article <m3************@g2ctech.com>, Jorge Godoy <go***@ieee.org>
wrote:

Just <ju**@xs4all.nl> writes:
But since .pyc's are always generated in the same directory as the .py
files, where else would you expect them to be generated?

At the directory where the zip archive is stored.

How does that follow? The zip archive _itself_ is the "directory" where
the .py files are, why would Python suddenly choose to write .pyc files
one level up? And what about packages? It simply doesn't work that way.

Just

Jul 18 '05 #38

Martin v. Löwis

Paul Rubin wrote:

so you could say
import x(signed)
or
import x(signed, certfile='mycerts.pem')

or whatever.

I believe that import is the wrong point in time for checking
signatures. You want to check the signature when the file is
added to sys.path, i.e.

imp.verify_signature(filename)
sys.path.append(filename)

or

imp.verify_all_signatures(sys.path)

That way, you can guarantee that trusted code is on sys.path
all the time. Then, you can also trust any import statement.

Regards,
Martin

Jul 18 '05 #39

Just <ju**@xs4all.nl> writes:

In article <m3************@g2ctech.com>, Jorge Godoy <go***@ieee.org>
wrote:
Just <ju**@xs4all.nl> writes:
> But since .pyc's are always generated in the same directory as the .py
> files, where else would you expect them to be generated?

At the directory where the zip archive is stored.

How does that follow? The zip archive _itself_ is the "directory" where
the .py files are, why would Python suddenly choose to write .pyc files
one level up? And what about packages? It simply doesn't work that way.

Because the implementation that allowed to unzip the "directory" and
find the files in there would also allow to place the '.pyc' in a
different place such as the real directory where the zip file resides.

Python didn't use to open zip files also, and now it does. I don't see
nothing wrong with making it also writing the '.pyc' files. Do you?

--
Godoy. <go***@ieee.org>

Jul 18 '05 #40

Just

In article <m3************@g2ctech.com>, Jorge Godoy <go***@ieee.org>
wrote:

How does that follow? The zip archive _itself_ is the "directory" where
the .py files are, why would Python suddenly choose to write .pyc files
one level up? And what about packages? It simply doesn't work that way.

Because the implementation that allowed to unzip the "directory" and
find the files in there would also allow to place the '.pyc' in a
different place such as the real directory where the zip file resides.

Python didn't use to open zip files also, and now it does. I don't see
nothing wrong with making it also writing the '.pyc' files. Do you?

Yes, because it's conceptually inconsistent with how "regular" imports
work.

Just

Jul 18 '05 #41

Just <ju**@xs4all.nl> writes:

In article <m3************@g2ctech.com>, Jorge Godoy <go***@ieee.org>
wrote:
Because the implementation that allowed to unzip the "directory" and
find the files in there would also allow to place the '.pyc' in a
different place such as the real directory where the zip file resides.

Python didn't use to open zip files also, and now it does. I don't see
nothing wrong with making it also writing the '.pyc' files. Do you?

Yes, because it's conceptually inconsistent with how "regular" imports
work.

So here, we have our thoughts taking two separate ways. I don't see too
much inconsistency here. You do. :-)

But, packaging the ".pyc" solves the problem... Just warning that they
are not generated after the packing is a good advice to have. It
wouldn't hurt anybody (and takes less time than we took talking about it
here).
Be seeing you,
--
Godoy. <go***@ieee.org>

Jul 18 '05 #42