would be nice: import from archive

Here is a python feature that I would like: to be able to import modules
from an archive like the jar files in Java. Maybe a regular tar file?
Maybe a python specific file type, let's call it a 'par' file?

It would be useful in packaging an python library. Sure, there's always the
python packages, but a single file instead of a whole directory tree would
be more convenient. I am particularly interested because I am working on a
framework/toolkit and I am using a configuration divided into several
modules. It would be nice to be able to save configurations (combinations
of several modules) into single archive files and then switch between
configurations by pointing to one such archive file.

I am quite new to python so I should ask first whether there is already
something like that, although I did a search already. Or maybe such a
feature has already been discussed somewhere? If this is an original idea,
how can I propose it for future releases?

Dan Perl
(yes, I'm Mr. Perl, but I'm using Python, after all Perl is not my middle
name, it's my last name)

re: would be nice: import from archive

On Fri, 27 Aug 2004 17:06:06 GMT, Dan Perl <dperl@rogers.com> wrote:[color=blue]
> Here is a python feature that I would like: to be able to import modules
> from an archive like the jar files in Java. Maybe a regular tar file?
> Maybe a python specific file type, let's call it a 'par' file?[/color]

Or a zip file, perhaps? See PEP-0273, which was implemented in Python 2.3.

(Hm. 273 isn't up-to-date. This is bad :-(

re: would be nice: import from archive

My bad. Right after posting the message I found the 'Index of Python
Enhancement Proposals (PEPs)'
and there is PEP 273, 'Import Modules from Zip Archives', submitted by
James C. Ahlstrom. Thank you, Jim! And there are even two implementations
already.

Dan

"Dan Perl" <dperl@rogers.com> wrote in message
news:2SJXc.79576$pTn.26490@news01.bloor.is.net.cab le.rogers.com...[color=blue]
> Here is a python feature that I would like: to be able to import modules
> from an archive like the jar files in Java. Maybe a regular tar file?
> Maybe a python specific file type, let's call it a 'par' file?
>
> It would be useful in packaging an python library. Sure, there's always[/color]
the[color=blue]
> python packages, but a single file instead of a whole directory tree would
> be more convenient. I am particularly interested because I am working on[/color]
a[color=blue]
> framework/toolkit and I am using a configuration divided into several
> modules. It would be nice to be able to save configurations (combinations
> of several modules) into single archive files and then switch between
> configurations by pointing to one such archive file.
>
> I am quite new to python so I should ask first whether there is already
> something like that, although I did a search already. Or maybe such a
> feature has already been discussed somewhere? If this is an original[/color]
idea,[color=blue]
> how can I propose it for future releases?
>
> Dan Perl
> (yes, I'm Mr. Perl, but I'm using Python, after all Perl is not my middle
> name, it's my last name)
>
>[/color]

re: would be nice: import from archive

Yes, I saw that after posting my initial message. Thanks though, Anthony!

Dan

"Anthony Baxter" <anthonybaxter@gmail.com> wrote in message
news:mailman.2538.1093626899.5135.python-list@python.org...[color=blue]
> On Fri, 27 Aug 2004 17:06:06 GMT, Dan Perl <dperl@rogers.com> wrote:[color=green]
> > Here is a python feature that I would like: to be able to import modules
> > from an archive like the jar files in Java. Maybe a regular tar file?
> > Maybe a python specific file type, let's call it a 'par' file?[/color]
>
> Or a zip file, perhaps? See PEP-0273, which was implemented in Python 2.3.
>
> (Hm. 273 isn't up-to-date. This is bad :-([/color]

re: would be nice: import from archive

"Dan Perl" <dperl@rogers.com> wrote in message
news:2SJXc.79576$pTn.26490@news01.bloor.is.net.cab le.rogers.com...[color=blue]
> Here is a python feature that I would like: to be able to import modules
> from an archive like the jar files in Java. Maybe a regular tar file?[/color]
[snip][color=blue]
> I am quite new to python so I should ask first whether there is already
> something like that, although I did a search already. Or maybe such a
> feature has already been discussed somewhere? If this is an original[/color]
idea,[color=blue]
> how can I propose it for future releases?[/color]


Well, you can use .zip files
http://www.python.org/doc/2.3.4/whatsnew/node5.html
HTH,
Sean

re: would be nice: import from archive

Dan Perl <dperl@rogers.com> wrote:
[color=blue]
> Here is a python feature that I would like: to be able to import modules
> from an archive like the jar files in Java. Maybe a regular tar file?[/color]

Python 2.3 lets you import modules from a zipfile. The zip format is
better than tar when you just need to get one file from it, which is why
java's jar files are also basically zipfiles.
[color=blue]
> I am quite new to python so I should ask first whether there is already
> something like that, although I did a search already. Or maybe such a
> feature has already been discussed somewhere? If this is an original idea,
> how can I propose it for future releases?[/color]

It's there already, and has been for over a year now;-).


Alex

re: would be nice: import from archive

On Fri, 27 Aug 2004 17:15:49 GMT, Dan Perl <dperl@rogers.com> wrote:[color=blue]
> My bad. Right after posting the message I found the 'Index of Python
> Enhancement Proposals (PEPs)'
> and there is PEP 273, 'Import Modules from Zip Archives', submitted by
> James C. Ahlstrom. Thank you, Jim! And there are even two implementations
> already.[/color]

Note that the PEP is not up-to-date. zipimport "just works" in Python
2.3 and Python 2.4. In the following example, we use the -v flag to
show where imports are coming from.

bonanza% cat hello.py
def hello():
print 'hello world'
bonanza% zip hello.zip hello.py
adding: hello.py (deflated 8%)
bonanza% rm hello.py
bonanza% python2.4 -v
Python 2.4a2 (#3, Aug 24 2004, 01:25:51)
[GCC 3.4.0 20040613 (Red Hat Linux 3.4.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
... lots and lots of lines showing default imports snipped ...[color=blue][color=green][color=darkred]
>>> import hello[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: No module named hello[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.path.append('hello.zip')
>>> import hello[/color][/color][/color]
# zipimport: found 1 names in hello.zip
dlopen("/usr/local/lib/python2.4/lib-dynload/zlib.so", 2);
import zlib # dynamically loaded from
/usr/local/lib/python2.4/lib-dynload/zlib.so
# zipimport: zlib available
import hello # loaded from Zip hello.zip/hello.py[color=blue][color=green][color=darkred]
>>> hello.hello()[/color][/color][/color]
hello world

At least on this box (linux) there's even a zip file on the default sys.path!

Anthony

re: would be nice: import from archive

aleaxit@yahoo.com (Alex Martelli) writes:[color=blue][color=green]
> > Here is a python feature that I would like: to be able to import modules
> > from an archive like the jar files in Java. Maybe a regular tar file?[/color]
>
> Python 2.3 lets you import modules from a zipfile. The zip format is
> better than tar when you just need to get one file from it, which is why
> java's jar files are also basically zipfiles.[/color]

Jar files are -signed- zip files. Is there some reason to not do that
for Python?

re: would be nice: import from archive

I am using 2.3 and I tried importing from a zip file in a script of my own
and, yes, it works.

I never used the -v flag before but it's great for debugging some problems.
Thanks for mentioning it!

Dan

"Anthony Baxter" <anthonybaxter@gmail.com> wrote in message
news:mailman.2541.1093628460.5135.python-list@python.org...[color=blue]
> Note that the PEP is not up-to-date. zipimport "just works" in Python
> 2.3 and Python 2.4. In the following example, we use the -v flag to
> show where imports are coming from.[/color]

re: would be nice: import from archive

----- Original Message -----
From: "Alex Martelli" <aleaxit@yahoo.com>
Newsgroups: comp.lang.python
Sent: Friday, August 27, 2004 1:37 PM
Subject: Re: would be nice: import from archive

[color=blue]
> Python 2.3 lets you import modules from a zipfile. The zip format is
> better than tar when you just need to get one file from it, which is why
> java's jar files are also basically zipfiles.[/color]


[...]
[color=blue]
> It's there already, and has been for over a year now;-).
>
> Alex[/color]

Alright, alright! And given the fact that it was added only in 2.3, that
explains why I couldn't find any mention of the feature in 'Python
Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
2nd ed. ;-)

Dan

re: would be nice: import from archive

Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
[color=blue]
> aleaxit@yahoo.com (Alex Martelli) writes:[color=green][color=darkred]
> > > Here is a python feature that I would like: to be able to import modules
> > > from an archive like the jar files in Java. Maybe a regular tar file?[/color]
> >
> > Python 2.3 lets you import modules from a zipfile. The zip format is
> > better than tar when you just need to get one file from it, which is why
> > java's jar files are also basically zipfiles.[/color]
>
> Jar files are -signed- zip files. Is there some reason to not do that
> for Python?[/color]

I know of no reason to forbid support for such "signing", no. If you
want to offer a patch to zipimport to let it support whatever signing,
encryption, or other devilry appeals to you, this is definitely the
right moment if you hope to see it happen in Python 2.4.

If you mean patching zipimport to _forbid_ importing from any zipfile
whatsoever, including a plain vanilla one, I think it's too late for
THAT for Python 2.4 -- not sure what backwards incompatibilities that
might cause, but it's definitely not the kind of thing you can spring on
the world in a release that's already fast moving towards its
hopefully-last alpha release. (Adding functionality is one thing,
breaking compatibility with something that a previous release allowed is
quite another...).


Alex

re: would be nice: import from archive

Dan Perl <dperl@rogers.com> wrote:
...[color=blue]
> Alright, alright! And given the fact that it was added only in 2.3, that
> explains why I couldn't find any mention of the feature in 'Python
> Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
> 2nd ed. ;-)[/color]

If somebody submits a good recipe about it, I'll be overjoyed to add it
(not sure what chapter -- 'files'? 'system administration'? 'programs
about programs'? -- ah well, I'll find a spot!-).

In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
breath!-), I'll write it up just like I will for all the new delights
since 2.2 -- but for the Cookbook I'm supposed to use recipes that
people do submit to the online cookbook site... OK, I and my co-editors
_do_ perform a lot of editing and merging, and occasionally do add a
recipe or three, but mostly it IS the book you all have written, with
over a hundred authors covering the subjects THEY think matter...


Alex

re: would be nice: import from archive

aleaxit@yahoo.com (Alex Martelli) writes:[color=blue]
> I know of no reason to forbid support for such "signing", no. If you
> want to offer a patch to zipimport to let it support whatever signing,
> encryption, or other devilry appeals to you, this is definitely the
> right moment if you hope to see it happen in Python 2.4.[/color]

I think the simplest is to just have zipimport understand jar files
and their signatures. There's enough supporting infrastructure needed
that getting it in 2.4 is probably asking a bit much, though.
[color=blue]
> If you mean patching zipimport to _forbid_ importing from any zipfile
> whatsoever, including a plain vanilla one, I think it's too late for
> THAT for Python 2.4[/color]

If signing is supported, then there has to be a way to reject imports
whose signatures don't verify. It could be a runtime option or
something, I guess. How does zipimport work anyway? I don't see it
in the library doc index for 2.3.

re: would be nice: import from archive

aleaxit@yahoo.com (Alex Martelli) writes:
[color=blue]
> Dan Perl <dperl@rogers.com> wrote:
> ...[color=green]
>> Alright, alright! And given the fact that it was added only in 2.3, that
>> explains why I couldn't find any mention of the feature in 'Python
>> Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in the
>> 2nd ed. ;-)[/color]
>
> If somebody submits a good recipe about it, I'll be overjoyed to add it
> (not sure what chapter -- 'files'? 'system administration'? 'programs
> about programs'? -- ah well, I'll find a spot!-).
>
> In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
> breath!-), I'll write it up just like I will for all the new delights
> since 2.2 -- but for the Cookbook I'm supposed to use recipes that
> people do submit to the online cookbook site... OK, I and my co-editors
> _do_ perform a lot of editing and merging, and occasionally do add a
> recipe or three, but mostly it IS the book you all have written, with
> over a hundred authors covering the subjects THEY think matter...[/color]

It would be great to have one example with more than one file.

From the discussion I got curious and tested it here and -- since
Python's so efficient I wasn't surprised that -- it worked.



$ cat test.py
def test():
print "Test from file 1"

$ cat test2.py
def test():
print "Test from file 2"

[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.path.append('test.zip')
>>> import test
>>> import test2
>>> test.test()[/color][/color][/color]
Test from file 1[color=blue][color=green][color=darkred]
>>> test2.test()[/color][/color][/color]
Test from file 2[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]


I also noticed that there was no '.pyc' created for that import, as is
usually done for uncompressed modules.



Be seeing you,
--
Godoy. <godoy@ieee.org>

re: would be nice: import from archive

I thought about submitting a recipe but I couldn't think of a way to use it
in a good code 'snippet'. I am using the zipimport feature now to save
several configuration files together in a zip file (so I can have many
configurations saved in a convenient format, better than a new directory for
each configuration). That would be more of an example for using multiple
configurations but, anyway, it doesn't make for a short, well-contained
example. Sorry, Alex.

BTW, there are two books on Python that I keep on my Safari bookshelf:
'Python Cookbook' and 'Learning Python'. So my jab was made will all the
respect I can muster. I am really looking forward to the second edition,
recipe on zipimport or not.

Dan

"Alex Martelli" <aleaxit@yahoo.com> wrote in message
news:1gj7a9r.g7ec5a1g89rj0N%aleaxit@yahoo.com...[color=blue]
> Dan Perl <dperl@rogers.com> wrote:
> ...[color=green]
> > Alright, alright! And given the fact that it was added only in 2.3,[/color][/color]
that[color=blue][color=green]
> > explains why I couldn't find any mention of the feature in 'Python
> > Cookbook', the 1st ed. I'm looking forward to seeing it mentioned in[/color][/color]
the[color=blue][color=green]
> > 2nd ed. ;-)[/color]
>
> If somebody submits a good recipe about it, I'll be overjoyed to add it
> (not sure what chapter -- 'files'? 'system administration'? 'programs
> about programs'? -- ah well, I'll find a spot!-).
>
> In the 2nd Edition of the _Nutshell_, when THAT comes (don't hold your
> breath!-), I'll write it up just like I will for all the new delights
> since 2.2 -- but for the Cookbook I'm supposed to use recipes that
> people do submit to the online cookbook site... OK, I and my co-editors
> _do_ perform a lot of editing and merging, and occasionally do add a
> recipe or three, but mostly it IS the book you all have written, with
> over a hundred authors covering the subjects THEY think matter...
>
>
> Alex[/color]

re: would be nice: import from archive

Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
[color=blue]
> aleaxit@yahoo.com (Alex Martelli) writes:[color=green]
> > I know of no reason to forbid support for such "signing", no. If you
> > want to offer a patch to zipimport to let it support whatever signing,
> > encryption, or other devilry appeals to you, this is definitely the
> > right moment if you hope to see it happen in Python 2.4.[/color]
>
> I think the simplest is to just have zipimport understand jar files
> and their signatures. There's enough supporting infrastructure needed
> that getting it in 2.4 is probably asking a bit much, though.
>[color=green]
> > If you mean patching zipimport to _forbid_ importing from any zipfile
> > whatsoever, including a plain vanilla one, I think it's too late for
> > THAT for Python 2.4[/color]
>
> If signing is supported, then there has to be a way to reject imports
> whose signatures don't verify. It could be a runtime option or
> something, I guess. How does zipimport work anyway? I don't see it
> in the library doc index for 2.3.[/color]

While the overall way the new import hooks work is well documented in
their PEP, zipimport is admittedly underdocumented. I suggest peeking
at the Python source distribution, files:
dist/src/Modules/zipimport.c
dist/src/Lib/test/test_zipimport.py

The excu^H^H^H^H reason for the documentation scarcity, you can read at
the end of the docstring for zipimport...:
"""
It is usually not needed to use the zipimport module explicitly; it is
used by the builtin import mechanism for sys.path items that are paths
to Zip archives.
"""


Alex

re: would be nice: import from archive

Dan Perl <dperl@rogers.com> wrote:
[color=blue]
> I thought about submitting a recipe but I couldn't think of a way to use it
> in a good code 'snippet'. I am using the zipimport feature now to save
> several configuration files together in a zip file (so I can have many
> configurations saved in a convenient format, better than a new directory for
> each configuration). That would be more of an example for using multiple
> configurations but, anyway, it doesn't make for a short, well-contained
> example. Sorry, Alex.[/color]

Alas! Your analysis seems spot-on and is pretty close to why I haven't
done a zipimport recipe myself -- it's a very useful feature but it's
not easy to think of a way to show it off in a recipe that's just how
recipes should be -- short, self-contained, readable. We can still hope
that somebody else does think of something...
[color=blue]
>
> BTW, there are two books on Python that I keep on my Safari bookshelf:
> 'Python Cookbook' and 'Learning Python'. So my jab was made will all the
> respect I can muster. I am really looking forward to the second edition,
> recipe on zipimport or not.[/color]

I did take your friendly jab as just that, friendly and pleasant, and
answered in just the same vein. (I do hope the reason the Nutshell
isn't on your Safari bookshelf is that you have the paper copy always at
hand, right?-)


Alex

re: would be nice: import from archive

aleaxit@yahoo.com (Alex Martelli) writes:[color=blue]
> While the overall way the new import hooks work is well documented in
> their PEP, zipimport is admittedly underdocumented. I suggest peeking[/color]

OK. I'll look at the PEP. I think adding signing is hairy enough
that it should have its own round of discussion; should I see about
editing the PEP to add something about signing?

re: would be nice: import from archive

Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
[color=blue]
> aleaxit@yahoo.com (Alex Martelli) writes:[color=green]
> > While the overall way the new import hooks work is well documented in
> > their PEP, zipimport is admittedly underdocumented. I suggest peeking[/color]
>
> OK. I'll look at the PEP. I think adding signing is hairy enough
> that it should have its own round of discussion; should I see about
> editing the PEP to add something about signing?[/color]

I think that would be an excellent idea. If it was just about allowing
import from signed zipfiles it might not be needed, but how best to let
the user optionally DIS-allow imports from UN-signed files does appear
to be something requiring a little debate. An environment variable
would have the advantage of letting the disallowing work even for the
early imports that Python does before application code gets control, but
some people dislike relying on environment variables particularly for
security-related configuration tasks. Would it make sense to rely on a
naming convention instead? I.e. foo.zip would be unsigned but bar.jar
would have to be signed or else no go. This would have the advantage of
allowing substantial granularity in controlling this.

Side question, does module zipfile already have the code to allow
reading such signed files? Otherwise a first, uncontentious step, in
parallel with the PEP, might be a patch to add that ability to zipfile.


Alex

re: would be nice: import from archive

aleaxit@yahoo.com (Alex Martelli) writes:[color=blue]
> Would it make sense to rely on a naming convention instead?
> I.e. foo.zip would be unsigned but bar.jar would have to be signed
> or else no go. This would have the advantage of allowing
> substantial granularity in controlling this.[/color]

I think this is reasonable, except what does the import statement look
like? Do you say something like "import frob from bar.jar"?
[color=blue]
> Side question, does module zipfile already have the code to allow
> reading such signed files?[/color]

I think jar files are just zip files containing an extra file (called
"manifest") that has signatures in it. So you can import from a jar
as if it were a zip.

re: would be nice: import from archive

In article <m3wtzk4562.fsf@g2ctech.com>, Jorge Godoy <godoy@ieee.org>
wrote:
[color=blue]
> It would be great to have one example with more than one file.
>
> From the discussion I got curious and tested it here and -- since
> Python's so efficient I wasn't surprised that -- it worked.
>
>
>
> $ cat test.py
> def test():
> print "Test from file 1"
>
> $ cat test2.py
> def test():
> print "Test from file 2"
>
>[color=green][color=darkred]
> >>> import sys
> >>> sys.path.append('test.zip')
> >>> import test
> >>> import test2
> >>> test.test()[/color][/color]
> Test from file 1[color=green][color=darkred]
> >>> test2.test()[/color][/color]
> Test from file 2[color=green][color=darkred]
> >>>[/color][/color]
>
>
> I also noticed that there was no '.pyc' created for that import, as is
> usually done for uncompressed modules.[/color]

The zipimport module will never write to the zip archive, so for most
efficient imports, you have to store .pyc data in there yourself.
zipimport is mostly meant as a repackaging tool, and typical zip files
only contain .pyc files.

Just

re: would be nice: import from archive

Paul Rubin <http://phr.cx@NOSPAM.invalid> writes:[color=blue][color=green]
> > Side question, does module zipfile already have the code to allow
> > reading such signed files?[/color]
>
> I think jar files are just zip files containing an extra file (called
> "manifest") that has signatures in it. So you can import from a jar
> as if it were a zip.[/color]

But to add to that, if module zipfile is going to eventually expect
jar files to be signed, the first patch needed is that if it doesn't
have code to actually check the signatures, it should refuse to load
jar files.

I guess I better check into what Java does about this. It's been a
while since I've used Java, but I seem to remember that signing is not
mandatory.

re: would be nice: import from archive

Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
[color=blue]
> Paul Rubin <http://phr.cx@NOSPAM.invalid> writes:[color=green][color=darkred]
> > > Side question, does module zipfile already have the code to allow
> > > reading such signed files?[/color]
> >
> > I think jar files are just zip files containing an extra file (called
> > "manifest") that has signatures in it. So you can import from a jar
> > as if it were a zip.[/color]
>
> But to add to that, if module zipfile is going to eventually expect
> jar files to be signed, the first patch needed is that if it doesn't
> have code to actually check the signatures, it should refuse to load
> jar files.[/color]

Presumably that would be an optional argument on the ZipFile constructor
specifying what to do about signatures -- defaulting to 'ignore' for
backwards compatibility, I guess, but possibly 'strict' or 'optional' or
something.
[color=blue]
>
> I guess I better check into what Java does about this. It's been a
> while since I've used Java, but I seem to remember that signing is not
> mandatory.[/color]

OK, but it might make for a nice optional feature anyway.


Alex

re: would be nice: import from archive

Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
[color=blue]
> aleaxit@yahoo.com (Alex Martelli) writes:[color=green]
> > Would it make sense to rely on a naming convention instead?
> > I.e. foo.zip would be unsigned but bar.jar would have to be signed
> > or else no go. This would have the advantage of allowing
> > substantial granularity in controlling this.[/color]
>
> I think this is reasonable, except what does the import statement look
> like? Do you say something like "import frob from bar.jar"?[/color]

No, you say, as always:

import frob

Importing looks at each item on sys.path, and each item can be:

1. a directory X -- then import looks for X/frob.py or a subdirectory
X/frob/ containing an __init__.py (or in either case .pyc or .pyo)

2. a zipfile X.zip -- then import looks inside (unsigned) file X.zip for
a frob.py, frob.pyc, etc

3. [only novelty...] a signed zipfile X.jar -- then import verifies the
signature then if valid proceed as in 2

[color=blue][color=green]
> > Side question, does module zipfile already have the code to allow
> > reading such signed files?[/color]
>
> I think jar files are just zip files containing an extra file (called
> "manifest") that has signatures in it. So you can import from a jar
> as if it were a zip.[/color]

But it might be nice to check signatures automatically if reading such
files is a common task.


Alex

re: would be nice: import from archive

aleaxit@yahoo.com (Alex Martelli) writes:[color=blue][color=green]
> > I guess I better check into what Java does about this. It's been a
> > while since I've used Java, but I seem to remember that signing is not
> > mandatory.[/color]
>
> OK, but it might make for a nice optional feature anyway.[/color]

Well, in Java, jars are the only thing you can import from, and you
need to be able to import from unsigned ones or else you have to sign
them all the time. If we use a naming convention, then if you want an
unsigned archive you can just name it .zip instead of .jar. But
still, I better check. I do remember that in Javascript (which also
used jar files under the Netscape browsers of the day), you could load
code from unsigned jars and the code could do normal operations. But
code that did "dangerous" operations wouldn't run unless it was loaded
from a signed jar file.

re: would be nice: import from archive

Alex Martelli wrote:[color=blue]
> Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
>
>[color=green]
>>aleaxit@yahoo.com (Alex Martelli) writes:
>>[color=darkred]
>>>While the overall way the new import hooks work is well documented in
>>>their PEP, zipimport is admittedly underdocumented. I suggest peeking[/color]
>>
>>OK. I'll look at the PEP. I think adding signing is hairy enough
>>that it should have its own round of discussion; should I see about
>>editing the PEP to add something about signing?[/color]
>
>
> I think that would be an excellent idea. If it was just about allowing
> import from signed zipfiles it might not be needed, but how best to let
> the user optionally DIS-allow imports from UN-signed files does appear
> to be something requiring a little debate. An environment variable
> would have the advantage of letting the disallowing work even for the
> early imports that Python does before application code gets control, but
> some people dislike relying on environment variables particularly for
> security-related configuration tasks. Would it make sense to rely on a
> naming convention instead? I.e. foo.zip would be unsigned but bar.jar
> would have to be signed or else no go. This would have the advantage of
> allowing substantial granularity in controlling this.[/color]
Isn't the purpose of signatures that the importing program can trust the
module? If it's implemented as you suggest, an attacker could just
inject path to an unsigned module into PYTHONPATH to fool a program. How
about something like

require_signature('mymodule')
import mymodule

or

import mymodule
verify_module(mymodule)

Another question is, where to place (require|verify)_signature() (that
could also take a CA key (or list of) as optional argument to only allow
modules signed by this CA). It must not be imported from an untrusted
module.
The whole signing thing probably make only sense, if python and it's
stdlib can be trusted (=signed).

Or am I missing other useful applications of signed archives?

[color=blue]
>
> Side question, does module zipfile already have the code to allow
> reading such signed files? Otherwise a first, uncontentious step, in
> parallel with the PEP, might be a patch to add that ability to zipfile.
>
>
> Alex
>[/color]

re: would be nice: import from archive

Benjamin Niemann <pink@odahoda.de> writes:[color=blue]
> import mymodule
> verify_module(mymodule)[/color]

This is no good. The import runs any code in the module, so the sig
has to verify BEFORE the module loads.
[color=blue]
> Another question is, where to place (require|verify)_signature() (that
> could also take a CA key (or list of) as optional argument to only
> allow modules signed by this CA). It must not be imported from an
> untrusted module.[/color]

Correct, that's the messy infrastructure I mentioned. My basic idea is
"do whatever Java does".

re: would be nice: import from archive

Benjamin Niemann <pink@odahoda.de> wrote:
...[color=blue]
> Isn't the purpose of signatures that the importing program can trust the
> module? If it's implemented as you suggest, an attacker could just
> inject path to an unsigned module into PYTHONPATH to fool a program. How[/color]

If the attacker is able to alter sys.path then it does not matter
whether zipfiles are even considered -- the attacker could simply
position a .pyc file early on the path.
[color=blue]
> about something like
>
> require_signature('mymodule')
> import mymodule[/color]

This could be made to work, but only if _every_ module was so checked
before importing it; otherwise, even just one unchecked module could
easily subvert __import__ or other aspects of the import hook mechanism.

So, if you're considering this approach, it makes more sense to switch
on module checking globally in an early phase of Python's startup
(because Python starts importing modules pretty early indeed). New
conventions will also be needed for signature of .py, .pyc, .pyo, and
..so (or other binary DLLoid files containing Python extensions).

It appears to me that this is a project of orders of magnitude more work
than the original idea, which didn't assume the attacker could freely
alter sys.path, and protected only against altered or replaced zipfiles
specifically -- presumably files that have been legitimately placed on
the path by authorized agents.
[color=blue]
>
> or
>
> import mymodule
> verify_module(mymodule)[/color]

Too late:'import mymodule' runs code in mymodule, shutting the barn door
after mymodule has tramped all over your system is little use.
[color=blue]
>
> Another question is, where to place (require|verify)_signature() (that
> could also take a CA key (or list of) as optional argument to only allow
> modules signed by this CA). It must not be imported from an untrusted
> module.
> The whole signing thing probably make only sense, if python and it's
> stdlib can be trusted (=signed).[/color]

The stdlib Python (.pyc) parts could be moved into a .jar (signed) just
as easily as into a .zip (unsigned). The EXE and DLL's involved may be
quite a problem, though, since for those you're in the hands of the
operating system -- what could Python itself possibly do to stop an
altered Python.Exe from running?!
[color=blue]
>
> Or am I missing other useful applications of signed archives?[/color]

Remote distribution of code. My program's startup checks if an updated
version of foobar.jar purports to be available, and if so downloads it
and places it where the previous version used to be. Admittedly, in
this case, checking once and for all right after the download might work
better than checking after each and every import. (Not sure why but
this reminds me of the old 'end to end approach' issue;-).

Unfortunately, Python currently doesn't have a working 'sandbox'
mechanism where code might run in a resricted way if it hadn't passed
all needed checks. This lack, among other things, may certainly lessen
the usefulness of checks performed at (or, rather, just before) import
time.


Alex

re: would be nice: import from archive

Alex Martelli wrote:
[color=blue]
> Benjamin Niemann <pink@odahoda.de> wrote:
> ...
>[color=green]
>>Isn't the purpose of signatures that the importing program can trust the
>>module? If it's implemented as you suggest, an attacker could just
>>inject path to an unsigned module into PYTHONPATH to fool a program. How[/color]
>
>
> If the attacker is able to alter sys.path then it does not matter
> whether zipfiles are even considered -- the attacker could simply
> position a .pyc file early on the path.
>
>[color=green]
>>about something like
>>
>>require_signature('mymodule')
>>import mymodule[/color]
>
>
> This could be made to work, but only if _every_ module was so checked
> before importing it; otherwise, even just one unchecked module could
> easily subvert __import__ or other aspects of the import hook mechanism.
>
> So, if you're considering this approach, it makes more sense to switch
> on module checking globally in an early phase of Python's startup
> (because Python starts importing modules pretty early indeed). New
> conventions will also be needed for signature of .py, .pyc, .pyo, and
> .so (or other binary DLLoid files containing Python extensions).
>
> It appears to me that this is a project of orders of magnitude more work
> than the original idea, which didn't assume the attacker could freely
> alter sys.path[/color]
Mmmm, seems I missed this point...
[color=blue]
> , and protected only against altered or replaced zipfiles
> specifically -- presumably files that have been legitimately placed on
> the path by authorized agents.
>
>[color=green]
>>or
>>
>>import mymodule
>>verify_module(mymodule)[/color]
>
>
> Too late:'import mymodule' runs code in mymodule, shutting the barn door
> after mymodule has tramped all over your system is little use.[/color]
correct
[color=blue][color=green]
>>Another question is, where to place (require|verify)_signature() (that
>>could also take a CA key (or list of) as optional argument to only allow
>>modules signed by this CA). It must not be imported from an untrusted
>>module.
>>The whole signing thing probably make only sense, if python and it's
>>stdlib can be trusted (=signed).[/color]
>
>
> The stdlib Python (.pyc) parts could be moved into a .jar (signed) just
> as easily as into a .zip (unsigned). The EXE and DLL's involved may be
> quite a problem, though, since for those you're in the hands of the
> operating system -- what could Python itself possibly do to stop an
> altered Python.Exe from running?![/color]
A use-case that came to my mind was a suid program that wants to verify
that everything that it imports is what it expects, specifically not to
allow the non-root user to inject any malicious replacement modules.
Doing this with module signatures is probably not the easiest way...
[color=blue][color=green]
>>Or am I missing other useful applications of signed archives?[/color]
>
>
> Remote distribution of code. My program's startup checks if an updated
> version of foobar.jar purports to be available, and if so downloads it
> and places it where the previous version used to be. Admittedly, in
> this case, checking once and for all right after the download might work
> better than checking after each and every import. (Not sure why but
> this reminds me of the old 'end to end approach' issue;-).[/color]
Yes, this could be handled by a generic 'file-signature-verification'
mechanism. No need to delay verification until import.
[color=blue]
>
> Unfortunately, Python currently doesn't have a working 'sandbox'
> mechanism where code might run in a resricted way if it hadn't passed
> all needed checks. This lack, among other things, may certainly lessen
> the usefulness of checks performed at (or, rather, just before) import
> time.[/color]
Yep. Python treats us as adults. But adults often have to deal with
children and the environment should give adults some kind of authority
and places where the kiddies can play without causing damage... ;) This
is currently not the case for Python.

[color=blue]
>
>
> Alex[/color]

re: would be nice: import from archive

"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7xd61b8uh2.fsf@ruckus.brouhaha.com...[color=blue]
> Benjamin Niemann <pink@odahoda.de> writes:[color=green]
>> import mymodule
>> verify_module(mymodule)[/color]
>
> This is no good. The import runs any code in the module, so the sig
> has to verify BEFORE the module loads.[/color]

'import x' is syntactic sugar for 'x = __import__('x')'. I do not see it
as necessary that sugar for the common case need cover every possible case.
So, how about giving __import__ had an optional param 'signed' defaulted to
False, to allow signed =True or signed = CA?

Terry J. Reedy

re: would be nice: import from archive

"Terry Reedy" <tjreedy@udel.edu> writes:[color=blue]
> 'import x' is syntactic sugar for 'x = __import__('x')'. I do not see it
> as necessary that sugar for the common case need cover every possible case.
> So, how about giving __import__ had an optional param 'signed' defaulted to
> False, to allow signed =True or signed = CA?[/color]

Man, that __import__ thing is ugly. I think it's better to extend the
syntax, e.g.
import x(a,b) => __import__('x', {'a':None, 'b':None})
import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})

so you could say
import x(signed)
or
import x(signed, certfile='mycerts.pem')

or whatever.

re: would be nice: import from archive

"Paul Rubin" <"http://phr.cx"@NOSPAM.invalid> wrote in message
news:7x1xhqg3kb.fsf@ruckus.brouhaha.com...[color=blue]
> "Terry Reedy" <tjreedy@udel.edu> writes:[color=green]
>> 'import x' is syntactic sugar for 'x = __import__('x')'. I do not see
>> it
>> as necessary that sugar for the common case need cover every possible
>> case.
>> So, how about giving __import__ had an optional param 'signed' defaulted
>> to
>> False, to allow signed =True or signed = CA?[/color]
>
> Man, that __import__ thing is ugly.[/color]

Yes... but importing from signed zips is sufficiently rare and esoteric
that I would not see surface ugliness that accompanies using current syntax
as the most important consideration.

[color=blue]
> I think it's better to extend the syntax, e.g.
> import x(a,b) => __import__('x', {'a':None, 'b':None})
> import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})[/color]

Identifier(args) is currently a call of identifier with args and
overloading that syntax to mean somthing similar but different is, to me,
even uglier in a different sort of way.

Terry J. Reedy

re: would be nice: import from archive

"Terry Reedy" <tjreedy@udel.edu> writes:[color=blue][color=green]
> > I think it's better to extend the syntax, e.g.
> > import x(a,b) => __import__('x', {'a':None, 'b':None})
> > import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})[/color]
>
> Identifier(args) is currently a call of identifier with args and
> overloading that syntax to mean somthing similar but different is, to me,
> even uglier in a different sort of way.[/color]

Ok, use brackets instead: import x[a,b].
Then it's just a matter of overloading the index operator on __import__.

re: would be nice: import from archive

Paul Rubin wrote:
[color=blue]
> "Terry Reedy" <tjreedy@udel.edu> writes:[color=green][color=darkred]
>> > I think it's better to extend the syntax, e.g.
>> > import x(a,b) => __import__('x', {'a':None, 'b':None})
>> > import x(a=v1,b=v2)=> __import__('x', {'a':v1, 'b':v2})[/color]
>>
>> Identifier(args) is currently a call of identifier with args and
>> overloading that syntax to mean somthing similar but different is, to me,
>> even uglier in a different sort of way.[/color]
>
> Ok, use brackets instead: import x[a,b].
> Then it's just a matter of overloading the index operator on __import__.[/color]

@signed
@certfile('mycerts.pem')
import x

anybody?


Peter

re: would be nice: import from archive

Just <just@xs4all.nl> writes:
[color=blue]
> The zipimport module will never write to the zip archive, so for most
> efficient imports, you have to store .pyc data in there yourself.
> zipimport is mostly meant as a repackaging tool, and typical zip files
> only contain .pyc files.[/color]

They aren't created even outside of the zip archive, this is what I
meant ;-)

--
Godoy. <godoy@ieee.org>

re: would be nice: import from archive

In article <m3sma5zqx0.fsf@g2ctech.com>, Jorge Godoy <godoy@ieee.org>
wrote:
[color=blue]
> Just <just@xs4all.nl> writes:
>[color=green]
> > The zipimport module will never write to the zip archive, so for most
> > efficient imports, you have to store .pyc data in there yourself.
> > zipimport is mostly meant as a repackaging tool, and typical zip files
> > only contain .pyc files.[/color]
>
> They aren't created even outside of the zip archive, this is what I
> meant ;-)[/color]

But since .pyc's are always generated in the same directory as the .py
files, where else would you expect them to be generated?

Just

re: would be nice: import from archive

Just <just@xs4all.nl> writes:
[color=blue]
> But since .pyc's are always generated in the same directory as the .py
> files, where else would you expect them to be generated?[/color]

At the directory where the zip archive is stored.

--
Godoy. <godoy@ieee.org>

re: would be nice: import from archive

In article <m3oektzkj2.fsf@g2ctech.com>, Jorge Godoy <godoy@ieee.org>
wrote:
[color=blue]
> Just <just@xs4all.nl> writes:
>[color=green]
> > But since .pyc's are always generated in the same directory as the .py
> > files, where else would you expect them to be generated?[/color]
>
> At the directory where the zip archive is stored.[/color]

How does that follow? The zip archive _itself_ is the "directory" where
the .py files are, why would Python suddenly choose to write .pyc files
one level up? And what about packages? It simply doesn't work that way.

Just

re: would be nice: import from archive

Paul Rubin wrote:[color=blue]
> so you could say
> import x(signed)
> or
> import x(signed, certfile='mycerts.pem')
>
> or whatever.[/color]

I believe that import is the wrong point in time for checking
signatures. You want to check the signature when the file is
added to sys.path, i.e.

imp.verify_signature(filename)
sys.path.append(filename)

or

imp.verify_all_signatures(sys.path)

That way, you can guarantee that trusted code is on sys.path
all the time. Then, you can also trust any import statement.

Regards,
Martin

re: would be nice: import from archive

Just <just@xs4all.nl> writes:
[color=blue]
> In article <m3oektzkj2.fsf@g2ctech.com>, Jorge Godoy <godoy@ieee.org>
> wrote:
>[color=green]
>> Just <just@xs4all.nl> writes:
>>[color=darkred]
>> > But since .pyc's are always generated in the same directory as the .py
>> > files, where else would you expect them to be generated?[/color]
>>
>> At the directory where the zip archive is stored.[/color]
>
> How does that follow? The zip archive _itself_ is the "directory" where
> the .py files are, why would Python suddenly choose to write .pyc files
> one level up? And what about packages? It simply doesn't work that way.[/color]

Because the implementation that allowed to unzip the "directory" and
find the files in there would also allow to place the '.pyc' in a
different place such as the real directory where the zip file resides.

Python didn't use to open zip files also, and now it does. I don't see
nothing wrong with making it also writing the '.pyc' files. Do you?

--
Godoy. <godoy@ieee.org>

re: would be nice: import from archive

In article <m3ekloont2.fsf@g2ctech.com>, Jorge Godoy <godoy@ieee.org>
wrote:
[color=blue][color=green]
> > How does that follow? The zip archive _itself_ is the "directory" where
> > the .py files are, why would Python suddenly choose to write .pyc files
> > one level up? And what about packages? It simply doesn't work that way.[/color]
>
> Because the implementation that allowed to unzip the "directory" and
> find the files in there would also allow to place the '.pyc' in a
> different place such as the real directory where the zip file resides.
>
> Python didn't use to open zip files also, and now it does. I don't see
> nothing wrong with making it also writing the '.pyc' files. Do you?[/color]

Yes, because it's conceptually inconsistent with how "regular" imports
work.

Just

re: would be nice: import from archive

Just <just@xs4all.nl> writes:
[color=blue]
> In article <m3ekloont2.fsf@g2ctech.com>, Jorge Godoy <godoy@ieee.org>
> wrote:
>[color=green]
>> Because the implementation that allowed to unzip the "directory" and
>> find the files in there would also allow to place the '.pyc' in a
>> different place such as the real directory where the zip file resides.
>>
>> Python didn't use to open zip files also, and now it does. I don't see
>> nothing wrong with making it also writing the '.pyc' files. Do you?[/color]
>
> Yes, because it's conceptually inconsistent with how "regular" imports
> work.[/color]

So here, we have our thoughts taking two separate ways. I don't see too
much inconsistency here. You do. :-)

But, packaging the ".pyc" solves the problem... Just warning that they
are not generated after the packing is a good advice to have. It
wouldn't hurt anybody (and takes less time than we took talking about it
here).


Be seeing you,
--
Godoy. <godoy@ieee.org>

re: would be nice: import from archive

"Martin v. Löwis" <martin@v.loewis.de> writes:[color=blue]
> I believe that import is the wrong point in time for checking
> signatures. You want to check the signature when the file is
> added to sys.path, i.e.
>
> imp.verify_signature(filename)
> sys.path.append(filename)[/color]

There's something to be said for that. Maybe you could append a tuple
to say how to verify signatures:

sys.add_library((filename, 'certfile.pem'))

checks the sig and updates sys.path. The whole notion of sys.path.append
(i.e. sys.path is just a naked Python list) is kludgy anyway.

re: would be nice: import from archive

In article <1gj8848.1bouuroizku25N%aleaxit@yahoo.com>, Alex Martelli wrote:[color=blue]
> Benjamin Niemann <pink@odahoda.de> wrote:
> ...[color=green]
>> Isn't the purpose of signatures that the importing program can trust the
>> module? If it's implemented as you suggest, an attacker could just
>> inject path to an unsigned module into PYTHONPATH to fool a program. How[/color]
>
> If the attacker is able to alter sys.path then it does not matter
> whether zipfiles are even considered -- the attacker could simply
> position a .pyc file early on the path.
>[color=green]
>> about something like
>>
>> require_signature('mymodule')
>> import mymodule[/color]
>
> This could be made to work, but only if _every_ module was so checked
> before importing it; otherwise, even just one unchecked module could
> easily subvert __import__ or other aspects of the import hook mechanism.
>
> So, if you're considering this approach, it makes more sense to switch
> on module checking globally in an early phase of Python's startup
> (because Python starts importing modules pretty early indeed). New
> conventions will also be needed for signature of .py, .pyc, .pyo, and
> .so (or other binary DLLoid files containing Python extensions).[/color]

It doesn't look like anyone has mentioned the Python Cryptography
Toolkit in this thread yet. (I have no affiliation with said project)

http://www.amk.ca/python/code/crypto.html

http://www.amk.ca/python/writing/pycrypt/pycrypt.html :

7.2 Demo 2: secimp and sign


secimp demonstrates an application of the Toolkit that may be useful
if Python is being used as an extension language for mail and Web
clients: secure importing of Python modules. To use it, run sign.py
in a directory with several compiled Python files present. It will
use the key in testkey.py to generate digital signatures for the
compiled Python code, and save both the signature and the code in a
file ending in ".pys". Then run python -i secimp.py, and import a
file by using secimport.


For example, if foo.pys was constructed, do secimport('foo'). The
import should succeed. Now fire up Emacs or some other editor, and
change a string in the code in foo.pys; you might try changing a
letter in the name of a variable. When you run secimport('foo'), it
should raise an exception reporting the failed signature. If you
execute the statement __import__ = secimport, the secure import will
be used by default for all future module imports. Alternatively, if
you were creating a restricted execution environment using rexec.py,
you could place secimport() in the restricted environment's
namespace as the default import function.



-Steve