By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,275 Members | 1,745 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,275 IT Pros & Developers. It's quick & easy.

importing pyc from memory?

P: n/a
Using the Python/C API, is there a way I can import a pyc file that I
have in memory (as opposed to loading from disk)?

I'm trying to save compiled python code in a proprietary file format to
cut reduce the overhead of compiling all my scripts when my app starts
up.

Derek

Jul 21 '05 #1
Share this Question
Share on Google+
10 Replies


P: n/a
Derek van Vliet enlightened us with:
I'm trying to save compiled python code in a proprietary file format
to cut reduce the overhead of compiling all my scripts when my app
starts up.


Why is that faster than having the .pyc files ready on your
filesystem? And why do you want it in a proprietary file format?

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Jul 21 '05 #2

P: n/a

"Derek van Vliet" <dv*******@3dna.net> schrieb im Newsbeitrag
news:11**********************@g49g2000cwa.googlegr oups.com...
Using the Python/C API, is there a way I can import a pyc file that I
have in memory (as opposed to loading from disk)?

I'm trying to save compiled python code in a proprietary file format to
cut reduce the overhead of compiling all my scripts when my app starts
up.

you can use a zip file as "proprietary file format"? with .pyc's or .py's in
and add the zippath to sys.path ?

other than that you can load the code object of .pyc's content in a string
like

co = marshal.loads( pyc_s[8:] )
mod=imp.new_module('xy')
exec co in mod.__dict__
Robert
Jul 21 '05 #3

P: n/a
Up to now, I've had all my python scripts defined in XML elements,
which were compiled when my program started up using Py_CompileString.
This has worked great, but I'm finding that most of the time my app
uses to start up is spent in that Py_CompileString, because of the
large number of scripts that I use.

Unfortunately, storing these scripts as .pyc files on disk is not an
option for me.

Jul 21 '05 #4

P: n/a
The second method you describe sounds like it is along the lines of
what I need to do. Is there a way to do this using the Python/C API?

For instance, if I have the contents of a pyc file loaded entirely into
memory, and I have a pointer to the beginning of the file and the size
in bytes at my disposal, can I import that by any means?

Jul 21 '05 #5

P: n/a
"Derek van Vliet" <dv*******@3dna.net> writes:
The second method you describe sounds like it is along the lines of
what I need to do. Is there a way to do this using the Python/C API?

For instance, if I have the contents of a pyc file loaded entirely into
memory, and I have a pointer to the beginning of the file and the size
in bytes at my disposal, can I import that by any means?


The .pyc file format is sort of 'documented' in lib/py_compile.py.
IIRC, it is a magic number, a timestamp, plus a marshalled code
object.

Thomas
Jul 21 '05 #6

P: n/a
This stupid code works for modules, but not for packages. It probably has bugs.
import marshal, types

class StringImporter:
def __init__(self, old_import, modules):
self._import = old_import
self._modules = modules

def __call__(self, name, *args):
module = self._modules.get(name, None)
if module is None:
return self._import(name, *args)
code = marshal.loads(module)
mod = types.ModuleType(name)
exec code in mod.__dict__
return mod

def test():
import __builtin__
__builtin__.__import__ = StringImporter(__builtin__.__import__,
{ 'test_importer': open("/usr/lib/python2.3/os.pyc").read()[8:] })

import test_importer
print test_importer.path.join("a", "b")
print test_importer.__doc__

if __name__ == '__main__': test()

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCyXlhJd01MZaTXX0RAgTkAJ9YBAd3ms+q1tWcjFe/jOkV0+pY4ACgn/RU
bakTEx933/aaDQa3Byp9204=
=uKAI
-----END PGP SIGNATURE-----

Jul 21 '05 #7

P: n/a
On Mon, 04 Jul 2005 19:25:13 +0200, Thomas Heller <th*****@python.net> wrote:
"Derek van Vliet" <dv*******@3dna.net> writes:
The second method you describe sounds like it is along the lines of
what I need to do. Is there a way to do this using the Python/C API?

For instance, if I have the contents of a pyc file loaded entirely into
memory, and I have a pointer to the beginning of the file and the size
in bytes at my disposal, can I import that by any means?


The .pyc file format is sort of 'documented' in lib/py_compile.py.
IIRC, it is a magic number, a timestamp, plus a marshalled code
object.

I'm wondering if what the OP has is really .pyc files "in memory" or code objects.
If he is using Py_CompileString to compile "scripts" extracted from XML,
wouldn't he get code objects? And if so, wouldn't he have to execute them
in a constructed module dict and be careful of normal imports in the "scripts"
etc etc to simulate import? Anyway, it feels like what he wants to do could be done,
but "the devil is in the details," which are missing ;-)

Regards,
Bengt Richter
Jul 21 '05 #8

P: n/a
Derek van Vliet enlightened us with:
Up to now, I've had all my python scripts defined in XML elements,
which were compiled when my program started up using
Py_CompileString. This has worked great, but I'm finding that most
of the time my app uses to start up is spent in that
Py_CompileString


It al makes sense now! Thanks for the explanation.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Jul 21 '05 #9

P: n/a
I do have to do everything you describe here in the current
implementation of my system. I realize that importing a pyc file and
compiling a string results in two different things. This is okay though
as I'm fairly sure this will still suit my needs.

As I said, the file format I plan to store these compiled scripts in is
proprietary, but it would be very comparable storing them in XML
elements as CDATA. I'm hoping that this would allow me to import and
run them, thereby removing the need for my app to compile scripts.

Jul 21 '05 #10

P: n/a
On Mon, 4 Jul 2005 13:01:06 -0500, Jeff Epler <je****@unpythonic.net> wrote:

--s2ZSL+KKDSLx8OML
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

This stupid code works for modules, but not for packages. It probably has bugs.
import marshal, types

class StringImporter:
def __init__(self, old_import, modules):
self._import = old_import
self._modules = modules

def __call__(self, name, *args):
module = self._modules.get(name, None)
if module is None:
return self._import(name, *args)
code = marshal.loads(module)
mod = types.ModuleType(name)
exec code in mod.__dict__
return mod

def test():
import __builtin__
__builtin__.__import__ = StringImporter(__builtin__.__import__,
{ 'test_importer': open("/usr/lib/python2.3/os.pyc").read()[8:] })

import test_importer
print test_importer.path.join("a", "b")
print test_importer.__doc__

if __name__ == '__main__': test()


I have a feeling that the OP might be looking for a way to convert
source -> source, e.g., (sketch only, not even remotely tested ;-)

# main.py
import some_module
import another

into

# main.py
# prelim
import marshal, sys # builtin
ModuleType = type(__builtins__) # avoid import types
def unscramble_proprietary_format(proprietary_string):
# ... whatever
def extract_code(proprietary_string):
return marshal.loads(unscramble_proprietary_format(propri etary_string))

# replacing each import with
try: some_module = sys.modules['some_module'] # don't re-"import"
except KeyError:
some_module = ModuleType('some_module')
exec extract_code(
"< ... string literal for proprietarily scrambled marshalled module body code"
" generated by source -> source rewrite utility ... >"
) in some_module.__dict__
sys.modules['some_module'] = some_module

try: another = sys.modules['another']
except NameError:
another = ModuleType('another')
exec extract_code('<proprietary marshalled code string for module source another.py>')
sys.modules['another'] = another

# etc

so that his some_module and another scripts can be pre-compiled by the rewrite utility and embedded in
a single source (e.g. main.py) so that he doesn't have to have external .pyc file accesses.

His startup will then still cost the execs, but not the compiles, and if he doesn't "import" stuff
he doesn't need, until he needs it, it should spread the import time load, if so desired.

All the proprietary string literal sources should become efficiently represented as part of the
single main.pyc IWT, assuming at least that one is used. Chasing various forms of import of
non-builtins recursively to eliminate imports in imported modules before they are converted to
marshalled form etc., all to avoid real imports, and statically determining that some imports
don't need to be converted to marshalled string import form because a prior import can be proved,
should be an interesting exercise, which I can't pursue at this point... ;-)

But I may be reading too much between the lines ;-)

Regards,
Bengt Richter
Jul 21 '05 #11

This discussion thread is closed

Replies have been disabled for this discussion.