473,390 Members | 1,405 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,390 software developers and data experts.

Import a module without executing it?

Is there a good way to import python files without executing their content?

I'm trying some relfection based stuff and I want to be able to import a
module dynamically to check it's contents (class ad functions defined)
but without having any of the content executed.

For example:
----------------------
def test(var):
print var
#main
test(3)
----------------------

I want to be able to import this module so I can see "ah ha, this module
defines a function called 'test'", but I don't want the code at the
bottom executed during the import.

Thanks

Take care,
Jay
Jul 18 '05 #1
14 10330
Hi

You could just parse the model file. Off the top of my head

***
f = open('ModuleYouWantToExamine.py','r')
for i in f:
if i.find('def ') > -1:
print 'Found a function!: '+i.replace('def ','')

f.close()
***

You would have to build this up for a more complete examination. Of
course, one of the guru's around here should be able to give you guidance
regarding actually parsing the file with the interpreter (not executing)
and building a dict or something with all the different types of
constructs. That's not me :)

I want to be able to import this module so I can see "ah ha, this module
defines a function called 'test'", but I don't want the code at the
bottom executed during the import.

Thanks

Take care,
Jay


Jul 18 '05 #2

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg
On Dec 7, 2004, at 10:56 PM, Caleb Hattingh wrote:
Hi

You could just parse the model file. Off the top of my head

***
f = open('ModuleYouWantToExamine.py','r')
for i in f:
if i.find('def ') > -1:
print 'Found a function!: '+i.replace('def ','')

f.close()
***

You would have to build this up for a more complete examination. Of
course, one of the guru's around here should be able to give you
guidance regarding actually parsing the file with the interpreter (not
executing) and building a dict or something with all the different
types of constructs. That's not me :)

I want to be able to import this module so I can see "ah ha, this
module defines a function called 'test'", but I don't want the code
at the bottom executed during the import.

Thanks

Take care,
Jay


--
http://mail.python.org/mailman/listinfo/python-list


Jul 18 '05 #3
Andy

thx for that. I had a file called 'tktest.py' lying around, and I did:

'>>> a = compiler.parseFile('tktest.py')

And "a" looks something like this:

***
Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
'titles', 'rows'], [], 0, None, Stmt([Discard(CallFunc(Getattr(Name('w'),
'configure'), [Keyword('state', Const('normal'))], None, None)),
For(AssName('r', 'OP_ASSIGN'), Name('rows'),
Stmt([For(AssTuple([AssName('t', 'OP_ASSIGN'), AssName('v',
'OP_ASSIGN')]), CallFunc(Name('zip'), [Name('titles'), Name('r')], None,
None), Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None, None))]),
None), Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Const('\n')], None, None))]), None), Discard(CallFunc(Getattr(Name('w'),
'configure'), [Keyword('state', Const('disabled'))], None, None))])),
Assign([AssName('app', 'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'),
'Tk'), [], None, None)), Assign([AssName('t', 'OP_ASSIGN')],
CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'), Keyword('state',
Const('disabled'))], None, None)), Discard(CallFunc(Getattr(Name('t'),
'pack'), [], None, None)), Assign([AssName('info', 'OP_ASSIGN')],
List([List([Const('Ali'), Const(18)]), List([Const('Zainab'), Const(16)]),
List([Const('Khalid'), Const(18)])])), Discard(CallFunc(Name('add_rows'),
[Name('t'), List([Const('Name'), Const('Age')]), Name('info')], None,
None)), Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None,
None))])
***

Pretty impressive :)

Do you know of more batteries that can process this stuff further, for
interest sake (and maybe the OP)?

thx again
Caleb
On Tue, 7 Dec 2004 15:57:16 -0500, Andy Gross <an**@andygross.org> wrote:

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg

Jul 18 '05 #4
Jay O'Connor wrote:
----------------------
def test(var):
print var
#main
test(3)
----------------------

I want to be able to import this module so I can see "ah ha, this module
defines a function called 'test'", but I don't want the code at the
bottom executed during the import.


If you have source control over this file, you could write it with the
more standard idiom:

def test(var):
print var

if __name__ == "__main__":
test(3)

Then when the module is imported, only the def statement gets executed,
not the 'test(3)'. Of course, if you don't have source control over the
file, you can't do this...
Also note that code that is not protected by an
if __name__ == "__main__":
test may be part of the module definition, so examining a module without
executing this code may be misleading, e.g.:

def test(var):
print var

globals()['test'] = type('C', (object,), dict(
f=lambda self, var, test=test: test(var)))

if __name__ == "__main__":
test().f(3)

The module above turns 'test' from a function into a class, so if all
you do is look at the def statements, you may misinterpret the module.

Steve
Jul 18 '05 #5
Here's a quick example that will pull out all functions defined in the
top-level of a module:

---
#/usr/bin/env python
from compiler import parse, walk
from compiler.visitor import ASTVisitor

testdata = r'''
def aFunction(anArg):
return anArg + 1
'''
class SimpleVisitor(ASTVisitor):
def visitFunction(self, parsedFunc):
print "Function %(name)s at %(lineno)s takes %(argnames)s " \
" with code %(code)s" % parsedFunc.__dict__

if __name__ == "__main__":
ast = parse(testdata)
walk(ast, SimpleVisitor(), verbose=True)
---

andy@alyosha:~$ ./test.py
Function aFunction at 2 takes ['anArg'] with code
Stmt([Return(Add((Name('anArg'), Const(1))))])

HTH,

/arg
On Dec 7, 2004, at 11:14 PM, Caleb Hattingh wrote:
Andy

thx for that. I had a file called 'tktest.py' lying around, and I did:

'>>> a = compiler.parseFile('tktest.py')

And "a" looks something like this:

***
Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
'titles', 'rows'], [], 0, None,
Stmt([Discard(CallFunc(Getattr(Name('w'), 'configure'),
[Keyword('state', Const('normal'))], None, None)), For(AssName('r',
'OP_ASSIGN'), Name('rows'), Stmt([For(AssTuple([AssName('t',
'OP_ASSIGN'), AssName('v', 'OP_ASSIGN')]), CallFunc(Name('zip'),
[Name('titles'), Name('r')], None, None),
Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None,
None))]), None), Discard(CallFunc(Getattr(Name('w'), 'insert'),
[Const('end'), Const('\n')], None, None))]), None),
Discard(CallFunc(Getattr(Name('w'), 'configure'), [Keyword('state',
Const('disabled'))], None, None))])), Assign([AssName('app',
'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'), 'Tk'), [], None,
None)), Assign([AssName('t', 'OP_ASSIGN')],
CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'),
Keyword('state', Const('disabled'))], None, None)),
Discard(CallFunc(Getattr(Name('t'), 'pack'), [], None, None)),
Assign([AssName('info', 'OP_ASSIGN')], List([List([Const('Ali'),
Const(18)]), List([Const('Zainab'), Const(16)]),
List([Const('Khalid'), Const(18)])])),
Discard(CallFunc(Name('add_rows'), [Name('t'), List([Const('Name'),
Const('Age')]), Name('info')], None, None)),
Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None, None))])
***

Pretty impressive :)

Do you know of more batteries that can process this stuff further, for
interest sake (and maybe the OP)?

thx again
Caleb
On Tue, 7 Dec 2004 15:57:16 -0500, Andy Gross <an**@andygross.org>
wrote:

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg

--
http://mail.python.org/mailman/listinfo/python-list


Jul 18 '05 #6
Functions and classes are created during the very execution you're
trying to skip so there's no precise way to do what you want.

That said, you can parse the code without executing it, and that will
give you some information about defined functions and classes. It will
_not_ give you actual function objects; the only way to get those is to
execute the code. It will also not catch anything created "on the
fly"*, e.g. "exec 'def func(x): pass'"
# example:

def defined_functions(code_text):
module_ast = parse(code_text)
return [statement for statement in
module_ast.node.nodes if
isinstance(statement, ast.Function)]

# read the module's source code
test_string = """
def foo(x):
pass
def bar(y):
pass
"""

# Which functions are defined in the test string?
function_asts = defined_functions(test_string)
print "Defined functions:", [f.name for f in function_asts]


* - Okay, everything in Python happens on-the-fly. But you know what I
mean.

Footnote: Wow. The new, "improved" google groups 2 beta has totally
annihilated my indentation. Sorry about that. Hopefully you can still
figure it out.

Jul 18 '05 #7

On Dec 7, 2004, at 4:20 PM, Steven Bethard wrote:

If you have source control over this file, you could write it with the
more standard idiom...


I should have mentioned this first. If you're just trying to avoid
existing top-level code from being executed, use the if __name__ ==
"__main__" idiom.

Jul 18 '05 #8
fi**************@gmail.com wrote:
Functions and classes are created during the very execution you're
trying to skip so there's no precise way to do what you want.
That said, you can parse the code without executing it, and that will
give you some information about defined functions and classes. It will
_not_ give you actual function objects; the only way to get those is to
execute the code. It will also not catch anything created "on the
fly"*, e.g. "exec 'def func(x): pass'"

OK, some more information on what I'm tryng to do would help.

I come from a strong Smalltalk background and in most Smalltalk IDEs is
the capability to browse "Implementors" (what classes implement a method
with a given name or like a given name) and "Senders" (from what methods
is a given method name called)

Given a set of code, not all of which I am familiar with, I wanted to
have similar explorative capabilities in understanding the code. grep
in Linux does a decent job, but under Windows, the file search (look for
a particular string in a py file) does not seem to be too..useful (I
don't get back the results I should)

I shied away from just hand-parsing the code because knowing that a file
has a string, or even a function with a given name, in it is not hard,
knowing that the module has a class that has the function (and knowing
the class as well) is more useful

So, in Smalltalk, the way you find out about what's in the system is
reflection...you ask the classes what methods they implement (if looking
for 'implementors'), you ask the methods for the source and can string
search the code (if looking for senders).

Knowing that both modules and classes (and functions) had reality enough
to be queried, that was my first line of thought, to load the module and
query the module and query the classes in the module, etc... and do
various matches against whatever I was looking for.

It worked fine for a file that just defined functions and classes.
However, files that had code not contained in classes or functions
caused a bit of a problem because loading the module would execute that
code and that's not what I wanted.

So therefore, my original question.

The real question, I suppose, is "what is a good technique to find what
modules and classes implement or refer to particular names"

Take care,
Jay
Jul 18 '05 #9
> So, in Smalltalk, the way you find out about what's in the system is
reflection...you ask the classes what methods they implement (if
looking for 'implementors'), you ask the methods for the source and
can string search the code (if looking for senders).
There's no real way to get the source code from a compiled code object
, although there are some tools, like decompyle, which will try for
you. I think some people on this list were discussing the potential
utility of having a __source__ attribute for code objects, but Python
currently doesn't do this. You can probably also get some mileage out
of the inspect module (try 'pydoc inspect'), but that *does* compile
the code you're inspecting.
It worked fine for a file that just defined functions and classes.
However, files that had code not contained in classes or functions
caused a bit of a problem because loading the module would execute
that code and that's not what I wanted.


To get at top-level module code like this, you'll have to parse the
file and walk the AST - there's no other way. In a well implemented
library or app, however, there typically shouldn't be much important
code being executed at the top level. Typically the top-level code
will be initialization of singleton globals or decorator-type function
wrappers, and you can get at most of the important names reflectively
through the inspect module.

/arg

Jul 18 '05 #10
> The real question, I suppose, is "what is a good technique to find
what
modules and classes implement or refer to particular names"


I think your best bet is still to import the module and introspect it.
It will execute some code, but (by convention) simply importing a
module doesn't usually unleash any spectacular computation. Scripts
meant to be executed from the command line will almost always have an
(if __name__ == '__main__' ) clause to prevent just this.

A search of this newsgroup for "duck" or "duck typing" will probably
turn up more info on how people approximate the "X implements Y"
relationship in Python.

On the other hand, if you're just trying to figure out what a black-box
object does from the interpreter's command line, you can use help(),
dir() or pydoc amongst others. "import pydoc; pydoc.gui()"

There are some functions that will be handy for determining properties
of an unknown object:
callable, eval, hasattr, isinstance, type, super

And most objects will have a few magic attributes-
def qux(x, y=3): pass .... dir(qux) ['__call__', '__class__', '__delattr__', '__dict__', '__doc__',
'__get__', '__getattribute__', '__hash__', '__init__', '__module__',
'__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__str__', 'func_closure', 'func_code', 'func_defaults',
'func_dict', 'func_doc', 'func_globals', 'func_name'] qux.func_defaults (3,) qux.func_name 'qux' qux.func_code.co_varnames

('x','y')

Jul 18 '05 #11
Lonnie Princehouse wrote:
The real question, I suppose, is "what is a good technique to find


what
modules and classes implement or refer to particular names"

I think your best bet is still to import the module and introspect it.
It will execute some code, but (by convention) simply importing a
module doesn't usually unleash any spectacular computation. Scripts
meant to be executed from the command line will almost always have an
(if __name__ == '__main__' ) clause to prevent just this.


Well, I sorta ran into that trouble testing some testfiles of
mine...little scripts for testing specific things that did'nt follow
convension in that manner

Thanks

Take care,
Jay
Jul 18 '05 #12
Jay O'Connor wrote:
The real question, I suppose, is "what is a good technique to find what
modules and classes implement or refer to particular names"


You might like to try ctags. I have had a good experience with it. It's not as automatic as I would
like - you have to build a cross-reference table before you can use it - and it only finds
implementors, not referrers. It integrates with lots of editors, I have used it with TextPad.
http://ctags.sourceforge.net/

Kent
Jul 18 '05 #13
Andy, this is a nice example. It prompted me to look at the docs for compiler.visitor. The docs are,
um, pretty bad. I'm going to attempt to clean them up a little. Would you mind if I include this
example?

Thanks,
Kent

Andy Gross wrote:
Here's a quick example that will pull out all functions defined in the
top-level of a module:

---
#/usr/bin/env python
from compiler import parse, walk
from compiler.visitor import ASTVisitor

testdata = r'''
def aFunction(anArg):
return anArg + 1
'''
class SimpleVisitor(ASTVisitor):
def visitFunction(self, parsedFunc):
print "Function %(name)s at %(lineno)s takes %(argnames)s " \
" with code %(code)s" % parsedFunc.__dict__

if __name__ == "__main__":
ast = parse(testdata)
walk(ast, SimpleVisitor(), verbose=True)
---

andy@alyosha:~$ ./test.py
Function aFunction at 2 takes ['anArg'] with code
Stmt([Return(Add((Name('anArg'), Const(1))))])

HTH,

/arg
On Dec 7, 2004, at 11:14 PM, Caleb Hattingh wrote:
Andy

thx for that. I had a file called 'tktest.py' lying around, and I did:

'>>> a = compiler.parseFile('tktest.py')

And "a" looks something like this:

***
Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
'titles', 'rows'], [], 0, None,
Stmt([Discard(CallFunc(Getattr(Name('w'), 'configure'),
[Keyword('state', Const('normal'))], None, None)), For(AssName('r',
'OP_ASSIGN'), Name('rows'), Stmt([For(AssTuple([AssName('t',
'OP_ASSIGN'), AssName('v', 'OP_ASSIGN')]), CallFunc(Name('zip'),
[Name('titles'), Name('r')], None, None),
Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None,
None))]), None), Discard(CallFunc(Getattr(Name('w'), 'insert'),
[Const('end'), Const('\n')], None, None))]), None),
Discard(CallFunc(Getattr(Name('w'), 'configure'), [Keyword('state',
Const('disabled'))], None, None))])), Assign([AssName('app',
'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'), 'Tk'), [], None,
None)), Assign([AssName('t', 'OP_ASSIGN')],
CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'),
Keyword('state', Const('disabled'))], None, None)),
Discard(CallFunc(Getattr(Name('t'), 'pack'), [], None, None)),
Assign([AssName('info', 'OP_ASSIGN')], List([List([Const('Ali'),
Const(18)]), List([Const('Zainab'), Const(16)]),
List([Const('Khalid'), Const(18)])])),
Discard(CallFunc(Name('add_rows'), [Name('t'), List([Const('Name'),
Const('Age')]), Name('info')], None, None)),
Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None, None))])
***

Pretty impressive :)

Do you know of more batteries that can process this stuff further, for
interest sake (and maybe the OP)?

thx again
Caleb
On Tue, 7 Dec 2004 15:57:16 -0500, Andy Gross <an**@andygross.org> wrote:

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg


--
http://mail.python.org/mailman/listinfo/python-list


Jul 18 '05 #14

On Dec 8, 2004, at 5:44 AM, Kent Johnson wrote:
Andy, this is a nice example. It prompted me to look at the docs for
compiler.visitor. The docs are, um, pretty bad. I'm going to attempt
to clean them up a little. Would you mind if I include this example?


Be my guest!

/arg

Jul 18 '05 #15

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Markus Doering | last post by:
Hello, I just switched from 2.2 to Python 2.3. I am developing an XML/CGI interface to databases as a python package called "unitWrapper" containing several modules which ran fine under v2.2. ...
5
by: Colin Brown | last post by:
I have instances where Python 2.3.2 import both does not work or error. It can be demonstrated under Win2k and Linux by doing the following: 1. Create a subdirectory "abc" 2. In directory "abc"...
0
by: Vio | last post by:
Hi, I've been trying to embed (statically) wxPy alongside an embedded py interpreter on a linux/gtk box. At one point, for some reason misc.o linking reported "multiple definitions of...
0
by: Simon John | last post by:
I have a program that consists of one main module and lots of small sub-modules. In the main module I open a text file and grep for a language setting, this language setting will then be used as...
79
by: pinkfloydhomer | last post by:
I want to scan a file byte for byte for occurences of the the four byte pattern 0x00000100. I've tried with this: # start import sys numChars = 0 startCode = 0 count = 0
6
by: robert | last post by:
I get python crashes and (in better cases) strange Python exceptions when (in most cases) importing and using cookielib lazy on demand in a thread. It is mainly with cookielib, but remember the...
5
by: Roland Hedberg | last post by:
Hi! I'm having a bit of a problem with import. I'm writing a marshalling system that based on a specification will create one or more files containing mostly class definitions. If there are...
7
by: Ron Adam | last post by:
from __future__ import absolute_import Is there a way to check if this is working? I get the same results with or without it. Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) on win 32 ...
7
by: bambam | last post by:
import works in the main section of the module, but does not work as I hoped when run inside a function. That is, the modules import correctly, but are not visible to the enclosing (global)...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.