Loading functions from a file during run-time

[Bryant Huang]

Hello!

I would like to read in files, during run-time, which contain plain
Python function definitions, and then call those functions by their
string name. In other words, I'd like to read in arbitrary files with
function definitions, using a typical 'open()' call, but then have
those functions available for use.

The 'import' keyword is not appropriate, AFAIK, because I want to be
able to open any file, not one that I know ahead of time (and thus can
import at design-time).

I already have some code that I cobbled together from many newsgroup
posts, but I wonder if there's a cleaner/simpler way to do it.

The following is a toy example of what I'm doing so far. I have a file
called 'bar.txt' that contains two function definitions. I have a main
driver program called 'foo.py' which reads in 'bar.txt' (but should be
able to read any file it hasn't seen before), then calls the two
functions specified in 'bar.txt'.

===== [bar.txt] =====

def negate(x):
return -x

def square(x):
return x*x
===== [foo.py] =====

# open functions file
foo_file = open("bar.txt")
foo_lines = foo_file.readlines()
foo_file.close()
foo_str = "".join(foo_lines)

# compile code
foo_code = compile(foo_str, "<string>", "exec")
foo_ns = {}
exec(foo_code) in foo_ns

# use functions
k = 5
print foo_ns["negate"](k) // outputs -5
print foo_ns["square"](k) // outputs 25
I'm not sure exactly what happens below the surface, but I'm guessing
the 'compile()' and 'exec()' commands load in 'negate()' and 'square()'
as functions in the global scope of 'foo.py'. I find that when I run
'compile()' and 'exec()' from within a function, say 'f()', the
functions I read in from 'bar.txt' are no longer accessible since they
are in global scope, and not in the scope of 'f()'.

Any pointers would be very welcome.

Thanks!
Bryant

[Grant Edwards]

On 2005-02-11, Bryant Huang <73****@gmail.com> wrote:

I would like to read in files, during run-time, which contain
plain Python function definitions, and then call those
functions by their string name. In other words, I'd like to
read in arbitrary files with function definitions, using a
typical 'open()' call, but then have those functions available
for use.
that's pretty simple:

$ cat foo.txt
def foo():
print "foo here"
def bar():
print "bar here"

$ cat foo.py
filename = 'foo.txt'
execfile(filename)
foo()
bar()

$ python foo.py
foo here
bar here
The 'import' keyword is not appropriate, AFAIK, because I want to be
able to open any file, not one that I know ahead of time (and thus can
import at design-time).
This will import a module name determined at run-time:

exec('import %s' % moduleName)
The following is a toy example of what I'm doing so far.
[...]
I'm not sure exactly what happens below the surface, but I'm
guessing the 'compile()' and 'exec()' commands load in
'negate()' and 'square()' as functions in the global scope of
'foo.py'. I find that when I run 'compile()' and 'exec()' from
within a function, say 'f()', the functions I read in from
'bar.txt' are no longer accessible since they are in global
scope, and not in the scope of 'f()'.

Huh? I'm lost. What, exactly, are you trying to accomplish?

Did your example program do what you intended or not?

Is this what you're trying to do?

$ cat foo.txt
def foo():
print "foo here"
def bar():
print "bar here"

$ cat bar.py
def testing():
filename = 'foo.txt'
execfile(filename,globals())
foo()
bar()
testing()
foo()
bar()

$ python bar.py
foo here
bar here
foo here
bar here

--
Grant Edwards grante Yow! Clear the
at laundromat!! This
visi.com whirl-o-matic just had a
nuclear meltdown!!

[Nick Coghlan]

Bryant Huang wrote:

Hello!

I would like to read in files, during run-time, which contain plain
Python function definitions, and then call those functions by their
string name. In other words, I'd like to read in arbitrary files with
function definitions, using a typical 'open()' call, but then have
those functions available for use.

The 'import' keyword is not appropriate, AFAIK, because I want to be
able to open any file, not one that I know ahead of time (and thus can
import at design-time).

This usage is one of the reasons why the __import__ function exists (despite
what its docs say). It's right there at the top of the library reference's
'builtin functions' section:

http://www.python.org/dev/doc/devel/...-in-funcs.html

Instead of writing "import foo as mod" you can write "mod = __import__('foo')".
The first is better when you do know the name of the module you want at coding
time, but the latter is handy when you want to be dynamic about it.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[Bryant Huang]

Ah, thanks a lot, Grant and Nick.

Let me try to clarify because I think I was unclear in specifying what
I want to do:

1. Read in a file containing a bunch of function definitions:

def f1(x):
...

def f2(x):
...

def f3(x):
...

def f4(x):
...

2. In wxPython, populate a CheckListBox with all the functions defined
in that file.

3. Allow the user to check some of the functions, say for example, f1()
and f3().

4. The program then executes f1() and f3() on some specified data.
The reason I asked these questions is because I don't know what
functions are contained in the function file ahead of time, but I still
want to be able to read those in, then based on which functions the
user selects, to run those accordingly, even though I still don't know,
at design-time, what functions are contained in the function file.

Does that make sense?

Thanks a lot!
Bryant

[Wensheng]

if the file you want to include is not txt, but instead py, it should
be easy.
for example you have fs.py

you just
-------------------------------------
fs=__import__("fs")
f=[a for a in dir(fs) if a[0:2]!='__']
#no you have f1(),f2(),f3() as f[0],f[1],f[2]
then excute desired function, for example f2():
exec("fs."+f[1]+"()")
if you have top level variable in that file, just modify this to test
typs.FunctionType(import types first)

[TJ's Projects]

Answer below...

"Bryant Huang" <73****@gmail.com> wrote in message news:<11*********************@g14g2000cwa.googlegr oups.com>...

Hello!

I would like to read in files, during run-time, which contain plain
Python function definitions, and then call those functions by their
string name. In other words, I'd like to read in arbitrary files with
function definitions, using a typical 'open()' call, but then have
those functions available for use.

<snip>

===== [bar.txt] =====

def negate(x):
return -x

def square(x):
return x*x
===== [foo.py] =====

# open functions file
foo_file = open("bar.txt")
foo_lines = foo_file.readlines()
foo_file.close()
foo_str = "".join(foo_lines)

# compile code
foo_code = compile(foo_str, "<string>", "exec")
foo_ns = {}
exec(foo_code) in foo_ns

# use functions
k = 5
print foo_ns["negate"](k) // outputs -5
print foo_ns["square"](k) // outputs 25
I'm not sure exactly what happens below the surface, but I'm guessing
the 'compile()' and 'exec()' commands load in 'negate()' and 'square()'
as functions in the global scope of 'foo.py'. I find that when I run
'compile()' and 'exec()' from within a function, say 'f()', the
functions I read in from 'bar.txt' are no longer accessible since they
are in global scope, and not in the scope of 'f()'.

Any pointers would be very welcome.

Thanks!
Bryant

You're actually very close here. The problem you are having is that
you are creating a local namespace (foo_ns) and executing that code
within that namespace. To "import" the functions into the global
namespace, exec them as:

exec foo_code in globals()
You will then be able to call the methods as if they had been declared
locally, ie. 'negate(5)' rather than 'foo_ns["negate"](5)'.

<plug type="shameless">

I've written a simple line editor intended to be used within the
Python interactive interpreter, based on the old DOS 'edlin' command.
This is how I enable the user to "import" the entered functions into
their local namespace. The actual "execution" of the code is actually
pretty simple, and not much more involved than what you see here. You
may be able to get some more ideas, however.

For more info (or comments, suggestions, or pointers), check out
http://pyedlin.sourceforge.net).

</plug>
Here's a simple example that shows a "good" input loop:

====== Start exec example ======
if __name__ == '__main__':
funcdata = ['def func():\n', "\tprint 'Hello, world!'\n", '\n']

codeline = ''
for line in funcdata:
codeline += line
execline = codeline
if execline.endswith('\n'):
execline = execline[:-1]

try:
obj = compile(execline, '<string>', 'exec')
except SyntaxError:
pass
else:
exec obj in globals()
codeline = ''

func()

====== End exec sample ======
Executing this code will print 'Hello, world!' to the console.
Hope this helps!

T.J.

[tjprojects_usenet]

I have to say, I was skeptical about your execution method at first --
simply joining all of the lines and then compiling them. My
understanding of the compile method from the documentation led me to
believe that you needed to process a line at a time. ''.join ing them
all seems to work very well, however, and it's a lot easier!

My comments about 'exec obj in globals()' still stand, however. Give
it a try and let me know if it works the way you expect.
T.J.

[Wensheng]

#--- file bar.py
def negate(n):
return -n

def square(n):
return n*n
#--- end bar.py

foo="bar"
fs=__import__(foo)
import types
f=[a for a in dir(fs) if a[0:2]!='__' and type(getattr(fs,a))==types.FunctionType] f ['negate', 'square'] n=5
exec("print fs."+f[0]+"(n)") -5 exec("print fs."+f[1]+"(n)")

25
Isn't the problem solved?

[Wensheng]

or don't use exec():
f=[getattr(fs,a) for a in dir(fs) if a[0:2]!='__' and
type(getattr(fs,a))==types.FunctionType]

f [<function negate at 0x100a00>, <function square at 0x144038>] f[0](n) -5 f[1](n)

25

[tjprojects_usenet]

Wensheng wrote:

#--- file bar.py

<snip>

foo="bar"
fs=__import__(foo)

<snip>

Wensheng:

The problem with this is that it assumes the text file is a valid
python file, and that the extension is ".py". This may work for the
OP's situation; he would need to weigh in. 'exec'ing the functions
into the global namespace allows you to create functions from any
text-based source -- registry keys, ini files, user input, etc.
T.J.

[Wensheng]

f=open("bar.txt")
import imp
fs=imp.new_module("fs")
exec f in fs.__dict__
..... rests are the same

althought why use anything other than .py, when you import .py, it get
compiled into .pyc and it load faster next time

[Bryant Huang]

Ah, thank you, Wensheng and T.J.! I have drawn bits and pieces from
what you have suggested. Both of your solutions work well.