Import module with non-standard file name

[Ben Finney]

Howdy all,

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

Background:

On Unix, I write programs intended to be run as commands to a file
with no extension. This allows other programs to use the command as an
interface, and I can re-write the program in some other language
without obsoleting the commandline interface.

e.g., I might write 'frobnicate-foo' as a shell program so that other
programs can 'frobnicate-foo --bar baz'. If I later decide to
re-implement 'frobnicate-foo' in Python, I'll save the top level
module to the same file name since it implements the same command-line
interface.

Now that I've got it written as a Python module, I'd like to write
unit tests for that module, which of course will need to import the
program module to test it. The unit test can explicitly add the
directory where the program module lives to 'sys.path' for the purpose
of importing that module.

However, the Python reference tells me that 'import' (specifically,
'__import__()') needs modules to live in files named a particular way:
with a '.py' suffix. But my module is in a file called
'frobnicate-foo', with no suffix, and that's part of the definition of
the program interface.

I don't want symbolic links, or anything else that presents two
filenames for the same module, because there's no need for that except
for Python's apparent insistence on a particular naming
convention. Also, avoiding symbolic links inside the source code tree
makes version control smoother.
What are my options to import a module from a file whose name can't
change?

--
\ "[W]e are still the first generation of users, and for all that |
`\ we may have invented the net, we still don't really get it." |
_o__) -- Douglas Adams |
Ben Finney

[John Machin]

Ben Finney wrote:

Howdy all,

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

Background:

On Unix, I write programs intended to be run as commands to a file
with no extension. This allows other programs to use the command as an
interface, and I can re-write the program in some other language
without obsoleting the commandline interface.

e.g., I might write 'frobnicate-foo' as a shell program so that other
programs can 'frobnicate-foo --bar baz'. If I later decide to
re-implement 'frobnicate-foo' in Python, I'll save the top level
module to the same file name since it implements the same command-line
interface.

Now that I've got it written as a Python module, I'd like to write
unit tests for that module, which of course will need to import the
program module to test it. The unit test can explicitly add the
directory where the program module lives to 'sys.path' for the purpose
of importing that module.

If it can do that, it can copy the MUT to some temp directory, adding
..py to the end of the name of the new file, and put the temp directory
in sys.path .... can't it?

Cheers,
John

[Ben Finney]

"John Machin" <sj******@lexicon.netwrites:

Ben Finney wrote:

Now that I've got it written as a Python module, I'd like to write
unit tests for that module, which of course will need to import
the program module to test it. The unit test can explicitly add
the directory where the program module lives to 'sys.path' for the
purpose of importing that module.

If it can do that, it can copy the MUT to some temp directory,
adding .py to the end of the name of the new file, and put the temp
directory in sys.path .... can't it?

Sounds like a nasty hack (not that fiddling sys.path isn't a hack, but
at least that one's addressed in Python 2.5 with relative and absolute
imports).

The problem with importing the program module from a file in a
different directory is that the program won't be able to find its own
relative modules. That leads to either *more* sys.path hackery, or
importing from a temporary file in the *same* directory.

Besides which, that's still two file names for the same module
code. The whole point of testing is to know that I'm testing the same
module; with a two-file shim, that's one step further away from that
ideal.

What you describe is possible, but leads to very smelly hacks. I'd
like to see what other options there are.

--
\ "It's not what you pay a man, but what he costs you that |
`\ counts." -- Will Rogers |
_o__) |
Ben Finney

[John Machin]

Ben Finney wrote:

"John Machin" <sj******@lexicon.netwrites:

Ben Finney wrote:

Now that I've got it written as a Python module, I'd like to write
unit tests for that module, which of course will need to import
the program module to test it. The unit test can explicitly add
the directory where the program module lives to 'sys.path' for the
purpose of importing that module.

If it can do that, it can copy the MUT to some temp directory,
adding .py to the end of the name of the new file, and put the temp
directory in sys.path .... can't it?

Sounds like a nasty hack (not that fiddling sys.path isn't a hack, but
at least that one's addressed in Python 2.5 with relative and absolute
imports).

The problem with importing the program module from a file in a
different directory is that the program won't be able to find its own
relative modules. That leads to either *more* sys.path hackery, or
importing from a temporary file in the *same* directory.

Please explain both the "own" and "relative" in "its own relative
modules". Do these modules not have names that end in ".py"?

>
Besides which, that's still two file names for the same module
code. The whole point of testing is to know that I'm testing the same
module; with a two-file shim, that's one step further away from that
ideal.

The two-file caper exists only for the duration of the test. You'll
have to trust yourselt to write and test a file copying gadget. :-)

>
What you describe is possible, but leads to very smelly hacks. I'd
like to see what other options there are.

Probably smellier ones ...:<)

Cheers,
John

[Ben Finney]

"John Machin" <sj******@lexicon.netwrites:

Ben Finney wrote:

"John Machin" <sj******@lexicon.netwrites:

If it can [modify sys.path], it can copy the MUT to some temp
directory, adding .py to the end of the name of the new file,
and put the temp directory in sys.path .... can't it?

The problem with importing the program module from a file in a
different directory is that the program won't be able to find its
own relative modules. That leads to either *more* sys.path
hackery, or importing from a temporary file in the *same*
directory.

Please explain both the "own" and "relative" in "its own relative
modules". Do these modules not have names that end in ".py"?

The program can import modules with relative paths, because it can
expect its position in the directory tree to remain the same relative
to those modules. If the program module suddenly exists in a different
directory, that assumption no longer holds and the relative imports
performed by the program will fail.

Thus to avoid that problem, testing needs to be done on the program
module with its position relative to all other modules the same as
when that program runs.

--
\ "It may be that our role on this planet is not to worship God |
`\ -- but to create him." -- Arthur C. Clarke |
_o__) |
Ben Finney

[Patrick Maupin]

Ben Finney wrote:

Howdy all,

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

This is a piece of cake in Python.

>>from types import ModuleType
x = ModuleType('myModName')
data = open('myfilename').read()
exec data in x.__dict__

Your output here...

This won't save a .pyc, but as your message later explains, this is for
unittesting, so this could probably be considered a feature for this
usage.

Regards,
Pat

[Simon Forman]

Ben Finney wrote:

Howdy all,

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

Background:

On Unix, I write programs intended to be run as commands to a file
with no extension. This allows other programs to use the command as an
interface, and I can re-write the program in some other language
without obsoleting the commandline interface.

e.g., I might write 'frobnicate-foo' as a shell program so that other
programs can 'frobnicate-foo --bar baz'. If I later decide to
re-implement 'frobnicate-foo' in Python, I'll save the top level
module to the same file name since it implements the same command-line
interface.

Now that I've got it written as a Python module, I'd like to write
unit tests for that module, which of course will need to import the
program module to test it. The unit test can explicitly add the
directory where the program module lives to 'sys.path' for the purpose
of importing that module.

However, the Python reference tells me that 'import' (specifically,
'__import__()') needs modules to live in files named a particular way:
with a '.py' suffix. But my module is in a file called
'frobnicate-foo', with no suffix, and that's part of the definition of
the program interface.

I don't want symbolic links, or anything else that presents two
filenames for the same module, because there's no need for that except
for Python's apparent insistence on a particular naming
convention. Also, avoiding symbolic links inside the source code tree
makes version control smoother.
What are my options to import a module from a file whose name can't
change?

--
\ "[W]e are still the first generation of users, and for all that |
`\ we may have invented the net, we still don't really get it." |
_o__) -- Douglas Adams |
Ben Finney

Leave your python module with the .py extension and create a small
python script without the .py extension to import and run your code
from the command line.

For example, on my [linux] system /usr/local/bin/idle contains this:

#!/usr/bin/python

from idlelib.PyShell import main
if __name__ == '__main__':
main()

You also get a modest performance boost because the interpreter will
only process the text of this small script but will use the precompiled
byte-code .pyc files (when available) of your main module, rather than
re-parsing its text.

HTH,
~Simon

[Ben Finney]

"Patrick Maupin" <pm*****@gmail.comwrites:

Ben Finney wrote:

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

This is a piece of cake in Python.

>from types import ModuleType
x = ModuleType('myModName')
data = open('myfilename').read()
exec data in x.__dict__

Your output here...

This won't save a .pyc, but as your message later explains, this is for
unittesting, so this could probably be considered a feature for this
usage.

Very nice. Okay, my unit testing scaffold module now has a new function:

def make_module_from_file(module_name, file_name):
""" Make a new module object from the code in specified file """

from types import ModuleType
module = ModuleType(module_name)

module_file = open(file_name, 'r')
exec module_file in module.__dict__

return module

The unit test now just imports that functionality, and then makes the
module object via that function:

import scaffold
module_name = 'frobnicate_foo'
module_file_under_test = os.path.join(scaffold.code_dir, 'frobnicate-foo')
frobnicate_foo = scaffold.make_module_from_file(
module_name, module_file_under_test)

The rest of the unit test then has 'frobnicate_foo' as a module to test.

It's working fine. Does anyone foresee any problems with doing it this way?

--
\ "Injustice is relatively easy to bear; what stings is justice." |
`\ -- Henry L. Mencken |
_o__) |
Ben Finney

[Miki]

Hello Ben,

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

http://docs.python.org/lib/module-imp.html (hint: load_source :)

HTH,
Miki
http://pythonwise.blogspot.com/

[Martin v. Löwis]

Ben Finney schrieb:

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

I recommend to use imp.load_module.

Regards,
Martin

[Ben Finney]

"Martin v. Löwis" <ma****@v.loewis.dewrites:

Ben Finney schrieb:

Question: I have Python modules named without '.py' as the extension,
and I'd like to be able to import them. How can I do that?

I recommend to use imp.load_module.

I've tried this; as Patrick Maupin alludes to, it compiles the module
leaving a strangely-named file behind.

Program in a file named 'frob_foo'; no other file names needed nor
desired.

import imp

file_name = "frob_foo"
module_name = 'frob_foo'

module_file = open(file_name, 'r')
module_desc = ("", 'r', imp.PY_SOURCE)
module = imp.load_module(module_name, module_file, file_name, module_desc)

Result: two files, 'frob_foo' and 'frob_fooc'. I can see why this
happens, but it's not what's desired. Currently I'm going with:

file_name = "frob_foo"
module_name = 'frob_foo'

from types import ModuleType
module = ModuleType(module_name)

module_file = open(file_name, 'r')
exec module_file in module.__dict__

Still, the purpose is simply to get a module object out, with a named
file as input. If the 'imp' module can do that without leaving
unwanted turds behind, it seems more elegant. Can anyone suggest a way
to get the same result as the above 'exec' method, using the 'imp'
module?

--
\ "...one of the main causes of the fall of the Roman Empire was |
`\ that, lacking zero, they had no way to indicate successful |
_o__) termination of their C programs." -- Robert Firth |
Ben Finney