distutils, sdist and tests

[Steven Bethard]

How do I get distutils to include my testing module in just the "sdist"
distribution? My current call to setup() looks like::

distutils.core.setup(
...
py_modules=['argparse'],
)

If change this to::

distutils.core.setup(
...
py_modules=['argparse', 'test_argparse'],
)

then test_argparse.py gets included in the source distribution, but it
also gets installed to site-packages like a normal module. I don't think
I want that. If I change it to::

distutils.core.setup(
...
py_modules=['argparse'],
scripts=['test_argparse.py'],
)

then test_argparse.py gets included in the source distribution, but it
also gets installed in the Python scripts directory. I don't think I
want that either.

I want test_argparse.py to be available in the source distribution, but
I don't think it should be included in the binary distributions.

Or am I just being to picky? Do folks normally include their tests in
the binary distributions?

STeVe

[Robert Kern]

Steven Bethard wrote:

How do I get distutils to include my testing module in just the "sdist"
distribution?

Use a MANIFEST.

http://docs.python.org/dist/source-dist.html

I want test_argparse.py to be available in the source distribution, but
I don't think it should be included in the binary distributions.

Or am I just being to picky? Do folks normally include their tests in
the binary distributions?

Some do, but only when they are packages and can thus hide.the.tests .

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Steven Bethard]

Robert Kern wrote:

Steven Bethard wrote:

>How do I get distutils to include my testing module in just the "sdist"
distribution?

Use a MANIFEST.

http://docs.python.org/dist/source-dist.html

>I want test_argparse.py to be available in the source distribution, but
I don't think it should be included in the binary distributions.

Using a MANIFEST appears to do the same thing as putting "test_argparse"
into py_modules -- that is, it puts "test_argparse.py" into both "sdist"
and "bdist" distributions. In "bdist" distributions it gets installed to
the site-packages directory like any other module.

STeVe

[Robert Kern]

Steven Bethard wrote:

Robert Kern wrote:

>Steven Bethard wrote:

>>How do I get distutils to include my testing module in just the "sdist"
distribution?

Use a MANIFEST.

http://docs.python.org/dist/source-dist.html

Also, I just noted this tidbit:

"""If you don't supply an explicit list of files (or instructions on how to
generate one), the sdist command puts a minimal default set into the source
distribution:
...
* anything that looks like a test script: test/test*.py (currently, the
Distutils don't do anything with test scripts except include them in source
distributions, but in the future there will be a standard for testing Python
module distributions)
...
"""

So you can just stick test_argparse.py into a test/ directory. Having tested
this, it appears to work (see below).

>>I want test_argparse.py to be available in the source distribution, but
I don't think it should be included in the binary distributions.

Using a MANIFEST appears to do the same thing as putting "test_argparse"
into py_modules -- that is, it puts "test_argparse.py" into both "sdist"
and "bdist" distributions. In "bdist" distributions it gets installed to
the site-packages directory like any other module.

Are you sure that you don't have changes left over in your setup.py when you
tested that? Also check that you clear out your build/ directory every time. I
think the bdist_* commands tend to just copy whatever is in the appropriate
build/lib.*/ directory even though you've changed how those files should be
generated.

This is what I get with setup.py unmodified. The only thing I added was
MANIFEST.in and an empty test_argparse.py file.

[argparse]$ ls
MANIFEST.in PKG-INFO README.txt argparse.py setup.py
test_argparse.py
[argparse]$ cat MANIFEST.in
include PKG-INFO
include *.txt
include *.py
[argparse]$ python setup.py bdist_dumb
running bdist_dumb
running build
running build_py
creating build
creating build/lib
copying argparse.py -build/lib
installing to build/bdist.macosx-10.3-fat/dumb
running install
running install_lib
creating build/bdist.macosx-10.3-fat
creating build/bdist.macosx-10.3-fat/dumb
creating build/bdist.macosx-10.3-fat/dumb/Library
creating build/bdist.macosx-10.3-fat/dumb/Library/Frameworks
creating build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages
copying build/lib/argparse.py ->
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages
byte-compiling
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse.py
to argparse.pyc
running install_egg_info
Writing
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse-0.5.0-py2.5.egg-info
creating /Users/rkern/src/argparse/dist
tar -cf /Users/rkern/src/argparse/dist/argparse-0.5.0.macosx-10.3-fat.tar .
gzip -f9 /Users/rkern/src/argparse/dist/argparse-0.5.0.macosx-10.3-fat.tar
removing 'build/bdist.macosx-10.3-fat/dumb' (and everything under it)
[argparse]$ tar ztf dist/argparse-0.5.0.macosx-10.3-fat.tar.gz
../
../Library/
../Library/Frameworks/
../Library/Frameworks/Python.framework/
../Library/Frameworks/Python.framework/Versions/
../Library/Frameworks/Python.framework/Versions/2.5/
../Library/Frameworks/Python.framework/Versions/2.5/lib/
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse-0.5.0-py2.5.egg-info
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse.py
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse.pyc
[argparse]$ python setup.py sdist
running sdist
reading manifest template 'MANIFEST.in'
writing manifest file 'MANIFEST'
creating argparse-0.5.0
making hard links in argparse-0.5.0...
hard linking PKG-INFO -argparse-0.5.0
hard linking README.txt -argparse-0.5.0
hard linking argparse.py -argparse-0.5.0
hard linking setup.py -argparse-0.5.0
hard linking test_argparse.py -argparse-0.5.0
tar -cf dist/argparse-0.5.0.tar argparse-0.5.0
gzip -f9 dist/argparse-0.5.0.tar
removing 'argparse-0.5.0' (and everything under it)
[argparse]$ tar ztf dist/argparse-0.5.0.tar.gz
argparse-0.5.0/
argparse-0.5.0/argparse.py
argparse-0.5.0/PKG-INFO
argparse-0.5.0/README.txt
argparse-0.5.0/setup.py
argparse-0.5.0/test_argparse.py
And this is what I get when I put test_argparse.py into a test/ directory:

[argparse]$ ls
PKG-INFO README.txt argparse.py setup.py test
[argparse]$ python setup.py bdist_dumb
running bdist_dumb
running build
running build_py
creating build
creating build/lib
copying argparse.py -build/lib
installing to build/bdist.macosx-10.3-fat/dumb
running install
running install_lib
creating build/bdist.macosx-10.3-fat
creating build/bdist.macosx-10.3-fat/dumb
creating build/bdist.macosx-10.3-fat/dumb/Library
creating build/bdist.macosx-10.3-fat/dumb/Library/Frameworks
creating build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
creating
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages
copying build/lib/argparse.py ->
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages
byte-compiling
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse.py
to argparse.pyc
running install_egg_info
Writing
build/bdist.macosx-10.3-fat/dumb/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse-0.5.0-py2.5.egg-info
creating /Users/rkern/src/argparse/dist
tar -cf /Users/rkern/src/argparse/dist/argparse-0.5.0.macosx-10.3-fat.tar .
gzip -f9 /Users/rkern/src/argparse/dist/argparse-0.5.0.macosx-10.3-fat.tar
removing 'build/bdist.macosx-10.3-fat/dumb' (and everything under it)
[argparse]$ tar ztf dist/argparse-0.5.0.macosx-10.3-fat.tar.gz
../
../Library/
../Library/Frameworks/
../Library/Frameworks/Python.framework/
../Library/Frameworks/Python.framework/Versions/
../Library/Frameworks/Python.framework/Versions/2.5/
../Library/Frameworks/Python.framework/Versions/2.5/lib/
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse-0.5.0-py2.5.egg-info
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse.py
../Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/argparse.pyc
[argparse]$ python setup.py sdist
running sdist
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default
file list)
writing manifest file 'MANIFEST'
creating argparse-0.5.0
creating argparse-0.5.0/test
making hard links in argparse-0.5.0...
hard linking README.txt -argparse-0.5.0
hard linking argparse.py -argparse-0.5.0
hard linking setup.py -argparse-0.5.0
hard linking test/test_argparse.py -argparse-0.5.0/test
tar -cf dist/argparse-0.5.0.tar argparse-0.5.0
gzip -f9 dist/argparse-0.5.0.tar
removing 'argparse-0.5.0' (and everything under it)
[argparse]$ tar ztf dist/argparse-0.5.0.tar.gz
argparse-0.5.0/
argparse-0.5.0/argparse.py
argparse-0.5.0/PKG-INFO
argparse-0.5.0/README.txt
argparse-0.5.0/setup.py
argparse-0.5.0/test/
argparse-0.5.0/test/test_argparse.py

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Steven Bethard]

Robert Kern wrote:

Steven Bethard wrote:

>Robert Kern wrote:

>>Steven Bethard wrote:
How do I get distutils to include my testing module in just the "sdist"
distribution?
Use a MANIFEST.

http://docs.python.org/dist/source-dist.html

Also, I just noted this tidbit:

"""If you don't supply an explicit list of files (or instructions on how to
generate one), the sdist command puts a minimal default set into the source
distribution:
...
* anything that looks like a test script: test/test*.py (currently, the
Distutils don't do anything with test scripts except include them in source
distributions, but in the future there will be a standard for testing Python
module distributions)
...
"""

So you can just stick test_argparse.py into a test/ directory. Having tested
this, it appears to work (see below).

Yeah, I was trying to avoid having to put it in a test/ directory. When
they're in the same directory, I can just run test_argparse.py and it
tests the local one instead of the one I have installed in
site-packages. When it's in a test/ directory, running
test/test_argparse.py tests the one in site-packages.

I did try putting test_argparse.py in a test/ directory though, and it
doesn't seem to solve the problem -- test_argparse.py gets put into
site-packages with the "bdist" command:

python setup.py sdist

running sdist
warning: sdist: manifest template 'MANIFEST.in' does not exist (using
default file list)
writing manifest file 'MANIFEST'
creating argparse-0.6.0
creating argparse-0.6.0\test
copying files to argparse-0.6.0...
copying README.txt -argparse-0.6.0
copying argparse.py -argparse-0.6.0
copying setup.py -argparse-0.6.0
copying test\test_argparse.py -argparse-0.6.0\test
creating 'dist\argparse-0.6.0.zip' and adding 'argparse-0.6.0' to it
adding 'argparse-0.6.0\argparse.py'
adding 'argparse-0.6.0\PKG-INFO'
adding 'argparse-0.6.0\README.txt'
adding 'argparse-0.6.0\setup.py'
adding 'argparse-0.6.0\test\test_argparse.py'
removing 'argparse-0.6.0' (and everything under it)

python setup.py bdist_dumb

running bdist_dumb
running build
running build_py
installing to build\bdist.win32\dumb
running install
running install_lib
creating build\bdist.win32\dumb
creating build\bdist.win32\dumb\Program Files
creating build\bdist.win32\dumb\Program Files\Python
creating build\bdist.win32\dumb\Program Files\Python\Lib
creating build\bdist.win32\dumb\Program Files\Python\Lib\site-packages
copying build\lib\argparse.py -build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages
copying build\lib\test_argparse.py -build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages
byte-compiling build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages\argparse.py to argparse.pyc
byte-compiling build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages\test_argparse.py to test_argparse.pyc
running install_egg_info
Writing build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages\argparse-0.6.0-py2.5.egg-info
creating 'dist\argparse-0.6.0.win32.zip' and adding '.' to it
adding 'Program
Files\Python\Lib\site-packages\argparse-0.6.0-py2.5.egg-info'
adding 'Program Files\Python\Lib\site-packages\argparse.py'
adding 'Program Files\Python\Lib\site-packages\argparse.pyc'
adding 'Program Files\Python\Lib\site-packages\test_argparse.py'
adding 'Program Files\Python\Lib\site-packages\test_argparse.pyc'
removing 'build\bdist.win32\dumb' (and everything under it)

>Using a MANIFEST appears to do the same thing as putting "test_argparse"
into py_modules -- that is, it puts "test_argparse.py" into both "sdist"
and "bdist" distributions. In "bdist" distributions it gets installed to
the site-packages directory like any other module.

Are you sure that you don't have changes left over in your setup.py when you
tested that?

Yep. (Though I still cleared everything out and tried it again.)
Here's what I got using an unmodified setup.py and the MANIFEST.in you
suggested. Note that the "bdist" version is putting test_argparse.py
into site-packages.

type MANIFEST.in

include PKG-INFO
include *.txt
include *.py

python setup.py sdist

running sdist
reading manifest template 'MANIFEST.in'
warning: no files found matching 'PKG-INFO'
writing manifest file 'MANIFEST'
creating argparse-0.6.0
copying files to argparse-0.6.0...
copying LICENSE.txt -argparse-0.6.0
copying README.txt -argparse-0.6.0
copying argparse.py -argparse-0.6.0
copying setup.py -argparse-0.6.0
copying test_argparse.py -argparse-0.6.0
creating 'dist\argparse-0.6.0.zip' and adding 'argparse-0.6.0' to it
adding 'argparse-0.6.0\argparse.py'
adding 'argparse-0.6.0\LICENSE.txt'
adding 'argparse-0.6.0\PKG-INFO'
adding 'argparse-0.6.0\README.txt'
adding 'argparse-0.6.0\setup.py'
adding 'argparse-0.6.0\test_argparse.py'
removing 'argparse-0.6.0' (and everything under it)

python setup.py bdist_dumb

running bdist_dumb
running build
running build_py
installing to build\bdist.win32\dumb
running install
running install_lib
creating build\bdist.win32\dumb
creating build\bdist.win32\dumb\Program Files
creating build\bdist.win32\dumb\Program Files\Python
creating build\bdist.win32\dumb\Program Files\Python\Lib
creating build\bdist.win32\dumb\Program Files\Python\Lib\site-packages
copying build\lib\argparse.py -build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages
copying build\lib\test_argparse.py -build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages
byte-compiling build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages\argparse.py to argparse.pyc
byte-compiling build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages\test_argparse.py to test_argparse.pyc
running install_egg_info
Writing build\bdist.win32\dumb\Program
Files\Python\Lib\site-packages\argparse-0.6.0-py2.5.egg-info
creating 'dist\argparse-0.6.0.win32.zip' and adding '.' to it
adding 'Program
Files\Python\Lib\site-packages\argparse-0.6.0-py2.5.egg-info'
adding 'Program Files\Python\Lib\site-packages\argparse.py'
adding 'Program Files\Python\Lib\site-packages\argparse.pyc'
adding 'Program Files\Python\Lib\site-packages\test_argparse.py'
adding 'Program Files\Python\Lib\site-packages\test_argparse.pyc'
removing 'build\bdist.win32\dumb' (and everything under it)
STeVe

[Robert Kern]

Steven Bethard wrote:

Robert Kern wrote:

>Are you sure that you don't have changes left over in your setup.py when you
tested that?

Yep. (Though I still cleared everything out and tried it again.)
Here's what I got using an unmodified setup.py and the MANIFEST.in you
suggested. Note that the "bdist" version is putting test_argparse.py
into site-packages.

python setup.py bdist_dumb

running bdist_dumb
running build
running build_py
installing to build\bdist.win32\dumb
running install
running install_lib

Are you sure that you removed the build/ directory? Because if it wasn't there,
distutils would say that it was creating one:

[argparse]$ python setup.py bdist_dumb
running bdist_dumb
running build
running build_py
creating build
creating build/lib
copying argparse.py -build/lib
installing to build/bdist.macosx-10.3-fat/dumb
running install
running install_lib

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Steven Bethard]

Robert Kern wrote:

Steven Bethard wrote:

>Robert Kern wrote:

>>Are you sure that you don't have changes left over in your setup.py when you
tested that?

Yep. (Though I still cleared everything out and tried it again.)
Here's what I got using an unmodified setup.py and the MANIFEST.in you
suggested. Note that the "bdist" version is putting test_argparse.py
into site-packages.

> python setup.py bdist_dumb

running bdist_dumb
running build
running build_py
installing to build\bdist.win32\dumb
running install
running install_lib

Are you sure that you removed the build/ directory?

Ahh thanks. Sorry I thought I was supposed to remove the old stuff from
dist/ not from build/. Yes, I get the same behavior as you do when I
remove build/. Sorry for the confusion!

Steve

[Steven Bethard]

Robert Kern wrote:

Steven Bethard wrote:

>How do I get distutils to include my testing module in just the "sdist"
distribution?

Use a MANIFEST.

Thanks again to Robert Kern for all the help. For the record, in the
end all I did was add a MANIFEST.in file with the single line:

include test_argparse.py

and "test_argparse.py" got included in the source distribution and left
out of the binary distributions.

It's a pity there's no way to specify something this simple in the
setup.py file, but since I'm not offering to maintain distutils, I guess
I don't really have a right to complain. ;-)

STeVe