baffling os.system behavior - differs from command line

[Ben Floyd]

It goes like this:
Im converting a perl script that executes a command on each result of
a directory listing. In perl, all is well. In Python, using
approximately
the same logic creates a seemingly random problem I can't pin down.
I hesitated to ask this question since it's so Slackware
Linux-centric,
and im new to Python. It's hard to ask in a generic way, but im out of
options...

The command is Slackware's package creation utility, makepkg. It uses
the
current working directory as package source and takes a filename as an
argument to output the newly created package to. Makepkg is
essentially
a tarball creator, wrapped in a nice command. Simple stuff. Here is my
code:

I have a for loop that reads a file list, like so:
pkg_srcdir = '/packages_source'
pkg_dir = '/packages_tgzs'
ls = os.listdir(pkg_srcdir)

for name in ls:
build_src = pkg_srcdir + '/' + name
new_filename = pkg_dir + '/' + name + '.tgz'

# Valid directory?
if not os.path.isdir(build_src):
sys.exit("\nInvalid directory \'" +
pkg_srcdir + "\'\n")

# Go into the new directory
try: os.chdir(build_src)
except os.error, value:
sys.exit("\nInvalid directory: \'" +
build_src + "\': " + value[1])

# I then execute the command on each result,
like so:
# new_filename is correctly assembled, which i
can verify
# from printing it out on each loop iteration
os.system('/sbin/makepkg --chown n --linkadd y
' + new_filename)

When all is done running, which take a good 7 minutes, I get varied
results.
It looks like its working... pauses, has lots of disk activity on the
drive
(verified with ps and vmstat), but in the end only a couple of
packages are
created. I can actually watch the output of makepkg and see it
creating the
tarballs. Here is the truly whacky part: Changing pkg_dir to anything
else,
say /tmp/beavis, will create the packages correctly for a while, then
mysteriously stop with no warning or change to the script - just run
it a
few times and it stops working.

Im running as root, so its not a permission problem - its far to
erratic
anyway.

I dont think this problem has anything to do with os.system, per say,
since it also happens with os.popen.

I thought it might be a buffering problem, so i tried using Python's
-u
switch and flushing stdout with sys.stdout.flush() - no go.

I tried sleeping for long periods of time in each loop iteration - no
go.

I tried writing each individual makepkg command out to a file and
running it with perl, doing all the proper filehandling. I also
tried bash for kicks - no go

In the end, I put an ampersand at the end of the makepkg command, like
so:
os.system('/sbin/makepkg --chown n --linkadd y ' +
new_filename + ' &')

This sort of works: it creates 95% of the packages most of the time -
results varies. It always seems to leave out the same packages.
The packages it misses are in the middle of the list, so its not
just skipping the first or the last few.

Im baffled.

Any ideas?

[Donn Cave]

In article <60**************************@posting.google.com >,
ev**@pondie.com (Ben Floyd) wrote:
....

The command is Slackware's package creation utility, makepkg. It uses
the
current working directory as package source and takes a filename as an
argument to output the newly created package to. Makepkg is
essentially
a tarball creator, wrapped in a nice command. Simple stuff. Here is my
code:

I have a for loop that reads a file list, like so:
pkg_srcdir = '/packages_source'
pkg_dir = '/packages_tgzs'
ls = os.listdir(pkg_srcdir)

for name in ls:
build_src = pkg_srcdir + '/' + name
new_filename = pkg_dir + '/' + name + '.tgz'

# Valid directory?
if not os.path.isdir(build_src):
sys.exit("\nInvalid directory \'" + pkg_srcdir + "\'\n")

# Go into the new directory
try: os.chdir(build_src)
except os.error, value:
sys.exit("\nInvalid directory: \'" + build_src + "\': " + value[1])

# I then execute the command on each result, like so:
# new_filename is correctly assembled, which i can verify
# from printing it out on each loop iteration
os.system('/sbin/makepkg --chown n --linkadd y ' + new_filename)

Well, I don't know. In casual inspection I don't notice
anything obviously wrong, but then for all I know this
isn't the exact code that has the problem anyway. What
I could suggest is substitute your own shell script for
makepkg, something that prints out its arguments, current
working directory and whatever you think might be interesting.
The object is to find out exactly what you're asking makepkg
to do, since that is evidently a problem.

Actually a Python program might be a little better, because
"print sys.argv" gives you less ambiguous information than
"echo $*", more like "for arg do echo $arg; done" but on
one line.

The two usage points that occur to me are
1. chdir() affects Python module path resolution. If you
chdir to a directory with an "os.py", that would be an
obvious if unlikely problem, but also some installations
rely on Python's library finding heuristics in a way
that is kind of fragile. I may be wrong about that,
I don't even want to think about wasting time on this
misguided feature.

2. system() can be replaced with spawnv() for an increase
in command parameter list reliability. What if some
file has spaces? In spawnv(), you don't care, but in
system() it's your job to quote it. This is also a
security related issue.

Donn Cave, do**@u.washington.edu