By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,908 Members | 1,892 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,908 IT Pros & Developers. It's quick & easy.

calling python scripts as a sub-process

P: n/a
I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Popen command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

and I'm calling it as:

sub1 = subprocess.Popen(command)

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Thanks for any help,

Catherine
Nov 19 '08 #1
Share this Question
Share on Google+
9 Replies


P: n/a

On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ', '--
file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf
', '--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--
chmetric='M2' ", "--use_textid='true '"]
[snip]
>
I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ', '--
file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ', '--
file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ', '--
block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

Nov 19 '08 #2

P: n/a
On Nov 19, 1:03*pm, Catherine Moroney
<Catherine.M.Moro...@jpl.nasa.govwrote:
I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. *The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Popen command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]

and I'm calling it as:

sub1 = subprocess.Popen(command)

I get the error below. *Does anybody know what this error refers
to and what I'm doing wrong? *Is it even allowable to call another
script as a sub-process rather than calling it directly?

* File "../src_python/Match4.py", line 24, in RunMatch4
* * *sub1 = subprocess.Popen(command1)
* *File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
* * *errread, errwrite)
* *File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
* * *raise child_exception
OSError: [Errno 2] No such file or directory

Thanks for any help,

Catherine
Try giving an absolute path to the python file rather than a relative
path. I don't think the subprocess module "knows" where to look
otherwise.

Mike
Nov 19 '08 #3

P: n/a
On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <ph****@semanchuk.comwrote:
>
On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]

[snip]
>>
I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]
I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)
Nov 19 '08 #4

P: n/a
I just tried that, and I get the same error.

Interestingly enough, a shorter (and incorrect) version
of the command works well enough so that it gets into the
Match1 code and does the argument check there.

The following code gets into Match1:
>>command = ['python', '../src_python/Match1.py','--filex="xyz"']
sub1 = subprocess.Popen(command)
whereas this doesn't even get to call Match1:

command =
['python','/data/svn_workspace/cmm/sieglind/USC/EE569/tpaper/test/../src_python/Match1.py
', '--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', '--chmetric=M2 ',
'--use_textid=true']

sub1 = subprocess.Popen(command)

Can anybody see a reason for why the abbreviated version works, and
the full-up one doesn't?

Catherine

Philip Semanchuk wrote:
>
On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ',
"--chmetric='M2' ", "--use_textid='true '"]

[snip]
>>
I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]
Nov 19 '08 #5

P: n/a
Dan Upton wrote:
On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <ph****@semanchuk.comwrote:
>On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>>The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]
[snip]
>>I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='true '"]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)
I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?

Catherine
Nov 19 '08 #6

P: n/a
On Wed, Nov 19, 2008 at 2:38 PM, Catherine Moroney
<Ca*****************@jpl.nasa.govwrote:
Dan Upton wrote:
>>
On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <ph****@semanchuk.com>
wrote:
>>>
On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]

[snip]

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)

I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?
Alternately, rather than using a list of arguments, have you tried
just using a string? (Again, that's the way I do it and I haven't
been having any problems recently, although I'm running shell scripts
or binaries with arguments rather than trying to invoke python on a
script.)

command = "python ../src_python/Match1.py
--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf
--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf
--block_start=62 --block_end=62 --istep=16 --chmetric='M2'
--use_textid=true"

proc = subprocess.Popen(command, shell=True)
Nov 19 '08 #7

P: n/a
Catherine Moroney wrote:
I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Popen command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='true '"]
If you want to avoid going by the shell, and you *should* for security
reasons, you need to have each of your arguments separately in the list
without the shell quoting and extra spaces, i.e.

['python', '../src_python/Match1.py',
'--file_ref=xxxx.hdf', '--file_cmp=yyyy.hdf',
'--block_start=xx', '--block_end=62', '--istep=16', '--chmetric=M2',
'--use_texid=true']

Jeremy

--
Jeremy Sanders
http://www.jeremysanders.net/
Nov 19 '08 #8

P: n/a
Dan Upton wrote:
I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)
That's really papering over the bug. You need to have the parameters
separately, including the name of the program, separately in the list. You
need to remove any shell quoting you may use on the unix/dos command line.

Jeremy

--
Jeremy Sanders
http://www.jeremysanders.net/
Nov 19 '08 #9

P: n/a
Dan Upton wrote:
On Wed, Nov 19, 2008 at 2:38 PM, Catherine Moroney
<Ca*****************@jpl.nasa.govwrote:
>Dan Upton wrote:
>>On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <ph****@semanchuk.com>
wrote:
On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:

The command (stored as an array of strings) that I'm executing is:
>
['python ../src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]
>
[snip]

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?
>
File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Popen(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf ',
'--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
",
"--use_textid='true '"]
I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Popen(command, shell=True)
I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?

Alternately, rather than using a list of arguments, have you tried
just using a string? (Again, that's the way I do it and I haven't
been having any problems recently, although I'm running shell scripts
or binaries with arguments rather than trying to invoke python on a
script.)

command = "python ../src_python/Match1.py
--file_ref=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_BF _F03_0024.hdf
--file_cmp=MISR_AM1_GRP_ELLIPSOID_GM_P228_O003571_DF _F03_0024.hdf
--block_start=62 --block_end=62 --istep=16 --chmetric='M2'
--use_textid=true"

proc = subprocess.Popen(command, shell=True)
Thanks - that did the trick. I just passed in one long string
and everything actually works. Wow! I had no idea if this was
even do-able.

This is so cool, and saves me a lot of code duplication. I can
spawn off half a dozen jobs at once and then just wait for them
to finish. It's great that python can function both as a
scripting language and also a full-blown programming language
at the same time.

Catherine
Nov 19 '08 #10

This discussion thread is closed

Replies have been disabled for this discussion.