Popen and wget, problems

[Jesse]

Hi all, I have a problem using wget and Popen. I hope someone can help.
-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org"
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is executed)

Popen doesn't work, while os.system and shell do. Popen will give the error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -"dir1/cpan.txt" [1]

-- Background info about wget --
-Option -nv: -nv, --no-verbose turn off verboseness, without
being quiet.
-Option -O: -O, --output-document=FILE write documents to FILE.

Note that wget requires any directories in the file-path of option -O to be
existing before the wget command is executed.
-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('dir')
except:
print 'dir already exists'
# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n%s" % (proc.stdout.read() + proc.stderr.read())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.read() +
proc.stdout.read())
# execute using os.system (does work)
os.system(cmd)
-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory
-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

[Rob Wolfe]

"Jesse" <ha**@spam.comwrites:

Hi all, I have a problem using wget and Popen. I hope someone can help.
-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org"
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is executed)

Popen doesn't work, while os.system and shell do. Popen will give the error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -"dir1/cpan.txt" [1]

[...]

-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('dir')
except:
print 'dir already exists'
# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n%s" % (proc.stdout.read() + proc.stderr.read())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.read() +
proc.stdout.read())
# execute using os.system (does work)
os.system(cmd)
-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory
-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

I don't know exactly why in this case Popen doesn't work,
but the counterpart of os.system is Popen with option shell=True
and the first parameter should be a string instead of list.
That seems to work:
proc = Popen("wget -nv -O dir/cpan.txt http://search.span.org",
shell=True, stdout=PIPE, stderr=PIPE)

and this variant seems to work too:
cmd_set = ['wget', '-nv', '-O', 'dir/cpan.txt', 'http://search.span.org']

--
HTH,
Rob

[Jesse]

Thx Rob!

Your solution works perfect!
"Rob Wolfe" <[e-mail] in message
news:87************@merkury.smsnet.pl...

"Jesse" <ha**@spam.comwrites:

>Hi all, I have a problem using wget and Popen. I hope someone can help.
-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org"
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is
executed)

Popen doesn't work, while os.system and shell do. Popen will give the
error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -"dir1/cpan.txt" [1]

[...]

>-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('dir')
except:
print 'dir already exists'
# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n%s" % (proc.stdout.read() + proc.stderr.read())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.read() +
proc.stdout.read())
# execute using os.system (does work)
os.system(cmd)
-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory
-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

I don't know exactly why in this case Popen doesn't work,
but the counterpart of os.system is Popen with option shell=True
and the first parameter should be a string instead of list.
That seems to work:
proc = Popen("wget -nv -O dir/cpan.txt http://search.span.org",
shell=True, stdout=PIPE, stderr=PIPE)

and this variant seems to work too:
cmd_set = ['wget', '-nv', '-O', 'dir/cpan.txt', 'http://search.span.org']

--
HTH,
Rob

[js]

Hi Jesse.

cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org']

[snip]

>proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)

wget will treat this as
$ wget -nv '-O dir/cpan.txt' "http://search.cpan.org"

And will emit the following error because there's no pathname ' dir/cpan.txt'.
(Note the pathname has a trailing space.)

dir/cpan.txt: No such file or directory

Replace '-O dir/cpan.txt" with '-Odir/cpan.txt' or '-O', 'dir/cpan.txt'
and it should work.

Hope this helps