Avoiding shell metacharacters in os.popen

[Nick Craig-Wood]

I'm trying to avoid using shell metacharacters in os.popen in a portable
fashion.

os.popen() only seems to take a string as the command which would need
tricky quoting.

os.popen2() can take a string or a list - the relevant code in Unix
python (in popen2.py) being...

def _run_child(self, cmd):
if isinstance(cmd, basestring):
cmd = ['/bin/sh', '-c', cmd]
for i in range(3, MAXFD):
try:
os.close(i)
except OSError:
pass
try:
os.execvp(cmd[0], cmd)
finally:
os._exit(1)

This is perfect behaviour as far as I'm concerned - if you pass a list
it doesn't go through the shell and if you pass a string it does.

eg

w, r = os.popen2(["ls", "-l"])
r.read()

This leads on to my questions :-

1) is this behaviour (string vs list) of popen2 intentional? Its not
documented anywhere and it doesn't work under Windows (gives
"TypeError: popen2() argument 1 must be string, not list")

2) is there anything similar for os.popen() planned?

3) is there an equivalent to the perl quotemeta() command. This
quotes meta-characters in a string for use in the shell. Here is a
suitable definition for Unix, but I don't think that \ quoting works
in Windows

cmd = re.sub(r"(\W)", r"\\\1", cmd)

Avoiding shell metacharacter attacks is a must for secure programs.
Python does pretty well with its os.exec* and os.spawn* functions, but
seems to be lacking in the os.popen* department!

Any insights appreciated!

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick

[Istvan Albert]

Nick Craig-Wood wrote:

Avoiding shell metacharacter attacks is a must for secure programs.

Not passing down commands into a shell is a must for secure programs.

What you should do is recognize a command, identify it as a
valid and allowed one, then call it yourself. If you think that
escaping metacharacters gives you any kind of security you are
deceiving yourself.

Istvan.

[Nick Craig-Wood]

Istvan Albert <ia*****@mailblocks.com> wrote:

Nick Craig-Wood wrote:

Avoiding shell metacharacter attacks is a must for secure programs.
Not passing down commands into a shell is a must for secure programs.

What you should do is recognize a command, identify it as a
valid and allowed one, then call it yourself.

I'm not running commands passed by the user - that would be nuts!

I'm running another program written by us. The program doing the
running is a CGI and it needs to pass parameters to the second program
which come from the user. It also needs to read the output of that
program - hence popen.

What my post was about was avoiding the shell completely. If you use
os.system(string) then you go via the shell. However if you use
os.spawnl(mode, file, *args) then it doesn't go anywhere near the
shell. As I pointed out in my post there isn't an equivalent for
os.popen* which doesn't go via the shell (except for undocumented
os.popen2).
If you think that escaping metacharacters gives you any kind of
security you are deceiving yourself.

As a second best escaping the metacharacters and using os.popen will
work, but AFAICS there isn't a portable metacharacter escaping routine
built into python.

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick

[Donn Cave]

In article <sl*****************@irishsea.home.craig-wood.com>,
Nick Craig-Wood <ni**@craig-wood.com> wrote:
....

What my post was about was avoiding the shell completely. If you use
os.system(string) then you go via the shell. However if you use
os.spawnl(mode, file, *args) then it doesn't go anywhere near the
shell. As I pointed out in my post there isn't an equivalent for
os.popen* which doesn't go via the shell (except for undocumented
os.popen2).

Well, it sounded to me like the real problem is that Microsoft
Windows doesn't support any functional equivalent to spawnv
for pipes. I don't know if that's true or not, I'm just
taking it from you that os.popen2 doesn't support a list of
parameters [1] on Microsoft Windows platforms, and inferring
that it doesn't because it can't - there isn't any specific
function that does it, and you can't just roll your own out
of pipe/fork/execve like you can on UNIX, as in fact popen2 does.

If that's really a question and not a well known fact, you
might pose it again with a subject line that would attract
more attention from Microsoft Windows developers, since
only they would know. The attempt to cast it as a general
Python problem could be counterproductive, if it means the
people who read about this problem tend to be those who
don't really suffer from it.

Donn Cave, do**@u.washington.edu

[Michael Hoffman]

Nick Craig-Wood wrote:

As I pointed out in my post there isn't an equivalent for
os.popen* which doesn't go via the shell (except for undocumented
os.popen2).

Is it the undocumented nature of it that is unnerving you?

I submitted a bug on the documentation more than six months ago but no
one has responded.

http://sourceforge.net/tracker/?func...70&atid=105470
--
Michael Hoffman

[Nick Craig-Wood]

Michael Hoffman <m.*********************************@example.com > wrote:

Nick Craig-Wood wrote:

As I pointed out in my post there isn't an equivalent for
os.popen* which doesn't go via the shell (except for undocumented
os.popen2).
Is it the undocumented nature of it that is unnerving you?

Yes! If it was documented then it would have to be made to work on
Windows too I gues.
I submitted a bug on the documentation more than six months ago but no
one has responded.

http://sourceforge.net/tracker/?func...70&atid=105470

At least it is in the system.

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick

[Nick Craig-Wood]

Donn Cave <do**@u.washington.edu> wrote:

Well, it sounded to me like the real problem is that Microsoft
Windows doesn't support any functional equivalent to spawnv for
pipes. I don't know if that's true or not,
No me neither!
I'm just taking it from you that os.popen2 doesn't support a list
of parameters [1] on Microsoft Windows platforms,
Well it didn't work when I tried it.
and inferring that it doesn't because it can't - there isn't any
specific function that does it, and you can't just roll your own
out of pipe/fork/execve like you can on UNIX, as in fact popen2
does.
It could always do " ".join(cmd) which would leave windows users
exactly where they are at the moment.

I'll probably end up writing a little wrapper to popen2 which does
something like that, possibly with a bit better quoting for windows.
If that's really a question and not a well known fact, you
might pose it again with a subject line that would attract
more attention from Microsoft Windows developers, since
only they would know. The attempt to cast it as a general
Python problem could be counterproductive, if it means the
people who read about this problem tend to be those who
don't really suffer from it.

Good point. I'll have a think about it (and a look at the Windows
source of popen2!)

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick

[Michael Hudson]

Nick Craig-Wood <ni**@craig-wood.com> writes:

Michael Hoffman <m.*********************************@example.com > wrote:

Nick Craig-Wood wrote:

As I pointed out in my post there isn't an equivalent for
os.popen* which doesn't go via the shell (except for undocumented
os.popen2).
Is it the undocumented nature of it that is unnerving you?

Yes! If it was documented then it would have to be made to work on
Windows too I gues.

I think this might be part of the reason it's underdocumented. I also
have a sneaking suspicion it's impossible on windows, but don't really
know...

Cheers,
mwh

-- Why are we talking about bricks and concrete in a lisp newsgroup?

After long experiment it was found preferable to talking about why
Lisp is slower than C++...
-- Duane Rettig & Tim Bradshaw, comp.lang.lisp

[David Fraser]

Nick Craig-Wood wrote:

I'm trying to avoid using shell metacharacters in os.popen in a portable
fashion.

os.popen() only seems to take a string as the command which would need
tricky quoting.

os.popen2() can take a string or a list - the relevant code in Unix
python (in popen2.py) being...

def _run_child(self, cmd):
if isinstance(cmd, basestring):
cmd = ['/bin/sh', '-c', cmd]
for i in range(3, MAXFD):
try:
os.close(i)
except OSError:
pass
try:
os.execvp(cmd[0], cmd)
finally:
os._exit(1)

This is perfect behaviour as far as I'm concerned - if you pass a list
it doesn't go through the shell and if you pass a string it does.

eg

w, r = os.popen2(["ls", "-l"])
r.read()

This leads on to my questions :-

1) is this behaviour (string vs list) of popen2 intentional? Its not
documented anywhere and it doesn't work under Windows (gives
"TypeError: popen2() argument 1 must be string, not list")

2) is there anything similar for os.popen() planned?

3) is there an equivalent to the perl quotemeta() command. This
quotes meta-characters in a string for use in the shell. Here is a
suitable definition for Unix, but I don't think that \ quoting works
in Windows

cmd = re.sub(r"(\W)", r"\\\1", cmd)

Avoiding shell metacharacter attacks is a must for secure programs.
Python does pretty well with its os.exec* and os.spawn* functions, but
seems to be lacking in the os.popen* department!

Any insights appreciated!

I can't exactly remember the details but I now I've had painful
experiences on Windows trying to do this kind of thing. It seems to mess
up quoting parameters / exe file names

David

[Nick Craig-Wood]

David Fraser <da****@sjsoft.com> wrote:

Nick Craig-Wood wrote:

3) is there an equivalent to the perl quotemeta() command. This
quotes meta-characters in a string for use in the shell. Here is a
suitable definition for Unix, but I don't think that \ quoting works
in Windows

cmd = re.sub(r"(\W)", r"\\\1", cmd)

I can't exactly remember the details but I now I've had painful
experiences on Windows trying to do this kind of thing. It seems to mess
up quoting parameters / exe file names

Last time I did this for Windows was in C (urk!). I put every
parameter in double quotes and then doubled up any " in the parameter.
This worked mostly but I don't think the OS parsed some bizarre cases
properly.

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick

[David Fraser]

Nick Craig-Wood wrote:

David Fraser <da****@sjsoft.com> wrote:

Nick Craig-Wood wrote:

3) is there an equivalent to the perl quotemeta() command. This
quotes meta-characters in a string for use in the shell. Here is a
suitable definition for Unix, but I don't think that \ quoting works
in Windows

cmd = re.sub(r"(\W)", r"\\\1", cmd)

I can't exactly remember the details but I now I've had painful
experiences on Windows trying to do this kind of thing. It seems to mess
up quoting parameters / exe file names

Last time I did this for Windows was in C (urk!). I put every
parameter in double quotes and then doubled up any " in the parameter.
This worked mostly but I don't think the OS parsed some bizarre cases
properly.

If I recall correctly (yikes) the problem was that Python's os.system
would handle either a program name with spaces in it or a parameter with
spaces in it, but quoting them both didn't work. Anyway if someone can
get it to work nicely please post the details here :-)

David