When argparse will be in the python standard installation

[PengYu.UT]

Hi,

I feel argparse has some useful things that optparse doesn't have. But
I can't find it argparse in python library reference. I'm wondering
when it will be available in the python standard installation.

Thanks,
Peng

[Fredrik Lundh]

Pe*******@gmail.com wrote:

I feel argparse has some useful things that optparse doesn't have. But
I can't find it argparse in python library reference. I'm wondering
when it will be available in the python standard installation.

there's already two different option parsing modules in the standard
library. maybe improving one of the existing ones might be a better
idea than adding a third one?

</F>

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Pe*******@gmail.com schrieb:

I feel argparse has some useful things that optparse doesn't have. But
I can't find it argparse in python library reference. I'm wondering
when it will be available in the python standard installation.

On its own, never. Somebody has to contribute it to Python, and that
somebody has to be the author.

However, it would be better if missing features where provided in
existing libraries, rather than introducing new libraries to replace
the existing ones (i.e. I would likely reject a patch to add yet another
argument parsing library - we already have getopt and optparse).

Regards,
Martin

[Steven Bethard]

Martin v. Löwis wrote:

Pe*******@gmail.com schrieb:

>I feel argparse has some useful things that optparse doesn't have. But
I can't find it argparse in python library reference. I'm wondering
when it will be available in the python standard installation.

On its own, never. Somebody has to contribute it to Python, and that
somebody has to be the author.

As the author of argparse, I'm happy to contribute it, though I've been
thinking that it would be more appropriate for the Python 3.0 line than
the Python 2.X line.

However, it would be better if missing features where provided in
existing libraries, rather than introducing new libraries to replace
the existing ones (i.e. I would likely reject a patch to add yet another
argument parsing library - we already have getopt and optparse).

That was my original intention, and argparse started out using the
optparse code. But optparse is quite difficult to extend, and in the
end, when I found that I had monkey-patched or rewritten just about
everything in optparse, I decided to drop the optparse code entirely.

If someone has an idea how to include argparse features into optparse,
I'm certainly all for it. But I tried and failed to do this myself, so I
don't know how to go about it.

STeVe

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Steven Bethard schrieb:

If someone has an idea how to include argparse features into optparse,
I'm certainly all for it. But I tried and failed to do this myself, so I
don't know how to go about it.

It's not necessary that the implementation is retained, only that the
interface is preserved. So if you can come up with an implementation
that supports all optparse applications, and adds the additional
features, your implementation could replace the current optparse.
If you need to make incompatible changes, it would be best if you
could produce a list of such changes, so that optparse users can
review them to find out whether they are affected.

Regards,
Martin

[Steven Bethard]

Martin v. Löwis wrote:

Steven Bethard schrieb:

>If someone has an idea how to include argparse features into optparse,
I'm certainly all for it. But I tried and failed to do this myself, so I
don't know how to go about it.

It's not necessary that the implementation is retained, only that the
interface is preserved. So if you can come up with an implementation
that supports all optparse applications, and adds the additional
features, your implementation could replace the current optparse.
If you need to make incompatible changes, it would be best if you
could produce a list of such changes, so that optparse users can
review them to find out whether they are affected.

FWIW, here's a short list of issues that could be easily addressed:

* alias ArgumentParser to OptionParser
* alias add_argument to add_option
* alias Values to Namespace
* alias OptionError and OptionValueError to ArgumentError
* alias add_help= keyword argument of ArgumentParser to add_help_option=
* alias namespace= keyword argument of parse_args to options=
* add the has_option, get_option and remove_option methods
* add the set_conflict_handler method
* add the destroy method
* add the set_usage method
* add the string names for types (e.g. "string", "str", "int", etc.) as
aliases to the type objects (argparse already has a type registry for
exactly these kinds of things)
Some slightly harder issues:

* ArgumentParser.parse_args returns a single namespace object, not an
(options, args) tuple, since argparse handles positional arguments.
This could probably be addressed by adding an __iter__ method to the
Namespace object which would return (self, [])

* argparse uses standard string formatting specifiers, e.g. %(default)s
and %(prog)s instead of optparse's %default and %prog. It could
probably be hacked to support both though.

* argparse doesn't support {enable,disable}_interspersed_args() because
their primary use case was for handling sub-parsers, which argparse
handles through the add_subparsers method. It could probably be hacked
to work though I guess.
And the issues that I don't see any good way to address:

* argparse makes the default value for a "store_true" action False, and
the default value for a "store_false" action True. This is what users
expect, but different from optparse where the default is always None.

* the ArgumentParser constructor doesn't accept the option_list=,
option_class= or formatter= keyword arguments. Since argparse uses
Action subclasses instead of a single Option class, the former two
don't make much sense. And formatter= has been replaced with
formatter_class= where the API of the formatter was dramatically
changed. (That said, the formatter API is undocumented in both
optparse and argparse.)

* the choices argument is now checked *after* arguments have been
type-converted. This is intentional, so that you can specify, say
xrange(100) instead of ["0", "1", "2", "3", ... "99"]. There is also
no "choices" type anymore since any action can also specify their
choices.

* argparse doesn't support ``callback`` actions because the same effects
are better handled by defining a simple type-checking function and
passing it as the type= keyword argument, or by subclassing
argparse.Action and passing this as the action= keyword argument.
I could probably add callback actions by creating an appropriate
Action subclass and registering it. However, any code relying on
parser.{largs,rargs,values} would break because the parsing algorithm
is completely different in argparse.

* The `Extending optparse`_ section in the docs is pretty much
completely inapplicable. One of the major goals of argparse was to get
rid of the need to hack class attributes like TYPES, STORE_ACTIONS,
etc. Instead, these things are handled by defining simple
string-to-object functions that are passed to type= or by defining
appropriate subclasses of argparse.Action which are passed to action=.
Trying to support these sorts of things would be nearly impossible.
I guess I don't know how to proceed from here. I'm reluctant to start
adding the code necessary to support even the easily solved issues when
the issues that I don't know how to solve seem like they could be deal
breakers.

STeVe

... _Extending optparse:
http://docs.python.org/lib/optparse-...-optparse.html

[John J. Lee]

Steven Bethard <st************@gmail.comwrites:

Martin v. Löwis wrote:

Steven Bethard schrieb:

If someone has an idea how to include argparse features into optparse,
I'm certainly all for it. But I tried and failed to do this myself, so I
don't know how to go about it.

It's not necessary that the implementation is retained, only that the
interface is preserved. So if you can come up with an implementation
that supports all optparse applications, and adds the additional
features, your implementation could replace the current optparse.
If you need to make incompatible changes, it would be best if you
could produce a list of such changes, so that optparse users can
review them to find out whether they are affected.

FWIW, here's a short list of issues that could be easily addressed:

* alias ArgumentParser to OptionParser

[...snip long list of proposed aliases and additions...]

Some slightly harder issues:

[...]

You propose here to attempt to merge optparse and argparse into a
single interface (single class, by the sound of it).

I stopped reading after a bit, but at least up to that point, all of
these issues could (and certainly should, if argparse is added to the
stdlib, even in Python 3) be addressed by instead providing both
OptionParser and an OptsAndArgsParser class (or whatever you'd call
the latter). OptionParser would then support exactly the current
optparse.OptionParser interface. OptsAndArgsProcessor would support
your new interface. Presumably most of the actual implementation code
would be shared.
John

[Steven Bethard]

Steven Bethard schrieb:

If someone has an idea how to include argparse features into optparse,
I'm certainly all for it. But I tried and failed to do this myself, so
I don't know how to go about it.

Martin v. Löwis wrote:

It's not necessary that the implementation is retained, only that the
interface is preserved.

[snip]

If you need to make incompatible changes, it would be best if you
could produce a list of such changes

Steven Bethard writes:

FWIW, here's a short list of issues that could be easily addressed:

* alias ArgumentParser to OptionParser

[snip]

John J. Lee wrote:

all of these issues could ... be addressed by instead providing both
OptionParser and an OptsAndArgsParser class (or whatever you'd call
the latter).

You're essentially proposing to move argparse.ArgumentParser to the
optparse module. (The ArgumentParser class handles both optional and
positional command-line args -- it *is* your OptsAndArgsProcessor).

That's certainly an easy option for me =) but does it address Martin and
Fredrik's concerns that there are already too many argument parsing
libraries in the stdlib?

Presumably most of the actual implementation code
would be shared.

Well, the "harder issues" I listed in the previous post were basically
the reasons that sharing the implementation code would be difficult. I
don't know how to do it without breaking OptionParser in backwards
incompatible ways. (The documented ways of extending optparse make this
particularly difficult since they make public a number of internal
optparse details.)

STeVe

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Steven Bethard schrieb:

* alias ArgumentParser to OptionParser
* alias add_argument to add_option
* alias Values to Namespace
* alias OptionError and OptionValueError to ArgumentError
* alias add_help= keyword argument of ArgumentParser to add_help_option=
* alias namespace= keyword argument of parse_args to options=

Not sure what you mean by "alias": I don't think the argparse names
should be retained (unless you feel they describe things better).
You don't need to aim for argparse compatibility: users of argparse
can continue to use the current (or future) version of argparse as
a separate module.

* ArgumentParser.parse_args returns a single namespace object, not an
(options, args) tuple, since argparse handles positional arguments.
This could probably be addressed by adding an __iter__ method to the
Namespace object which would return (self, [])

I find that I'm not that familiar with command line arguments :-)

To understand that difference, I would have to see examples of both,
and why it is better to return a single result.

In any case, if the argparse interface is more powerful, and the
old interface cannot be emulated transparently, a new interface could
be added, with the old one deprecated and phased out (preferably with
some conversion guideline). It would be possible to drop the old one
in Py3k, I guess.

* argparse uses standard string formatting specifiers, e.g. %(default)s
and %(prog)s instead of optparse's %default and %prog. It could
probably be hacked to support both though.

That would be good.

* argparse makes the default value for a "store_true" action False, and
the default value for a "store_false" action True. This is what users
expect, but different from optparse where the default is always None.

Could we deprecate "store_false" with no default? It may be that users
expect to see True and False, but how can you find out whether the
option was given or not afterwards?

Alternatively, could the actions become keyword arguments (with the
action argument only preserved for backwards compatibility)? E.g.

parser.add_option("-v", store_true_into="verbose")
parser.add_option("-q", store_false_into="verbose")
(same for store, append, count, callback)

* the choices argument is now checked *after* arguments have been
type-converted. This is intentional, so that you can specify, say
xrange(100) instead of ["0", "1", "2", "3", ... "99"]. There is also
no "choices" type anymore since any action can also specify their
choices.

As an incompatible change, this could be warned-about if the type is not
string, yet the first choice is.

Alternatively, we could require users that pass converted values to
specify converted_choices=True in 2.6, and warn all users that don't
pass a converted_choices flag, and then flip the default in 2.7 and 3.0.
Then current users had a quick fix to silence the warning, and new
users could drop the extra flag in the future.

I could probably add callback actions by creating an appropriate
Action subclass and registering it. However, any code relying on
parser.{largs,rargs,values} would break because the parsing algorithm
is completely different in argparse.

If you can find a way to make callbacks work in the "common case",
this flag could be deprecated and removed.

I guess I don't know how to proceed from here. I'm reluctant to start
adding the code necessary to support even the easily solved issues when
the issues that I don't know how to solve seem like they could be deal
breakers.

This asks for a PEP. The views above are mine only, others may feel
differently.

Regards,
Martin

[Steven Bethard]

[Thanks for looking through all these Martin!]

Martin v. Löwis wrote:

Steven Bethard schrieb:

>* alias ArgumentParser to OptionParser
* alias add_argument to add_option
* alias Values to Namespace
* alias OptionError and OptionValueError to ArgumentError
* alias add_help= keyword argument of ArgumentParser to add_help_option=
* alias namespace= keyword argument of parse_args to options=

Not sure what you mean by "alias": I don't think the argparse names
should be retained (unless you feel they describe things better).

They do describe things a bit better. Given a command line like::

prog --opt1 o --opt2 pos1 pos2 pos3

optparse can only parse out the optional command line arguments,
``--opt1 o --opt2``, hence it uses names like ``add_option``. On the
other hand, argparse can parse the entire command line (including the
positional arguments ``pos1 pos2 pos3``) so it uses ``add_argument``
instead of ``add_option``.

>* ArgumentParser.parse_args returns a single namespace object, not an
(options, args) tuple, since argparse handles positional arguments.
This could probably be addressed by adding an __iter__ method to the
Namespace object which would return (self, [])

I find that I'm not that familiar with command line arguments :-)

To understand that difference, I would have to see examples of both,
and why it is better to return a single result.

A brief example to illustrate the differences. Here is a program that
reads the command-line args and writes the sum to a log file specified
by --log. With optparse, this would look something like::

parser = optparse.OptionParser()
parser.add_option('--log')
options, args = parser.parse_args()
if options.log is not None:
logfile = open(options.log)
else:
logfile = sys.stdout
try:
integers = [int(arg) for arg in args]
except ValueError:
parser.error('positional arguments must be integers')
logfile.write('%s\n' % sum(integers))
logfile.close()

Since argparse handles positional arguments as well as optional
arguments (and since it accepts type='outfile') the argparse version is
much more concise::

parser = argparse.ArgumentParser()
parser.add_argument('integers', type=int, nargs='+')
parser.add_argument('--log', type='outfile', default=sys.stdout)
args = parser.parse_args()
args.log.write('%s\n' % sum(args.integers))
args.log.close()

So basically, since argparse parses the entire command line (instead of
just the optional parts) there is no concept of "remaining command line
arguments" -- *all* command line arguments have become attributes of the
namespace returned by parse_args(). Thus, while optparse has to return
both a namespace and the unparsed portion of the command line, argparse
only has to return the namespace.

>* argparse makes the default value for a "store_true" action False, and
the default value for a "store_false" action True. This is what users
expect, but different from optparse where the default is always None.

Could we deprecate "store_false" with no default?

That's probably reasonable. In Python 3.0, the default could then
become the appropriate one.

It may be that users expect to see True and False, but how can you find
out whether the option was given or not afterwards?

For both optparse and argparse, with "store_false", you know the option
was not given if the stored value is something other than ``False``, and
with "store_true", you know the option was not given if the stored value
is something other than ``True``.

With optparse, that "something other" is always ``None`` which is
unfortunate because it makes the following code fail::

parser = optparse.OptionParser()
parser.add_option('--not-foo', action='store_false')
options, args = parser.parse_args()
if options.not_foo:
# do something

The ``do something`` block will execute every time because
``options.not_foo`` is either ``False`` or ``None``, both of which
indicate that the if-statement should not be executed.

In argparse, the code above works fine because the default is ``True``
instead of ``None``.

Alternatively, could the actions become keyword arguments (with the
action argument only preserved for backwards compatibility)? E.g.

parser.add_option("-v", store_true_into="verbose")
parser.add_option("-q", store_false_into="verbose")
(same for store, append, count, callback)

It looks like you're trying to merge the ``dest=`` and ``action=``
functionality, but I'm not sure I understand where this is going...

>* the choices argument is now checked *after* arguments have been
type-converted. This is intentional, so that you can specify, say
xrange(100) instead of ["0", "1", "2", "3", ... "99"]. There is also
no "choices" type anymore since any action can also specify their
choices.

As an incompatible change, this could be warned-about if the type is not
string, yet the first choice is.

Alternatively, we could require users that pass converted values to
specify converted_choices=True in 2.6, and warn all users that don't
pass a converted_choices flag, and then flip the default in 2.7 and 3.0.
Then current users had a quick fix to silence the warning, and new
users could drop the extra flag in the future.

That seems reasonable.

> I could probably add callback actions by creating an appropriate
Action subclass and registering it. However, any code relying on
parser.{largs,rargs,values} would break because the parsing algorithm
is completely different in argparse.

If you can find a way to make callbacks work in the "common case",
this flag could be deprecated and removed.

Sounds good.

>I guess I don't know how to proceed from here. I'm reluctant to start
adding the code necessary to support even the easily solved issues when
the issues that I don't know how to solve seem like they could be deal
breakers.

This asks for a PEP. The views above are mine only, others may feel
differently.

Ok. I'll try to write a PEP up this week.

Thanks again for all your useful feedback!

STeVe