Passing options around your program

[Leo Breebaart]

I have another question where I am not so much looking for a
solution but rather hoping to get some feedback on *which*
solutions people here consider good Pythonic ways to approach a
issue.

The situation is this: I am writing fairly large console scripts
in Python. They have quite a few command-line options, which lead
to configuration variables that are needed all over the program
(e.g. the "--verbose" option alone is used by just about every
function and method).

I am using optparse to handle the actual option parsing, so I
have a main function that contains:

opts,args = parser.parse_args()

My question in a nutshell is: what is the best way to make the
parsed options in 'opts' easily available to the rest of my
program?

- Having 'opts' trickle down by passing it as an explicit
parameter is the most flexible option. It allows me to easily
separate the options from the command-line parsing -- if I want
to call my script interactively or from a GUI instead, I can just
craft my own 'opts' object and pass that.

But on the other hand, I feel that the proliferation of 'opts'
parameters gets ugly, and does not do wonders for the readability
of the program.

- So especially for shorter scripts, a solution I've used is to
define a global config class, and transfer the options to it:

class Config:
pass

With in main():

for opt in vars(opts):
setattr(Config, opt, getattr(opts, opt))

or:

Config.cmd_opts = opts

Now every method that needs it can just call Config.verbose or
Config.cmd_opts.verbose or whatever.

A variant I have also used occasionally is to derive Config from
a Martelli Borg parent class with shared-state instances, so that
every function or class that needs access to the config
parameters can just instantiate its own 'local' Config() object.
I find this works nicely if the program is spread out over more
than one file.

- A third option I have seen mentioned (possibly even here on
c.l.p.), but have not actually tried yet, is that if you decide
to have something 'global' anyway, why not just use a module
instead of a class to store the options, so that every file can
do "import config" and then say "config.verbose" etc.
So, does anybody have any particular positive or negative
opinions on any of these approaches, and/or are there perhaps
possibilities I have overlooked?

Many thanks in advance for your feedback.

--
Leo Breebaart <le*@lspace.org>

[jeffshannon]

When I have done this, I've used the last option -- creating a "config"
module which can be imported wherever the configuration info is needed.
(This module also handles any config files and/or Windows registry
keys needed.) Using a Borg is a nice idea... but that ConfigBorg is
probably going to be in its own module, which will need to be imported
anyhow... so why not let sys.__import__() do your singleton-enforcement
for you?

Of course, I hesitate to claim anything that I do as being necessarily
related to "best practices", but what would Usenet be without swarms of
half-informed opinions? ;)

Jeff Shannon

[alisonken1]

Leo Breebaart wrote:

I have another question where I am not so much looking for a
solution but rather hoping to get some feedback on *which*
solutions people here consider good Pythonic ways to approach a
issue.

The situation is this: I am writing fairly large console scripts
in Python. They have quite a few command-line options, which lead
to configuration variables that are needed all over the program
(e.g. the "--verbose" option alone is used by just about every
function and method).

<SNIP>

One question I have is about the "--verbose" option.

If you're doing something that is equivalent to logging to <file |
console>, rather than continuing to pass the '--verbose' flag around,
why not just use the built-in logging facility to manage the extra
output?

By having the different modules call logging( <level>, "<message">)
throughout your programs, you only need to set the initial level from
where you're checking options (whether from command line or
configuration file) rather than having to pass the '--verbose' option
around.

[Leo Breebaart]

"alisonken1" <al********@gmail.com> writes:

Leo Breebaart wrote:

I am writing fairly large console scripts in Python. They
have quite a few command-line options, which lead to
configuration variables that are needed all over the program
(e.g. the "--verbose" option alone is used by just about
every function and method).

<SNIP>

One question I have is about the "--verbose" option.

If you're doing something that is equivalent to logging to
<file | console>, rather than continuing to pass the
'--verbose' flag around, why not just use the built-in logging
facility to manage the extra output?

Good point. It's not a real conscious decision, and I have one or
two other projects lying around where I do in fact use logging. I
think the main reason why I am not using it by default is
because, when all is said and done, it still comes easier to me
to resort to guarded print statements then to set up and use the
logging machinery.

This may well be a false economy in the long run, but it is
nevertheless how I perceive it in day-to-day programming.

(Related to this is that while I found e.g. the optparse
documentation very clear and relevant to my needs, the logging
documentation and examples, while readable, just never seem to
relate to what I actually need. This is not a complaint -- just
a subjective observation.)

But you are right. Especially for the larger projects I really
should bite the bullet and start using the logging module.

--
Leo Breebaart <le*@lspace.org>

[Sybren Stuvel]

Leo Breebaart enlightened us with:

I think the main reason why I am not using it by default is because,
when all is said and done, it still comes easier to me to resort to
guarded print statements then to set up and use the logging
machinery.

The logging "machinery" isn't that huge nor is it difficult to set up:

==================================================
import logging

logging.basicConfig()
log = logging.getLogger('your.module.name')
==================================================

and that's all there is to it.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa

[Leo Breebaart]

Sybren Stuvel <sy*******@YOURthirdtower.com.imagination> writes:

Leo Breebaart enlightened us with:

I think the main reason why I am not using [logging] by
default is because, when all is said and done, it still comes
easier to me to resort to guarded print statements then to
set up and use the logging machinery.

The logging "machinery" isn't that huge nor is it difficult to set up:

==================================================
import logging

logging.basicConfig()
log = logging.getLogger('your.module.name')
==================================================

and that's all there is to it.

I did say: "it comes easier *to me*" -- I am aware that my own
tastes, which may not be shared by others, or be entirely
logical, feature into this.

Having said that, I do think it's also not *quite* as trivial as
you claim. In order to duplicate what I already have ("if config.verbose:
print msg"), I probably need to log at the DEBUG level, and I
need to change the format so that that output looks like the
'print' output (no timestamping cruft etc.)

Okay, you say, that's still easy. It's just:

logging.basicConfig(level=logging.DEBUG,
format='%(message)s')

logging.debug('A debug message')

And that's true, but I did have to go and look this up, and I
made a couple of false starts first. I can only repeat that the
workings of the logging framework are rarely intuitive to me.

Also, assume that I have set it up as above. Now I want certain
other print statements to go to sys.stderr alone. If I understand
the docs correctly (possibly I don't), the way to do this is to
start attaching explicit StreamHandlers and whatnot. Whereas with
print, I can say "print >>sys.stderr, msg".

I know that it won't take much for a script to become so complex
that the extra stuff logging buys you vastly overpowers the
convenience of the print statements. I have just not needed that
often enough for it to become second nature to use in smaller
scripts as well.

--
Leo Breebaart <le*@lspace.org>

[Sybren Stuvel]

Leo Breebaart enlightened us with:

Okay, you say, that's still easy. It's just:

logging.basicConfig(level=logging.DEBUG,
format='%(message)s')
I always use a separate logger, as per my example. That would then
just require an additional line:

log.setLeveL(logging.DEBUG)
And that's true, but I did have to go and look this up, and I made a
couple of false starts first. I can only repeat that the workings of
the logging framework are rarely intuitive to me.
If you're just writing software for yourself, that's fine. If you want
to share your software, things that are intuitive to you might not be
to others. In such a case (and usually software is going to be shared
at some point or another) it's generally a good idea to spend a few
minutes learning how to use such a system, because then other people
will see your code using things they are already familiar with.

Reusing other people's code instead of writing your own logging, also
means that other people will fix bugs for you.
I know that it won't take much for a script to become so complex
that the extra stuff logging buys you vastly overpowers the
convenience of the print statements. I have just not needed that
often enough for it to become second nature to use in smaller
scripts as well.

I don't use the logging module in small scripts either, but as soon as
it gets over 100 lines, I do. It's quite easy to get it going, once
you get the hang of it.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa

[alisonken1]

Leo Breebaart wrote:

<SNIP>

Also, assume that I have set it up as above. Now I want certain
other print statements to go to sys.stderr alone. If I understand
the docs correctly (possibly I don't), the way to do this is to
start attaching explicit StreamHandlers and whatnot. Whereas with
print, I can say "print >>sys.stderr, msg".

<SNIP>

Something else to consider, every time your script calls "print

sys.stderr, msg", it is making another file open request.

I've run across situations where this format will actually cause a
system to return "too many open files" error.

As was pointed out earlier, it's much easier to learn how to use the
logging facility and create a default stdout logger as well as a
secondary stderr logger that only maintains one file handle than to try
to find out where you're getting some errors that are not script errors
but system limit errors.

The other nice aspect of using the logging facility is the availability
of changing the logger to save output to a file or stdout/stderr than
to try and go through a more-than-one-file program.

It's easy to learn how to redirect stdout/stderr within a script, but
it's usually more flexible to use the logging facility that's already
been included. Before the logging module was included in the library, I
usually ended up writing my own logging module to do the same thing.
Especially since the logger allows you to specify levels.

For example, I typically have my logging facility setup as follows:

10 - Very basic logging - typically main routine changes
20 - Log imports
30 - Log class instantiation
40 - Log module calls
50 - Log function calls

The other aspect of using the logging facility, you can also define
your own in-between log levels:

51 - Entering function
52 - Exiting function
53 - Function sub-routines
60 - Everyhing under the sun

As part of the configParser options:
-v : Increment logging level by 1 level for every -v option on command
line
-loglevel=<level> : Specify log level
-logfile=<filename> : File to save stdout messages
-errfile=<filename> : File to save stderr messages

Once you get used to the logging module, it's hard to go back to using
file redirects and the <sometimes> system limits troubleshooting in
larger programs.