selective logger disable/enable

[Gary Jefferson]

Suppose I have 3 modules that belong to a project, 'A', 'A.a' and 'B',
and each module has its own logger, created with:

module1logger = logging.getLogger('project.A')

and

module2logger = logging.getLogger('project.A.a')

and

module3logger = logging.getLogger('project.B')
And I want to selectively enable/disable these, per module (for
example, I only want logging from A and A.a, or just from B, etc). It
seems like logging.Filter is supposed to let me do this, but I can't
see how to apply it globally. Is there really no other way that to add
a addFilter(filter) call to each of these loggers individually?

logging.basicConfig gets inherited by all loggers, but it doesn't seem
capable of giving a Filter to all loggers. Is there any way to do this?

[Vinay Sajip]

Gary Jefferson wrote:

Suppose I have 3 modules that belong to a project, 'A', 'A.a' and 'B',
and each module has its own logger, created with:

module1logger = logging.getLogger('project.A')

and

module2logger = logging.getLogger('project.A.a')

and

module3logger = logging.getLogger('project.B')
And I want to selectively enable/disable these, per module (for
example, I only want logging from A and A.a, or just from B, etc). It
seems like logging.Filter is supposed to let me do this, but I can't
see how to apply it globally. Is there really no other way that to add
a addFilter(filter) call to each of these loggers individually?

The documentation for Logger - see

http://docs.python.org/lib/node406.html

- shows that there are addFilter() and removeFilter() methods on the
Logger class which you can use to add or remove filters from individual
Logger instances. From the above page (entitled "14.5.1 Logger
Objects"):

addFilter(filt)
Adds the specified filter filt to this logger.

removeFilter(filt)
Removes the specified filter filt from this logger.

The parent section of Section 14.5.1, which is Section 14.5, was the
first search result when I just searched Google for "python logging".

Best regards,

Vinay Sajip

[Gary Jefferson]

Vinay Sajip wrote:

>
The documentation for Logger - see

http://docs.python.org/lib/node406.html

- shows that there are addFilter() and removeFilter() methods on the
Logger class which you can use to add or remove filters from individual
Logger instances. From the above page (entitled "14.5.1 Logger
Objects"):

addFilter(filt)
Adds the specified filter filt to this logger.

removeFilter(filt)
Removes the specified filter filt from this logger.

The parent section of Section 14.5.1, which is Section 14.5, was the
first search result when I just searched Google for "python logging".

Thanks for the reply Vinay. I had been reading those docs prior to
posting, but addFilter/removeFilter only work on individual logger
instances and handlers. I want to globally enable/disable multiple
logger instances, based on their namespaces (see the example I
provided). I cannot find a way to do this. I can certainly call
addFilter on every instance of every logger throughout the source code,
but this is more than inconvenient -- it breaks when new modules are
added with their own loggers in their own namespaces (until fixed by
manually adding addFilter() to those instances).

e.g., suppose you have a source tree with a several components that do
network access, and several components that do other things like saving
state to disk, accessing the filesystem, etc. And suppose you find a
bug that you think is in the networking code. It would be nice to be
able to GLOBALLY enable just the loggers that belong in the networking
namespace, and disable all others (similar to how Filters are supposed
to work for individual loggers). And then to be able to do this with
any component, by providing, for example, a command line switch or
environment variable. Otherwise, the poor programmer is forced to go
and edit every module in the source tree to selectively turn on/off
their respecitve loggers. Or am I missing something really obvious
about how this is done with the logging module?

thanks,
Gary

[Vinay Sajip]

Gary Jefferson wrote:

Thanks for the reply Vinay. I had been reading those docs prior to
posting, but addFilter/removeFilter only work on individual logger
instances and handlers. I want to globally enable/disable multiple
logger instances, based on their namespaces (see the example I
provided). I cannot find a way to do this. I can certainly call
addFilter on every instance of every logger throughout the source code,
but this is more than inconvenient -- it breaks when new modules are
added with their own loggers in their own namespaces (until fixed by
manually adding addFilter() to those instances).

e.g., suppose you have a source tree with a several components that do
network access, and several components that do other things like saving
state to disk, accessing the filesystem, etc. And suppose you find a
bug that you think is in the networking code. It would be nice to be
able to GLOBALLY enable just the loggers that belong in the networking
namespace, and disable all others (similar to how Filters are supposed
to work for individual loggers). And then to be able to do this with
any component, by providing, for example, a command line switch or
environment variable. Otherwise, the poor programmer is forced to go
and edit every module in the source tree to selectively turn on/off
their respecitve loggers. Or am I missing something really obvious
about how this is done with the logging module?

I don't know enough about your target environment and application to
necessarily give you the best advice, but I'll make some general
comments which I hope are useful. You seem to be thinking that loggers
are binary - i.e. you turn them on or off. But they can be controlled
more finely than that; you should be able to get the effect that you
want by using the different logging levels available judiciously, as
well as using the fact that loggers inhabit a named hierarchy. Note
that loggers, by default, inherit the level of the first ancestor
logger which has an explicitly set level (by ancestor, I mean in the
name hierarchy). The root logger's default level is WARNING, so by
default all loggers will work at this level. [N.B. You can also set
levels for individual handlers, e.g. to ensure that only CRITICAL
conditions are emailed to a specified email address using SMTPHandler,
or that ERROR conditions and above are written to file but not to
console.]

So, for your networking scenario, let's suppose you do the following:
Have all network loggers live in the hierarchy namespace below
"network" (e.g. "network", "network.tcp", "network.http" etc.). By
default, all of these will only log events of severity WARNING and
above. Also, suppose you log events in your application code, which are
sometimes but not always of interest, at level DEBUG or level INFO.
Then, these events will never show up in the logging output, since they
are below WARNING in severity. Subsequently, if you want to turn on
logging verbosity for the network code only, you can arrange, via a
command-line switch or environment variable or configuration file, to
do

logging.getLogger("network").setLevel(logging.DEBU G)

whereupon you will start seeing events from the networking code at
severity DEBUG and INFO. This will affect all loggers in the "network"
hierarchy whose levels you have not explicitly set (so that they will
get the effective level of the first ancestor which has a level
explicitly set - the logger named "network").

If all you are interested in is turning on verbosity based on different
event severities (levels), you should not need to use or set Filters.
Filters are for use only when levels don't meet your use case
requirements.

Best regards,

Vinay Sajip

[Gary Jefferson]

Vinay Sajip wrote:

>
I don't know enough about your target environment and application to
necessarily give you the best advice, but I'll make some general
comments which I hope are useful. You seem to be thinking that loggers
are binary - i.e. you turn them on or off. But they can be controlled
more finely than that; you should be able to get the effect that you
want by using the different logging levels available judiciously, as
well as using the fact that loggers inhabit a named hierarchy. Note
that loggers, by default, inherit the level of the first ancestor
logger which has an explicitly set level (by ancestor, I mean in the
name hierarchy). The root logger's default level is WARNING, so by
default all loggers will work at this level. [N.B. You can also set
levels for individual handlers, e.g. to ensure that only CRITICAL
conditions are emailed to a specified email address using SMTPHandler,
or that ERROR conditions and above are written to file but not to
console.]

So, for your networking scenario, let's suppose you do the following:
Have all network loggers live in the hierarchy namespace below
"network" (e.g. "network", "network.tcp", "network.http" etc.). By
default, all of these will only log events of severity WARNING and
above. Also, suppose you log events in your application code, which are
sometimes but not always of interest, at level DEBUG or level INFO.
Then, these events will never show up in the logging output, since they
are below WARNING in severity. Subsequently, if you want to turn on
logging verbosity for the network code only, you can arrange, via a
command-line switch or environment variable or configuration file, to
do

logging.getLogger("network").setLevel(logging.DEBU G)

whereupon you will start seeing events from the networking code at
severity DEBUG and INFO. This will affect all loggers in the "network"
hierarchy whose levels you have not explicitly set (so that they will
get the effective level of the first ancestor which has a level
explicitly set - the logger named "network").

If all you are interested in is turning on verbosity based on different
event severities (levels), you should not need to use or set Filters.
Filters are for use only when levels don't meet your use case
requirements.

Best regards,

Vinay Sajip

Vinay, okay, I think what you described will work out for me -- thank
you very much for the explanation.

I am still a bit confused about Filters, though. It seems they are a
bit of an anomoly in the hierarchical view of loggers that the API
supports elsewhere, i.e., filters don't seem to inherit... Or am I
missing something again? Here's a quick example:

import logging

log1 = logging.getLogger("top")
log2 = logging.getLogger("top.network")
log3 = logging.getLogger("top.network.tcp")
log4 = logging.getLogger("top.network.http")
log5 = logging.getLogger("top.config")
log6 = logging.getLogger("top.config.file")

logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s')

filter = logging.Filter("top.network")
log1.addFilter(filter) # only affects log1, do this for each of log2-7
too?

log1.debug("I'm top")
log2.debug("I'm top.network")
log3.debug("I'm top.network.tcp")
log4.debug("I'm top.network.http")
log5.debug("I'm top.config")
log6.debug("I'm top.config.file")
This is only for the binary case (and I think if I ignore the binary
case and filters altogether as you suggested), but it really would be
nice to be able to squelch /all/ output from loggers that belong to
certain parts of the namespace by using a filter as above (which I
can't get to work).

Perhaps if I set up a basicConfig with a loglevel of nothing, I can get
this to approximate squelching of everything but that which I
explicitly setLevel (which does inherit properly).

In other words, the addFilter/removeFilter part of the API seems rather
impotent if it can't be inherited in the logging namespaces. In fact,
I can't really figure out a use case where I could possibly want to use
it without it inheriting. Obviously I'm missing something. I'm sure
I've consumed more attention that I deserve already in this thread,
but, do you have any pointers which can enlighten me as to how to
effectively use addFilter/removeFilter?

many thanks,
Gary

[Vinay Sajip]

Gary Jefferson wrote:

I am still a bit confused about Filters, though. It seems they are a
bit of an anomoly in the hierarchical view of loggers that the API
supports elsewhere, i.e., filters don't seem to inherit... Or am I
missing something again? Here's a quick example:

import logging

log1 = logging.getLogger("top")
log2 = logging.getLogger("top.network")
log3 = logging.getLogger("top.network.tcp")
log4 = logging.getLogger("top.network.http")
log5 = logging.getLogger("top.config")
log6 = logging.getLogger("top.config.file")

logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s')

filter = logging.Filter("top.network")
log1.addFilter(filter) # only affects log1, do this for each of log2-7
too?

log1.debug("I'm top")
log2.debug("I'm top.network")
log3.debug("I'm top.network.tcp")
log4.debug("I'm top.network.http")
log5.debug("I'm top.config")
log6.debug("I'm top.config.file")
This is only for the binary case (and I think if I ignore the binary
case and filters altogether as you suggested), but it really would be
nice to be able to squelch /all/ output from loggers that belong to
certain parts of the namespace by using a filter as above (which I
can't get to work).

Perhaps if I set up a basicConfig with a loglevel of nothing, I can get
this to approximate squelching of everything but that which I
explicitly setLevel (which does inherit properly).

In other words, the addFilter/removeFilter part of the API seems rather
impotent if it can't be inherited in the logging namespaces. In fact,
I can't really figure out a use case where I could possibly want to use
it without it inheriting. Obviously I'm missing something. I'm sure
I've consumed more attention that I deserve already in this thread,
but, do you have any pointers which can enlighten me as to how to
effectively use addFilter/removeFilter?

I don't really think there's a problem with the logging API - it went
through a fair amount of peer review on python-dev before making it
into the Python distribution. You need to use what's there rather than
shoehorn it into how you think it ought to be. Filters are for more
esoteric requirements - you can see some examples of filters in the old
(out of date) standalone distribution at
http://www.red-dove.com/python_logging.html (download and examine some
of the test scripts). Loggers aren't binary - levels are there to be
used. Most people get by with judicious use of levels, using Filters on
loggers only for unusual cases. Since Filters are only meant to be used
in unusual situations, there is no need to think about inheriting them.

Filters can be set on Handlers so that even if Loggers log the events,
they don't go to any output if filtered out at the handlers for that
output.

BTW I would also advise reading PEP-282 to understand more about the
logging approach.

Best regards,

Vinay Sajip

[Gary Jefferson]

Vinay Sajip wrote:

>
BTW I would also advise reading PEP-282 to understand more about the
logging approach.

You've been most helpful, Vinay. The PEP section on Filters states
that I can do what I've been trying to do with filters, but doesn't
provide enough information to do it (or, at least, I'm too thick to
guess at how it would work by looking at the API and PEP and trying a
dozen different ways). Luckily, the examples you point to from your
original package do provide enough info; log_test15.py held the key.

I still feel like it would be more intuitive if filters were inherited
down the hierarchy instead of having to go through the extra steps of
getting at the root handler first, but I'm sure there are good reasons
for not doing this.

One more question, is there any way to get the list of all named
loggers (from the Manager, perhaps)? Or... maybe I don't need this,
either, as MatchFilter (log_test18.py) seems to do what I was thinking
I need the list of logger names for... most excellent.

Thanks,
Gary

BTW, the python logging module is one of the best readily available
loggers I've come across in any language.

[Peter Otten]

Gary Jefferson wrote:

Suppose I have 3 modules that belong to a project, 'A', 'A.a' and 'B',
and each module has its own logger, created with:

module1logger = logging.getLogger('project.A')

and

module2logger = logging.getLogger('project.A.a')

and

module3logger = logging.getLogger('project.B')
And I want to selectively enable/disable these, per module (for
example, I only want logging from A and A.a, or just from B, etc). It
seems like logging.Filter is supposed to let me do this, but I can't
see how to apply it globally. Is there really no other way that to add
a addFilter(filter) call to each of these loggers individually?

logging.basicConfig gets inherited by all loggers, but it doesn't seem
capable of giving a Filter to all loggers. Is there any way to do this?

An alternative approach might be to set the 'propagate' flag:

import logging

def warn_all(loggers, message):
for lgr in loggers:
lgr.warn(message)

logging.basicConfig()
root = logging.getLogger()
root.manager.emittedNoHandlerWarning = True
loggers = [logging.getLogger(name) for name in """
alpha
alpha.1
alpha.2
alpha.2.1
alpha.2.2
alpha.2.2.1
beta
beta.1
beta.2
""".split()]

warn_all(loggers, "all on")

print "---"
logging.getLogger("alpha").propagate = False
warn_all(loggers, "alpha off")
logging.getLogger("alpha").propagate = True

print "---"
logging.getLogger("alpha.2").propagate = False
warn_all(loggers, "alpha.2 off")

Peter

[Vinay Sajip]

Gary Jefferson wrote:

Vinay Sajip wrote:

BTW I would also advise reading PEP-282 to understand more about the
logging approach.

You've been most helpful, Vinay. The PEP section on Filters states
that I can do what I've been trying to do with filters, but doesn't
provide enough information to do it (or, at least, I'm too thick to
guess at how it would work by looking at the API and PEP and trying a
dozen different ways). Luckily, the examples you point to from your
original package do provide enough info; log_test15.py held the key.

I still feel like it would be more intuitive if filters were inherited
down the hierarchy instead of having to go through the extra steps of
getting at the root handler first, but I'm sure there are good reasons
for not doing this.

One more question, is there any way to get the list of all named
loggers (from the Manager, perhaps)? Or... maybe I don't need this,
either, as MatchFilter (log_test18.py) seems to do what I was thinking
I need the list of logger names for... most excellent.

Thanks,
Gary

BTW, the python logging module is one of the best readily available
loggers I've come across in any language.

Thanks. Glad the tests/examples (log_testxx.py) helped. When I get a
chance, I will try to work some of them into the docs...

Best regards,
Vinay Sajip

[Gary Jefferson]

So maybe I don't have all this figured out quite as well as I thought.
What I really want to do is set an environment variable, MYDEBUG, which
contains a list of wildcarded logger names, such as "a.*.c a.d" (which
becomes ['a.*.c', 'a.d'], and then selectively crank the loglevel up to
DEBUG for those that match.

In order to do that, I think I need something kind of like filters, but
for logger name... I'm not seeing how to do this, even after playing
with variations of test15, 18, and 21.

What I do have working at the moment is passing a list of non-wildcard
logger names, i.e., doing the wildcard expansion manually such as
"['a.b.c', 'a.c.c', 'a.f.c', 'a.d']. Is there anyway to automate this
dynamically?

BTW, I do understand that 'a.d' is essentially equivalent to 'a.d.*',
but I'm dealing with a hierarchy that doesn't always tidy up like that.
For example, I have top.network.server.http, top.network.server.smtp,
top.network.client.http, and top.network.client.smtp. Obviously, if I
only want server or client DEBUG msgs, this is easy. And sometimes
that's exactly what I want. Other times, I want only smtp DEBUG msgs,
and the hierarchy won't help me get that (unless I break it for just
getting client or server msgs), etc. So I would really like to figure
out how to do 'a.*.c'.

Any ideas?

Thanks again,
Gary
On Jan 23, 3:01 am, "Vinay Sajip" <[e-mail]:

Glad the tests/examples (log_testxx.py) helped. When I get a
chance, I will try to work some of them into the docs...

Best regards,

Vinay Sajip

[Gary Jefferson]

Gary Jefferson wrote:

So maybe I don't have all this figured out quite as well as I thought.
What I really want to do is set an environment variable, MYDEBUG, which
contains a list of wildcarded logger names, such as "a.*.c a.d" (which
becomes ['a.*.c', 'a.d'], and then selectively crank the loglevel up to
DEBUG for those that match.

In order to do that, I think I need something kind of like filters, but
for logger name... I'm not seeing how to do this, even after playing
with variations of test15, 18, and 21.

What I do have working at the moment is passing a list of non-wildcard
logger names, i.e., doing the wildcard expansion manually such as
"['a.b.c', 'a.c.c', 'a.f.c', 'a.d']. Is there anyway to automate this
dynamically?

BTW, I do understand that 'a.d' is essentially equivalent to 'a.d.*',
but I'm dealing with a hierarchy that doesn't always tidy up like that.
For example, I have top.network.server.http, top.network.server.smtp,
top.network.client.http, and top.network.client.smtp. Obviously, if I
only want server or client DEBUG msgs, this is easy. And sometimes
that's exactly what I want. Other times, I want only smtp DEBUG msgs,
and the hierarchy won't help me get that (unless I break it for just
getting client or server msgs), etc. So I would really like to figure
out how to do 'a.*.c'.

Any ideas?

Thanks again,
Gary

Okay, I think I'm back on track again: I can do regex comparisons
against the name field of the LogRecord instance handed to me by
filter().

Gary