By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
429,087 Members | 1,513 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 429,087 IT Pros & Developers. It's quick & easy.

namespaces module (a.k.a. bunch, struct, generic object, etc.) PEP

P: n/a
In the "empty classes as c structs?" thread, we've been talking in some
detail about my proposed "generic objects" PEP. Based on a number of
suggestions, I'm thinking more and more that instead of a single
collections type, I should be proposing a new "namespaces" module
instead. Some of my reasons:

(1) Namespace is feeling less and less like a collection to me. Even
though it's still intended as a data-only structure, the use cases seem
pretty distinct from other collections.

(2) I think there are a couple different types that should be provided,
not just one. For example:

* The suggested namespace view (which allows, for example, globals() to
be manipulated with the getattr protocol) should probably be its own
class, NamespaceView, since the behavior of a NamespaceView is
substantially different from that of a Namespace. (Modifying a
NamespaceView modifies the dict, while modifying a Namespace doesn't.)
This also allows NamespaceView to have a separate __repr__ to indicate
some of the differences.

* The suggested namespace chain, if included, should also probably be
its own class, NamespaceChain. As defined, it doesn't depend on
Namespace at all -- it could be used just as well for other
non-Namespace objects...
I've updated the PEP to reflect these changes. Comments and suggestions
greatly appreciated! (Note that I've included the current
implementation of the module at the end of the PEP.)

----------------------------------------------------------------------
PEP: XXX
Title: Attribute-Value Mapping Data Type
Version:
Last-Modified:
Author: Steven Bethard <st************@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 10-Feb-2005
Python-Version: 2.5
Post-History: 10-Feb-2005
Abstract
========

This PEP proposes a standard library addition to support the simple
creation of attribute-value mappping objects which can be given named
attributes without the need to declare a class. Such attribute-value
mappings are intended to complement the name-value mappings provided
by Python's builtin dict objects.
Motivation
==========

Python's dict objects provide a simple way of creating anonymous
name-value mappings. These mappings use the __getitem__ protocol to
access the value associated with a name, so that code generally
appears like::

mapping['name']

Occasionally, a programmer may decide that dotted-attribute style
access is more appropriate to the domain than __getitem__ style
access, and that their mapping should be accessed like::

mapping.name

Currently, if a Python programmer makes this design decision, they
are forced to declare a new class, and then build instances of this
class. When no methods are to be associated with the attribute-value
mappings, declaring a new class can be overkill. This PEP proposes
adding a new module to the standard library to provide a few simple
types that can be used to build such attribute-value mappings.

Providing such types allows the Python programmer to determine which
type of mapping is most appropriate to their domain and apply this
choice with minimal effort. Some of the suggested uses include:
Returning Named Results
-----------------------

It is often appropriate for a function that returns multiple items to
give names to the different items returned. The types suggested in
this PEP provide a simple means of doing this that allows the
returned values to be accessed in the usual attribute-style access::
def f(x): ... return Namespace(double=2*x, squared=x**2)
... y = f(10)
y.double 20 y.squared 100
Representing Hierarchical Data
------------------------------

The types suggested in this PEP also allow a simple means of
representing hierarchical data that allows attribute-style access::
x = Namespace(spam=Namespace(rabbit=1, badger=[2]), ham='3')
x.spam.badger [2] x.ham '3'
Manipulating Dicts through Attributes
-------------------------------------

Sometimes it is desirable to access the items of an existing dict
object using dotted-attribute style access instead of __getitem__
style access. The types suggested in this PEP provide a simple means
of doing so::
d = {'name':'Pervis', 'lumberjack':True}
ns = NamespaceView(d)
ns NamespaceView(lumberjack=True, name='Pervis') ns.rugged = False
del ns.lumberjack
d {'rugged': False, 'name': 'Pervis'}
Rationale
=========

As Namespace objects are intended primarily to replace simple,
data-only classes, simple Namespace construction was a primary
concern. As such, the Namespace constructor supports creation from
keyword arguments, dicts, and sequences of (attribute, value) pairs::
Namespace(eggs=1, spam=2, ham=3)

Namespace(eggs=1, ham=3, spam=2) Namespace({'eggs':1, 'spam':2, 'ham':3}) Namespace(eggs=1, ham=3, spam=2) Namespace([('eggs',1), ('spam',2), ('ham',3)]) Namespace(eggs=1, ham=3, spam=2)

To allow attribute-value mappings to be easily combined, the
Namespace type provides a update staticmethod that supports similar
arguments::
ns = Namespace(eggs=1)
Namespace.update(ns, [('spam', 2)], ham=3)
ns Namespace(eggs=1, ham=3, spam=2)

Note that update should be used through the class, not through the
instances, to avoid the confusion that might arise if an 'update'
attribute added to a Namespace instance hid the update method.
Namespace objects support also object equality, comparing Namespace
objects by attributes recursively::
x = Namespace(parrot=Namespace(lumberjack=True, spam=42), peng='shrub') y = Namespace(peng='shrub', parrot=Namespace(spam=42, lumberjack=True)) z = Namespace(parrot=Namespace(lumberjack=True), peng='shrub')
x == y True x == z False
NamespaceView objects are intended to allow manipulating dict objects
through the getattr protocol instead of the getitem protocol. For
example usage, see "Viewing Dict Items as Attributes" above.
Note that support for the various mapping methods, e.g.
__(get|set|del)item__, __len__, __iter__, __contains__, items, keys,
values, etc. was intentionally omitted as these methods did not seem
to be necessary for the core uses of an attribute-value mapping. If
such methods are truly necessary for a given use case, this may
suggest that a dict object is a more appropriate type for that use.
Examples
=========

Converting an XML DOM tree into a tree of nested Namespace objects::
import xml.dom.minidom
def gettree(element): ... result = Namespace()
... if element.attributes:
... Namespace.update(result, element.attributes.items())
... children = {}
... for child in element.childNodes:
... if child.nodeType == xml.dom.minidom.Node.TEXT_NODE:
... children.setdefault('text', []).append(
... child.nodeValue)
... else:
... children.setdefault(child.nodeName, []).append(
... gettree(child))
... Namespace.update(result, children)
... return result
... doc = xml.dom.minidom.parseString("""\ ... <xml>
... <a attr_a="1">
... a text 1
... <b attr_b="2" />
... <b attr_b="3"> b text </b>
... a text 2
... </a>
... <c attr_c="4"> c text </c>
... </xml>""") tree = gettree(doc.documentElement)
tree.a[0].b[1]

Namespace(attr_b=u'3', text=[u' b text '])

Reference Implementation
========================

::
class Namespace(object):
"""Namespace([namespace|dict|seq], **kwargs) -> Namespace object

The new Namespace object's attributes are initialized from (if
provided) either another Namespace object's attributes, a
dictionary, or a sequence of (name, value) pairs, then from the
name=value pairs in the keyword argument list.
"""

def __init__(*args, **kwargs):
"""Initializes a Namespace instance."""
# inheritance-friendly update call
type(args[0]).update(*args, **kwargs)

def __eq__(self, other):
"""x.__eq__(y) <==> x == y

Two Namespace objects are considered equal if they have the
same attributes and the same values for each of those
attributes.
"""
return (other.__class__ == self.__class__ and
self.__dict__ == other.__dict__)

def __repr__(self):
"""x.__repr__() <==> repr(x)

If all attribute values in this namespace (and any nested
namespaces) are reproducable with eval(repr(x)), then the
Namespace object is also reproducable for eval(repr(x)).
"""
return '%s(%s)' % (
type(self).__name__,
', '.join('%s=%r' % (k, v) for k, v in sorted(
self.__dict__.iteritems())))

def update(*args, **kwargs):
"""Namespace.update(ns, [ns|dict|seq,] **kwargs) -> None

Updates the first Namespace object's attributes from (if
provided) either another Namespace object's attributes, a
dictionary, or a sequence of (name, value) pairs, then from
the name=value pairs in the keyword argument list.
"""
if not 1 <= len(args) <= 2:
raise TypeError('expected 1 or 2 arguments, got %i' %
len(args))
self = args[0]
if not isinstance(self, Namespace):
raise TypeError('first argument to update should be '
'Namespace, not %s' %
type(self).__name__)
if len(args) == 2:
other = args[1]
if isinstance(other, Namespace):
other = other.__dict__
try:
self.__dict__.update(other)
except (TypeError, ValueError):
raise TypeError('cannot update Namespace with %s' %
type(other).__name__)
self.__dict__.update(kwargs)
class NamespaceView(Namespace):
"""NamespaceView(dict) -> new Namespace view of the dict

Creates a Namespace that is a view of the original dictionary,
that is, changes to the Namespace object will be reflected in
the dictionary, and vice versa.
"""
def __init__(self, d):
self.__dict__ = d
Open Issues
===========
What should the types be named? Some suggestions include 'Bunch',
'Record', 'Struct' and 'Namespace'.

Where should the types be placed? The current suggestion is a new
"namespaces" module.

Should namespace chaining be supported? One suggestion would add a
NamespaceChain object to the module::

class NamespaceChain(object):
"""NamespaceChain(*objects) -> new attribute lookup chain

The new NamespaceChain object's attributes are defined by the
attributes of the provided objects. When an attribute is
requested, the sequence is searched sequentially for an object
with such an attribute. The first such attribute found is
returned, or an AttributeError is raised if none is found.

Note that this chaining is only provided for getattr and delattr
operations -- setattr operations must be applied explicitly to
the appropriate objects.

The list of objects is stored in a NamespaceChain object's
__namespaces__ attribute.
"""
def __init__(self, *objects):
"""Initialize the NamespaceChain object"""
self.__namespaces__ = objects

def __getattr__(self, name):
"""Return the first such attribute found in the object list
"""
for obj in self.__namespaces__:
try:
return getattr(obj, name)
except AttributeError:
pass
raise AttributeError(name)

def __delattr__(self, name):
"""Delete the first such attribute found in the object list
"""
for obj in self.__namespaces__:
try:
return delattr(obj, name)
except AttributeError:
pass
raise AttributeError(name)
References
==========


...
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
Jul 18 '05 #1
Share this Question
Share on Google+
18 Replies


P: n/a
On Thu, 10 Feb 2005 11:56:45 -0700, Steven Bethard wrote:
In the "empty classes as c structs?" thread, we've been talking in some
detail about my proposed "generic objects" PEP. Based on a number of
suggestions, I'm thinking more and more that instead of a single
collections type, I should be proposing a new "namespaces" module instead.


Context: I've never been excited by this proposal.

But I am intrigued to note that with a couple of differences, you've now
converged on a module I've already written for my own use, which I called
a Registry. I actually subclassed 'dict', added some methods to use
dotted-access as you describe, and added some methods for accessing and
initializing a deeper key (basically a "get" that can handle hierarchy)
and a couple of other things.

There are worse things for application config storing than a pickled
Namespace. (Not human readable, but if you don't care about that, as I
generally don't, it's close to usable.)
Jul 18 '05 #2

P: n/a
Jeremy Bowers wrote:
On Thu, 10 Feb 2005 11:56:45 -0700, Steven Bethard wrote:
In the "empty classes as c structs?" thread, we've been talking in some
detail about my proposed "generic objects" PEP. Based on a number of
suggestions, I'm thinking more and more that instead of a single
collections type, I should be proposing a new "namespaces" module instead.


Context: I've never been excited by this proposal.

But I am intrigued to note that with a couple of differences, you've now
converged on a module I've already written for my own use, which I called
a Registry.


Yeah, I guess that was really the motivation of this module. I
personally wouldn't use it all that often -- certainly not with the
frequency that I use, say, Python 2.4's set type -- but I think there
are enough of us out here who have written essentially the same module
enough times that it merits a little bit of standardization and addition
to the standard library.

Steve
Jul 18 '05 #3

P: n/a
On Thu, 10 Feb 2005 13:39:29 -0700, Steven Bethard wrote:
Yeah, I guess that was really the motivation of this module. I
personally wouldn't use it all that often -- certainly not with the
frequency that I use, say, Python 2.4's set type -- but I think there
are enough of us out here who have written essentially the same module
enough times that it merits a little bit of standardization and addition
to the standard library.


If it expands past what was basically a slight elaboration on

class Bunch: pass

to actually do real work, I'd be much more impressed, as long as simple
things stay simple. A hierarchy-aware get might be a good start:
config = Namespace()
myParm = config.get("myapp.wrapping.line", constants.STRICT)
someParms = config.get("myapp.settings.printer", Namespace())


which would create the necessary namespaces or something.
Jul 18 '05 #4

P: n/a
I like it alot! My only minor complaint is that the name is to long.
Also I *really wish* the Namespace could do this:

r, g, b = col = Namespace(r = 4, g = 3, b = 12)

But alas, I guess that's not doable within the scope of the Namespace PEP.

--
mvh Björn
Jul 18 '05 #5

P: n/a
BJörn Lindqvist wrote:
I like it alot! My only minor complaint is that the name is to long.
Also I *really wish* the Namespace could do this:

r, g, b = col = Namespace(r = 4, g = 3, b = 12)

But alas, I guess that's not doable within the scope of the Namespace PEP.


You can almost spell that already, it just won't do what you want:

r, g, b = col = dict(r = 4, g = 3, b = 12).values()

The dict-based implementation means named tuples they ain't :)

However, the namespaces module *would* provide a natural location for a
NamedTuple class.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #6

P: n/a
Jeremy Bowers <je**@jerf.org> wrote in message news:<pa****************************@jerf.org>...
On Thu, 10 Feb 2005 11:56:45 -0700, Steven Bethard wrote:
In the "empty classes as c structs?" thread, we've been talking in some
detail about my proposed "generic objects" PEP. Based on a number of
suggestions, I'm thinking more and more that instead of a single
collections type, I should be proposing a new "namespaces" module instead.


Context: I've never been excited by this proposal.

But I am intrigued to note that with a couple of differences, you've now
converged on a module I've already written for my own use, which I called
a Registry. I actually subclassed 'dict', added some methods to use
dotted-access as you describe, and added some methods for accessing and
initializing a deeper key (basically a "get" that can handle hierarchy)
and a couple of other things.

There are worse things for application config storing than a pickled
Namespace. (Not human readable, but if you don't care about that, as I
generally don't, it's close to usable.)


FWIW, you can count me about the people who (re)wrote this same thing
(actually with some difference, since I wanted to keep the order, so
I used nested lists instead of nested dictionaries, but the idea was
similar). I would welcome some module in the standard library to store
hierarchical data.

Michele Simionato
Jul 18 '05 #7

P: n/a
Steven Bethard wrote:
In the "empty classes as c structs?" thread, we've been talking in some
detail about my proposed "generic objects" PEP. Based on a number of
suggestions, I'm thinking more and more that instead of a single
collections type, I should be proposing a new "namespaces" module
instead. Some of my reasons:

(1) Namespace is feeling less and less like a collection to me. Even
though it's still intended as a data-only structure, the use cases seem
pretty distinct from other collections.
Particularly since the collections module is billed as a location for
purpose-specific optimised data structures where some general-purpose features
have been dropped in order to make other aspects faster.

All the ideas here relate to conveniently manipulating namespaces, and are
independent enough to make sense as separate classes. I think a new module is a
good call.

It also allows a Python implementation :)
>>> ns = Namespace(eggs=1)
>>> Namespace.update(ns, [('spam', 2)], ham=3)
>>> ns

Namespace(eggs=1, ham=3, spam=2)

Note that update should be used through the class, not through the
instances, to avoid the confusion that might arise if an 'update'
attribute added to a Namespace instance hid the update method.


I'd like to see the PEP text itself encourage the more inheritance friendly
style used in the __init__ method:
type(ns).update(ns, [('spam', 2)], ham=3)

Accessing the class directly means that you may get incorrect behaviour if ns is
actually an instance of a subclass of Namespace that overrides update.

It may also be worth mentioning that the standard library applies this technique
in order to reliably access the magic methods of an instance's type, rather than
the versions shadowed in the instance (this is important when trying to pickle
or copy instances of 'type' for example).
Note that support for the various mapping methods, e.g.
__(get|set|del)item__, __len__, __iter__, __contains__, items, keys,
values, etc. was intentionally omitted as these methods did not seem
to be necessary for the core uses of an attribute-value mapping. If
such methods are truly necessary for a given use case, this may
suggest that a dict object is a more appropriate type for that use.
The 'vars' builtin also makes it trivial to use dictionary style operations to
manipulate the contents of a Namespace.
class Namespace(object):
def __eq__(self, other):
"""x.__eq__(y) <==> x == y

Two Namespace objects are considered equal if they have the
same attributes and the same values for each of those
attributes.
"""
return (other.__class__ == self.__class__ and
self.__dict__ == other.__dict__)
Hmm, the exact class check strikes me as being rather strict. Although I guess
it makes sense, as the loose check is easily spelt:
if vars(ns1) == vars(ns2): pass

*shrug* I'll let other people argue about this one :)
def update(*args, **kwargs):
"""Namespace.update(ns, [ns|dict|seq,] **kwargs) -> None
Same comment as above about encouraging the inheritance friendly style.
Open Issues
===========
What should the types be named? Some suggestions include 'Bunch',
'Record', 'Struct' and 'Namespace'.
Where should the types be placed? The current suggestion is a new
"namespaces" module.
If there aren't any objections to the current answers to these questions in this
thread, you may finally get to move them to a 'Resolved Issues' section :)
Should namespace chaining be supported? One suggestion would add a
NamespaceChain object to the module::
This does have the advantage of keeping the basic namespace simple. However, it
may also be worth having native chaining support in Namespace:

class Namespace(object):
def __init__(*args, **kwds):
if not 1 <= len(args) <= 3:
raise TypeError('expected 1 to 3 non-keyword arguments,'
'got %i' % len(args))
self = args[0]
# Set __fallback__ first, so keyword arg can override
if len(args) > 2:
self.__fallback__ = args[2]
else:
self.__fallback__ = None
# Update __dict__. OK to use normal method calls, since dict hasn't
# been updated yet
if len(args) > 1 and args[1] is not None:
self.update(args[1], **kwds)
elif kwds:
self.update(**kwds)

def __getattr__(self, attr):
if self.__fallback__ is not None:
return getattr(self.__fallback__, attr)
raise AttributeError("No attribute named " + attr)

# Otherwise unchanged
class NamespaceChain(object):
"""NamespaceChain(*objects) -> new attribute lookup chain

The new NamespaceChain object's attributes are defined by the
attributes of the provided objects. When an attribute is
requested, the sequence is searched sequentially for an object
with such an attribute. The first such attribute found is
returned, or an AttributeError is raised if none is found.

Note that this chaining is only provided for getattr and delattr
operations -- setattr operations must be applied explicitly to
the appropriate objects.


Hmm, I'm not so sure about this. I think that the right model is the way that a
class instance is currently chained with its class.

That is, assume we have the following:
c = cls()
ns = Namespace(vars(c), vars(cls)) # Using modified NS above
nc = NamespaceChain(Namespace(vars(c)), Namespace(vars(cls)))
I would expect modification of attributes on ns or nc to behave similarly to
modification of attributes on c - attribute retrieval follows the chain, but
attribute modification (set/del) always operates on the first namespace in the
chain.

Cheers,
Nick.
--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #8

P: n/a
Michele Simionato wrote:
FWIW, you can count me about the people who (re)wrote this same thing
(actually with some difference, since I wanted to keep the order, so
I used nested lists instead of nested dictionaries, but the idea was
similar). I would welcome some module in the standard library to store
hierarchical data.


This is one of the reasons why Steven's idea of switching to proposing a new
module is a good one. It then provides a natural location for any future
extensions of the idea such as Records (i.e. namespaces with a defined set of
legal fields) and NamedTuples and NamedLists (i.e. namespaces with a defined
field order)

Cheers,
Nick

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #9

P: n/a
On Fri, 11 Feb 2005 22:23:58 +1000, Nick Coghlan wrote:
This is one of the reasons why Steven's idea of switching to proposing a
new module is a good one. It then provides a natural location for any
future extensions of the idea such as Records (i.e. namespaces with a
defined set of legal fields) and NamedTuples and NamedLists (i.e.
namespaces with a defined field order)


I'm not very good at reading Guido's mind, but it might be worth feeling
out whether mentioning this will improve the chances of this passing or
not, because while I do not know, can not know, and am not trying to
predict, it is at least possible that Guido may feel as I have been: That
this proposal on its own isn't that exciting, but as the foundation of
some *other* standard functionality as described it might be very good. In
that case that should be emphasized.

See, now it still doesn't make me jump up and down because I can write
what I need fairly easily, but it would be a great boon to beginners or
people who just want to do work and re-write this again, but slightly
differently, and would also provide some standardization where otherwise
we all roll our not-quite-similar-enough implementations, which would help
us read each other's code.

Y'all are bringing me around, slowly but surely :-)
Jul 18 '05 #10

P: n/a
Jeremy Bowers <je**@jerf.org> wrote in
news:pa****************************@jerf.org:
On Fri, 11 Feb 2005 22:23:58 +1000, Nick Coghlan wrote:
This is one of the reasons why Steven's idea of switching to
proposing a new module is a good one. It then provides a
natural location for any future extensions of the idea such as
Records (i.e. namespaces with a defined set of legal fields)
and NamedTuples and NamedLists (i.e. namespaces with a defined
field order)


I'm not very good at reading Guido's mind, but it might be worth
feeling out whether mentioning this will improve the chances of
this passing or not, because while I do not know, can not know,
and am not trying to predict, it is at least possible that Guido
may feel as I have been: That this proposal on its own isn't
that exciting, but as the foundation of some *other* standard
functionality as described it might be very good. In that case
that should be emphasized.

See, now it still doesn't make me jump up and down because I can
write what I need fairly easily, but it would be a great boon to
beginners or people who just want to do work and re-write this
again, but slightly differently, and would also provide some
standardization where otherwise we all roll our
not-quite-similar-enough implementations, which would help us
read each other's code.

Y'all are bringing me around, slowly but surely :-)


Yes, it seems that the main benefit is to provide a Python-standard
way of doing something that many individuals see as useful. Like
many others, I've implemented my own variation (a class I call
Data) to do essentially the same thing as the bunch/namespace
proposal. If there had already been an existing standard way to do
it, I would at most have needed to subclass the standard method.

I would bet that subclassing is *still* going to be common, though,
as each of us individually roll our own version to get that one
vital feature the standard doesn't cover (for me, it's update with
numerous other types), so the net effect may actually not be all
that different from what happens now. Still, I could happily give
up some features as long as the functionality is still available
and presuming I intended the code for public consumption, and for
that, Namespace would be most useful.

Though I'd like it to have a shorter name. I'm lazy.

--
rzed

Jul 18 '05 #11

P: n/a
Nick Coghlan wrote:
Steven Bethard wrote:
>>> ns = Namespace(eggs=1)
>>> Namespace.update(ns, [('spam', 2)], ham=3)
>>> ns

Namespace(eggs=1, ham=3, spam=2)

Note that update should be used through the class, not through the
instances, to avoid the confusion that might arise if an 'update'
attribute added to a Namespace instance hid the update method.


I'd like to see the PEP text itself encourage the more inheritance
friendly style used in the __init__ method:
type(ns).update(ns, [('spam', 2)], ham=3)


Where in the PEP do you think this belongs? Or did you mean you wanted
it in the docstrings of the Namespace functions?
Note that support for the various mapping methods, e.g.
__(get|set|del)item__, __len__, __iter__, __contains__, items, keys,
values, etc. was intentionally omitted as these methods did not seem
to be necessary for the core uses of an attribute-value mapping. If
such methods are truly necessary for a given use case, this may
suggest that a dict object is a more appropriate type for that use.


The 'vars' builtin also makes it trivial to use dictionary style
operations to manipulate the contents of a Namespace.


Right, I had meant to mention that. Thanks!
Should namespace chaining be supported? One suggestion would add a
NamespaceChain object to the module::


This does have the advantage of keeping the basic namespace simple.
However, it may also be worth having native chaining support in Namespace:


I think I prefer the separate NamespaceChain object because it allows
you to chain namespaces other than just Namespace objects -- any object
that supports getattr is okay. If chaining is builtin, all namespaces
(except the last one?) have to be Namespace objects...
class NamespaceChain(object):
"""NamespaceChain(*objects) -> new attribute lookup chain

The new NamespaceChain object's attributes are defined by the
attributes of the provided objects. When an attribute is
requested, the sequence is searched sequentially for an object
with such an attribute. The first such attribute found is
returned, or an AttributeError is raised if none is found.

Note that this chaining is only provided for getattr and delattr
operations -- setattr operations must be applied explicitly to
the appropriate objects.


Hmm, I'm not so sure about this. I think that the right model is the way
that a class instance is currently chained with its class.

That is, assume we have the following:
c = cls()
ns = Namespace(vars(c), vars(cls)) # Using modified NS above
nc = NamespaceChain(Namespace(vars(c)), Namespace(vars(cls)))

I would expect modification of attributes on ns or nc to behave
similarly to modification of attributes on c - attribute retrieval
follows the chain, but attribute modification (set/del) always operates
on the first namespace in the chain.


Yeah, that makes sense. I hadn't quite wrapped my head around what
setattr _aught_ to do in these cases, so I wasn't willing to commit to
anything yet. ;) I'll update NamespaceChain to work like classes do...

Steve
Jul 18 '05 #12

P: n/a
rzed wrote:
I would bet that subclassing is *still* going to be common, though,
as each of us individually roll our own version to get that one
vital feature the standard doesn't cover (for me, it's update with
numerous other types)
This is certainly what I expect to happen. It's the main reason I think the
relevant documentation (including the PEP and docstrings) should encourage the
type(self).method() style of accessing class methods to avoid shadowing problems
while still properly supporting inheritance.

The other important aspect is for the classes to be designed to be inheritance
friendly.

Though I'd like it to have a shorter name. I'm lazy.


'from namespaces import Namespace as ns' :)

I thought about suggesting simply 'space' as a name, but I think that's way too
vague. We're using it as a namespace, so we might as well call it one :)

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #13

P: n/a
Steven Bethard wrote:
Should namespace chaining be supported? One suggestion would add a
NamespaceChain object to the module::

This does have the advantage of keeping the basic namespace simple.
However, it may also be worth having native chaining support in
Namespace:

I think I prefer the separate NamespaceChain object because it allows
you to chain namespaces other than just Namespace objects -- any object
that supports getattr is okay. If chaining is builtin, all namespaces
(except the last one?) have to be Namespace objects...


I like NamespaceChain too - I was simply thinking it might be good to have a
recursive chaining method associated with actual Namespace objects as well.

However, now that I look more closely at NamespaceChain, it makes more sense to
me to explicitly make the 'head' of the chain a namespace view in its own right.
This should also make the local binding, chained lookup behaviour fairly obvious
(since it will work just as it does for any class with __getattr__ defined, but
not __setattr__ or __delattr__).
That is, something like:

class NamespaceChain(NamespaceView):
def __init__(self, head, *args):
NamespaceView.__init__(self, head)
self.__namespaces__ = args

def __getattr__(self, name):
"""Return the first such attribute found in the object list

This is only invoked for attributes not found in the head
namespace.
"""
for obj in self.__namespaces__:
try:
return getattr(obj, name)
except AttributeError:
pass
raise AttributeError(name)
Python gives us the local set and local del for free.

The 'nested namespaces' approach that prompted my original suggestion can then
be spelt by using:

parent = Namespace()
child1 = NamespaceChain({}, parent)
child2 = NamespaceChain({}, child1)

There *is* a problem with using __getattr__ though - any attribute in the
chained namespaces that is shadowed by a class attribute (like 'update') will be
picked up from the class, not from the chained namespaces. So we do need to use
__getattribute__ to change that lookup order. However, given the amount of grief
the default lookup behaviour causes, I think the place to do that is in
Namespace itself:

class Namespace(object):
# otherwise unchanged
def __getattribute__(self, name):
"""Namespaces do NOT default to looking in their class dict
for non-magic attributes
"""
getattribute = super(Namespace, self).__getattribute__
try:
return getattribute("__dict__")[name]
except KeyError:
if name.startswith('__') and name.endswith('__'):
# Employ the default lookup system for magic names
return getattribute(name)
else:
# Skip the default lookup system for normal names
if hasattr(self, "__getattr__"):
return getattribute("__getattr__")(name)
else:
raise AttributeError(name)

The above is a pretty simple approach - it completely bypasses the descriptor
machinery for non-magic names. This means that the instance namespace does NOT
get polluted by any non-magic names in the class dictionary, but magic names can
be accessed normally. And subclasses can add their own methods, and this feature
will continue to 'just work'. For example:

Py> ns = namespaces.Namespace()
Py> ns.update
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "namespaces.py", line 30, in __getattribute__
raise AttributeError(name)
AttributeError: update
Py> type(ns).update
<unbound method Namespace.update>
Py> ns.__init__
<bound method Namespace.__init__ of Namespace()>

(side note: I don't think I have ever written a __getattribute__ method without
introducing an infinite recursion on the first attempt. . .)

Anyway, it is probably worth digging into the descriptor machinery a bit further
in order to design a lookup scheme that is most appropriate for namespaces, but
the above example has convinced me that object.__getattribute__ is NOT it :)

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Jul 18 '05 #14

P: n/a
Nick Coghlan wrote:
There *is* a problem with using __getattr__ though - any attribute in
the chained namespaces that is shadowed by a class attribute (like
'update') will be picked up from the class, not from the chained
namespaces. So we do need to use __getattribute__ to change that lookup
order. However, given the amount of grief the default lookup behaviour
causes, I think the place to do that is in Namespace itself:

class Namespace(object):
# otherwise unchanged
def __getattribute__(self, name):
"""Namespaces do NOT default to looking in their class dict
for non-magic attributes
"""
getattribute = super(Namespace, self).__getattribute__
try:
return getattribute("__dict__")[name]
except KeyError:
if name.startswith('__') and name.endswith('__'):
# Employ the default lookup system for magic names
return getattribute(name)
else:
# Skip the default lookup system for normal names
if hasattr(self, "__getattr__"):
return getattribute("__getattr__")(name)
else:
raise AttributeError(name)
Hmmm... This does seem sensible. And sidesteps the issue about
suggesting that subclasses use Namespace.update instead of
namespaceinstance.update -- the latter just won't work! (This is a Good
Thing, IMHO.)
Anyway, it is probably worth digging into the descriptor machinery a bit
further in order to design a lookup scheme that is most appropriate for
namespaces, but the above example has convinced me that
object.__getattribute__ is NOT it :)


Yeah, I'm going to look into this a bit more too, but let me know if you
have any more insights in this area.

Steve
Jul 18 '05 #15

P: n/a
Steven Bethard wrote:
Hmmm... This does seem sensible. And sidesteps the issue about
suggesting that subclasses use Namespace.update instead of
namespaceinstance.update -- the latter just won't work! (This is a Good
Thing, IMHO.)


Yeah, I thought so too. It also crystallised for me that the real problem was
that we were trying to use the attribute namespace for something different from
what it is usually used for, and so Python's normal lookup sequence was a
hindrance rather than a help.

I guess Guido knew what he was doing when he provided the __getattribute__ hook ;)

One other point is that dealing correctly with meta-issues like this greatly
boosts the benefits from having this module in the standard library. I would
expect most of the 'roll-your-own' solutions that are out there deal badly with
this issue. "Inherit from namespaces.Namespace" is a simpler instruction than
"figure out how to write an appropriate __getattribute__ method".
Anyway, it is probably worth digging into the descriptor machinery a
bit further in order to design a lookup scheme that is most
appropriate for namespaces, but the above example has convinced me
that object.__getattribute__ is NOT it :)

Yeah, I'm going to look into this a bit more too, but let me know if you
have any more insights in this area.


My gut feel is that we want to get rid of all the decriptor behaviour for normal
names, but keep it for special names. After all, if people actually wanted that
machinery for normal attributes, they'd be using a normal class rather than a
namespace.

I'm also interested in this because I'd like to let the 'Record' class (see
below) be able to store anything in the defaults, including descriptors and have
things 'just work'.

The __getattribute__ I posted comes close to handling that, but we need
__setattr__ and __delattr__ as well in order to deal fully with the issue, since
override descriptors like property() can affect those operations. Fortunately,
they should be relatively trivial:

# Module helper function
def _isspecial(name):
return name.startswith("__") and name.endswith("__")

# And in Namespace
def __getattribute__(self, name):
"""Namespaces only use __dict__ and __getattr__
for non-magic attribute names.
Class attributes and descriptors like property() are ignored
"""
getattribute = super(Namespace, self).__getattribute__
try:
return getattribute("__dict__")[name]
except KeyError:
if _isspecial(name)
# Employ the default lookup system for magic names
return getattribute(name)
else:
# Skip the default lookup system for normal names
if hasattr(self, "__getattr__"):
return getattribute("__getattr__")(name)
else:
raise AttributeError('%s instance has no attribute %s'
% (type(self).__name__, name))

def __setattr__(self, name, val):
"""Namespaces only use __dict__ for non-magic attribute names.
Descriptors like property() are ignored"""
if _isspecial(name):
super(Namespace, self).__setattr__(name, val)
else:
self.__dict__[name] = val

def __delattr__(self, name):
"""Namespaces only use __dict__ for non-magic attribute names.
Descriptors like property() are ignored"""
if _isspecial(name):
super(Namespace, self).__delattr__(name)
else:
del self.__dict__[name]

In action:

Py> def get(self): print "Getting"
....
Py> def set(self, val): print "Setting"
....
Py> def delete(self): print "Deleting"
....
Py> prop = property(get, set, delete)
Py> class C(object):
.... x = prop
....
Py> c = C()
Py> c.x
Getting
Py> c.x = 1
Setting
Py> del c.x
Deleting
Py> class NS(namespaces.Namespace):
.... x = prop
....
Py> ns = NS()
Py> ns.x
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "namespaces.py", line 40, in __getattribute__
raise AttributeError('%s instance has no attribute %s'
AttributeError: NS instance has no attribute x
Py> ns.x = 1
Py> del ns.x
Py> ns.__x__
Getting
Py> ns.__x__ = 1
Setting
Py> del ns.__x__
Deleting

I've attached my latest local version of namespaces.py (based on the recent PEP
draft) which includes all of the above. Some other highlights are:

NamespaceView supports Namespace instances in the constructor

NamespaceChain inherits from NamespaceView (as per my last message about that)

LockedView is a new class that supports 'locking' a namespace, so you can only
modify existing names, and cannot add or remove them

NamespaceChain and LockedView are the main reason I modified NamespaceView to
directly support namespaces - so that subclassed could easily support using either.

Record inherits from LockedView and is designed to make it easy to define and
create fully specified "data container" objects.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

def _isspecial(name):
return name.startswith("__") and name.endswith("__")

class Namespace(object):
"""Namespace([namespace|dict|seq], **kwargs) -> Namespace object

The new Namespace object's attributes are initialized from (if
provided) either another Namespace object's attributes, a
dictionary, or a sequence of (name, value) pairs, then from the
name=value pairs in the keyword argument list.
"""

def __init__(*args, **kwargs):
"""Initializes the attributes of a Namespace instance.

Calling __init__ from a subclass is optional. Any _required_
initialisation will be done in __new__. Subclasses should
preserve this characteristic.
"""
# inheritance-friendly update call
type(args[0]).update(*args, **kwargs)

def __getattribute__(self, name):
"""Namespaces only use __dict__ and __getattr__
for non-magic attribute names.
Class attributes and descriptors like property() are ignored
"""
getattribute = super(Namespace, self).__getattribute__
try:
return getattribute("__dict__")[name]
except KeyError:
if _isspecial(name):
# Employ the default lookup system for magic names
return getattribute(name)
else:
# Skip the default lookup system for normal names
if hasattr(self, "__getattr__"):
return getattribute("__getattr__")(name)
else:
raise AttributeError('%s instance has no attribute %s'
% (type(self).__name__, name))

def __setattr__(self, name, val):
"""Namespaces only use __dict__ for non-magic attribute names.
Descriptors like property() are ignored"""
if _isspecial(name):
super(Namespace, self).__setattr__(name, val)
else:
self.__dict__[name] = val

def __delattr__(self, name):
"""Namespaces only use __dict__ for non-magic attribute names.
Descriptors like property() are ignored"""
if _isspecial(name):
super(Namespace, self).__delattr__(name)
else:
del self.__dict__[name]

def __eq__(self, other):
"""x.__eq__(y) <==> x == y

Two Namespace objects are considered equal if they have the
same attributes and the same values for each of those
attributes.
"""
return (other.__class__ == self.__class__ and
self.__dict__ == other.__dict__)

def __repr__(self):
"""x.__repr__() <==> repr(x)

If all attribute values in this namespace (and any nested
namespaces) are reproducable with eval(repr(x)), then the
Namespace object is also reproducable for eval(repr(x)).
"""
return '%s(%s)' % (
type(self).__name__,
', '.join('%s=%r' % (k, v) for k, v in
sorted(self.__dict__.iteritems())))

def update(*args, **kwargs):
"""Namespace.update(ns, [ns|dict|seq,] **kwargs) -> None

Updates the first Namespace object's attributes from (if
provided) either another Namespace object's attributes, a
dictionary, or a sequence of (name, value) pairs, then from
the name=value pairs in the keyword argument list.
"""
if not 1 <= len(args) <= 2:
raise TypeError('expected 1 or 2 arguments, got %i' %
len(args))
self = args[0]
if not isinstance(self, Namespace):
raise TypeError('first argument to update should be '
'Namespace, not %s' %
type(self).__name__)
if len(args) == 2:
other = args[1]
if isinstance(other, Namespace):
other = other.__dict__
try:
self.__dict__.update(other)
except (TypeError, ValueError):
raise TypeError('cannot update Namespace with %s' %
type(other).__name__)
self.__dict__.update(kwargs)
class NamespaceView(Namespace):
"""NamespaceView([ns|dict]) -> new Namespace view of the dict

Creates a Namespace that is a view of the original dictionary,
that is, changes to the Namespace object will be reflected in
the dictionary, and vice versa.
For namespaces, the view is of the other namespace's attributes.
"""
def __init__(self, orig):
if isinstance(orig, Namespace):
self.__dict__ = orig.__dict__
else:
self.__dict__ = orig

class LockedView(NamespaceView):
"""LockedView(dict) -> new Namespace view of the dict

Creates a NamespaceView that prevents addition of deletion
of names in the namespace. Existing names can be rebound.
"""
def __setattr__(self, name, val):
getattr(self, name)
self.__dict__[name] = val

def __delattr__(self, name):
raise TypeError("%s does not permit deletion of attributes"
% type(self).__name__)

class NamespaceChain(NamespaceView):
"""NamespaceChain(dict, *chain) -> new attribute lookup chain

The new NamespaceChain is firstly a standard NamespaceView for
the supplied dictionary.

However, when an attribute is looked up and is not found in the
main dictionary, the sequence of chained objects is searched
sequentially for an object with such an attribute. The first
such attribute found is returned, or an AttributeError is raised
if none is found.

The list of chained objects is stored in the __namespaces__
attribute.
"""
def __init__(self, head, *args):
NamespaceView.__init__(self, head)
self.__namespaces__ = args

def __getattr__(self, name):
"""Return the first such attribute found in the object list

This is only invoked for attributes not found in the head
namespace.
"""
for obj in self.__namespaces__:
try:
return getattr(obj, name)
except AttributeError:
pass
raise AttributeError('%s instance has no attribute %s'
% (type(self).__name__, name))

class Record(LockedView):
_subfield_prefix = "_sub_"
def __init__(self):
"""Record() -> Record instance

Instantiates a Namespace populated based on a Record
subclass definition.
Normal attributes are placed in the instance dictionary
on initialisation.
Attributes whose names start with '_sub_' are called, and
the result placed in the instance dictionary using a name
without the subfield prefix.
The subfield prefix used can be changed by setting the
_subfield_prefix attribute in the subclass
Setting the subfield prefix to None means no subfields will be
automatically instantiated, and setting it to "" means that
every field will be automatically instantiated.
Any fields starting with an underscore are ignored (this includes
subfields which start with an underscore after the subfield prefix
has been stripped).

For example:
Py> from namespaces import Record
Py> class Example(Record):
... a = 1
... b = ""
... class _sub_sf(Record):
... c = 3
... def _sub_calc(): return "Calculated value!"
...
Py> x = Example()
Py> x
Example(a=1, b='', calc='Calculated value!', sf=_sub_sf(c=3))
Py> class Example2(namespaces.Record):
... _subfield_prefix = ""
... a = str
... b = int
... class x(namespaces.Record): pass
... def f(): return "Hi!"
...
Py> x = Example2()
Py> x
Example2(a='', b=0, f='Hi!', x=x())
Py> x.c = 1
Traceback (most recent call last):
...
AttributeError: Example2 instance has no attribute c
"""
definition = type(self)
prefix = definition._subfield_prefix
prefix_len = len(prefix)
for field, value in definition.__dict__.iteritems():
if field.startswith(prefix):
subfield = field[prefix_len:]
if not subfield.startswith("_"):
# Set the calculated value
self.__dict__[subfield] = value()
elif not field.startswith("_"):
# Set the value
self.__dict__[field] = value

Jul 18 '05 #16

P: n/a
Nick Coghlan wrote:
Py> class NS(namespaces.Namespace):
... x = prop
...


Oops - c&p error here. This was actually:

Py> class NS(namespaces.Namespace):
.... x = prop
.... __x__ = prop
....

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #17

P: n/a
Yeah, talking to myself again. I had a couple of further thoughts on how to do
things in an inheritance friendly way. . .

Firstly, for Namespaces handling of special names, I think it would be good to
make it easy for subclasses to change the sets of names that are handled using
either Namespace lookup or standard Python lookup.

The simplest way is to make the method that performs the check an instance method.

class Namespace(object):
# etc
def _is_special_name(name):
return name.startswith("__") and name.endswith("__")

I also think subclasses should be able to affect update() just by defining
__setattr__ (for example, my LockedView class forgot to deal with update()). So
I'd replace the calls to self.__dict__.update(other) with calls to
type(self)._update_dict(self, other):

class Namespace(object):
# etc
def _update_dict(self, other):
for k in other:
setattr(self, k, other[k])

Finally, it might be nice if the get/set/del methods special-cased names with
dots in them. Consider the following with a normal class:

Py> class C(object): pass
....
Py> c = C()
Py> c.x = C()
Py> setattr(c, "x.y", 1)
Py> getattr(c, "x.y")
1
Py> vars(c.x)
{}
Py> vars(c)
{'x.y': 1, 'x': <__main__.C object at 0x009E0A10>}
Py> c.x.y
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'C' object has no attribute 'y'

With a namespace, we can make the dotted access 'just work':

Py> ns = namespaces.Namespace()
Py> ns.x = C()
Py> ns.y = namespaces.Namespace()
Py> setattr(ns, "x.y", C())
Py> setattr(ns, "x.y.z", 1)
Py> setattr(ns, "y.x", C())
Py> setattr(ns, "y.x.z", 2)
Py> vars(ns)
{'y': Namespace(x=<__main__.C object at 0x009EB8D0>), 'x': <__main__.C object at
0x009EB1F0>}
Py> vars(ns.x)
{'y': <__main__.C object at 0x009EB890>, 'y.z': 1}
Py> vars(ns.y)
{'x': <__main__.C object at 0x009EB8D0>}
Py> vars(ns.y.x)
{'z': 2}

Combined with the update change, it means this code does the right thing:

Py> settings = {
.... "x" : 1,
.... "y" : namespaces.Namespace(),
.... "y.z" : 1,
.... "y.x" : "Hi there!",
.... }
Py> ns = namespaces.Namespace(settings)
Py> ns
Namespace(x=1, y=Namespace(x='Hi there!', z=1))

Rather than sending it again, I've posted my current code at:
http://members.iinet.net.au/~ncoghla.../namespaces.py

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #18

P: n/a
Nick Coghlan wrote:
class Namespace(object):
# etc
def _update_dict(self, other):
for k in other:
setattr(self, k, other[k])


This doesn't work, as it doesn't allow the sequence of 2-tuples. So I copied the
relevant check for a keys() attribute from dict.update:

def _update_dict(self, other):
"""Allow subclasses to easily override handling of dict updates
Also allows dotted names in the source to be handled correctly

Uses the "keys" attribute to identify mappings
"""
try:
items = other.keys()
except AttributeError:
for k, v in other:
setattr(self, k, v)
else:
for k in items:
setattr(self, k, other[k])

Another change I made was to the Record class. It now uses __new__ to initialise
the instance dictionary with the defaults from the subclass definition, leaving
__init__ free to be used to invoke update() like it is for a standard namespace:

Py> from namespaces import Record
Py> class Example(Record):
... a = 1
... b = ""
... class _sub_sf(Record):
... c = 3
... def _sub_calc(): return "Calculated value!"
...
Py> x = Example()
Py> x
Example(a=1, b='', calc='Calculated value!', sf=_sub_sf(c=3))
Py> x = Example(a=3)
Py> x
Example(a=3, b='', calc='Calculated value!', sf=_sub_sf(c=3))
Py> x = Example([("sf.c", 5)], a=3, b="altered")
Py> x
Example(a=3, b='altered', calc='Calculated value!', sf=_sub_sf(c=5))
Py> x = Example({"sf.c": 5}, a=3, b="altered")
Py> x
Example(a=3, b='altered', calc='Calculated value!', sf=_sub_sf(c=5))

I also moved the module to my normal site:
http://boredomandlaziness.skystorm.n.../namespaces.py

Steven: Feel free to pick through this for anything you want to add to the PEP.
That's why I'm posting it, after all :)

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Jul 18 '05 #19

This discussion thread is closed

Replies have been disabled for this discussion.