Weakrefs to classes that derive from str

[Ron Garret]

Why doesn't this work?

from weakref import ref
class C(str): pass .... ref(C()) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: cannot create weak reference to 'C' object
Note that this does work:
class D(int): pass .... ref(D()) <weakref at 0x53e10; dead>

Likewise for floats, lists, etc. Everything but strs.

rg

[Steven Bethard]

Ron Garret wrote:

Why doesn't this work?

from weakref import ref
class C(str): pass ...ref(C())

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: cannot create weak reference to 'C' object

Note that you don't need the class redirection:

py> ref('')
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: cannot create weak reference to 'str' object

But I don't know why strings aren't valid arguments to ref...

STeVe

[Ron Garret]

In article <bv********************@comcast.com>,
Steven Bethard <st************@gmail.com> wrote:

Ron Garret wrote:

Why doesn't this work?

>from weakref import ref
>class C(str): pass

...

>ref(C())

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: cannot create weak reference to 'C' object

Note that you don't need the class redirection:

py> ref('')
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: cannot create weak reference to 'str' object

But I don't know why strings aren't valid arguments to ref...

None of the native types (int, float, list, tuple, etc.) can have weak
references, but wrapping them in a class is supposed to get around that.
And it does -- for all classes except str.

rg

[Steven Bethard]

Ron Garret wrote:

Note that you don't need the class redirection:

py> ref('')
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: cannot create weak reference to 'str' object

But I don't know why strings aren't valid arguments to ref...

None of the native types (int, float, list, tuple, etc.) can have weak
references, but wrapping them in a class is supposed to get around that.
And it does -- for all classes except str.

Interesting. Is the wrapping thing documented somewhere? I didn't see
it in the documentation for weakref.ref (though I have been known to be
blind occasionally) ;)

STeVe

[Peter Hansen]

Steven Bethard wrote:

Ron Garret wrote:

None of the native types (int, float, list, tuple, etc.) can have weak
references, but wrapping them in a class is supposed to get around
that. And it does -- for all classes except str.

Interesting. Is the wrapping thing documented somewhere? I didn't see
it in the documentation for weakref.ref (though I have been known to be
blind occasionally) ;)

I believe it's here: http://docs.python.org/lib/module-weakref.html
if you search for the string "Not all" and read the next two
paragraphs.

On the other hand, it says (there) only that "several builtin
types such as list and dict ... can add support through
subclassing", and does not say anything about int, str, etc...

-Peter

[Steven Bethard]

Peter Hansen wrote:

I believe it's here: http://docs.python.org/lib/module-weakref.html
if you search for the string "Not all" and read the next two
paragraphs.

On the other hand, it says (there) only that "several builtin
types such as list and dict ... can add support through
subclassing", and does not say anything about int, str, etc...

Ahh, thanks for the help. I guess there are at least two solutions to
the OP's problem then:

(1) Document that str and subclasses of str can't be weakreffed (easy)
(2) Change str so that it (or at least its subclasses) can be weakreffed
(hard)

Probably either way a feature request should be filed.

STeVe

[Ron Garret]

In article <tu********************@powergate.ca>,
Peter Hansen <pe***@engcorp.com> wrote:

Steven Bethard wrote:

Ron Garret wrote:

None of the native types (int, float, list, tuple, etc.) can have weak
references, but wrapping them in a class is supposed to get around
that. And it does -- for all classes except str.

Interesting. Is the wrapping thing documented somewhere? I didn't see
it in the documentation for weakref.ref (though I have been known to be
blind occasionally) ;)

I believe it's here: http://docs.python.org/lib/module-weakref.html
if you search for the string "Not all" and read the next two
paragraphs.

On the other hand, it says (there) only that "several builtin
types such as list and dict ... can add support through
subclassing", and does not say anything about int, str, etc...

Empirically:

from weakref import ref
def foo(c): .... class C(c): pass
.... ref(C())
.... foo(int)
foo(float)
foo(dict)
foo(list)
foo(str) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in foo
TypeError: cannot create weak reference to 'C' object foo(tuple) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in foo
TypeError: cannot create weak reference to 'C' object foo(long) Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in foo
TypeError: cannot create weak reference to 'C' object

Ah, it appears that non-immediate immutable types don't support
weakrefs. Hm...

rg

[Peter Hansen]

Ron Garret wrote:

foo(int)
foo(float)
foo(dict)
foo(list)
foo(str) TypeError: cannot create weak reference to 'C' object
foo(tuple) TypeError: cannot create weak reference to 'C' object
foo(long)

TypeError: cannot create weak reference to 'C' object

Ah, it appears that non-immediate immutable types don't support
weakrefs. Hm...

I see the same results you do, and yet I don't understand
the comment. Can you please explain what "immediate"
means in this context?

-Peter

[Raymond Hettinger]

[Ron Garret]

Why doesn't this work?

from weakref import ref
class C(str): pass ... ref(C())

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: cannot create weak reference to 'C' object

. . . Everything but strs.

Also subclasses of tuple are not weak referencable.

The issue is in the design of the C structure as a variable-sized immutable
object. Both strings and tuples allocate as a single unit of memory that holds
both the header information and the content information (the characters in a
string or the array of object pointers for a tuple). Since the size varies from
one string or tuple to the next, there is no straight-forward way for a subclass
to add an additional header field pointing to a list of weak references.

For lists and dicts, this is not a problem because the object is allocated in
two sections, a fixed size header component and a pointer to another area of
memory to hold the contents of the collection. This makes it possible for a
subclass to graft-on a weak reference pointer at a known, fixed offset from the
beginning of the header.

There are two ways to fix this. One is to add a weak reference pointer to every
string object -- that way you wouldn't even have to subclass it. Another way is
to break the object into two pieces as is done for mutable containers like dicts
and lists.

Both approaches consume time and space. In general, that is not a big deal, but
fast, memory efficient strings and tuples are at the center of all things
Python. The need to weak reference this objects is somewhat rare in comparison
to the frequency of their other uses. It did not make sense to always pay a
time/space penalty just to create the possibility of weak referencing.

While the design decision is unlikely to change, the docs could certainly be
improved. A doc patch would be welcome.

FWIW, the work-arounds are to weak-reference instances of UserString or to
create a custom class with a has-a relationship instead of an is-a relationship.
Raymond Hettinger

[Ron Garret]

In article <-4********************@powergate.ca>,
Peter Hansen <pe***@engcorp.com> wrote:

Ron Garret wrote:

>foo(int)
>foo(float)
>foo(dict)
>foo(list)
>foo(str)

TypeError: cannot create weak reference to 'C' object

>foo(tuple)

TypeError: cannot create weak reference to 'C' object

>foo(long)

TypeError: cannot create weak reference to 'C' object

Ah, it appears that non-immediate immutable types don't support
weakrefs. Hm...

I see the same results you do, and yet I don't understand
the comment. Can you please explain what "immediate"
means in this context?

"Immediate" means a data type that will fit entirely in a machine
register, and therefore doesn't need to actually be stored in memory.
Ints and floats are the only two immediate types in Python. They are
immutable, but you can still create weakrefs to classes that derive from
them.

rg

[Ron Garret]

In article <RbB2e.52616$u76.11019@trndny08>,
"Raymond Hettinger" <vz******@verizon.net> wrote:

[Ron Garret]

Why doesn't this work?

>> from weakref import ref
>> class C(str): pass

...

>> ref(C())

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: cannot create weak reference to 'C' object

. . .

Everything but strs.

Also subclasses of tuple are not weak referencable.

The issue is in the design of the C structure as a variable-sized immutable
object. Both strings and tuples allocate as a single unit of memory that
holds
both the header information and the content information (the characters in a
string or the array of object pointers for a tuple). Since the size varies
from
one string or tuple to the next, there is no straight-forward way for a
subclass
to add an additional header field pointing to a list of weak references.

For lists and dicts, this is not a problem because the object is allocated in
two sections, a fixed size header component and a pointer to another area of
memory to hold the contents of the collection. This makes it possible for a
subclass to graft-on a weak reference pointer at a known, fixed offset from
the
beginning of the header.

There are two ways to fix this. One is to add a weak reference pointer to
every
string object -- that way you wouldn't even have to subclass it. Another way
is
to break the object into two pieces as is done for mutable containers like
dicts
and lists.

Both approaches consume time and space. In general, that is not a big deal,
but
fast, memory efficient strings and tuples are at the center of all things
Python. The need to weak reference this objects is somewhat rare in
comparison
to the frequency of their other uses. It did not make sense to always pay a
time/space penalty just to create the possibility of weak referencing.

While the design decision is unlikely to change, the docs could certainly be
improved. A doc patch would be welcome.

FWIW, the work-arounds are to weak-reference instances of UserString or to
create a custom class with a has-a relationship instead of an is-a
relationship.

Thanks for the detailed explanation. I understand now why you can't
create weakrefs to these types. What I don't understand still is why
you can't create weakrefs to user-defined classes that inherit from
these types. I would think that instances of user-defined classes have
the same header structure regardless of what they inherit from. This
would seem to be supported by the fact that you can create weakrefs to
instances of user-defined classes that inherit from int and float.

rg

[Raymond Hettinger]

[Ron Garret]

Thanks for the detailed explanation. I understand now why you can't
create weakrefs to these types. What I don't understand still is why
you can't create weakrefs to user-defined classes that inherit from
these types. I would think that instances of user-defined classes have
the same header structure regardless of what they inherit from. This
would seem to be supported by the fact that you can create weakrefs to
instances of user-defined classes that inherit from int and float.

It is an over-statement to say that it can't be done. In fact, Michael Hudson
is already working on a patch.

It is more accurate to say that the current mechanism doesn't allow it.
Michael's solution is to build a new mechanism.

The existing mechanism has a subclass extend the superclass's structure:

[--someobj--][--subclassdata--]
^
|
|---- offset to wr table ---

The offset is fixed for the type and must be the same across instances.

This is a problem for tuples and ints because someobj is of varying length:

[--tuple header, elem0, elem1, elem2--]
[--tuple header, elem0 ]

In contrast, ints and floats floats have no problem because they are always the
same size:

[--int header, int value--]
Raymond Hettinger

[Ron Garret]

In article <RgJ2e.26268$I16.3183@trndny03>,
"Raymond Hettinger" <vz******@verizon.net> wrote:

[Ron Garret]

Thanks for the detailed explanation. I understand now why you can't
create weakrefs to these types. What I don't understand still is why
you can't create weakrefs to user-defined classes that inherit from
these types. I would think that instances of user-defined classes have
the same header structure regardless of what they inherit from. This
would seem to be supported by the fact that you can create weakrefs to
instances of user-defined classes that inherit from int and float.

It is an over-statement to say that it can't be done. In fact, Michael
Hudson
is already working on a patch.

It is more accurate to say that the current mechanism doesn't allow it.
Michael's solution is to build a new mechanism.

The existing mechanism has a subclass extend the superclass's structure:

[--someobj--][--subclassdata--]
^
|
|---- offset to wr table ---

The offset is fixed for the type and must be the same across instances.

This is a problem for tuples and ints because someobj is of varying length:

[--tuple header, elem0, elem1, elem2--]
[--tuple header, elem0 ]

In contrast, ints and floats floats have no problem because they are always
the
same size:

[--int header, int value--]
Raymond Hettinger

I see. Thanks!

rg