Object Reference?

[Chris S.]

I'm trying to make a graphical editor and browser for Pickled files. One
aspect I'm not sure about is how to detect multiple references to the
same data.

For instance, say I had the Pickled data:
a=[1,2,3]
b=[a,4,5]
c=[b,6,7]
d=[a,b,c]

The idea is to allow the user to browse this data and indicate
references. In this case, if 'a' was selected, the browser should show
that 'b', 'c', and 'd' contain a reference to 'a'. Inversely, if 'c'
were selected, it should indicate that it's first element just isn't a
list, but a reference to a list defined elsewhere, namely 'b'.

Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less obtrusive
method? Python figures out when to delete objects based on the remaining
references to an object. Is there a way to access this information to
automatically lookup these references? Any help is greatly appreciated.

[Anthony Baxter]

On Fri, 06 Aug 2004 05:39:51 GMT, Chris S. <ch*****@nospamudel.edu> wrote:

Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less obtrusive
method? Python figures out when to delete objects based on the remaining
references to an object. Is there a way to access this information to
automatically lookup these references? Any help is greatly appreciated.

Use the "id()" of the objects?

[Chris S.]

Anthony Baxter wrote:

On Fri, 06 Aug 2004 05:39:51 GMT, Chris S. <ch*****@nospamudel.edu> wrote:

Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less obtrusive
method? Python figures out when to delete objects based on the remaining
references to an object. Is there a way to access this information to
automatically lookup these references? Any help is greatly appreciated.

Use the "id()" of the objects?

Well, I was thinking of a function that takes an object's id and returns
the ids of all the objects that reference it. Given that Python keeps
this information internally (I believe), my question is how do I access
his data and what would be the easiest way to code this function?

[Michael Hudson]

"Chris S." <ch*****@NOSPAMudel.edu> writes:

I'm trying to make a graphical editor and browser for Pickled
files. One aspect I'm not sure about is how to detect multiple
references to the same data.

For instance, say I had the Pickled data:
a=[1,2,3]
b=[a,4,5]
c=[b,6,7]
d=[a,b,c]

The idea is to allow the user to browse this data and indicate
references. In this case, if 'a' was selected, the browser should show
that 'b', 'c', and 'd' contain a reference to 'a'. Inversely, if 'c'
were selected, it should indicate that it's first element just isn't a
list, but a reference to a list defined elsewhere, namely 'b'.
I would recommend getting initmately familiar with the format of
pickles (something I can't claim to be). The information must be in
there somewhere.

Actually, maybe you could write a custom unpickler that keeps track of
this kind of information. Something else I haven't tried :-) But,
seriously, I think the knowledge that these objects came out of a
particular pickle is going to be a powerful advantage for this sort of
thing, and you should use it for everything it's worth.
Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less
obtrusive method? Python figures out when to delete objects based on
the remaining references to an object. Is there a way to access this
information to automatically lookup these references? Any help is
greatly appreciated.

Well, there's gc.get_referrers() and gc.get_referrents(), but I think
they are probably overkill (see above).

Cheers,
mwh

--
(ps: don't feed the lawyers: they just lose their fear of humans)
-- Peter Wood, comp.lang.lisp

[Chris S.]

Michael Hudson wrote:

Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less
obtrusive method? Python figures out when to delete objects based on
the remaining references to an object. Is there a way to access this
information to automatically lookup these references? Any help is
greatly appreciated.

Well, there's gc.get_referrers() and gc.get_referrents(), but I think
they are probably overkill (see above).

Thanks, these are exactly what I need. Pickle is just the file format.
The data are Python constructs, so I shouldn't have to bother with the
format at all, since I'm simply converting the Pickled file into generic
Python data. Plus, writing my own Pickle parser would be hugely
complicated since Pickle is essentially its own small programming language.

[Gandalf]

> I'm trying to make a graphical editor and browser for Pickled files.

One aspect I'm not sure about is how to detect multiple references to
the same data.
....
For instance, say I had the Pickled data:
a=[1,2,3]
b=[a,4,5]
c=[b,6,7]
d=[a,b,c]

The idea is to allow the user to browse this data and indicate
references. In this case, if 'a' was selected, the browser should show
that 'b', 'c', and 'd' contain a reference to 'a'. Inversely, if 'c'
were selected, it should indicate that it's first element just isn't a
list, but a reference to a list defined elsewhere, namely 'b'.

Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less
obtrusive
method? Python figures out when to delete objects based on the
remaining
references to an object. Is there a way to access this information to
automatically lookup these references? Any help is greatly appreciated.

Use the "id()" of the objects?

Well, I was thinking of a function that takes an object's id and
returns the ids of all the objects that reference it. Given that
Python keeps this information internally (I believe), my question is
how do I access his data and what would be the easiest way to code
this function?

Yes, Python figures out when to delete an object, based on its reference
count. But Python only counts the references. It keeps track of the
number of references for the object. It does not know where the
reference is located in memory. The short answer is that you cannot get
this information effectively.

Long answer follows.

Question: There can be many objects in the structure without a name
(e.g. not referenced from the value part of a dictionary).
How do you want to display them? As an example:

class A:
pass

a = A() # THE object
b = [a,a,a,a]
del a

# At this point, THE object does not have a named reference at all but
it is referenced four times. Do you really want to identify the objects
with their memory address? The id function does this to you (Anthony
Baxter said the same.) But this will be a useless pickle browser since
nobody will notice memory address duplications.

Another big problem here. In Python, the only things you can have are an
object references. You cannot name the object itself. There are no
'object variables', only 'reference variables'. For example:

class A:
pass

a = A() # THE object we are talking about
b = [a]
c = [b]

Here is the question: does the object b "references" a? The answer is
that the question is wrong because 'a' is a reference itself - a
reference can only reference one object at a time. Good questions are:

- does 'a' and 'b' reference to the same object? (No)
- does 'b[0]' and 'a' reference to the same object? (Yes)

You may think that a reference of 'a' is contained in 'b'. But this is
not a general assumption. For example:

a = A()
b = A()
a.o = b
b.o = a

Now which is contained in which?

Sorry for the long e-mail.

G

[Chris S.]

Gandalf wrote:

Yes, Python figures out when to delete an object, based on its reference
count. But Python only counts the references. It keeps track of the
number of references for the object. It does not know where the
reference is located in memory. The short answer is that you cannot get
this information effectively.

The gc module has methods that list an object's referrers and referents.
Although you're correct in that we're unable to access the total number
of references since duplicate references are not counted. For instance

import gc
a=[1,2,3]
b=[a,a,a,a]
print gc.get_referrers(a)

would only print out one instance of b, even though it references 'a'
four times. However, this only seems to apply for direct references, so
the total number of references can be determined by scanning all
referrers and counting the duplicate ids.