How properly manage memory of this PyObject* array?? (C extension)

[seberino]

Suppose a C extension locally built an array of PyObject* 's as
follows...

my_array = malloc(n * sizeof(PyObject *));
for (i = 0; i < n; i++) {
my_array[i] = PyList_New(0);
}

Q1: Must I do a Py_DECREF(my_ar ray[i]) on all elements
before exiting this C extension function? (What if
the elements got used in other objects?)

Q2: Must I do free(my_array); at end of function?? What if
my_array[i]'s are
used in other objects so that I can't necessarily just
nuke it!!!

Chris

[Fredrik Lundh]

se******@spawar .navy.mil wrote:

Suppose a C extension locally built an array of PyObject* 's as
follows...

my_array = malloc(n * sizeof(PyObject *));
for (i = 0; i < n; i++) {
my_array[i] = PyList_New(0);
}

Q1: Must I do a Py_DECREF(my_ar ray[i]) on all elements
before exiting this C extension function?

if you're releasing my_array before existing, yes.

(What if the elements got used in other objects?)

if other parts of your program storing pointers to the elements in your
array, those parts must make sure to increment the reference count when
copying the pointer.

that's the whole point of reference counting, of course: the count for
an object should, at all times, match the number of *active* references
your program has to that object.

Q2: Must I do free(my_array); at end of function??

unless some other part of your program holds on to it, of course you
have to release it. it's a bit surprising that you have to ask this,
really -- any C tutorial should explain how malloc/free works.

What if my_array[i]'s are used in other objects so that I can't
necessarily just nuke it!!!

if those other objects do proper reference counting, everything will
work find. if they don't, your program will crash sooner or later, no
matter what you do in the function that allocates my_array.

</F>

[seberino]

Q2: Must I do free(my_array); at end of function??

unless some other part of your program holds on to it

F.

Thanks! If I understand you correctly then I never have
to do free(my_array); because all the elements
of my_array are still being used and appended to other
structures elsewhere right?

As long as each element of my_array is managed
properly and freed properly there is NEVER any
reason to worry about fact that free(my_array)
will never get run right?
cs

[John Machin]

On 11/07/2006 4:39 AM, se******@spawar .navy.mil wrote:

>>Q2: Must I do free(my_array); at end of function??

unless some other part of your program holds on to it

F.

Thanks! If I understand you correctly then I never have
to do free(my_array); because all the elements
of my_array are still being used and appended to other
structures elsewhere right?

*WRONG* -- reread /F's response:
"""
unless some other part of your program holds on to it, of course you
have to release it. it's a bit surprising that you have to ask this,
really -- any C tutorial should explain how malloc/free works.
"""

>
As long as each element of my_array is managed
properly and freed properly there is NEVER any
reason to worry about fact that free(my_array)
will never get run right?

*WRONG* -- the reason to worry is that the memory occupied by the
my_array itself is never handed back. If you don't free it, repeated
calls to your function will cause your app to run out of memory.

It's quite simple, really: You malloc it, you free it. I share the
effbot's surprise: If you (or the taxpayers!) paid money for a
course/book/tutorial that didn't include that advice, a claim for a
refund is definitely indicated.

HTH,
John

[seberino]

It's quite simple, really: You malloc it, you free it.

John - I hope you don't mind that I really want to make sure I
understand your
good wisdom in this area by asking for clarification.. ..

ASSUMPTIONS:

1. As long as we properly handle the reference counting of PyObjects
then
memory management is taken care of for us. (i.e. You don't ever
use the 'free' command explicitly on a PyObject* but let Python
garbage collector
do the final freeing.)

2. All malloc'd stuff in C must be freed with free command. (Python
garbage
collector is limited to PyObjects. It won't free non-Python stuff
for us
like int arrays, char* strings, etc.)

.....

Now it would appear that if you **malloc an array of PyObjects**
(call it FOO[]) then you have an ambiguity....

The PyObject elements will be freed *for us* eventually by garbage
collector.
Hence, we can't ever do 'free(FOO); ' because we don't know when
garbage collector will free FOO[0], FOO[1], FOO[2], etc.

Chris

[Fredrik Lundh]

se******@spawar .navy.mil wrote:

Now it would appear that if you **malloc an array of PyObjects**
(call it FOO[]) then you have an ambiguity....

The PyObject elements will be freed *for us* eventually by garbage
collector.

Hence, we can't ever do 'free(FOO); ' because we don't know when
garbage collector will free FOO[0], FOO[1], FOO[2], etc.

you're not quite getting how reference counting works. the fact that
you have a pointer to a Python object stuffed away somewhere is
completely irrelevant to the Python garbage collector; all it cares
about is the reference count of the object itself. if the reference
count is not zero, someone somewhere might have a pointer to the object,
so the object must be preserved. if the reference count drops to zero,
nobody is supposed to have a pointer anymore, so the object can be removed.

if you use malloc to allocate a memory block and store PyObject pointers
in it, you must

1) make sure that the reference count is *incremented* (Py_INCREF) when
you add the objects (unless the object is new; when you create an
object, the reference count is set to 1). this tells Python that you
have a pointer to an object that you plan to use some time in the future.

2) make sure that the reference count is *decremented* (Py_DECREF) if
you remove the objects (this tells Python that *you* won't access the
object any more; if nobody else uses it either, it can safely be removed).

required reading:

http://docs.python.org/api/objects.html

</F>

[John Machin]

On 14/07/2006 4:16 PM, se******@spawar .navy.mil wrote:

>It's quite simple, really: You malloc it, you free it.

John - I hope you don't mind that I really want to make sure I
understand your
good wisdom in this area by asking for clarification.. ..

ASSUMPTIONS:

1. As long as we properly handle the reference counting of PyObjects
then
memory management is taken care of for us. (i.e. You don't ever
use the 'free' command explicitly on a PyObject* but let Python
garbage collector
do the final freeing.)

That's correct: You didn't malloc it, you don't free it.

BTW, free is a function, not a "command".

>
2. All malloc'd stuff in C must be freed with free command. (Python
garbage
collector is limited to PyObjects. It won't free non-Python stuff
for us
like int arrays, char* strings, etc.)

Correct.

>
Now it would appear that if you **malloc an array of PyObjects**
(call it FOO[]) then you have an ambiguity....

No, not at all; there is no such action as "malloc an array of
PyObjects". Firstly, malloc does not produce an array of anything; it
allocates a chunk of memory and returns the address of the start of the
chunk. secondly, what you stuffing into that memory is not Python
objects but addresses of (a.k.a. pointers to) Python objects.

>
The PyObject elements will be freed *for us* eventually by garbage
collector.
Hence, we can't ever do 'free(FOO); ' because we don't know when
garbage collector will free FOO[0], FOO[1], FOO[2], etc.

Let's try reductio ad adsurdum on that one. Suppose that instead of
filling in a malloced chunk of memory, you had stored those gizmoids in
local variables foo0, foo1, foo2, etc. Using your reasoning: we can't
ever return from our function (which frees up the stack memory
containing foo0 etc) because we don't know when garbage collector will
free foo0 etc. Avoiding that problem would require a Python/C API
function with a name like Py_AwaitGarbage Collection() ... but there
isn't one.

HTH,
John

[seberino]

Let's try reductio ad adsurdum on that one. Suppose that instead of
filling in a malloced chunk of memory, you had stored those gizmoids in
local variables foo0, foo1, foo2, etc. Using your reasoning: we can't
ever return from our function (which frees up the stack memory
containing foo0 etc) because we don't know when garbage collector will
free foo0 etc. Avoiding that problem would require a Python/C API
function with a name like Py_AwaitGarbage Collection() ... but there
isn't one.

There are 2 memory components that make up a Python object.
First, there is the object in memory itself. Secondly, there
is the 4 bytes or so that hold the address to the object in memory.
(the 'pointer')

I think what I hear you saying is that garbage collection has
to automagically take care of my second component above as well.

So can I assume you are also saying our original malloc'd pointer
to Python objects will be take care of as well as the objects?
(Hence, no free(my_array) necessary?)
(Only Py_INCREFs and Py_DECREFs need apply?)

Chris

[seberino]

if you use malloc to allocate a memory block and store PyObject pointers
in it, you must

1) make sure that the reference count is *incremented* (Py_INCREF) when
you add the objects (unless the object is new; when you create an
object, the reference count is set to 1). this tells Python that you
have a pointer to an object that you plan to use some time in the future.

2) make sure that the reference count is *decremented* (Py_DECREF) if
you remove the objects (this tells Python that *you* won't access the
object any more; if nobody else uses it either, it can safely be removed).

OK, I'll read your 'required reading' I promise. Let me just make the
obvious
obvious for a minute. Your 'requirements' above only mention
Py_INCREFs and Py_DECREFs and never mention free(my_array).
Hence, I will assume the answer to my previous question is that I do
NOT
have to do free(my_array) because Py_INCREFs and Py_DECREFs will
allow garbage collection to take care of even the malloc'd part that
hold the pointers to the objects.

By the way, we are allowing garbage collector to free different
elements (pointers)
of my_array one at a time here. I just found out that curiously
enough, you CANNOT
do this in C with primitive types.

i.e. my_int_array = malloc(5 * sizeof(int));

You CANNOT do this...
free(&my_int_ar ray[1]);
free(&my_int_ar ray[2]);
free(&my_int_ar ray[3]);
...etc.
You can ONLY free the entire enchilada all at the same time...
free(my_int_arr ay);

Chris

[John Machin]

On 18/07/2006 1:45 PM, se******@spawar .navy.mil wrote:

>Let's try reductio ad adsurdum on that one. Suppose that instead of
filling in a malloced chunk of memory, you had stored those gizmoids in
local variables foo0, foo1, foo2, etc. Using your reasoning: we can't
ever return from our function (which frees up the stack memory
containing foo0 etc) because we don't know when garbage collector will
free foo0 etc. Avoiding that problem would require a Python/C API
function with a name like Py_AwaitGarbage Collection() ... but there
isn't one.

There are 2 memory components that make up a Python object.
First, there is the object in memory itself. Secondly, there
is the 4 bytes or so that hold the address to the object in memory.
(the 'pointer')

*WRONG*. The object exists in and of itself. There may be one *or more*
references to it, via pointers, scattered about in memory; they are
*NOT* components of the object. A reference count is maintained inside
the object and manipulated by Py_INCREF etc. The Python garbage
collector knows *nothing* about the memory occupied by those pointers;
it is *your* responsibility to allocate and free them. If the pointers
are in a function's local variables, that memory is automatically
returned when you return from the function. If the pointers are in a
chunk of memory that you grabbed using malloc(), then you must free()
the chunk as soon as you are finished with it.

Like I said:

You malloced it? You free it.
You didn't malloc it? You don't free it.

>
I think what I hear you saying is that garbage collection has
to automagically take care of my second component above as well.

I'm terribly sorry, but there is no magic involved with memory
management at this level; only hard, tedious, error-prone work.

Python garbage collection has neither the responsibility nor the
necessary information for carrying out that task. *YOU* have that
information; it is *YOUR* responsibility.

>
So can I assume you are also saying our original malloc'd pointer
to Python objects will be take care of as well as the objects?

*NO* You must free() any memory that you malloc()ed.

(Hence, no free(my_array) necessary?)

*NO* You must free() any memory that you malloc()ed.

(Only Py_INCREFs and Py_DECREFs need apply?)

*NO* You must free() any memory that you malloc()ed.

"What I tell you three times is true" -- the Bellman in "The Hunting of
the Snark", by Lewis Carroll.

HTH,
John