releasing memory to malloc

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
>
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
>
2) Releases all the Python objects.
>
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
>
I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).
>

>Is there any way to get Python to release memory back to the
>C allocator? I'm currently running a script that goes through
>the following steps:
>
>1) Creates a very large number of Python objects to produce
>a relatively small data structure that sits in a C extension.
>The Python objects consume quite a bit of memory.
>
>2) Releases all the Python objects.
>
>3) Invokes a function of said C extension for further
>processing. This step needs as much memory as possible.
>
>I'd like step 2 to return memory to the C allocator so that it
>is available to the extension in step 3 (which uses malloc).

Can you modify the C source? If you can, use the Python memory
allocation functions PyMem_Malloc/PyMem_Realloc/PyMem_Free.
>
>
>
Gabriel Genellina
Softlab SRL
>
>
>
>
>
__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.
>

>I can, but the extension is only a thin wrapper around a general
>purpose C library which is also used independently of Python.

iker.arizmendi@gmail.com wrote:>
So change the library to use xmalloc etc and add something like this to
the .h file:
>
#ifdef PYMEM
#define xmalloc PyMem_Malloc
etc
#else
#define xmalloc malloc
etc
#endif

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
>
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
>
2) Releases all the Python objects.
>
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
...
>
I happen to have the code for the C library in question, but I
don't think this is the way to go in general. If there's a way to
get Python to give memory back to the C allocator I can avoid
touching the library at all.
>
A cave-man approach might be to fork a new process after step 1, pass
the small data structure to it, and have the old process exit
(releasing all its memory back to the OS). The new process then
carries out the remaining steps.

I think I see what you're doing, but fork() after step 1 will
create a child process with the same memory allocated.
>
I think it would make more sense to do step 1 in a subprocess.
Use the subprocess module or one of the older popen()s to create
a process that builds the target object, pickles it and pipes
it back to the main process, then exits.

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
>
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
>
2) Releases all the Python objects.
>
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
>
I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).
>
Regards,
Iker Arizmendi

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
>
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
>
2) Releases all the Python objects.
>
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
>
I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).
>
Regards,
Iker Arizmendi

The workaround I went with made use of the shelve module and
calls to gc.collect() to cap the memory consumed by the Python
allocator. It was a bit intrusive but it got the job done.
>
Would a method in the gc module that released memory to malloc
be something that could get added to Python? Or are there some
reasons why allowing that would be a bad idea?
>
Regards,
Iker
>
P.S.
This may be a repeat of an earlier message - it seems that
google groups may have discarded my earlier post.
>
>
iker.arizmendi@gmail.com wrote: