releasing memory to malloc

[iker.arizmendi]

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.

I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).

Regards,
Iker Arizmendi

[John Machin]

ik************@gmail.com wrote:

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.

I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).

Huh-uh. Other way up is better.

If you have access to the source of the extension, change it so that it
uses PyMem_Malloc, PyMem_Free, etc

Read this; it tells you why:

http://docs.python.org/api/memoryOverview.html

Cheers,
John

[Gabriel Genellina]

At Tuesday 26/9/2006 21:34, ik************@gmail.com wrote:

>Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.

I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).

Can you modify the C source? If you can, use the Python memory
allocation functions PyMem_Malloc/PyMem_Realloc/PyMem_Free.

Gabriel Genellina
Softlab SRL

__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[iker.arizmendi]

I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.

Iker

Gabriel Genellina wrote:

Can you modify the C source? If you can, use the Python memory
allocation functions PyMem_Malloc/PyMem_Realloc/PyMem_Free.

Gabriel Genellina
Softlab SRL

__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[John Machin]

ik************@gmail.com wrote:

I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.

So change the library to use xmalloc etc and add something like this to
the .h file:

#ifdef PYMEM
#define xmalloc PyMem_Malloc
etc
#else
#define xmalloc malloc
etc
#endif

[Gabriel Genellina]

At Tuesday 26/9/2006 22:17, ik************@gmail.com wrote:

>I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.

If you can recompile a specific version for using with Python, you
can play with a few macros like

#ifdef USE_PYTHON_ALLOCATOR
#define my_malloc(s) PyMem_Malloc(s)
#else
#define my_malloc(s) malloc(s)
#endif

and change all bare malloc/realloc/free along the code to
my_malloc/etc. (Chances are the code is using a special malloc
spelling already, C guys tend to do such things...)
Then you can recompile it for use with or without Python.

Gabriel Genellina
Softlab SRL

__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

[iker.arizmendi]

I happen to have the code for the C library in question, but I don't
think this is the way to go in general. If there's a way to get Python
to give memory back to the C allocator I can avoid touching the
library at all.

Regards,
Iker

John Machin wrote:

ik************@gmail.com wrote:

I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.

So change the library to use xmalloc etc and add something like this to
the .h file:

#ifdef PYMEM
#define xmalloc PyMem_Malloc
etc
#else
#define xmalloc malloc
etc
#endif

[Paul Rubin]

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
...

I happen to have the code for the C library in question, but I
don't think this is the way to go in general. If there's a way to
get Python to give memory back to the C allocator I can avoid
touching the library at all.

A cave-man approach might be to fork a new process after step 1, pass
the small data structure to it, and have the old process exit
(releasing all its memory back to the OS). The new process then
carries out the remaining steps.

[Bryan Olson]

Paul Rubin wrote:

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
...

I happen to have the code for the C library in question, but I
don't think this is the way to go in general. If there's a way to
get Python to give memory back to the C allocator I can avoid
touching the library at all.

A cave-man approach might be to fork a new process after step 1, pass
the small data structure to it, and have the old process exit
(releasing all its memory back to the OS). The new process then
carries out the remaining steps.

I think I see what you're doing, but fork() after step 1 will
create a child process with the same memory allocated.

I think it would make more sense to do step 1 in a subprocess.
Use the subprocess module or one of the older popen()s to create
a process that builds the target object, pickles it and pipes
it back to the main process, then exits.
--
--Bryan

[Paul Rubin]

Bryan Olson <fa*********@nowhere.orgwrites:

I think I see what you're doing, but fork() after step 1 will
create a child process with the same memory allocated.

I think it would make more sense to do step 1 in a subprocess.
Use the subprocess module or one of the older popen()s to create
a process that builds the target object, pickles it and pipes
it back to the main process, then exits.

Sorry, yes, that's what I meant. I shouldn't have used the word fork
without further qualification, which specifically means the child
process has all the same data. What's needed would traditionally have
been done by a fork followed by an exec, and popen would have done
something like that. These days I think there's some more streamlined
ways to do it.

[iker.arizmendi]

The workaround I've settled for uses the shelve module and calls
to gc.collect() to put a cap on the amount of memory the Python
allocator consumes. A bit more intrusive but it gets the job done.

Would a gc method that released memory to malloc be something
worth adding to Python? Or are there reasons why allowing this is
a bad idea?

Iker

ik************@gmail.com wrote:

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.

I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).

Regards,
Iker Arizmendi

[iker.arizmendi]

The workaround I went with made use of the shelve module and
calls to gc.collect() to cap the memory consumed by the Python
allocator. It was a bit intrusive but it got the job done.

Would a method in the gc module that released memory to malloc
be something that could get added to Python? Or are there some
reasons why allowing that would be a bad idea?

Regards,
Iker

P.S.
This may be a repeat of an earlier message - it seems that
google groups may have discarded my earlier post.
ik************@gmail.com wrote:

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.

I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).

Regards,
Iker Arizmendi

[MrJean1]

The memory manager in the latest Python release 2.5 does return freed
memory to the underlying system, if possible. For more details, see the
5th bullet on this page

<http://docs.python.org/whatsnew/ports.html>.

/Jean Brouwers

ik************@gmail.com wrote:

The workaround I went with made use of the shelve module and
calls to gc.collect() to cap the memory consumed by the Python
allocator. It was a bit intrusive but it got the job done.

Would a method in the gc module that released memory to malloc
be something that could get added to Python? Or are there some
reasons why allowing that would be a bad idea?

Regards,
Iker

P.S.
This may be a repeat of an earlier message - it seems that
google groups may have discarded my earlier post.
ik************@gmail.com wrote:

Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:

1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.

2) Releases all the Python objects.

3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.

I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).

Regards,
Iker Arizmendi