Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
2) Releases all the Python objects.
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).
Regards,
Iker Arizmendi
12  3163  ik************@ gmail.com wrote:
Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
2) Releases all the Python objects.
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).
Huh-uh. Other way up is better.
If you have access to the source of the extension, change it so that it
uses PyMem_Malloc, PyMem_Free, etc
Read this; it tells you why: http://docs.python.org/api/memoryOverview.html
Cheers,
John
At Tuesday 26/9/2006 21:34, ik************@ gmail.com wrote:
>Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
2) Releases all the Python objects.
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
I'd like step 2 to return memory to the C allocator so that it
is available to the extension in step 3 (which uses malloc).
Can you modify the C source? If you can, use the Python memory
allocation functions PyMem_Malloc/PyMem_Realloc/PyMem_Free.
Gabriel Genellina
Softlab SRL
_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya! http://www.yahoo.com.ar/respuestas
I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.
Iker
Gabriel Genellina wrote:
Can you modify the C source? If you can, use the Python memory
allocation functions PyMem_Malloc/PyMem_Realloc/PyMem_Free.
Gabriel Genellina
Softlab SRL
_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas ik************@ gmail.com wrote:
I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.
So change the library to use xmalloc etc and add something like this to
the .h file:
#ifdef PYMEM
#define xmalloc PyMem_Malloc
etc
#else
#define xmalloc malloc
etc
#endif
At Tuesday 26/9/2006 22:17, ik************@ gmail.com wrote:
>I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.
If you can recompile a specific version for using with Python, you
can play with a few macros like
#ifdef USE_PYTHON_ALLO CATOR
#define my_malloc(s) PyMem_Malloc(s)
#else
#define my_malloc(s) malloc(s)
#endif
and change all bare malloc/realloc/free along the code to
my_malloc/etc. (Chances are the code is using a special malloc
spelling already, C guys tend to do such things...)
Then you can recompile it for use with or without Python.
Gabriel Genellina
Softlab SRL
_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya! http://www.yahoo.com.ar/respuestas
I happen to have the code for the C library in question, but I don't
think this is the way to go in general. If there's a way to get Python
to give memory back to the C allocator I can avoid touching the
library at all.
Regards,
Iker
John Machin wrote:
ik************@ gmail.com wrote:
I can, but the extension is only a thin wrapper around a general
purpose C library which is also used independently of Python.
So change the library to use xmalloc etc and add something like this to
the .h file:
#ifdef PYMEM
#define xmalloc PyMem_Malloc
etc
#else
#define xmalloc malloc
etc
#endif
Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
2) Releases all the Python objects.
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
...
I happen to have the code for the C library in question, but I
don't think this is the way to go in general. If there's a way to
get Python to give memory back to the C allocator I can avoid
touching the library at all.
A cave-man approach might be to fork a new process after step 1, pass
the small data structure to it, and have the old process exit
(releasing all its memory back to the OS). The new process then
carries out the remaining steps.
Paul Rubin wrote:
Is there any way to get Python to release memory back to the
C allocator? I'm currently running a script that goes through
the following steps:
1) Creates a very large number of Python objects to produce
a relatively small data structure that sits in a C extension.
The Python objects consume quite a bit of memory.
2) Releases all the Python objects.
3) Invokes a function of said C extension for further
processing. This step needs as much memory as possible.
...
I happen to have the code for the C library in question, but I
don't think this is the way to go in general. If there's a way to
get Python to give memory back to the C allocator I can avoid
touching the library at all.
A cave-man approach might be to fork a new process after step 1, pass
the small data structure to it, and have the old process exit
(releasing all its memory back to the OS). The new process then
carries out the remaining steps.
I think I see what you're doing, but fork() after step 1 will
create a child process with the same memory allocated.
I think it would make more sense to do step 1 in a subprocess.
Use the subprocess module or one of the older popen()s to create
a process that builds the target object, pickles it and pipes
it back to the main process, then exits.
--
--Bryan
Bryan Olson <fa*********@no where.orgwrites :
I think I see what you're doing, but fork() after step 1 will
create a child process with the same memory allocated.
I think it would make more sense to do step 1 in a subprocess.
Use the subprocess module or one of the older popen()s to create
a process that builds the target object, pickles it and pipes
it back to the main process, then exits.
Sorry, yes, that's what I meant. I shouldn't have used the word fork
without further qualification, which specifically means the child
process has all the same data. What's needed would traditionally have
been done by a fork followed by an exec, and popen would have done
something like that. These days I think there's some more streamlined
ways to do it. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: lebo |
last post by:
Hi
I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.
So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
|
by: jimjim |
last post by:
Hello,
This is a simple question for you all, I guess .
int main(){
double *g= new double;
*g = 9;
delete g;
cout<< sizeof(g)<<" "<<sizeof(double)<<" "<<sizeof(*g)<<" "<<*g<<" "<<endl;
*g = 111;
cout<< sizeof(g)<<" "<<sizeof(double)<<" "<<sizeof(*g)<<" "<<*g<<" "<<endl;
|
by: lovecreatesbeauty |
last post by:
Hello experts,
1. Does C guarantee the data layout of the memory allocated by malloc
function on the heap. I mean, for example, if I allocate a array of 100
elements of structure, can I always reference a correct/valid structure
member upon that allocated memory?
If I allocate memory, for example like this way:
|
by: trialproduct2004 |
last post by:
Hi all
I am having application in c# where i am loading one table of database
into dataset.
My table is of large size. so whenever i am loading that into dataset
my memory size is getting increased.
So what i want is as soon as i am setting dataset to null, memory
should be released.
But when i am looking into task manager for memory useage what i am
|
by: mangesh |
last post by:
Is there any function in standard library that can forcefully release
memory associated with a process(application) ?
Regards
Mangesh .
| | |
by: Mike |
last post by:
Hi,
I am new to C and having problems with the following program.
Basically I am trying to read some files, loading data structures into
memory for latter searching.
I am trying to use structres and arrays of pointers to them.
I have gotten the program to compile with gcc on WinXP.
If the file i read doesnt have alot of records, it runs thru. But once
i add more, it dies. In this program i have 4 files setup to read. The
|
by: Peterwkc |
last post by:
Hello all expert, i have two program which make me desperate bu after i have noticed the forum, my future is become brightness back.
By the way, my problem is like this i the first program was compiled and run without any erros but the second program has a run time error when the function return from allocate and the ptr become NULL. How to fixed this?
Second Program:
/* Best Method to allocate memory for 2D Array because it's
...
|
by: jeungster |
last post by:
Hello,
I'm trying to track down a memory issue with a C++ application that
I'm working on:
In a nutshell, the resident memory usage of my program continues to
grow as the program runs. It starts off at a nice 4% of memory, then
slowly grows up to 50% and beyond. This translates to around 2 gigs
of physical memory, and that's really way more memory than this
program should be taking up.
|
by: skip |
last post by:
I am starting to experiment with ctypes. I have a function which returns a
pointer to a struct allocated in heap memory. There is a corresponding free
function for that sort of struct, e.g.:
from ctypes import *
cdll.LoadLibrary("libthing.so")
c_thing = CDLL("libthing.so")
class THING(Structure):
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| | |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
