Hi,
I have a Python script interacting with a specialized
C numerical library. The whole program does quite a lot
of number crunching as should be running for a couple
of hours. However, it always seems to run out of memory
after maybe 40-45 minutes (I get a MemoryError from
the python interpreter). I wanted to see what took
that much memory so I printed a reference count
(using sys.getrefcount()) and I was suprised to
see that a global variable called 'myflag' has litteraly
millions of references to it. myflag appears maybe
8 or 10 times in the script, always something like
if myflag:
#do something
else:
#do something else
It appears that every time the interpreter tests for
the value of 'myflag', it keeps a reference to it.
I don't know whether this has something to do with
the garbage collector not doing its job correctly
of me doing something wrong in the code, but I'd really
like to fix this thing.
Any ideas, suggestions or comments greatly appreciated,
as always.
Thanks in advance,
J-P 4 2565
Alexander Schmolck wrote: J-P <me@home.net> writes:
It appears that every time the interpreter tests for the value of 'myflag', it keeps a reference to it.
What makes you think so? This seems rather unlikely to me (not that the references themselves should eat your memory anyway!).
Well, sys.getrefcount() does tell me there are 3 millions references
to it and other globals. Even though this doesn't eat up all my memory
(how large is a reference object in Python?), I definitely think there's
something fishy with keeping that many references to global variables
that appear here and there in the script.
Chances are, the C extension code doesn't work correctly (C extensions to python code have to do memory management by hand; increasing and decreasing reference counts for the python objects they deal with as appropriate; so if a bit of code forgets to decrease the refcount, the object will stay alive forever; my guess would be that this it what happens here).
Might be, but the ref count for the objects interacting with the C
library are pretty much what I expect them to be, i.e. a few dozens.
I don't think there are other memory leaks in the bindings to the
library. I've passed it through Purify a couple of times and everything
seems clean.
J-P
J-P wrote: Well, sys.getrefcount() does tell me there are 3 millions references to it and other globals. Even though this doesn't eat up all my memory (how large is a reference object in Python?), ...
Reference counts are just maintained internally with a single number
that's incremented or decremented.
I definitely think there's something fishy with keeping that many references to global variables that appear here and there in the script.
Yes, it does. It strongly suggests that the fishiness is in your C
extension.
--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Now I must follow them!
\__/ Beowulf, King of the Geats
J-P <me@home.net> writes: Alexander Schmolck wrote: J-P <me@home.net> writes:
It appears that every time the interpreter tests for the value of 'myflag', it keeps a reference to it. What makes you think so? This seems rather unlikely to me (not that the
references themselves should eat your memory anyway!).
Well, sys.getrefcount() does tell me there are 3 millions references to it and other globals. Even though this doesn't eat up all my memory (how large is a reference object in Python?), I definitely think there's something fishy with keeping that many references to global variables that appear here and there in the script.
Erik Max Francis has hopefully already sorted your reference count confusion
out (if not, maybe a look under "reference counts" in the C API/extending bits
of the python docu might clarify matters), so I'll just give you a simple
practical tip:
Take one of the suspicious C extension functions, and call it repeatedly from
python (passing and returning data structures that are as large as possible
and that you discard immediately aftewerwards). Then using 'top' or something
equivalent, look at how the memory consumption of your program changes: if you
find that with each couple of calls python swallows a few megabytes, you can
be pretty sure that something is going wrong (at least if you force gc with
gc.collect()).
Once you've isolated the function(s) you're in for some fun debugging the
corresponding C code, paying particular attention to PY_DECREFs and
PY_INCREFs. Chances are, the C extension code doesn't work correctly (C extensions to python code have to do memory management by hand; increasing and decreasing reference counts for the python objects they deal with as appropriate; so if a bit of code forgets to decrease the refcount, the object will stay alive forever; my guess would be that this it what happens here).
Might be, but the ref count for the objects interacting with the C library are pretty much what I expect them to be, i.e. a few dozens.
I don't think there are other memory leaks in the bindings to the library. I've passed it through Purify a couple of times and everything seems clean.
Purify is unlikely to have a deep understanding of python's internal reference
counting (memory management) mechanism, right? So while it will be helpful for
finding memory *violations* (and leaks not due to refcounts) it's quite
unlikely to find problems due to the C extension not *decreasing reference
counts*, which is what I bet is happening in your case.
'as
Thank you both for the advice, I appreciate it!
J-P This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Dave |
last post by:
Hello all,
Please consider the code below. It is representative of a problem I am
having.
foo_t needs to contain a bar_t which is a class without a copy constructor
or operator=. It is not...
|
by: Mark D. Anderson |
last post by:
About a month ago Richard Cornford did an interesting analysis of a
memory leak
in jscript (internet explorer) when there are "circular" references
between
DOM objects and (real) jscript objects:...
|
by: Steve Jorgensen |
last post by:
Yup, Steve's full of tips, but hey, it makes him feel important, right?
Ok, here goes. I've been trying to improve encapsulation by putting code
in the same object as the stuff it affects, so I...
|
by: MLH |
last post by:
I've read some posts indicating that having tons of GV's in
an Access app is a bad idea. Personally, I love GVs and I
use them (possibly abuse them) all the time for everything
imaginable - have...
|
by: bytecolor |
last post by:
I'm working on a simple graphics package. I've got a function show()
that the user needs to call at the end of the script to actually
display the points, lines and circles that have been defined in...
|
by: CDMAPoster |
last post by:
About a year ago there was a thread about the use of global variables
in A97:
http://groups.google.com/group/comp.databases.ms-access/browse_frm/thread/fedc837a5aeb6157
Best Practices by Kang...
|
by: igor.kulkin |
last post by:
References is a relatively basic feature of C++ language.
It might be a good thing to think of references as aliases to the
variables.
However it's good to think of references this way when you...
|
by: weaknessforcats |
last post by:
C++: The Case Against Global Variables
Summary
This article explores the negative ramifications of using global variables. The use of global variables is such a problem that C++ architects have...
|
by: Rahul |
last post by:
Hi Everyone,
I was wondering if references to a function defined in another file
is resolved at linker time, why isn't the same extended for global
variables? I mean, without the extern keyword...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM).
In this month's session, the creator of the excellent VBE...
|
by: DolphinDB |
last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation.
Take...
|
by: Aftab Ahmad |
last post by:
Hello Experts!
I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
|
by: ryjfgjl |
last post by:
ExcelToDatabase: batch import excel into database automatically...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: marcoviolo |
last post by:
Dear all,
I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: jfyes |
last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
|
by: ArrayDB |
last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
| |