473,287 Members | 1,463 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

Why so many references to global variables?

J-P

Hi,

I have a Python script interacting with a specialized
C numerical library. The whole program does quite a lot
of number crunching as should be running for a couple
of hours. However, it always seems to run out of memory
after maybe 40-45 minutes (I get a MemoryError from
the python interpreter). I wanted to see what took
that much memory so I printed a reference count
(using sys.getrefcount()) and I was suprised to
see that a global variable called 'myflag' has litteraly
millions of references to it. myflag appears maybe
8 or 10 times in the script, always something like

if myflag:
#do something
else:
#do something else

It appears that every time the interpreter tests for
the value of 'myflag', it keeps a reference to it.
I don't know whether this has something to do with
the garbage collector not doing its job correctly
of me doing something wrong in the code, but I'd really
like to fix this thing.

Any ideas, suggestions or comments greatly appreciated,
as always.

Thanks in advance,
J-P

Jul 18 '05 #1
4 2565
J-P
Alexander Schmolck wrote:
J-P <me@home.net> writes:

It appears that every time the interpreter tests for
the value of 'myflag', it keeps a reference to it.

What makes you think so? This seems rather unlikely to me (not that the
references themselves should eat your memory anyway!).


Well, sys.getrefcount() does tell me there are 3 millions references
to it and other globals. Even though this doesn't eat up all my memory
(how large is a reference object in Python?), I definitely think there's
something fishy with keeping that many references to global variables
that appear here and there in the script.
Chances are, the C extension code doesn't work correctly (C extensions to
python code have to do memory management by hand; increasing and decreasing
reference counts for the python objects they deal with as appropriate; so if a
bit of code forgets to decrease the refcount, the object will stay alive
forever; my guess would be that this it what happens here).


Might be, but the ref count for the objects interacting with the C
library are pretty much what I expect them to be, i.e. a few dozens.
I don't think there are other memory leaks in the bindings to the
library. I've passed it through Purify a couple of times and everything
seems clean.
J-P

Jul 18 '05 #2
J-P wrote:
Well, sys.getrefcount() does tell me there are 3 millions references
to it and other globals. Even though this doesn't eat up all my memory
(how large is a reference object in Python?), ...
Reference counts are just maintained internally with a single number
that's incremented or decremented.
I definitely think
there's
something fishy with keeping that many references to global variables
that appear here and there in the script.


Yes, it does. It strongly suggests that the fishiness is in your C
extension.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Now I must follow them!
\__/ Beowulf, King of the Geats
Jul 18 '05 #3
J-P <me@home.net> writes:
Alexander Schmolck wrote:
J-P <me@home.net> writes:
It appears that every time the interpreter tests for
the value of 'myflag', it keeps a reference to it.

What makes you think so? This seems rather unlikely to me (not that the

references themselves should eat your memory anyway!).

Well, sys.getrefcount() does tell me there are 3 millions references
to it and other globals. Even though this doesn't eat up all my memory
(how large is a reference object in Python?), I definitely think there's
something fishy with keeping that many references to global variables that
appear here and there in the script.

Erik Max Francis has hopefully already sorted your reference count confusion
out (if not, maybe a look under "reference counts" in the C API/extending bits
of the python docu might clarify matters), so I'll just give you a simple
practical tip:

Take one of the suspicious C extension functions, and call it repeatedly from
python (passing and returning data structures that are as large as possible
and that you discard immediately aftewerwards). Then using 'top' or something
equivalent, look at how the memory consumption of your program changes: if you
find that with each couple of calls python swallows a few megabytes, you can
be pretty sure that something is going wrong (at least if you force gc with
gc.collect()).

Once you've isolated the function(s) you're in for some fun debugging the
corresponding C code, paying particular attention to PY_DECREFs and
PY_INCREFs.

Chances are, the C extension code doesn't work correctly (C extensions to
python code have to do memory management by hand; increasing and decreasing
reference counts for the python objects they deal with as appropriate; so if a
bit of code forgets to decrease the refcount, the object will stay alive
forever; my guess would be that this it what happens here).


Might be, but the ref count for the objects interacting with the C library are
pretty much what I expect them to be, i.e. a few dozens.

I don't think there are other memory leaks in the bindings to the
library. I've passed it through Purify a couple of times and everything
seems clean.


Purify is unlikely to have a deep understanding of python's internal reference
counting (memory management) mechanism, right? So while it will be helpful for
finding memory *violations* (and leaks not due to refcounts) it's quite
unlikely to find problems due to the C extension not *decreasing reference
counts*, which is what I bet is happening in your case.

'as
Jul 18 '05 #4
J-P
Thank you both for the advice, I appreciate it!

J-P

Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

37
by: Dave | last post by:
Hello all, Please consider the code below. It is representative of a problem I am having. foo_t needs to contain a bar_t which is a class without a copy constructor or operator=. It is not...
4
by: Mark D. Anderson | last post by:
About a month ago Richard Cornford did an interesting analysis of a memory leak in jscript (internet explorer) when there are "circular" references between DOM objects and (real) jscript objects:...
25
by: Steve Jorgensen | last post by:
Yup, Steve's full of tips, but hey, it makes him feel important, right? Ok, here goes. I've been trying to improve encapsulation by putting code in the same object as the stuff it affects, so I...
33
by: MLH | last post by:
I've read some posts indicating that having tons of GV's in an Access app is a bad idea. Personally, I love GVs and I use them (possibly abuse them) all the time for everything imaginable - have...
6
by: bytecolor | last post by:
I'm working on a simple graphics package. I've got a function show() that the user needs to call at the end of the script to actually display the points, lines and circles that have been defined in...
9
by: CDMAPoster | last post by:
About a year ago there was a thread about the use of global variables in A97: http://groups.google.com/group/comp.databases.ms-access/browse_frm/thread/fedc837a5aeb6157 Best Practices by Kang...
9
by: igor.kulkin | last post by:
References is a relatively basic feature of C++ language. It might be a good thing to think of references as aliases to the variables. However it's good to think of references this way when you...
1
weaknessforcats
by: weaknessforcats | last post by:
C++: The Case Against Global Variables Summary This article explores the negative ramifications of using global variables. The use of global variables is such a problem that C++ architects have...
2
by: Rahul | last post by:
Hi Everyone, I was wondering if references to a function defined in another file is resolved at linker time, why isn't the same extended for global variables? I mean, without the extern keyword...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.