Hello,
I have programmed some python script that loads a graph (the
mathemical one with vertices and edges) into memory, does some
transformations on it, and then tries to find shortest paths in this
graph, typically several tens of thousands. This works fine.
Then I made a test for this, so I could time it, run it several times
and take a look at the best time, et cetera. But it so happens that
the first time the test is run, is always the fastest. If I track
memory usage of Python in top, I see it starts out with around 80 MB
and slowly grows to 500MB. This might cause the slowdown (which is
about a factor 5 for large graphs).
When I run a test, I disable the garbage collection during the test
run (as is adviced), but just before starting a test I instruct the
garbage collector to collect. Running the test without disabling the
garbage collect doesn't show any difference though.
Where possible I explicitly 'del' some of the larger data structures
that have been created after I don't need them anymore. I furthermore
don't really see why there would be references to these larger objects
left. (I can be mistaken of course).
I understand this might be a bit of a vague problem, but does someone
have any idea why the memory usage keeps growing? And whether there is
some tool that assists me in keeping track of the objects currently
alive and the amount of memory they occupy?
The best I now can do is run the whole script several times (from a
shell script) -- but this also forces Python to reparse the graph
input again, and do some other stuff it only has to do once. And it's
also more difficult to examine values and results this way.
Berteun
3  1663 
Berteun Damman wrote:
When I run a test, I disable the garbage collection during the
test run (as is adviced), but just before starting a test I
instruct the garbage collector to collect. Running the test
without disabling the garbage collect doesn't show any difference
though.
Did you check the return value of gc.collect? Also, try using
other "insight" facilities provided by the gc module.
Where possible I explicitly 'del' some of the larger data
structures that have been created after I don't need them anymore.
You cannot "del" structures, you only "del" names. Objects are
deleted when they are not bound to any names when and if the
garbage collector "wants" to delete them.
I furthermore don't really see why there would be references to
these larger objects left. (I can be mistaken of course).
Be sure to check for cyclic references, they can be a problem for
the GC.
Regards,
Björn
--
BOFH excuse #343:
The ATM board has run out of 10 pound notes. We are having a whip
round to refill it, care to contribute ?
On Sep 26, 2:31 pm, Bjoern Schliessmann <usenet-
mail-0306.20.chr0n.. .@spamgourmet.c omwrote:
Did you check the return value of gc.collect? Also, try using
other "insight" facilities provided by the gc module.
gc.collect states it cannot find any unreachable objects. Meanwhile
the number of objects the garbage collector has to keep track of keeps
increasing.
You cannot "del" structures, you only "del" names. Objects are
deleted when they are not bound to any names when and if the
garbage collector "wants" to delete them.
I understand, but just before I del the name, I ask for the refererres
to the object the name indicates, and there's only one object. Since
it is a local variable, I think this is logical.
This object is a dictionary which contains strings as keys, and heaps
as values. This heap consists of tuples. Every string is referenced
more than once (that's logical), the heaps are only referenced once.
So I would expect those to be destroyed if I destroy the dictionary. I
furthermore assume that if I call gc.collect() I force the garbage
collector to collect? Even if it wouldn't "want" to collect
otherwise?
Be sure to check for cyclic references, they can be a problem for
the GC.
I don't see how these could occur. It's basically something like list
(of lists possibly) of ints/strings. No list containing itself. I'll
see whether I can make a stripped down version which exhibits the same
memory growth.
Berteun
On Sep 26, 8:06 am, Berteun Damman <bert...@gmail. comwrote:
that have been created after I don't need them anymore. I furthermore
don't really see why there would be references to these larger objects
left. (I can be mistaken of course).
This could be tricky because you have a graph that (probably) allows
you to walk its nodes, thus even having a single other reference to
any of the nodes could keep the entire graph "alive"
The best I now can do is run the whole script several times (from a
shell script) -- but this also forces Python to reparse the graph
input again, and do some other stuff it only has to do once. A
you could pickle and save the graph once the initial processing is
done. That way subsequent runs will load substantially faster.
i. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics | |
by: lawrence |
last post by:
I'm not sure how this is normally done, on a large site, perhaps one
running Phorum. Occassionally a thread will have hundreds of entries,
perhaps a meg or two worth of data. You won't necessarily print all
that to the screen, but PHP has to hold it in memory. Run some
operations on it, or, more likely, have an array that you keep adding
things to, and very soon you run into the 8 meg limit that is the
default limit for PHP scripts.
...
|
by: Ian |
last post by:
Hi all,
I have a problem. I have an application which needs to work with a lot of
data, but not all at the same time. It is arranged as a set of objects, each
with lots of data that is created when the object is instantiated.
I'd ideally like to keep as many objects as possible in memory, but I can
get rid of any object the program isn't currently using.
Is there any way I can access the amount of memory python is using? I can
|
by: Ian Taite |
last post by:
Hello,
I'm exploring why one of my C# .NET apps has "high" memory usage, and
whether I can reduce the memory usage.
I have an app that wakes up and processes text files into a database
periodically. What happens, is that the app reads the contents of a
text file line by line into an ArrayList. Each element of the ArrayList
is a string representing a record from the file. The ArrayList is then
processed, and the arraylist goes out of...
|
by: Ganesan selvaraj |
last post by:
I using C# .net. i want to split the text files based of the some condition.
my source text file size may be 4 kb to 4 gb. some time when i split the
i got the "out of memory exception. when i read from the file i read the
10mb and split the this content and clear the buffer and read the second 10
mb and so on.
How can i rectify the problem?
|
by: Clement |
last post by:
hi,
i have an asp.net site and is using SQL Server 2k. i realize the
aspnet_wp.exe memory usage keep growing and i will receive an error for the
pages that call the sql connection. others page with no sql connection is
fined.
At the time when i encounter the error, i check the memory usage for
aspnet_wp.exe = 60000kb
sqlmangr.exe = 40000kb
| | |
by: Alex Martelli |
last post by:
Having fixed a memory leak (not the leak of a Python reference, some
other stuff I wasn't properly freeing in certain cases) in a C-coded
extension I maintain, I need a way to test that the leak is indeed
fixed. Being in a hurry, I originally used a q&d hack...:
if sys.platform in ('linux2', 'darwin'):
def _memsize():
""" this function tries to return a measurement of how much memory
this process is consuming, in some arbitrary unit...
|
by: Gawain Lavers |
last post by:
I have a script which uses a lot of object augmentation (in order to
extend the functionality of DOM elements), and I'm clearly stressing
out my browser (I rapidly get to a point where Firefox is telling me
that the script is "unresponsive", often more than once. I'm not sure
if it's simply execution time or memory usage.
My question is this: if I augment an object thusly:
function TestAug() {
this.fun = function() {
|
by: Frank Rizzo |
last post by:
I have an object tree that is pretty gigantic and it holds about 100mb
of data. When I set the top object to null, I expect that the .NET
framework will clean up the memory at some point. However, I am looking
at the Task Manager and I don't see the MemUsage column decreasing even
after an hour or two. I know that TaskManager may not be the best place
to see what is the true way to gauge memory usage and/or presense of
memory leaks. So...
|
by: Mohitz |
last post by:
Hi,
Need views on the following..
I need to design a data structure to store the values of n attributes,
where n is something near 20.
Now, all the attributes are optional. So, at times, i may have to
store the values of all n of them and
at times not.
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| | |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| | |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |
