473,395 Members | 1,941 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,395 software developers and data experts.

Memory Control - Can you get the memory usage of the interpreter?

Ian
Hi all,

I have a problem. I have an application which needs to work with a lot of
data, but not all at the same time. It is arranged as a set of objects, each
with lots of data that is created when the object is instantiated.

I'd ideally like to keep as many objects as possible in memory, but I can
get rid of any object the program isn't currently using.

Is there any way I can access the amount of memory python is using? I can
then decide when to give up objects to the gc.

I don't want to use weakref because the gc simply collects weakref'ed stuff
whether memory is tight or not, and then I have to recreate it (which is
costly). I only have one internal consumer for the data, and it is only
working with one object at once, but may change to a different object at any
time.

Thanks in advance

Ian.
Jul 18 '05 #1
3 2572

Assuming you are using Linux, below is an example for the memory usage
and stack size of the current process. See the man page for proc for
more details.

/Jean Brouwers
ProphICy Semiconductor, Inc.

<pre>

import os

_proc_status = '/proc/%d/status' % os.getpid() # Linux only?
_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
global _scale
try: # get the /proc/<pid>/status pseudo file
t = open(_proc_status)
v = [v for v in t.readlines() if v.startswith(VmKey)]
t.close()
# convert Vm value to bytes
if len(v) == 1:
t = v[0].split() # e.g. 'VmRSS: 9999 kB'
if len(t) == 3: ## and t[0] == VmKey:
return float(t[1]) * _scale.get(t[2], 0.0)
except:
pass
return 0.0

def memory(since=0.0):
'''Return process memory usage in bytes.
'''
return _VmB('VmSize:') - since

def stacksize(since=0.0):
'''Return process stack size in bytes.
'''
return _VmB('VmStk:') - since

</pre>
In article <40*********************@ptn-nntp-reader02.plus.net>, Ian
<in**@fretfarm.co.uk> wrote:
Hi all,

I have a problem. I have an application which needs to work with a lot of
data, but not all at the same time. It is arranged as a set of objects, each
with lots of data that is created when the object is instantiated.

I'd ideally like to keep as many objects as possible in memory, but I can
get rid of any object the program isn't currently using.

Is there any way I can access the amount of memory python is using? I can
then decide when to give up objects to the gc.

I don't want to use weakref because the gc simply collects weakref'ed stuff
whether memory is tight or not, and then I have to recreate it (which is
costly). I only have one internal consumer for the data, and it is only
working with one object at once, but may change to a different object at any
time.

Thanks in advance

Ian.

Jul 18 '05 #2
Ian
Thats great, thanks, and it seems to work.

Unfortunately, my client needs to run on windows and mac (osX) too. I'm
presuming osX is similar to linux: I'll need to have a play to see if I can
get it working.

Is there a windows friendly way to do it?

Ian.

"Jean Brouwers" <JB***********************@no.spam.net> wrote in message
news:090720041815251751%JB***********************@ no.spam.net...

Assuming you are using Linux, below is an example for the memory usage
and stack size of the current process. See the man page for proc for
more details.

/Jean Brouwers
ProphICy Semiconductor, Inc.

<pre>

import os

_proc_status = '/proc/%d/status' % os.getpid() # Linux only?
_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
global _scale
try: # get the /proc/<pid>/status pseudo file
t = open(_proc_status)
v = [v for v in t.readlines() if v.startswith(VmKey)]
t.close()
# convert Vm value to bytes
if len(v) == 1:
t = v[0].split() # e.g. 'VmRSS: 9999 kB'
if len(t) == 3: ## and t[0] == VmKey:
return float(t[1]) * _scale.get(t[2], 0.0)
except:
pass
return 0.0

def memory(since=0.0):
'''Return process memory usage in bytes.
'''
return _VmB('VmSize:') - since

def stacksize(since=0.0):
'''Return process stack size in bytes.
'''
return _VmB('VmStk:') - since

</pre>
In article <40*********************@ptn-nntp-reader02.plus.net>, Ian
<in**@fretfarm.co.uk> wrote:
Hi all,

I have a problem. I have an application which needs to work with a lot of data, but not all at the same time. It is arranged as a set of objects, each with lots of data that is created when the object is instantiated.

I'd ideally like to keep as many objects as possible in memory, but I can get rid of any object the program isn't currently using.

Is there any way I can access the amount of memory python is using? I can then decide when to give up objects to the gc.

I don't want to use weakref because the gc simply collects weakref'ed stuff whether memory is tight or not, and then I have to recreate it (which is
costly). I only have one internal consumer for the data, and it is only
working with one object at once, but may change to a different object at any time.

Thanks in advance

Ian.

Jul 18 '05 #3

Here is a slighly better version which avoids breaking the pseudo file
up into lines and parsing all lines. But still Linux only.

/Jean Brouwers
PropICy Semiconductor, Inc.

<pre>

import os

_proc_status = '/proc/%d/status' % os.getpid() # Linux only?

_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
'''Private.
'''
global _proc_status, _scale
# get pseudo file /proc/<pid>/status
try:
t = open(_proc_status)
v = t.read()
t.close()
except:
return 0.0 # non-Linux?
# get VmKey line e.g. 'VmRSS: 9999 kB\n ...'
t = v.index(VmKey)
v = v[t:].split(None, 3) # whitespace
if len(v) < 3:
return 0.0 # invalid format?
# convert Vm value to bytes
return float(v[1]) * _scale[v[2]]

def memory(since=0.0):
'''Return memory usage in bytes.
'''
return _VmB('VmSize:') - since

def resident(since=0.0):
'''Return resident memory usage in bytes.
'''
return _VmB('VmRSS:') - since

def stacksize(since=0.0):
'''Return stack size in bytes.
'''
return _VmB('VmStk:') - since

</pre>
In article <09******************************************@no.s pam.net>,
Jean Brouwers <JB***********************@no.spam.net> wrote:
Assuming you are using Linux, below is an example for the memory usage
and stack size of the current process. See the man page for proc for
more details.

/Jean Brouwers
ProphICy Semiconductor, Inc.

<pre>

import os

_proc_status = '/proc/%d/status' % os.getpid() # Linux only?
_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
global _scale
try: # get the /proc/<pid>/status pseudo file
t = open(_proc_status)
v = [v for v in t.readlines() if v.startswith(VmKey)]
t.close()
# convert Vm value to bytes
if len(v) == 1:
t = v[0].split() # e.g. 'VmRSS: 9999 kB'
if len(t) == 3: ## and t[0] == VmKey:
return float(t[1]) * _scale.get(t[2], 0.0)
except:
pass
return 0.0

def memory(since=0.0):
'''Return process memory usage in bytes.
'''
return _VmB('VmSize:') - since

def stacksize(since=0.0):
'''Return process stack size in bytes.
'''
return _VmB('VmStk:') - since

</pre>
In article <40*********************@ptn-nntp-reader02.plus.net>, Ian
<in**@fretfarm.co.uk> wrote:
Hi all,

I have a problem. I have an application which needs to work with a lot of
data, but not all at the same time. It is arranged as a set of objects, each
with lots of data that is created when the object is instantiated.

I'd ideally like to keep as many objects as possible in memory, but I can
get rid of any object the program isn't currently using.

Is there any way I can access the amount of memory python is using? I can
then decide when to give up objects to the gc.

I don't want to use weakref because the gc simply collects weakref'ed stuff
whether memory is tight or not, and then I have to recreate it (which is
costly). I only have one internal consumer for the data, and it is only
working with one object at once, but may change to a different object at any
time.

Thanks in advance

Ian.

Jul 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

21
by: Glen Wheeler | last post by:
Hello all, My program uses many millions of integers, and Python is allocating way too much memory for these. I can't have the performance hit by using disk, so I figured I'd write a C...
4
by: MSUTech | last post by:
Hello All, I am getting an ASP error.. that tells the server it is OUT OF MEMORY.. then the server displays a message about the DLLHOST.exe and it waits for an "OK" from me... after hitting...
3
by: Robert | last post by:
Hi all, I am writing programs of algorithms using C++ and C. When I run the programs, I found the programs take 100 percent of the CPU usage and I cannot run any other program. How to control...
5
by: Sharon | last post by:
I’m writing a Windows application. In the form I have a Panel and inside the panel I have a PictureBox control. I’m loading the PictureBox control with BMP image that has the following...
1
by: hape | last post by:
Hi all, i am looking for a way to get measurements for cpu and memory usage for ..net programs. The measurements have to be taken programmatically (monitoring). Thats why i cannot use tools...
1
by: Daniel Mark | last post by:
Hello all: I have a python program and would like to find out the maximum memory used by this program. Does Python provide such module so I could check it easily?
0
by: dariophoenix | last post by:
Does anyboy know if there is a way, in C++, to know how much free space left is there in the heap? (During runtime)
4
by: Tomassus | last post by:
Hi there, I have a problem with dynamic memory allocation. I know that it would have been easier to use vectors methods, but i want to know what i do here wrong. This is one of my methods in...
2
by: =?Utf-8?B?SXJmYW4=?= | last post by:
Hello, It may be a repeated question but I don't find the solution to the situation that I encounter in it. My application is monitoring another application that is built in VB6. The...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.