Useful? __slots__ for functions

[Nick Jacobson]

The __slots__ attribute of new-style classes can reduce memory usage
when there are millions of instantiations of a class.

So would a __slots__ attribute for functions/methods have a similar
benefit? i.e. could a function using __slots__ use significantly less
memory, and therefore run faster, if called millions of times?

If so, it will hopefully be in a future version of Python.

[Jean Brouwers]

[[ This message was both posted and mailed: see
the "To," "Cc," and "Newsgroups" headers for details. ]]
Nick,

I do not know the answer to your question, but the impact of __slots__
is very significant. Below is a message I posted a few weeks ago.

In addition to being faster (to create) and smaller, __slots__ objects
have another major advantage, at least in my view. They are 'frozen'
and can not be extended dynamically, not intentionally and -even more
importantly- not by accident. There are situations where the latter is
a major benefit, for example in applications where extensibility is not
a requirement.

/Jean Brouwers
ProphICy Semiconductor, Inc.

Path: attbi_s04!attbi_s03!attbi_s01!attbi_s02!attbi_slav e12!attbi_master11!wn14feed!worldnet.att.net!199.2 18.7.141!news.glorb.com!border1.nntp.dca.giganews. com!nntp.giganews.com!local1.nntp.dca.giganews.com !nntp.comcast.com!news.comcast.com.POSTED!not-for-mail
NNTP-Posting-Date: Wed, 12 May 2004 15:05:16 -0500
Subject: __slots__ vs __dict__
Date: Wed, 12 May 2004 13:14:48 -0700
From: Jean Brouwers <JB*******@ProphICy.com>
Newsgroups: comp.lang.python
Reply-To: JB*******@ProphICy.com
Message-ID: <120520041314481389%JB*******@ProphICy.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-transfer-encoding: 8bit
User-Agent: Thoth/1.7.2 (Carbon/OS X)
Lines: 209
NNTP-Posting-Host: 67.161.46.201
X-Trace: sv3-IgpWl13JKku68lEaAxfUeuMDddQ2Hb+wpcL1Xy3D1hWFRMKqKk QK8oYKWO6fJa6w0ibSPg2HtHvHitY!pbEuqBO6sUAMUZawbd1z xH3XYX1WmCBcg/PRch8EUhprgUWBOELz1iGE9uwWRe93JDJDgO8=
X-Complaints-To: ab***@comcast.net
X-DMCA-Complaints-To: dm**@comcast.net
X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
X-Postfilter: 1.1
Xref: attbi_master11 comp.lang.python:158864
X-Received-Date: Wed, 12 May 2004 20:05:16 GMT (attbi_s04)
Classes using __slots__ seem to be quite a bit smaller and faster
to instantiate than regular Python classes using __dict__.

Below are the results for the __slots__ and __dict__ version of a
specific class with 16 attributes. Each line in the tables shows the
number of instances created so far, the total memory usage in Bytes,
the CPU time in secs, the average size per instance in Bytes and the
average CPU time per instance in micseconds.

Instances of this particular class with __slots__ are almost 6x
smaller and nearly 3x faster to create than intances of the __dict__
version. Results for other classes will vary, obviously.

Comments?

/Jean Brouwers
ProphICy Semiconductor, Inc.
PS) The tests were run on a dual 2.4 GHz Xeon system with RedHat
8.0 and Python 2.3.2. The test script is attached but keep in mind
that it only has been tested on Linux. It will not work elsewhere
due to the implementation of the memory() function.
testing __slots__ version ...
4096 insts so far: 3.0e+05 B 0.030 sec 73.0 B/i 7.3 usec/i
8192 insts so far: 8.8e+05 B 0.070 sec 107.5 B/i 8.5 usec/i
16384 insts so far: 1.5e+06 B 0.150 sec 92.2 B/i 9.2 usec/i
32768 insts so far: 3.3e+06 B 0.280 sec 101.0 B/i 8.5 usec/i
65536 insts so far: 6.6e+06 B 0.560 sec 101.2 B/i 8.5 usec/i
131072 insts so far: 1.4e+07 B 1.200 sec 103.4 B/i 9.2 usec/i
262144 insts so far: 2.7e+07 B 2.480 sec 103.4 B/i 9.5 usec/i
524288 insts so far: 5.5e+07 B 5.630 sec 104.0 B/i 10.7 usec/i
1048576 insts so far: 1.1e+08 B 13.980 sec 104.0 B/i 13.3 usec/i
1050000 insts total: 1.1e+08 B 14.000 sec 103.9 B/i 13.3 usec/i
testing __dict__ version ...
4096 insts so far: 2.4e+06 B 0.050 sec 595.0 B/i 12.2 usec/i
8192 insts so far: 4.6e+06 B 0.090 sec 564.5 B/i 11.0 usec/i
16384 insts so far: 9.5e+06 B 0.180 sec 581.8 B/i 11.0 usec/i
32768 insts so far: 1.9e+07 B 0.370 sec 582.2 B/i 11.3 usec/i
65536 insts so far: 3.8e+07 B 0.830 sec 582.6 B/i 12.7 usec/i
131072 insts so far: 7.6e+07 B 1.760 sec 582.7 B/i 13.4 usec/i
262144 insts so far: 1.5e+08 B 4.510 sec 582.8 B/i 17.2 usec/i
524288 insts so far: 3.1e+08 B 12.820 sec 582.8 B/i 24.5 usec/i
1048576 insts so far: 6.1e+08 B 38.370 sec 583.1 B/i 36.6 usec/i
1050000 insts total: 6.1e+08 B 38.380 sec 583.1 B/i 36.6 usec/i
-------------------------------slots.py-------------------------------
<pre>

from time import clock as time_clock
def cputime(since=0.0):
'''Return CPU in secs.
'''
return time_clock() - since
import os
_proc_status = '/proc/%d/status' % os.getpid() # Linux only
_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0,
'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
global _scale
try: # get the /proc/<pid>/status pseudo file
t = open(_proc_status)
v = [v for v in t.readlines() if v.startswith(VmKey)]
t.close()
# convert Vm value to bytes
if len(v) == 1:
t = v[0].split() # e.g. 'VmRSS: 9999 kB'
if len(t) == 3: ## and t[0] == VmKey:
return float(t[1]) * _scale.get(t[2], 0.0)
except:
pass
return 0.0

def memory(since=0.0):
'''Return process memory usage in bytes.
'''
return _VmB('VmSize:') - since

def stacksize(since=0.0):
'''Return process stack size in bytes.
'''
return _VmB('VmStk:') - since

def slots(**kwds):
'''Return the slots names as sequence.
'''
return tuple(kwds.keys())

# __slots__ version
class SlotsClass(object):
__slots__ = slots(_attr1= False,
_attr2= None,
_attr3= None,
_attr4= None,
_attr5= None,
_attr6= None,
_attr7= 0,
_attr8= None,
_attr9= None,
_attr10=None,
_attr11=None,
_attr12=None,
_attr13=None,
_attr14=None,
_attr15=None,
_attr16=None)

def __init__(self, tuple4, parent):
self._attr1 = False
self._attr2 = None
self._attr3 = None
self._attr4 = None
self._attr5 = None
self._attr6 = None
if parent:
self._attr7 = parent._attr7 + 1
self._attr8 = parent._attr8
self._attr9 = parent._attr9
self._attr10 = parent
self._attr11 = parent._attr11
self._attr12 = parent._attr12
else:
self._attr7 = 0
self._attr8 = None
self._attr9 = None
self._attr10 = None
self._attr11 = self
self._attr12 = None
self._attr13, self._attr14, self._attr15, self._attr16 = tuple4
# __dict__ version
class DictClass(object):
_attr1 = None
_attr2 = None
_attr3 = None
_attr4 = None
_attr5 = None
_attr6 = None
_attr7 = 0
_attr8 = None
_attr9 = None
_attr10 = None
_attr11 = None
_attr12 = None
_attr13 = None
_attr14 = None
_attr15 = None
_attr16 = None

def __init__(self, tuple4, parent):
if parent:
self._attr7 = parent._attr7 + 1
self._attr8 = parent._attr8
self._attr9 = parent._attr9
self._attr10 = parent
self._attr11 = parent._attr11
self._attr12 = parent._attr12
else:
self._attr11 = self
self._attr13, self._attr14, self._attr15, self._attr16 = tuple4
if __name__ == '__main__':

import sys

def report(txt, n, b0, c0):
c = cputime(c0);
b = memory(b0)
print "%8d insts %s: %8.1e B %7.3f sec %6.1f B/i %6.1f usec/i" \
% (n, txt, b, c, b/n, 1.0e6*c/n)

if not sys.platform.startswith('linux'):
raise NotImplementedError, "%r not supported" % sys.platform

if 'dict' in sys.argv[1:]:
print 'testing __dict__ version ...'
testClass = DictClass
else:
print 'testing __slots__ version ...'
testClass = SlotsClass

t4 = ('', 0, 0, [])
b0 = memory()
c0 = cputime()
p = testClass(t4, None)
n, m = 1, 4096
# generate 1+ M instances
while n < 1050000: # 1048576:
p = testClass(t4, p)
n += 1
if n >= m: # occasionally print stats
m += m
report('so far', n, b0, c0)
report(' total', n, b0, c0)

</pre>

In article <f8**************************@posting.google.com >, Nick
Jacobson <ni***********@yahoo.com> wrote:
The __slots__ attribute of new-style classes can reduce memory usage
when there are millions of instantiations of a class.

So would a __slots__ attribute for functions/methods have a similar
benefit? i.e. could a function using __slots__ use significantly less
memory, and therefore run faster, if called millions of times?

If so, it will hopefully be in a future version of Python.

[Michele Simionato]

Jean Brouwers <mr*****@comcast.net> wrote in message news:<240520042236356939%mr*****@comcast.net>...

In addition to being faster (to create) and smaller, __slots__ objects
have another major advantage, at least in my view. They are 'frozen'
and can not be extended dynamically, not intentionally and -even more
importantly- not by accident. There are situations where the latter is
a major benefit, for example in applications where extensibility is not
a requirement.

Aarghh! This is a wart of __slots__, not an advantage. See
http://aspn.activestate.com/ASPN/Coo.../Recipe/252158
and google the newsgroup for __slots__.

Michele Simionato

[A. Lloyd Flanagan]

Jean Brouwers <mr*****@comcast.net> wrote in message news:<240520042236356939%mr*****@comcast.net>...
[...]

In addition to being faster (to create) and smaller, __slots__ objects
have another major advantage, at least in my view. They are 'frozen'
and can not be extended dynamically, not intentionally and -even more
importantly- not by accident. There are situations where the latter is
a major benefit, for example in applications where extensibility is not
a requirement.

/Jean Brouwers
ProphICy Semiconductor, Inc.

I would have to consider that a _major_ disadvantage!

There are a number of good reasons why users of a class might want to
add attributes dynamically. You can implement all sorts of useful
global mechanisms, like logging, by just adding them to classes or
instances.

Unless you've got a class that uses __slots__. Then all these schemes
fail miserably. This makes your class way less useful. I would never
use __slots__ except as a last resort for performance enhancement
(which is what it was added for).