Problem of Readability of Python

[Licheng Fang]

Python is supposed to be readable, but after programming in Python for
a while I find my Python programs can be more obfuscated than their C/C
++ counterparts sometimes. Part of the reason is that with
heterogeneous lists/tuples at hand, I tend to stuff many things into
the list and *assume* a structure of the list or tuple, instead of
declaring them explicitly as one will do with C structs. So, what used
to be

struct nameval {
char * name;
int val;
} a;

a.name = ...
a.val = ...

becomes cryptic

a[0] = ...
a[1] = ...

Python Tutorial says an empty class can be used to do this. But if
namespaces are implemented as dicts, wouldn't it incur much overhead
if one defines empty classes as such for some very frequently used
data structures of the program?

Any elegant solutions?

[John Machin]

On 8/10/2007 4:14 AM, Steven Bethard wrote:

Licheng Fang wrote:

>Python is supposed to be readable, but after programming in Python for
a while I find my Python programs can be more obfuscated than their C/C
++ counterparts sometimes. Part of the reason is that with
heterogeneou s lists/tuples at hand, I tend to stuff many things into
the list and *assume* a structure of the list or tuple, instead of
declaring them explicitly as one will do with C structs. So, what used
to be

struct nameval {
char * name;
int val;
} a;

a.name = ...
a.val = ...

becomes cryptic

a[0] = ...
a[1] = ...

Python Tutorial says an empty class can be used to do this. But if
namespaces are implemented as dicts, wouldn't it incur much overhead
if one defines empty classes as such for some very frequently used
data structures of the program?

Any elegant solutions?

You can use __slots__ to make objects consume less memory and have
slightly better attribute-access performance. Classes for objects that
need such performance tweaks should start like::

class A(object):
__slots__ = 'name', 'val'

The recipe below fills in the obvious __init__ method for such classes
so that the above is pretty much all you need to write:

http://aspn.activestate.com/ASPN/Coo.../Recipe/502237

If not needing/wanting __slots__, something simpler (no metaclasses!)
like the following helps the legibility/utility:

<file record.py>

class BaseRecord(obje ct):

def __init__(self, **kwargs):
for k, v in kwargs.iteritem s():
setattr(self, k, v)

def dump(self, text=''):
print '== dumping %s instance: %s' % (self.__class__ .__name__,
text)
for k, v in sorted(self.__d ict__.iteritems ()):
print ' %s: %r' % (k, v)
</file record.py>
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.

>>from record import BaseRecord
class A(BaseRecord):

.... pass
....

>>class B(BaseRecord):

.... pass
....

>>a1 = A(foo=1, bar='rab', zot=(1, 2))
a2 = A(foo=2, bar='xxx', zot=(42, 666))
a1.dump()

== dumping A instance:
bar: 'rab'
foo: 1
zot: (1, 2)

>>a2.dump('mo re of the same')

== dumping A instance: more of the same
bar: 'xxx'
foo: 2
zot: (42, 666)

>>a1.ugh = 'poked in'
a1.dump('afte r poking')

== dumping A instance: after poking
bar: 'rab'
foo: 1
ugh: 'poked in'
zot: (1, 2)

>>b1 = B()
b1.dump()

== dumping B instance:

>>b1.esrever = 'esrever'[::-1]
b1.dump()

== dumping B instance:
esrever: 'reverse'

>>>

HTH,
John

[Aahz]

In article <2t************ *************** ***@comcast.com >,
Steven Bethard <st************ @gmail.comwrote :

>
You can use __slots__ [...]

Aaaugh! Don't use __slots__!

Seriously, __slots__ are for wizards writing applications with huuuge
numbers of object instances (like, millions of instances). For an
extended thread about this, see

http://groups.google.com/group/comp....25f368e23ab058
--
Aahz (aa**@pythoncra ft.com) <* http://www.pythoncraft.com/

The best way to get information on Usenet is not to ask a question, but
to post the wrong information.

[Michele Simionato]

On Oct 8, 12:27 am, a...@pythoncraf t.com (Aahz) wrote:

>
Aaaugh! Don't use __slots__!

+1 QOTW ;)

Michele Simionato

[Michele Simionato]

On Oct 7, 7:58 pm, [e-mail] (Alex Martelli) wrote:

If you REALLY pine for Pascal's records, you might choose to inherit
from ctypes.Structur e, which has the additional "advantages " of
specifying a C type for each field and (a real advantage;-) creating an
appropriate __init__ method.

>import ctypes
class Record(ctypes.S tructure):

... _fields_ =
(('x',ctypes.c_ float),('y',cty pes.c_float),(' z',ctypes.c_flo at)
)
...>>r=Record()

>r.x

0.0

>r=Record(1,2,3 )
r.x

1.0

>r=Record('zip' ,'zop','zap')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: float expected instead of str instance

See? You get type-checking too -- Pascal looms closer and closer!-)

And if you need an array of 1000 such Records, just use as the type
Record*1000 -- think of the savings in memory (no indirectness, no
overallocations as lists may have...).

That's very cool Alex! I have just a question: suppose I want
to measure the memory allocation of a million of records made
with ctypes vs the memory allocation of equivalent
records made with __slots__, how do I measure it? Say on Linux,
Mac and Windows?
If ctypes records are more efficient than __slots__ records,
I will ask for deprecation of __slots__ (which is something
I wanted from the beginning!).
Michele Simionato

[Bruno Desthuilliers]

Brian Elmegaard a écrit :

Bruno Desthuilliers <br************ ********@wtf.we bsiteburo.oops. com>
writes:

>Use dicts, not lists or tuples:

a = dict(name='yadd a', val=42)
print a['name']
print a['val']

I guess you will then need a list or tuple to store the dicts?

Should be a list then IMHO. But then it's the correct use of a list : an
homegenous collection.

[Steven D'Aprano]

On Sun, 07 Oct 2007 21:27:31 -0700, Aahz wrote:

In article <2t************ *************** ***@comcast.com >, Steven
Bethard <st************ @gmail.comwrote :

>>
You can use __slots__ [...]

Aaaugh! Don't use __slots__!

Seriously, __slots__ are for wizards writing applications with huuuge
numbers of object instances (like, millions of instances). For an
extended thread about this, see

http://groups.google.com/group/comp....browse_thread/

thread/8775c70565fb4a6 5/0e25f368e23ab05 8

Well, I've read the thread, and I've read the thread it links to, and for
the life of me I'm still no clearer as to why __slots__ shouldn't be used
except that:

1 Aahz and Guido say __slots__ are teh suxxor;

2 rumour (?) has it that __slots__ won't make it into Python 3.0;

3 inheritance from classes using __slots__ doesn't inherit the slot-
nature of the superclass.
Point 1 is never to be lightly dismissed, but on the other hand Guido
doesn't like reduce(), and I'm allergic to "Cos I Said So" arguments.

History is full of things which were invented for one purpose being used
for something else. So, that being the case, suppose I accept that using
__slots__ is not the best way of solving the problem, and that people of
the skill and experience of Guido and Aahz will roll their eyes and
snicker at me.

But is there actually anything *harmful* that can happen if I use
__slots__?

--
Steven.

[Steven Bethard]

Aahz wrote:

In article <2t************ *************** ***@comcast.com >,
Steven Bethard <st************ @gmail.comwrote :

>You can use __slots__ [...]

Aaaugh! Don't use __slots__!

Seriously, __slots__ are for wizards writing applications with huuuge
numbers of object instances (like, millions of instances).

You clipped me saying that __slots__ are for performance tweaks:

You can use __slots__ to make objects consume less memory and have
slightly better attribute-access performance. Classes for objects
that need such performance tweaks should start like...

I fully agree that __slots__ are for applications with huge numbers of
instances. But if you have that situation, you really do want to be
using __slots__.

STeVe

[Hrvoje Niksic]

Steven D'Aprano <st***@REMOVE-THIS-cybersource.com .auwrites:

Well, I've read the thread, and I've read the thread it links to,
and for the life of me I'm still no clearer as to why __slots__
shouldn't be used except that:

[...]

But is there actually anything *harmful* that can happen if I use
__slots__?

Here is one harmful consequence: __slots__ breaks multiple
inheritance:

class A(object):
__slots__ = ['a', 'b']

class B(object):
__slots__ = ['c']

class AB(A, B):
pass

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
multiple bases have instance lay-out conflict

Even if A and B had the exact same slots, for example ['a', 'b'], it
wouldn't make a difference. AB explicitly setting __slots__ to
something like ['a', 'b', 'c'] doesn't help either. But that is only
a technical answer to your technical question which misses the real
problem people like Aahz and Guido have with __slots__. (I don't
claim to represent them, of course, the following is my
interpretation. )

The backlash against __slots__ is a consequence of it being so easy to
misunderstand what __slots__ does and why it exists. Seeing __slots__
has led some people to recommend __slots__ to beginners as a way to
"catch spelling mistakes", or as a way to turn Python's classes into
member-declared structures, a la Java. For people coming from Java
background, catching mistakes as early as possible is almost a dogma,
and they are prone to accept the use of __slots__ (and living with the
shortcomings) as a rule.

Python power users scoff at that because it goes against everything
that makes Python Python. Use of __slots__ greatly reduces class
flexibility, by both disabling __dict__ and __weakref__ by default,
and by forcing a tight instance layout that cripples inheritance.
With people using __slots__ for the majority of their classes, it
becomes much harder for 3rd-party code to attach an unforeseen
attribute to an existing object. Even with single inheritance,
__slots__ has unintuitive semantics because subclasses automatically
get __dict__ and __weakref__, thereby easily breaking the "benefits"
of their use.

__slots__ is a low-level tool that allows creation of dict-less
objects without resorting to Python/C. As long as one understands it
as such, there is no problem with using it.

[Licheng Fang]

On Oct 8, 4:24 am, [e-mail] (Alex Martelli) wrote:

Licheng Fang <fanglich...@gm ail.comwrote:

...

Python Tutorial says an empty class can be used to do this. But if
namespaces are implemented as dicts, wouldn't it incur much overhead
if one defines empty classes as such for some very frequently used
data structures of the program?

Just measure:

$ python -mtimeit -s'class A(object):pass' -s'a=A()' 'a.zop=23'
1000000 loops, best of 3: 0.241 usec per loop

$ python -mtimeit -s'a=[None]' 'a[0]=23'
10000000 loops, best of 3: 0.156 usec per loop

So, the difference, on my 18-months-old laptop, is about 85 nanoseconds
per write-access; if you have a million such accesses in a typical run
of your program, it will slow the program down by about 85 milliseconds.
Is that "much overhead"? If your program does nothing else except those
accesses, maybe, but then why are your writing that program AT ALL?-)

And yes, you CAN save about 1/3 of those 85 nanoseconds by having
'__slots__=["zop"]' in your class A(object)... but that's the kind of
thing one normally does only to tiny parts of one's program that have
been identified by profiling as dramatic bottlenecks, to shave off the
last few nanoseconds in the very last stages of micro-optimization of a
program that's ALMOST, but not QUITE, fast enough... knowing about such
"extreme last-ditch optimization tricks" is of very doubtful value (and
I think I'm qualified to say that, since I _do_ know many of them...:-).
There ARE important performance things to know about Python, but those
worth a few nanoseconds don't matter much.

Alex

This is enlightening. Surely I shouldn't have worried too much about
performance before doing some measurement.

[Kevin]

Am I missing something, or am I the only one who explicitly declares
structs in python?

For example:
FileObject = {
"filename" : None,
"path" : None,
}

fobj = FileObject.copy ()
fobj["filename"] = "passwd"
fobj["path"] = "/etc/"

Kevin Kelley

On 10/7/07, Licheng Fang <fa*********@gm ail.comwrote:

Python is supposed to be readable, but after programming in Python for
a while I find my Python programs can be more obfuscated than their C/C
++ counterparts sometimes. Part of the reason is that with
heterogeneous lists/tuples at hand, I tend to stuff many things into
the list and *assume* a structure of the list or tuple, instead of
declaring them explicitly as one will do with C structs. So, what used
to be

struct nameval {
char * name;
int val;
} a;

a.name = ...
a.val = ...

becomes cryptic

a[0] = ...
a[1] = ...

Python Tutorial says an empty class can be used to do this. But if
namespaces are implemented as dicts, wouldn't it incur much overhead
if one defines empty classes as such for some very frequently used
data structures of the program?

Any elegant solutions?

--
http://mail.python.org/mailman/listinfo/python-list

--
Kevin Kelley
http://technogeek.org/