473,702 Members | 2,436 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

New-style classes, iter and PySequence_Chec k

Someone pasted the original version of the following code snippet on
#python today. I started investigating why the new-style class didn't
work as expected, and found that at least some instances of new-style
classes apparently don't return true for PyInstance_Chec k, which causes
a problem in PySequence_Chec k, since it will only do an attribute lookup
for instances.

Things probably shouldn't be this way. Should I go to python-dev with this?

Demonstration snippet:

args={'a':0}

class Args(object):
def __getattr__(sel f,attr):
print "__getattr_ _:", attr
return getattr(args,at tr)

class ClassicArgs:
def __getattr__(sel f, attr):
print "__getattr_ _:", attr
return getattr(args, attr)

if __name__ == '__main__':
c = ClassicArgs()
i = c.__iter__()
print i
i = iter(c)
print i

a = Args()
i = a.__iter__()
print i
i = iter(a)
print i
Jul 19 '05 #1
4 2078
Tuure Laurinolli wrote:
Someone pasted the original version of the following code snippet on
#python today. I started investigating why the new-style class didn't
work as expected, and found that at least some instances of new-style
classes apparently don't return true for PyInstance_Chec k, which causes
a problem in PySequence_Chec k, since it will only do an attribute lookup
for instances.

Things probably shouldn't be this way. Should I go to python-dev with this?

Demonstration snippet:


For anyone who's curious, here's what the code actually does:

py> args={'a':0}
py> class Args(object):
.... def __getattr__(sel f,attr):
.... print "__getattr_ _:", attr
.... return getattr(args,at tr)
....
py> class ClassicArgs:
.... def __getattr__(sel f, attr):
.... print "__getattr_ _:", attr
.... return getattr(args, attr)
....
py> c = ClassicArgs()
py> i = c.__iter__()
__getattr__: __iter__
py> print i
<dictionary-keyiterator object at 0x0115D920>
py> i = iter(c)
__getattr__: __iter__
py> print i
<dictionary-keyiterator object at 0x01163CA0>
py> a = Args()
py> i = a.__iter__()
__getattr__: __iter__
py> print i
<dictionary-keyiterator object at 0x01163D20>
py> i = iter(a)
Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "D:\Steve\M y Programming\pyt honmodules\site customize.py", line
37, in iter
return orig(*args)
TypeError: iteration over non-sequence

STeVe
Jul 19 '05 #2
[Tuure Laurinolli]
Someone pasted the original version of the following code snippet on
#python today. I started investigating why the new-style class didn't
work as expected
classes apparently don't return true for PyInstance_Chec k, which causes
a problem in PySequence_Chec k, since it will only do an attribute lookup
for instances.

Things probably shouldn't be this way. Should I go to python-dev with this?


PyInstance_Chec k is defined as checking for instances of old style classes.
Hence, it appropriate that it does not return True for instances of new-style
classes.

PySequence_Chec k is pretty good shape. In general, it is not always possible to
tell a mapping from a sequence, but the function does try to take advantage of
all available information. In particular, it has a separate code path for
new-style instances which sometimes have additional information (i.e. dict and
its subclasses fill the tp_as_mapping->mp_subscript slot instead of the
tp_as_sequence->sq_slice slot).

Old style classes never inherit from dict, tuple, list, etc. so that slot
distinction isn't meaningful. Accordingly, PySequence_Chec k simply checks to
see if an object defines __getitem__ and that check entails an attribute lookup.
The posted code snippet uses a __getattr__ gimmick to supply an affirmative
answer to that check.

The docs do not make guarantees about how iter(o) will determine whether object
o is a sequence. Currently, it does the best it can and that entails having
different implementation logic for new-style and old-style objects as described
above. The provided snippet uses the __getattr__ gimmick to reveal the
implementation specific behavior. Mystery solved.

In contrast, the docs do make guarantees about explicit attribute lookup (i.e.
that an attribute lookup will actually occur and that slot checking logic will
not be substituted). The posted snippet reveals that these guarantees are
intact for both new and old-style classes.

Knowing all of this reveals the behavior to be a feature rather than a bug --
the feature being that operator.IsMapp ingType and operator.IsSequ enceType do a
remarkable job (few false positives and no false negatives) across a huge
variety of object types. It has been tested with instances of set, int, float,
complex, long, bool, str, unicode, UserString, list, UserList, tuple, deque,
NoneType, NotImplemented, new and old style instances with and without defining
__getitem__. Also, tested were subclasses of the above types. The false
positives are limited to user defined classes (new or old-style) defining
__getitem__. Since the mapping API overlaps with the sequence API, it is not
always possible to distinguish the two.

Of course, you found that it is possible to throw a monkey wrench in the process
using __getattr__ to exploit undefined, implementation specific behavior. Don't
do that. If a class needs to be iterable, then supply an __iter__ method. If
that method needs to be retargeted dynamically, then
take advantage of the guarantee that iter(o) will always find a defined __iter__
method. Perhaps, define __iter__ as lambda obj: obj(obj.__getat tr__(obj,
'__iter__')) or somesuch. IOW, to guarantee that iter() performs an __iter__
lookup, fill the slot with something that does that lookup.
Raymond Hettinger
E = mc**2 # Einstein
E = IR # Ohm's law
therefore
IR = mc**2 # Raymond's grand unified theory
Jul 19 '05 #3
On Thu, 14 Apr 2005 15:18:20 -0600, Steven Bethard <st************ @gmail.com> wrote:
Tuure Laurinolli wrote:
Someone pasted the original version of the following code snippet on
#python today. I started investigating why the new-style class didn't
work as expected, and found that at least some instances of new-style
classes apparently don't return true for PyInstance_Chec k, which causes
a problem in PySequence_Chec k, since it will only do an attribute lookup
for instances.

Things probably shouldn't be this way. Should I go to python-dev with this?

Demonstration snippet:


For anyone who's curious, here's what the code actually does:

py> args={'a':0}
py> class Args(object):
... def __getattr__(sel f,attr):
... print "__getattr_ _:", attr
... return getattr(args,at tr)
...
py> class ClassicArgs:
... def __getattr__(sel f, attr):
... print "__getattr_ _:", attr
... return getattr(args, attr)
...
py> c = ClassicArgs()
py> i = c.__iter__()
__getattr__: __iter__
py> print i
<dictionary-keyiterator object at 0x0115D920>
py> i = iter(c)
__getattr__: __iter__
py> print i
<dictionary-keyiterator object at 0x01163CA0>
py> a = Args()
py> i = a.__iter__()
__getattr__: __iter__
py> print i
<dictionary-keyiterator object at 0x01163D20>
py> i = iter(a)
Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "D:\Steve\M y Programming\pyt honmodules\site customize.py", line
37, in iter
return orig(*args)
TypeError: iteration over non-sequence

I think this is a known thing about the way several builtin functions like iter work.
I'm not sure, but I think it may be expedient optimization practicality beating purity.
IOW, I think maybe iter(a) skips the instance logic altogether and goes right to
type(a).__iter_ _(a) instead of looking for something on the instance shadowing __iter__.
And I suspect iter(a) does its own internal mro chase for __iter__, bypassing even a
__getattribute_ _ in a metaclass of Args, as it appears if you try to monitor that way. E.g.,
class Show(object): ... class __metaclass__(t ype):
... def __getattribute_ _(cls, attr):
... print 'Show.__getattr ibute__:', attr
... return type.__getattri bute__(cls, attr)
... def __getattribute_ _(self, attr):
... print 'self.__getattr ibute__:', attr
... return object.__getatt ribute__(self, attr)
...
... show = Show()
Show.__module__ Show.__getattri bute__: __module__
'__main__' show.__module__ self.__getattri bute__: __module__
'__main__' show.__iter__ self.__getattri bute__: __iter__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 8, in __getattribute_ _
AttributeError: 'Show' object has no attribute '__iter__' Show.__iter__ Show.__getattri bute__: __iter__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 5, in __getattribute_ _
AttributeError: type object 'Show' has no attribute '__iter__'

But no interception this way: iter(show) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: iteration over non-sequence
Naturally __getattr__ gets bypassed too, if there's no instance attribute lookup even tried.

If we actually supply an __iter__ method for iter(a) to find as type(a).__iter_ _, we can see:
class Args(object): ... def __getattr__(sel f, attr):
... print "__getattr_ _:", attr
... return getattr(args, attr)
... def __iter__(self):
... print "__iter__"
... return iter('Some kind of iterator')
... a = Args()
a.__iter__ <bound method Args.__iter__ of <__main__.Arg s object at 0x02EF8BEC>> a.__iter__() __iter__
<iterator object at 0x02EF888C> iter(a) __iter__
<iterator object at 0x02EF8CAC>
type(a).__iter_ _ <unbound method Args.__iter__> type(a).__iter_ _(a) __iter__
<iterator object at 0x02EF8CAC>

Now if we get rid of the __iter__ method, __getattr__ can come into play again:
del Args.__iter__
a.__iter__ __getattr__: __iter__
<method-wrapper object at 0x02EF8C0C> a.__iter__() __getattr__: __iter__
<dictionary-keyiterator object at 0x02EF8CE0> type(a).__iter_ _ Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: type object 'Args' has no attribute '__iter__' iter(a)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: iteration over non-sequence

It's kind of like iter(a) assuming that type(a).__iter_ _ is a data descriptor
delivering a bound method instead of an ordinary method, and therefore it can
assume that the data descriptor always trumps instance attribute lookup, and
it can optimize with that assumption. I haven't walked the relevant iter() code.
That would be too easy ;-)

Regards,
Bengt Richter
Jul 19 '05 #4
[Bengt Richter]
I'm not sure, but I think it may be expedient optimization practicality beating purity.

It is more a matter of coping with the limitations of the legacy API where want
to wrap a sequence iterator around __getitem__ in sequences but not in
mappings.
<snipped code experiments>
I haven't walked the relevant iter() code.
That would be too easy ;-)


The actual code for Object/abstract.c's PyObject_GetIte r() is somewhat more
straight-forward and less mysterious than those experiments suggest.

Raymond
Jul 19 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
1820
by: Madestro | last post by:
Hi guys, I am making a small program to retrieve e-mails from POP accounts. I got all the e-mail parsing stuff figured out, but I cannot seem to come up with a way to find out which e-mails are NEW so I don't have to retrieve them all. If you have experience with this kind of thing, you know that the server creates unique IDs for all the messages, but this IDs are not guaranteed to be unique, since they can be reused once a message is...
3
9405
by: Nimmi Srivastav | last post by:
There's a rather nondescript book called "Using Borland C++" by Lee and Mark Atkinson (Que Corporation) which presents an excellent discussion of overloaded new and delete operators. I am presenting below a summary of what I have gathered. I would appreciate if someone could point out to something that is specific to Borland C++ and is not supported by the ANSI standard. I am also concerned that some of the information may be outdated...
20
1827
by: Razvan | last post by:
Hi ! Is there any difference between new X and new X() ?! Regards, Razvan
2
7665
by: ip4ram | last post by:
I used to work with C and have a set of libraries which allocate multi-dimensional arrays(2 and 3) with single malloc call. data_type **myarray = (data_type**)malloc(widht*height*sizeof(data_type)+ height* sizeof(data_type*)); //allocate individual addresses for row pointers. Now that I am moving to C++,am looking for something by which I can
11
912
by: Jonan | last post by:
Hello, For several reasons I want to replace the built-in memory management with some custom built. The mem management itlsef is not subject to my question - it's ok to the point that I have nice and working allocation deallocation routines. However, I don't want to loose the nice extras of new operator, like - constructor calling, typecasting the result, keeping the array size, etc. For another bunch of reasons, outside this scope I...
5
2398
by: kUfa.scoopex | last post by:
Hi there! I have a small problem, and i really dont see any convenient way to fix it. Basically, i'd like to overload global new operators, into something like this: void *operator new( size_t size ); void *operator new( size_t size, AllocationType allocType ); void *operator new( size_t size, MemoryHandler* memHandler ); void *operator new( size_t size, MemoryHandler* memHandler, AllocationType
2
2082
by: Dave | last post by:
Hello all, I'd like to find a source on the web that discusses, in a comprehensive manner and in one place, everything about new / delete. It should include overloading operator new, the new operator, placement, nothrow, arrays, etc... My books cover the topic, I've found FAQs on the web that cover the topic, and so on, but all the sources I've found are disjointed. There's a bit on this page, a bit on that page, and so on. The...
15
4655
by: Steve | last post by:
I have a form with about 25 fields. In the BeforeUpdate event of the form, I have code that sets the default value of each field to its current value. For a new record, I can put the focus in any field to start. If I edit that field and then click on the new record button in the navigation buttons, the form goes to a new record and each field has the default value of the previous record. If I put the focus in any field to start, edit that...
7
2472
by: Mike Bulava | last post by:
I have created a base form that I plan to use throughout my application let call the form form1. I have Built the project then add another form that inherits from form1, I add a few panel controls each with a couple of controls in them I then rebuilt my project and my new panels and all controls they contained are gone... I've looked through the Auto generated code but don't see anything that looks wrong Any body have any idea why this...
3
1885
by: Grizlyk | last post by:
Hi, people. Can anybody explain me "multiple 'new' at single line" behavior. Consider: p::p(void*); p::p(void*,void*); new A( p(new B), p( new C(p(new D), p(new E)) ), p(new F));
0
8738
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
8652
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9234
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
8979
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8939
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
4412
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4667
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3104
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
2036
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.