How to create a limited set of instanceses of a class

[madpython]

For the pure theory sake and mind expansion i got a question.
Experimenting with __new__ i found out how to create a singleton.
class SingleStr(object):
def __new__(cls,*args,**kwargs):
instance=cls.__dict__.get('instance')
if instance:
return instance
cls.instance=object.__new__(cls,*args,**kwargs)
return cls.instance

What if i need to create no more than 5 (for example) instances of a
class. I guess that would be not much of a problema but there is a
complication. If one of the instances is garbagecollected I want to be
able to spawn another instance. Maybe it's difficult to perceive what i
said so think of a limited number of sockets. Wherenever one is freed
it can be captured again.
I believe that this type of the design pattern was described in
"Thinking in Java" but there was a special method finalize()
utilized, that is called every time when gc() is about to clean up an
unreferenced object. By using this method a counter of active instances
could be easily implemented. What is the pythonic approach to this
problema?

[Alex Martelli]

madpython <ma*******@gmail.com> wrote:

For the pure theory sake and mind expansion i got a question.
Experimenting with __new__ i found out how to create a singleton.
class SingleStr(object):
def __new__(cls,*args,**kwargs):
instance=cls.__dict__.get('instance')
if instance:
return instance
cls.instance=object.__new__(cls,*args,**kwargs)
return cls.instance

What if i need to create no more than 5 (for example) instances of a
And what if somebody asks for a 6th one -- raise an exception? or
return a random one of the 5? Ah well, anyway...:
class. I guess that would be not much of a problema but there is a
complication. If one of the instances is garbagecollected I want to be
able to spawn another instance. Maybe it's difficult to perceive what i
said so think of a limited number of sockets. Wherenever one is freed
it can be captured again.

Have the class hold weak references to the instances. E.g.:

import weakref
import random

class JustFive(object):
_counter = 0
_insts = weakref.WeakKeyDictionary()

def __new__(cls):
if len(cls._insts) < 5:
inst = object.__new__(cls)
inst.id = cls._counter
cls._insts[inst] = None
cls._counter += 1
else:
inst = random.choice(cls._insts.keys())
return inst

Testing this for the specified behavior should be done more
systematically, of course, but here's a snippet based only on showing
exactly what's going on...:
if __name__ == '__main__':
samp = [JustFive() for i in range(7)]
def showsamp():
for s in samp: print s.id,
ss = set(s.id for s in samp)
print '{',
for s in sorted(ss): print '%d,'%s,
print '} #%d'%len(ss)
print 'Start with:',
showsamp()
for x in range(10):
i = random.randrange(len(samp))
print ('@%d,'%i), samp[i].id, '->',
samp[i] = None
newone = JustFive()
samp[i] = newone
print '%d,' % samp[i].id,
showsamp()
A sample run might go, for example...:

Start with: 0 1 2 3 4 4 1 { 0, 1, 2, 3, 4, } #5
@0, 0 -> 5, 5 1 2 3 4 4 1 { 1, 2, 3, 4, 5, } #5
@1, 1 -> 3, 5 3 2 3 4 4 1 { 1, 2, 3, 4, 5, } #5
@6, 1 -> 6, 5 3 2 3 4 4 6 { 2, 3, 4, 5, 6, } #5
@2, 2 -> 7, 5 3 7 3 4 4 6 { 3, 4, 5, 6, 7, } #5
@4, 4 -> 7, 5 3 7 3 7 4 6 { 3, 4, 5, 6, 7, } #5
@4, 7 -> 6, 5 3 7 3 6 4 6 { 3, 4, 5, 6, 7, } #5
@1, 3 -> 7, 5 7 7 3 6 4 6 { 3, 4, 5, 6, 7, } #5
@3, 3 -> 8, 5 7 7 8 6 4 6 { 4, 5, 6, 7, 8, } #5
@6, 6 -> 5, 5 7 7 8 6 4 5 { 4, 5, 6, 7, 8, } #5
@2, 7 -> 6, 5 7 6 8 6 4 5 { 4, 5, 6, 7, 8, } #5

and the point is that the sample always has a set of objects with
exactly 5 separate id values, but the set "migrates" upwards as some
objects happen to be dropped. Well, not always "upwards" as smoothly as
luck would have it in this case, e.g., another run goes:

Start with: 0 1 2 3 4 1 2 { 0, 1, 2, 3, 4, } #5
@1, 1 -> 3, 0 3 2 3 4 1 2 { 0, 1, 2, 3, 4, } #5
@2, 2 -> 4, 0 3 4 3 4 1 2 { 0, 1, 2, 3, 4, } #5
@3, 3 -> 1, 0 3 4 1 4 1 2 { 0, 1, 2, 3, 4, } #5
@1, 3 -> 5, 0 5 4 1 4 1 2 { 0, 1, 2, 4, 5, } #5
@5, 1 -> 5, 0 5 4 1 4 5 2 { 0, 1, 2, 4, 5, } #5
@6, 2 -> 6, 0 5 4 1 4 5 6 { 0, 1, 4, 5, 6, } #5
@2, 4 -> 1, 0 5 1 1 4 5 6 { 0, 1, 4, 5, 6, } #5
@4, 4 -> 7, 0 5 1 1 7 5 6 { 0, 1, 5, 6, 7, } #5
@6, 6 -> 8, 0 5 1 1 7 5 8 { 0, 1, 5, 7, 8, } #5
@3, 1 -> 8, 0 5 1 8 7 5 8 { 0, 1, 5, 7, 8, } #5

here, the objects with ids 0 and 1 just never happen to get dropped.
Alex

[Scott David Daniels]

madpython wrote:

For the pure theory sake and mind expansion i got a question.
Experimenting with __new__ i found out how to create a singleton.
class SingleStr(object):
def __new__(cls,*args,**kwargs):
instance=cls.__dict__.get('instance')
if instance:
return instance
cls.instance=object.__new__(cls,*args,**kwargs)
return cls.instance

What if i need to create no more than 5 (for example) instances of a
class. I guess that would be not much of a problema but there is a
complication. If one of the instances is garbagecollected I want to be
able to spawn another instance....

This should help. You didn't really say how to pick the result
if there are duplicates, so I just decided for myself. Read up
on the weakref module, it is your friend in this kind of problem.

import weakref, random

class Limited(any_new_style_class):
_held_instances = weakref.WeakValueDictionary()
_held_limit = 5
def __new__(class_, *args, **kwargs):
if len(class_._held_instances) < class_._held_limit:
# You could go over here if threading is an issue
instance = super(class_, Limited).__new__(
class_, *args, **kwargs)
class_._held_instances[id(instance)] = instance
return instance
return random.choice(class_._held_instances.values())
--Scott David Daniels
sc***********@acm.org

[madpython]

Thanks Alex and Scott for your lead. It would've taken me forever
trying to figure it out by myself :)

I am affraid I didn't specify initially one thing and that led to a
confusion: there is no need to pick an instance from the weakref
dictionary, just return None if there are already 5 instances. But on
the other hand if a hardref to an object was deleted, it's place can be
taken by another one.
Here's what i mean (and where the problem is):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
#build a list of 5 elements
instList=[]
for i in range(7):
ainst=A()
if ainst:
instList.append(ainst)

for i in range(5):
instList.remove(instList[0]) #here the hardref is deleted
ainst=A()
while not ainst:
#make shure that ainst is not NoneType
gc.collect()
time.sleep(1) #wait 1 sec for gc() to clean the memory
ainst=A() #try again
instList.append(ainst) #new object added
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
the priblem is that ~3 out of 10 times the test part run into infinite
loop because of unsatisfied condition (while not ainst:) - memory
cannot be freed therefore new instance of A isn't permitted.
#!/usr/bin/env python
import weakref,random
import types,gc
import time
class Limited5(object):
__weakrefdict=weakref.WeakValueDictionary()
def __new__(cls,*args,**kwargs):
if len(cls.__weakrefdict)<5:
instance=super(Limited5,cls).__new__(cls,*args,**k wargs)
cls.__weakrefdict[id(instance)]=instance
return instance
# return random.choice(cls.__weakrefdict.values())
return None #no need to pick an instance
class A(Limited5):
counter=0
def __init__(self):
self.instCounter=self.counter
A.counter+=1
def getId(self):
return id(self)

if __name__=="__main__":
instList=[]
# populate the initial list of objects
#make shure that there are only 5 elements
for item in range(7):
ainst=A()
if hasattr(ainst,"getId"):
print ainst.getId()," -- ",ainst.instCounter
instList.append(ainst)
print "---------------------------------------------"

#delete and recreate an arbitrary element in the instList
for i in range(len(instList)):
instList.remove(instList[random.choice(range(len(instList)))])
ainst=A()
while not ainst: #here is an unstable part
ainst=A() #sometimes the loop becomes infinite
print gc.collect() #decpite the explicit call for gc() to
start
time.sleep(1)
print "*",
instList.append(ainst)
for item in instList:
print item.getId()," -- ",item.instCounter
#print "-------",item
print "++++++++++++++++++++++++++++"

[Alex Martelli]

madpython <ma*******@gmail.comwrote:

Thanks Alex and Scott for your lead. It would've taken me forever
trying to figure it out by myself :)

I am affraid I didn't specify initially one thing and that led to a
confusion: there is no need to pick an instance from the weakref
dictionary, just return None if there are already 5 instances. But on
the other hand if a hardref to an object was deleted, it's place can be
taken by another one.

And the latter issue is what the use of weakref in both responses was
about.

Here's what i mean (and where the problem is):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
#build a list of 5 elements
instList=[]
for i in range(7):
ainst=A()
if ainst:
instList.append(ainst)

Note that this is quite sloppy:

-- test "if ainst is None:" -- no excuse to use just "if ainst:"
-- at the end of the loop ainst remains bound -- "del ainst" to avoid
ainst being an extra reference to the instance

neither of these explains your bug, but -- just tighten up your code,
it's a good idea!-)

>
for i in range(5):
instList.remove(instList[0]) #here the hardref is deleted

SUPER sloppy! Just "del instList[0]" for the same effect much better
obtained.

ainst=A()
while not ainst:
#make shure that ainst is not NoneType

Again, "while ainst is None:" would be far better.

gc.collect()
time.sleep(1) #wait 1 sec for gc() to clean the memory

Useless, gc.collect() is synchronous *AND* only cleans up CYCLIC garbage
anyway -- unless instances of A have reference loops, both lines are
useless (the sleep is useless in ANY case).

ainst=A() #try again
instList.append(ainst) #new object added
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
the priblem is that ~3 out of 10 times the test part run into infinite
loop because of unsatisfied condition (while not ainst:) - memory
cannot be freed therefore new instance of A isn't permitted.

Your test code below does NOT do what your code here does! Instead it
removes (in the worst possible way, rather than cleanly, but, no matter)
a RANDOM reference -- which may happen to be the same as "item" had left
over from the previous run... because of a printing loop that is in your
sample below and not here... which is where the sloppiness catches up on
you. Specifically, look at this code from the sample that you had
below:

#delete and recreate an arbitrary element in the instList
for i in range(len(instList)):
instList.remove(instList[random.choice(range(len(instList)))])
ainst=A()
while not ainst: #here is an unstable part
ainst=A() #sometimes the loop becomes infinite
print gc.collect() #decpite the explicit call for gc() to start
time.sleep(1)
print "*", len(instList), len(A._weakrefdict)
instList.append(ainst)
for item in instList:
print item.getId()," -- ",item.instCounter
#print "-------",item
print "++++++++++++++++++++++++++++"

after the printing loop, name 'item' remains bound to the object that is
last element of instList -- so if that one just happens to be the
element you remove in that horrid first line of the main (outer) loop, 5
instances of class A nevertheless remain alive and therefore ainst will
be None forevermore, despite all the useless calls to gc.collect and
sleep.

A decent way to code this functionality would be:

# delete and recreate an arbitrary element in the instList
for i in range(len(instList)):
del instList[random.randrange(len(instList))]
instList.append(A())
for item in instList:
print item.getId()," -- ",item.instCounter
del item
print "++++++++++++++++++++++++++++"

It would be nice to also print len(instList) and the length of the
weakref dictionary of class A, to doublecheck things, but you've made
life extremely hard for yourself by naming the latter with TWO leading
underscores instead of one, thus asking Python to name-mangle it to make
it very inconvenient to access. Avoid the two-leading-underscores
construct (use just ONE leading underscore -- no mangling, no problems)
unless and until you are positive that you know exactly what you're
doing and are certain that you need the two underscores in a given
specific case -- do *NOT* "default" to using two underscores and make
life uselessly hard for yourself in terms of introspection and
debugging.

And, read up on such issues as "del somelist[index]" being WAY better
than "somelist.remove(somelist[index])" and the way names, references,
objects and garbage collection work in Python...
Alex

[madpython]

Thanks, Alex, again. The lesson has been taught. I appreciate very much
you spent time trying to help. Indeed the culprit of that infrequent
infinite loops was that bound reference "item" in the printing
loop. But frankly i thought that it only existed inside that loop.
Apparently I was wrong and glad that it was revealed.
I guess unless there is something to add here the problem is solved
and subject is closed. Thanks everyone who took their time to ponder on
it. I hope it was as much educational for you as it was for me.

[Alex Martelli]

madpython <ma*******@gmail.comwrote:

Thanks, Alex, again. The lesson has been taught. I appreciate very much
you spent time trying to help. Indeed the culprit of that infrequent
infinite loops was that bound reference "item" in the printing
loop. But frankly i thought that it only existed inside that loop.
Apparently I was wrong and glad that it was revealed.

Right -- a loop per se (be it a while loop or a for loop), just like an
if statement or a try statement, is NOT a separate scope from the code
around it (neither is a list comprehension); _functions_ are separate
scopes, and so are genexps.
Alex