Simple thread-safe counter?

[Paul Rubin]

I'd like to have a function (or other callable object) that returns
0, 1, 2, etc. on repeated calls. That is:

print f() # prints 0
print f() # prints 1
print f() # prints 2
# etc.

There should never be any possibility of any number getting returned
twice, or getting skipped over, even if f is being called from
multiple threads.

What's the simplest and most natural way to do this? I can think of a
few but am not sure that they work. And I can think of some ways that
are sure to work, but are messier than I'd like.

The GIL is your friend here:

import itertools
f = itertools.count().next

A similar thing can be done with xrange. But either way sucks if you
call it often enough to exceed the size of a Python short int
(platform C long). The obvious way with an explicit mutex doesn't
have that problem.

Paul> I'd like to have a function (or other callable object) that
Paul> returns 0, 1, 2, etc. on repeated calls.
...
Paul> There should never be any possibility of any number getting
Paul> returned twice, or getting skipped over, even if f is being called
Paul> from multiple threads.

How about (untested):

import Queue

counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get()
counter.put(i+1)
return i

Obviously, if you want multiple counters for some reason a little
information hiding with a class would help (also untested):

import Queue

class Counter:
def __init__(self, start=0):
self.counter = Queue.Queue()
self.counter.put(start)

def __call__(self):
i = self.counter.get()
self.counter.put(i+1)
return i

Skip

Tim Peters <ti********@gmail.com> writes:

The GIL is your friend here:

import itertools
f = itertools.count().next
Thanks, I was hoping something like this would work but was not sure I
could rely on it.
A similar thing can be done with xrange. But either way sucks if you
call it often enough to exceed the size of a Python short int
(platform C long). The obvious way with an explicit mutex doesn't
have that problem.

Xrange, of course :). I don't need to exceed the size of a short int,
so either of these should work fine. I wonder what measures the Pypy
implementers will take (if any) to make sure these things keep
working, but for now I won't worry about it.

Out of interest, are the above guaranteed to work under Jython? What
I'm doing right now is a short-term thing that will only have to run
under CPython, but I like to do things the right way when I can.

Skip Montanaro wrote:

Paul> I'd like to have a function (or other callable object) that
Paul> returns 0, 1, 2, etc. on repeated calls.
...
Paul> There should never be any possibility of any number getting
Paul> returned twice, or getting skipped over, even if f is being called
Paul> from multiple threads.

How about (untested):

import Queue

counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get() I think you need:
i = counter.get(True)
for this to work; otherwise a race condition would raise an exception. counter.put(i+1)
return i

[snip]

This is, of course dependent upon counter.get() being guaranteed to be
thread safe. (I haven't found anything in the docs specifically relating
to that. Perhaps it's implicit?)

Thanks,
--ag

--
Artie Gold -- Austin, Texas
http://it-matters.blogspot.com (new post 12/5)
http://www.cafepress.com/goldsays

Skip Montanaro <sk**@pobox.com> writes:

How about (untested):

import Queue
counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get()
counter.put(i+1)
return i

Hmmm, that's a bit messier than I hoped for, but it looks sure to work.

I think for my immediate requirement, I'm going to use xrange as Tim
suggested. However, I can't be sure that will work in non-GIL
implementations.

Tim Peters wrote in news:mailman.1223.1112417955.1799.python-
li**@python.org in comp.lang.python:

[Paul Rubin]

I'd like to have a function (or other callable object) that returns
0, 1, 2, etc. on repeated calls. That is:

print f() # prints 0
print f() # prints 1
print f() # prints 2
# etc.

There should never be any possibility of any number getting returned
twice, or getting skipped over, even if f is being called from
multiple threads.

What's the simplest and most natural way to do this? I can think of a
few but am not sure that they work. And I can think of some ways that
are sure to work, but are messier than I'd like.
The GIL is your friend here:

Yes but IIUC the GIL is only CPython, will this work with Jython,
IornPython etc ? (*)
import itertools
f = itertools.count().next

*) I'm either being rhetorical or stupid, not sure which :).

Rob.
--
http://www.victim-prime.dsl.pipex.com/

Artie Gold wrote:

Skip Montanaro wrote:

counter = Queue.Queue()
def f():
i = counter.get()

I think you need:
i = counter.get(True)

The default value for the "block" argument to Queue.get is True.

Paul Rubin wrote:

Skip Montanaro <sk**@pobox.com> writes:

How about (untested):

import Queue
counter = Queue.Queue()
counter.put(0)
def f():
i = counter.get()
counter.put(i+1)
return i

Hmmm, that's a bit messier than I hoped for, but it looks sure to work.

I think for my immediate requirement, I'm going to use xrange as Tim
suggested. However, I can't be sure that will work in non-GIL
implementations.

Hello,

Simple xrange stuff won't work in Jython as Sython is running
on a JVM which doesn't have a GIL kinda thing. If you wanna return a
sequential number then you'll need something like :

<PSEUDO CODE>

class COUNTER:
DOC : THIS WILL RETURN AN INCRMENTING LIST OF NUMBERS - THREAD SAFE.

FUNCTION INIT:
OBJRLOCK= RLOCK()
INTRETURNNUM = 0

FUNCTION NEXT
TRY{
RLOCK.AQUIRE
}FINALLY{
RLOCK.RELEASE
}

</PSUEDO CODE>

That basically all you'll need, if you make it iteratable of
whatever - you need to wrap the business end of what you are doing
around a recursive lock. Personally I dislike the GIL so I avoid
writing code that takes advantages of it.

Why psuedo code - this is similar to python code I know but it means
I'm not posting untested python code!!

Cheers,

Neil

Leif K-Brooks wrote:

Artie Gold wrote:

Skip Montanaro wrote:

counter = Queue.Queue()
def f():
i = counter.get()

I think you need:
i = counter.get(True)

The default value for the "block" argument to Queue.get is True.

Right. I misparsed the entry in the documentation:

"If optional args block is true and timeout is None (the default), block
if necessary..."

Thanks,
--ag

--
Artie Gold -- Austin, Texas
http://it-matters.blogspot.com (new post 12/5)
http://www.cafepress.com/goldsays

In article <ma**************************************@python.o rg>,
Skip Montanaro <sk**@pobox.com> wrote:

Obviously, if you want multiple counters for some reason a little
information hiding with a class would help (also untested):

import Queue

class Counter:
def __init__(self, start=0):
self.counter = Queue.Queue()
self.counter.put(start)

def __call__(self):
i = self.counter.get()
self.counter.put(i+1)
return i

This is one case where'd recommend using a plan RLock() instead of using
Queue -- the RLock() will be more efficient:

import threading

class Counter:
def __init__(self, start=0, increment=1):
self.counter = start
self.increment = increment
self.lock = threading.RLock()
def __call__(self):
self.lock.acquire()
self.counter += self.increment
i = self.counter
self.lock.release()
return i

There are several tricks one can use to squeeze further efficiency gains
from this, including using Lock() instead of RLock().
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR

aa**@pythoncraft.com (Aahz) writes:

This is one case where'd recommend using a plan RLock() instead of using
Queue -- the RLock() will be more efficient...

I'm starting to believe the GIL covers up an awful lot of sloppiness
in Python. I wonder if there could be a decorator approach:

@synchronized
def counter():
t = itertools.count()
while True:
yield t.next()

or better yet,

counter = synchronized(itertools.count())

[Paul Rubin]

I'm starting to believe the GIL covers up an awful lot of sloppiness
in Python.

The GIL is highly exploitable, and much of CPython does exploit it.

If you don't want to exploit it, that's fine: there was always an
obvious approach using an explicit mutex here, and the only thing
stopping you from using it is a desire to be clever. Exploiting the
GIL in CPython is clever; using an explicit mutex is utterly
straightforward. Pick your poison.

Am Samstag, 2. April 2005 22:28 schrieb Paul Rubin:

I'm starting to believe the GIL covers up an awful lot of sloppiness
in Python. I wonder if there could be a decorator approach:

@synchronized
def counter():
t = itertools.count()
while True:
yield t.next()

Of course there could:

def synchronized_iterator(f):
def wrapper(*args,**kwargs):
class iterator(object):
def __init__(self,f,args,kwargs):
self.iter = f(*args,**kwargs)
self.lock = threading.RLock()
def __iter__(self):
return self
def next(self):
self.lock.acquire()
try:
return self.iter.next()
finally:
self.lock.release()
return iterator(f,args,kwargs)
return wrapper

@synchronized_iterator
def create_counter():
t = itertools.count()
while True:
yield t.next()

or

counter = synchronized_iterator(itertools.count)

I used a class-based approach, as I don't want to destroy the semantics of
calling the returned wrapper(), which should've already instantiated the
wrapped generator object (which doesn't happen when making wrapper() a
generator itself).

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCTyN0f0bpgh6uVAMRAhl4AJ9eityDJSuzkfywFGC0DG UOiz4hXACePWFP
fi1F0tFXekyTzWbvbHi8zh8=
=flQB
-----END PGP SIGNATURE-----

Am Sonntag, 3. April 2005 00:57 schrieb Heiko Wundram:

<snip>
or

Make that:

create_counter = syncronized_iterator(itertools.count)

and

counter = create_counter()

to create the actual counter regardless of iterator.

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCTyQNf0bpgh6uVAMRAqdJAJ9Efwzkf9xNau6LaRgcOT KwOpz95ACfe1Y7
iHWep1RM157Dd/pvxlhtYEg=
=zRYe
-----END PGP SIGNATURE-----

Tim Peters <ti********@gmail.com> writes:

If you don't want to exploit it, that's fine: there was always an
obvious approach using an explicit mutex here, and the only thing
stopping you from using it is a desire to be clever. Exploiting the
GIL in CPython is clever; using an explicit mutex is utterly
straightforward. Pick your poison.

For my immediate requirement I'm going to use xrange as you suggested
(thanks!). It will work exactly as I want. I'd be uncomfortable
publishing such a program for long-term use by other people, but this
is just for a short-term hack (famous last words).

I think using an explicit mutex goes against Guido's quote mentioned
yesterday, about avoiding tedious code. But relying on the GIL
results in a brittle program that only works correctly in one Python
implementation. As use of other implementations becomes more
widespread, one of three things has to happen: 1) Python programs will
become uglier and more mistake-prone (from using explicit mutexes all
over the place); 2) Python programs will become unreliable (from
depending on a GIL that doesn't exist any more); or 3) Python will
acquire some new mechanisms for handling reliable synchronization
easily and cleanly, as other languages already have.

I think (3) is the preferable choice and it's reasonable to try to
figure out how such mechanisms should work.

Paul Rubin <http://ph****@NOSPAM.invalid> writes:

Tim Peters <ti********@gmail.com> writes:

The GIL is your friend here:

import itertools
f = itertools.count().next

Thanks, I was hoping something like this would work but was not sure I
could rely on it.

A similar thing can be done with xrange. But either way sucks if you
call it often enough to exceed the size of a Python short int
(platform C long). The obvious way with an explicit mutex doesn't
have that problem.

Xrange, of course :). I don't need to exceed the size of a short int,
so either of these should work fine. I wonder what measures the Pypy
implementers will take (if any) to make sure these things keep
working, but for now I won't worry about it.

Well, for now we don't support threads. Easy!

Cheers,
mwh
(no, really, this is for the future)

--
Two things I learned for sure during a particularly intense acid
trip in my own lost youth: (1) everything is a trivial special case
of something else; and, (2) death is a bunch of blue spheres.
-- Tim Peters, 1 May 1998