removeall() in list

[castironpi]

Any ideas for a thread-safe list.removeall( X ): removing all
occurrences of X within list L, when L might be modified concurrently?

Sincerely,
Aaron

[Paul Rubin]

ca********@gmail.com writes:

Any ideas for a thread-safe list.removeall( X ): removing all
occurrences of X within list L, when L might be modified concurrently?

That way lies madness. Do something sensible instead. Put a lock
around the list, or put all mutators for the list into a single
thread, or whatever. Don't do what you're describing.

[castironpi]

On Jan 11, 2:57 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Any ideas for a thread-safe list.removeall( X ): removing all
occurrences of X within list L, when L might be modified concurrently?

That way lies madness. Do something sensible instead. Put a lock
around the list, or put all mutators for the list into a single
thread, or whatever. Don't do what you're describing.

This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.

[Paul Rubin]

ca********@gmail.com writes:

This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.

If you're asking in a general way how to do something like that,
there are several basic methods:

1. Put a single thread in charge of the list, and communicate with it
by message passing through Queues. To get X out of the list, you'd
send the mutator thread a message asking for removal. The mutator
thread would loop reading and processing messages from the queue,
blocking when no requests are pending. This is sort of the preferred
Python style and is pretty simple to get correct, but if there are
many such objects you can end up with more threads than you really
want.

2. Associate a lock with the list. Anything wanting to access the
list should acquire the lock, do its stuff, then release the lock.
This gets confusing after a while.

3. Various other schemes involving finer grained locks etc. that
get horrendously error prone (race conditions etc).

There is probably a good tutorial somewhere about programming with
threads. It's sometimes a tricky subject, so it's worth taking
the time to study various approaches rather than re-inventing the wheeel.

[castironpi]

On Jan 11, 5:26 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.

2. Associate a lock with the list. Anything wanting to access the
list should acquire the lock, do its stuff, then release the lock.
This gets confusing after a while.

To keep it generic, how about:

listA.op( insert, x )
listA.op( remove, x )

[castironpi]

On Jan 11, 5:43 pm, [e-mail] wrote:

On Jan 11, 5:26 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.

2. Associate a lock with the list. Anything wanting to access the
list should acquire the lock, do its stuff, then release the lock.
This gets confusing after a while.

To keep it generic, how about:

listA.op( insert, x )
listA.op( remove, x )

However, in reality, your rock and hard place are:
listA.op( listA.insert, x )
listA.op( listA.remove, x )

or

listA.op( 'insert', x )
listA.op( 'remove', x )

[Paul Rubin]

ca********@gmail.com writes:

Could you:

lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )

Where lockerA ops acquire the locks on all its threads?

I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.

Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.

You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).

What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?

[castironpi]

On Jan 11, 8:04 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Could you:

lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )

Where lockerA ops acquire the locks on all its threads?

I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.

Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.

You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).

What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass

[castironpi]

On Jan 12, 2:37 am, [e-mail] wrote:

On Jan 11, 8:04 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Could you:

lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )

Where lockerA ops acquire the locks on all its threads?

I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.

Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.

You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).

What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass

Though:

class A:
@synch( 'lockA' )
def Connect( self, target ): ...
@synch( 'lockA' )
def Disconnect( self, target ): ...

where decorator 'synch' is defined as:

take the 'lockA' attribute of the first function parameter, call
acquire(), call the function, call release(); has the right idea.
Another is on __init__, go through and wrap every member function in
self.lockA.acquire() and .release().

[Paul Rubin]

ca********@gmail.com writes:

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

I think the Pythonic way to do this is have the targets communicate
with the observers through queues rather than with callbacks.

[Fredrik Lundh]

ca********@gmail.com wrote:

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass

so use a lock. it's a whopping two lines of code:

creation:

lock = threading.Lock()

usage:

with lock:
for emelem in ...
...

more here:

http://effbot.org/zone/thread-synchronization.htm

and btw, looping over a list to figure out what you want to remove from
that list is a bit pointless. better just create a new list:

with lock:
# get rid of all func instances
emlist = [e for e in emlist if e.func is not func]

an alternative approach would be to replace emlist with a dictionary,
keyed on func objects. that'll let you remove all items associated with
a given function with a single atomic operation:

del emdict[func]

</F>

[Rhamphoryncus]

On Jan 12, 1:37 am, [e-mail] wrote:

On Jan 11, 8:04 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Could you:

lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )

Where lockerA ops acquire the locks on all its threads?

I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.

Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.

You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).

What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass

Is there a reason you're using a list, rather than a dict? Note that
each call to list.remove() is O(n), whereas deleting a key from a dict
is O(1).

[castironpi]

On Jan 12, 12:26 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

2) List is referenced by others; concurrent modifications may be going
on; can not replace it. Can I make asynchronous modifications and
merge the changes, SCM-style?

Nothing else should have direct access to the list.

Impossible to guarantee in Python. If you do, the reference to you
does.

[Paul Rubin]

ca********@gmail.com writes:

Nothing else should have direct access to the list.

Impossible to guarantee in Python. If you do, the reference to you does.

Well, ok. Nothing else should USE that access.

[castironpi]

On Jan 12, 1:03 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Nothing else should have direct access to the list.

Impossible to guarantee in Python. If you do, the reference to you does.

Well, ok. Nothing else should USE that access.

Ah, very agreed. Access directly at your own risk.

Will you engage with me over e-mail to discuss the Locker
implementation I'm developing? Aaron

[Paul Rubin]

ca********@gmail.com writes:

Will you engage with me over e-mail to discuss the Locker
implementation I'm developing? Aaron

I really can't, sorry. I'm finding it hard enough to follow over the
newsgroup. If you just have a single one of these lists, it's
probably simplest to do what Frederik Lundh suggested. The other
stuff we've discussed is mostly about how to organize having a lot of
them.

[castironpi]

On Jan 12, 1:28 pm, Paul Rubin <http://[e-mail]:

[e-mail] writes:

Will you engage with me over e-mail to discuss the Locker
implementation I'm developing? Aaron

I really can't, sorry. I'm finding it hard enough to follow over the
newsgroup. If you just have a single one of these lists, it's
probably simplest to do what Frederik Lundh suggested. The other
stuff we've discussed is mostly about how to organize having a lot of
them.

Highlight from the working solution:

def onfulfill( self, func, *args, **kwargs ):
'''decorator launches its function in separate thread
upon completion of func. todo: cancel and reference
check.'''
locfunc= Ref()
def honorth():
result= self.op( func, *args, **kwargs )
locfunc.val( result )
def prefulfill( func ):
locfunc.val= func
th= threading.Thread( target= honorth )
th.start()
return prefulfill