471,344 Members | 984 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,344 software developers and data experts.

removeall() in list

Any ideas for a thread-safe list.removeall( X ): removing all
occurrences of X within list L, when L might be modified concurrently?

Sincerely,
Aaron
Jan 11 '08 #1
16 5992
ca********@gmail.com writes:
Any ideas for a thread-safe list.removeall( X ): removing all
occurrences of X within list L, when L might be modified concurrently?
That way lies madness. Do something sensible instead. Put a lock
around the list, or put all mutators for the list into a single
thread, or whatever. Don't do what you're describing.
Jan 11 '08 #2
On Jan 11, 2:57 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
Any ideas for a thread-safe list.removeall( X ): removing all
occurrences of X within list L, when L might be modified concurrently?

That way lies madness. Do something sensible instead. Put a lock
around the list, or put all mutators for the list into a single
thread, or whatever. Don't do what you're describing.
This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.
Jan 11 '08 #3
ca********@gmail.com writes:
This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.
If you're asking in a general way how to do something like that,
there are several basic methods:

1. Put a single thread in charge of the list, and communicate with it
by message passing through Queues. To get X out of the list, you'd
send the mutator thread a message asking for removal. The mutator
thread would loop reading and processing messages from the queue,
blocking when no requests are pending. This is sort of the preferred
Python style and is pretty simple to get correct, but if there are
many such objects you can end up with more threads than you really
want.

2. Associate a lock with the list. Anything wanting to access the
list should acquire the lock, do its stuff, then release the lock.
This gets confusing after a while.

3. Various other schemes involving finer grained locks etc. that
get horrendously error prone (race conditions etc).

There is probably a good tutorial somewhere about programming with
threads. It's sometimes a tricky subject, so it's worth taking
the time to study various approaches rather than re-inventing the wheeel.
Jan 11 '08 #4
On Jan 11, 5:26 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.

2. Associate a lock with the list. Anything wanting to access the
list should acquire the lock, do its stuff, then release the lock.
This gets confusing after a while.
To keep it generic, how about:

listA.op( insert, x )
listA.op( remove, x )
Jan 11 '08 #5
On Jan 11, 5:43 pm, castiro...@gmail.com wrote:
On Jan 11, 5:26 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
This function just wants X out of the list. It doesn't matter if this
happens before, during, or after something else; so long as it happens.
2. Associate a lock with the list. Anything wanting to access the
list should acquire the lock, do its stuff, then release the lock.
This gets confusing after a while.

To keep it generic, how about:

listA.op( insert, x )
listA.op( remove, x )
However, in reality, your rock and hard place are:
listA.op( listA.insert, x )
listA.op( listA.remove, x )

or

listA.op( 'insert', x )
listA.op( 'remove', x )
Jan 11 '08 #6
ca********@gmail.com writes:
Could you:

lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )

Where lockerA ops acquire the locks on all its threads?
I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.

Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.

You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).

What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?
Jan 12 '08 #7
On Jan 11, 8:04 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
Could you:
lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )
Where lockerA ops acquire the locks on all its threads?

I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.

Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.

You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).

What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?
I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass
Jan 12 '08 #8
On Jan 12, 2:37 am, castiro...@gmail.com wrote:
On Jan 11, 8:04 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
Could you:
lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )
Where lockerA ops acquire the locks on all its threads?
I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.
Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.
You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).
What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass
Though:

class A:
@synch( 'lockA' )
def Connect( self, target ): ...
@synch( 'lockA' )
def Disconnect( self, target ): ...

where decorator 'synch' is defined as:

take the 'lockA' attribute of the first function parameter, call
acquire(), call the function, call release(); has the right idea.
Another is on __init__, go through and wrap every member function in
self.lockA.acquire() and .release().
Jan 12 '08 #9
ca********@gmail.com writes:
I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.
I think the Pythonic way to do this is have the targets communicate
with the observers through queues rather than with callbacks.
Jan 12 '08 #10
ca********@gmail.com wrote:
I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass
so use a lock. it's a whopping two lines of code:

creation:

lock = threading.Lock()

usage:

with lock:
for emelem in ...
...

more here:

http://effbot.org/zone/thread-synchronization.htm

and btw, looping over a list to figure out what you want to remove from
that list is a bit pointless. better just create a new list:

with lock:
# get rid of all func instances
emlist = [e for e in emlist if e.func is not func]

an alternative approach would be to replace emlist with a dictionary,
keyed on func objects. that'll let you remove all items associated with
a given function with a single atomic operation:

del emdict[func]

</F>

Jan 12 '08 #11
On Jan 12, 1:37 am, castiro...@gmail.com wrote:
On Jan 11, 8:04 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
Could you:
lockerA= Locker( listA, listB )
lockerA.op( listB.reverse )
lockerA.op( listA.pop )
Where lockerA ops acquire the locks on all its threads?
I don't understand that question. The main thing to understand is
that multi-threaded programming is complicated (especially if you're
after high performance), and very difficult to get right without
knowing exactly what you're doing. The generally preferred approach
in Python is to keep things simple at some performance cost.
Your Locker approach above looks sort of reasonable if you can be
absolutely sure that nothing else can mess with listA or listB
concurrently with those locker operations. Normally you would put
listA and listB into a single object along with a lock, then do
operations on that object.
You might check the Python Cookbook for some specific recipes and
sample code for this stuff. If you've used Java, Python's general
threading mechanisms are similar, but they are in the library rather
than built into the language (i.e. there is no "synchronized"
keyword, you have to do that locking explicitly).
What is the actual application, if you don't mind saying? Are you
sure that you really need concurrency?

I'm writing an NxN observer pattern, mostly for my own personal
exploration. Two threads -might- be calling 'Disconnect' at the same
time, and I can't even guarantee that the function runs properly.

for emelem in [ e for e in emlist if e.func is func ]:
try:
emlist.remove( emelem )
except ValueError:
pass
Is there a reason you're using a list, rather than a dict? Note that
each call to list.remove() is O(n), whereas deleting a key from a dict
is O(1).
Jan 12 '08 #12
On Jan 12, 12:26 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
2) List is referenced by others; concurrent modifications may be going
on; can not replace it. Can I make asynchronous modifications and
merge the changes, SCM-style?

Nothing else should have direct access to the list.
Impossible to guarantee in Python. If you do, the reference to you
does.
Jan 12 '08 #13
ca********@gmail.com writes:
Nothing else should have direct access to the list.
Impossible to guarantee in Python. If you do, the reference to you does.
Well, ok. Nothing else should USE that access.
Jan 12 '08 #14
On Jan 12, 1:03 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
Nothing else should have direct access to the list.
Impossible to guarantee in Python. If you do, the reference to you does.

Well, ok. Nothing else should USE that access.
Ah, very agreed. Access directly at your own risk.

Will you engage with me over e-mail to discuss the Locker
implementation I'm developing? Aaron
Jan 12 '08 #15
ca********@gmail.com writes:
Will you engage with me over e-mail to discuss the Locker
implementation I'm developing? Aaron
I really can't, sorry. I'm finding it hard enough to follow over the
newsgroup. If you just have a single one of these lists, it's
probably simplest to do what Frederik Lundh suggested. The other
stuff we've discussed is mostly about how to organize having a lot of
them.
Jan 12 '08 #16
On Jan 12, 1:28 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
castiro...@gmail.com writes:
Will you engage with me over e-mail to discuss the Locker
implementation I'm developing? Aaron

I really can't, sorry. I'm finding it hard enough to follow over the
newsgroup. If you just have a single one of these lists, it's
probably simplest to do what Frederik Lundh suggested. The other
stuff we've discussed is mostly about how to organize having a lot of
them.
Highlight from the working solution:

def onfulfill( self, func, *args, **kwargs ):
'''decorator launches its function in separate thread
upon completion of func. todo: cancel and reference
check.'''
locfunc= Ref()
def honorth():
result= self.op( func, *args, **kwargs )
locfunc.val( result )
def prefulfill( func ):
locfunc.val= func
th= threading.Thread( target= honorth )
th.start()
return prefulfill
Jan 12 '08 #17

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by massimo | last post: by
1 post views Thread by John Young | last post: by
reply views Thread by hn | last post: by
1 post views Thread by Guadala Harry | last post: by
3 posts views Thread by msnews.microsoft.com | last post: by
1 post views Thread by Nathan Sokalski | last post: by
4 posts views Thread by Michael Nesslinger | last post: by
reply views Thread by Ronak mishra | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.