473,320 Members | 1,867 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Threads modify "global" variable -- asking for trouble?

I have been experimenting with some thread programming, but as I'm
doing this on my own I am worried I might be making a major mistake.

Here's a brief rundown of what I am working on. Multiple threads, via
Queue, are used to perform RBL checks on an IP. The threads are passed
a defined class (ConnectionScore) in their init. This is used to
collect the results, including a method to add the result. If I run
the program, everything goes fine and once all the threads complete the
main program has all the data from their RBL checks in the
ConnectionScore class.

It seems to work, even with an obscene (and silly) number of WORKERS
and RBLs to check. But I was reading in the Python Cookbook this
evening and it raised the issue in my mind of what happens if multiple
threads try to add data to the class at the same time.

I've attached the code below. The ConnectionClass.addRBL method just
takes the result and adds it to a dictionary object attached to the
class.
#!/usr/bin/python
from UCE_weight import ConnectionScore
from rbl import *
import Queue
import threading
import time, random

starttime = time.time()

WORKERS=5

class Worker(threading.Thread):

def __init__(self,queue,score):
self.__queue = queue
threading.Thread.__init__(self)
self.score = score

def run(self):
while 1:
RBL = self.__queue.get()
# Are we at the end of the queue?
if RBL is None: break
self.result = checkRBL("127.0.0.2",RBL,True,5)
score.addRBL(RBL,self.result[0],self.result[1])

queue = Queue.Queue(0)
score = ConnectionScore()

f = open("rbl.list",'r')
MEGARBL = []
while 1:
line = f.readline().strip()
if not line: break
MEGARBL.append(line)
f.close()
for i in range(WORKERS):
Worker(queue,score).start() # start a worker

for i in MEGARBL:
queue.put(i)

for i in range(WORKERS):
queue.put(None) # add end of queue marker

while 1:
# wait for the queue???
if queue.empty():
break
else:
time.sleep(0.1)
elapsed = time.time() - starttime
rate = len(MEGARBL)/elapsed
rate = str(round(rate,2))
elapsed = str(round(elapsed,2))

score.showRBL()
print "Checked",str(len(MEGARBL)),"zones in",elapsed,"seconds.
(",rate.strip(),"per second )"

Mar 17 '06 #1
4 2477

My apologizes, I missed the newish FAQ entry on this. The addrbl()
method looks like this:

def addRBL(self, testname, result, info=""):
self.testresultsRBL[testname] = result, info

So according to the FAQ, D[x] = y, where D is a dictionary, is atomic
and therefore thread-safe. Right?

Mar 17 '06 #2
J Rice <ri**********@gmail.com> wrote:
My apologizes, I missed the newish FAQ entry on this. The addrbl()
method looks like this:

def addRBL(self, testname, result, info=""):
self.testresultsRBL[testname] = result, info

So according to the FAQ, D[x] = y, where D is a dictionary, is atomic
and therefore thread-safe. Right?


In the current implementation specifically, for CPython exclusively, if
the hashing of testname is itself somehow 'atomic', and so is the access
to attribute testresultsRBL of object self, I do believe you might luck
out and end up "atomic" (by accident of implementation, only) if
everything's just right. Of course, any tiny change (any different
implementation of Python, any type that's not a pure primitive, any
future version of Python, etc, etc) could break this extremely fragile
set of circumstances -- Python as a language makes no guarantees of
atomicity for anything except the synchronization primitives (of which
Queue is the most powerful). There was a rather heated exchange on the
subject quite recently on this group.

Just put a lock acquire/release around this assignment (and any _use_ of
the same dictionary) and you should be vastly safer, IMNSHO.
Alex
Mar 17 '06 #3

"J Rice" <ri**********@gmail.com> wrote in message
news:11********************@i40g2000cwc.googlegrou ps.com...

My apologizes, I missed the newish FAQ entry on this. The addrbl()
method looks like this:

def addRBL(self, testname, result, info=""):
self.testresultsRBL[testname] = result, info

So according to the FAQ, D[x] = y, where D is a dictionary, is atomic
and therefore thread-safe. Right?


I believe an alternative, mentioned elsewhere by Tim Peters, is for workers
to send results (here (testname,result,info) ) back in a results queue read
by the main thread.

tjr

tjr

Mar 17 '06 #4
Thank you. Implementing a results queue was much simpler than I
expected, and I think as I add this into the rest of the program it
will avoid a lot of potential problems later too.

Mar 18 '06 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: mrbog | last post by:
I have an array/hash that stores path information for my app. As in, what directory this is in, what directory that's in, what the name of the site is, what the products are called, etc. It's...
1
by: Chris Stromberger | last post by:
This doesn't seem like it should behave as it does without using "global d" in mod(). d = {} def mod(): d = 3 mod() print d
7
by: Lyn | last post by:
Hi and Season's Greetings to all. I have a question regarding the use of a qualifier word "Global". I cannot find any reference to this in Access help, nor in books or on the Internet. "Global"...
4
by: BB | last post by:
Hello all, I might be missing something here, but am trying to understand the difference between using application-level variables--i.e. Application("MyVar")--and global variables--i.e. public...
3
by: Pierre | last post by:
Hello, In an aspx page (mypage.aspx) from a web projet, I would like to get the value of a variable of the projet that is declared as public in a module. The variable can be called from...
8
by: Hans Greif | last post by:
Hallo, hello, i try to implement a global dataset - singleton based. On www.bakterienforum.de i found a singleton implementation: sealed class SingletonCounter { public int Counter = 0;
11
by: eBob.com | last post by:
I have this nasty problem with Shared methods and what I think of as "global storage" - i.e. storage declared outside of any subroutines or functions. In the simple example below this "global"...
1
by: sap0321 | last post by:
This should be kindergarten stuff but for some reason I am having trouble with it. I do 99.9% web programming and this is the first windows app i've done in years - probably the first ever in C# ......
4
ChrisWang
by: ChrisWang | last post by:
Hi, I am having trouble understanding the use of 'global' variables I want to use a global variable with the same name as a parameter of a function. But I don't know how to use them at the same...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.