By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,769 Members | 1,151 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,769 IT Pros & Developers. It's quick & easy.

Using multiprocessing inside decorator

falekmarcin
P: 2
I came across a problem that I can't solve and it's associated with multiprocessing and use it inside the decorator.

When I'm calling the method run_in_parallels using multiprocessing I 'm getting the error:

Expand|Select|Wrap|Line Numbers
  1. Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase
The call takes place inside the decorator, then followed the above-mentioned problem. At the time of calling the same method run_in_parallels without a decorator all working properly.

What is the reason of this problem?

file: w_PythonHelper.py
desc: Function 'run_in_parallel' is used to run multiple processes simultaneously. The first method, which will end operation stops the others.

Expand|Select|Wrap|Line Numbers
  1. from multiprocessing import Process,Event
  2.  
  3. class ExtProcess(Process):
  4.     def __init__(self, event,*args,**kwargs):
  5.         self.event=event
  6.         Process.__init__(self,*args,**kwargs)
  7.  
  8.     def run(self):
  9.         Process.run(self)
  10.         self.event.set()
  11.  
  12. class PythonHelper(object):
  13.     @staticmethod
  14.     def run_in_parallel(*functions):
  15.         event=Event()
  16.         processes=dict()
  17.         for function in functions:
  18.             fname=function[0]
  19.             try:fargs=function[1]
  20.             except:fargs=list()
  21.             try:fproc=function[2]
  22.             except:fproc=1
  23.             for i in range(fproc):
  24.                 process=ExtProcess(event,target=fname,args=fargs)
  25.                 process.start()
  26.                 processes[process.pid]=process
  27.         event.wait()
  28.         for process in processes.values():
  29.             process.terminate()
  30.         for process in processes.values():
  31.             process.join()
  32.  
file: w_Recorder.py
desc: function 'capture' is used to grab a screenshot

Expand|Select|Wrap|Line Numbers
  1. from PIL import ImageGrab
  2.  
  3. def capture(self,filename=time.time(),extension="png"):
  4.     ImageGrab.grab().save("{f}.{e}".format(f=filename,e=extension))
  5.  
file: w_Decorators.py
desc: Running parallel a given function along with a method 'capture' of class 'Recorder'

Expand|Select|Wrap|Line Numbers
  1. from w_Recorder import Recorder
  2. from w_PythonHelper import PythonHelper
  3.  
  4. def check(function):
  5.     def wrapper(*args):
  6.         try:
  7.             recorder=Recorder()
  8.             PythonHelper.run_in_parallel([function,args],[recorder.capture])
  9.             print("success")
  10.         except Exception as e:
  11.             print("failure: {}".format(e))
  12.         return function
  13.     return wrapper
  14.  
file: w_Logger.py
desc: Main program (generates error)

Expand|Select|Wrap|Line Numbers
  1. from w_Decorators import check
  2. import time
  3.  
  4. class Logger(object):
  5.  
  6.     @check
  7.     def run_testcase(self):
  8.         # example function (runtime: 20s)
  9.         for i in range(20):
  10.             print("number: {}".format(i))
  11.             time.sleep(1)
  12.  
  13.     def run_logger(self):
  14.         self.run_testcase()
  15.  
  16.  
  17. if __name__=="__main__":
  18.     logger=Logger()
  19.     logger.run_logger()
  20.  
file: w_Logger.py
desc: Main program (works corectly)

Expand|Select|Wrap|Line Numbers
  1. from w_PythonHelper import PythonHelper
  2. from w_Recorder import Recorder
  3. import time
  4.  
  5. class Logger(object):
  6.  
  7.     def run_testcase(self):
  8.         # example function (runtime: 20s)
  9.         for i in range(20):
  10.             print("number: {}".format(i))
  11.             time.sleep(1)
  12.  
  13.     def run_logger(self):
  14.         recorder=Recorder()
  15.         PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])
  16.  
  17. if __name__=="__main__":
  18.     logger=Logger()
  19.     logger.run_logger()
  20.  
What is the difference that these same methods presented in the two cases work differently?

I got only one answer on this strange question:

Expand|Select|Wrap|Line Numbers
  1. import traceback
  2. def wrapper(function,*args):
  3.     try:
  4.         recorder=Recorder()
  5.         PythonHelper().run_in_parallel([function,args],[recorder.capture])
  6.         print("success")
  7.     except Exception,e:
  8.         print("failure: "+traceback.format_exc(10))
  9.  
from w_Decorators import wrapper

Expand|Select|Wrap|Line Numbers
  1. if __name__=="__main__":
  2.     logger=Logger()
  3.     wrapper(logger.run_testcase)
  4.  
I can use wrapper as a function, but can't use it as decorator. I have many functions decorated by this decorator and the simpliest way is to use multiprocessing inside decorator. But unfortunatly I can't solve this problem. Maybe someone has already solved a similar problem. I would be grateful for any hint.

I've trying to make run_testcase a top level function but still nothing.
Apr 29 '12 #1
Share this Question
Share on Google+
2 Replies

Expert 100+
P: 626
"pickle errors" generally means that multiprocessing can not pickle some method (you pass as class method as the target), which it uses to pass things around. The problem here is not the pickle error. The problem here is code that isn't tested as you go along. Start with smaller pieces and test each piece as you go along as you would then know that it is a multiprocessing problem and not a decorator problem, at least. And read the multiprocessing docs-->hint: you want to pass a function that is part of the class which will do the processing. Also this code
Expand|Select|Wrap|Line Numbers
  1. def capture(self,filename=time.time(),extension="png"):
  2.      ImageGrab.grab().save("{f}.{e}".format(f=filename,e=extensiom)) 
will use the same file every time since time.time() will not change. I will leave it up to you to discover why as you should acquire some more knowledge about Python before you attempt a project like this.
Apr 29 '12 #2

falekmarcin
P: 2
Yes. You are right about the timestamp (this is only example and doesn't affect the issue) but decorator works ok - decorator is working properly, is used for a long time and takes a lot of methods (including static methods). The problem occurred when using multiprocessing.

My Traceback:

Expand|Select|Wrap|Line Numbers
  1.     Traceback (most recent call last):
  2.       File "C:\Interpreters\Python32\lib\pickle.py", line 679, in save_global
  3.         klass = getattr(mod, name)
  4.     AttributeError: 'module' object has no attribute 'run_testcase'
  5.  
  6.     During handling of the above exception, another exception occurred:
  7.  
  8.     Traceback (most recent call last):
  9.       File "C:\EskyTests\w_Logger.py", line 19, in <module>
  10.         logger.run_logger()
  11.       File "C:\EskyTests\w_Logger.py", line 14, in run_logger
  12.         self.run_testcase()
  13.       File "C:\EskyTests\w_Decorators.py", line 14, in wrapper
  14.         PythonHelper.run_in_parallel([function,args],[recorder.capture])
  15.       File "C:\EskyTests\w_PythonHelper.py", line 25, in run_in_parallel
  16.         process.start()
  17.       File "C:\Interpreters\Python32\lib\multiprocessing\process.py", line 130, in start
  18.         self._popen = Popen(self)
  19.       File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 267, in __init__
  20.         dump(process_obj, to_child, HIGHEST_PROTOCOL)
  21.       File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 190, in dump
  22.         ForkingPickler(file, protocol).dump(obj)
  23.       File "C:\Interpreters\Python32\lib\pickle.py", line 237, in dump
  24.         self.save(obj)
  25.       File "C:\Interpreters\Python32\lib\pickle.py", line 344, in save
  26.         self.save_reduce(obj=obj, *rv)
  27.       File "C:\Interpreters\Python32\lib\pickle.py", line 432, in save_reduce
  28.         save(state)
  29.       File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
  30.         f(self, obj) # Call unbound method with explicit self
  31.       File "C:\Interpreters\Python32\lib\pickle.py", line 623, in save_dict
  32.         self._batch_setitems(obj.items())
  33.       File "C:\Interpreters\Python32\lib\pickle.py", line 656, in _batch_setitems
  34.         save(v)
  35.       File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
  36.         f(self, obj) # Call unbound method with explicit self
  37.       File "C:\Interpreters\Python32\lib\pickle.py", line 683, in save_global
  38.         (obj, module, name))
  39.     _pickle.PicklingError: Can't pickle <function run_testcase at 0x00000000027725C8>: it's not found as __main__.run_testcase
Apr 29 '12 #3

Post your reply

Sign in to post your reply or Sign up for a free account.