Using multiprocessing inside decorator

[falekmarcin]

I came across a problem that I can't solve and it's associated with multiprocessing and use it inside the decorator.

When I'm calling the method run_in_parallels using multiprocessing I 'm getting the error:

Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase

The call takes place inside the decorator, then followed the above-mentioned problem. At the time of calling the same method run_in_parallels without a decorator all working properly.

What is the reason of this problem?

file: w_PythonHelper.py
desc: Function 'run_in_parallel' is used to run multiple processes simultaneously. The first method, which will end operation stops the others.

from multiprocessing import Process,Event
 
class ExtProcess(Process):
    def __init__(self, event,*args,**kwargs):
        self.event=event
        Process.__init__(self,*args,**kwargs)
 
    def run(self):
        Process.run(self)
        self.event.set()
 
class PythonHelper(object):
    @staticmethod
    def run_in_parallel(*functions):
        event=Event()
        processes=dict()
        for function in functions:
            fname=function[0]
            try:fargs=function[1]
            except:fargs=list()
            try:fproc=function[2]
            except:fproc=1
            for i in range(fproc):
                process=ExtProcess(event,target=fname,args=fargs)
                process.start()
                processes[process.pid]=process
        event.wait()
        for process in processes.values():
            process.terminate()
        for process in processes.values():
            process.join()
 

file: w_Recorder.py
desc: function 'capture' is used to grab a screenshot

from PIL import ImageGrab
 
def capture(self,filename=time.time(),extension="png"):
    ImageGrab.grab().save("{f}.{e}".format(f=filename,e=extension))
 

file: w_Decorators.py
desc: Running parallel a given function along with a method 'capture' of class 'Recorder'

from w_Recorder import Recorder
from w_PythonHelper import PythonHelper
 
def check(function):
    def wrapper(*args):
        try:
            recorder=Recorder()
            PythonHelper.run_in_parallel([function,args],[recorder.capture])
            print("success")
        except Exception as e:
            print("failure: {}".format(e))
        return function
    return wrapper
 

file: w_Logger.py
desc: Main program (generates error)

from w_Decorators import check
import time
 
class Logger(object):
 
    @check
    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)
 
    def run_logger(self):
        self.run_testcase()
 
 
if __name__=="__main__":
    logger=Logger()
    logger.run_logger()
 

file: w_Logger.py
desc: Main program (works corectly)

from w_PythonHelper import PythonHelper
from w_Recorder import Recorder
import time
 
class Logger(object):
 
    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)
 
    def run_logger(self):
        recorder=Recorder()
        PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])
 
if __name__=="__main__":
    logger=Logger()
    logger.run_logger()
 

What is the difference that these same methods presented in the two cases work differently?

I got only one answer on this strange question:

import traceback
def wrapper(function,*args):
    try:
        recorder=Recorder()
        PythonHelper().run_in_parallel([function,args],[recorder.capture])
        print("success")
    except Exception,e:
        print("failure: "+traceback.format_exc(10))
 

from w_Decorators import wrapper

if __name__=="__main__":
    logger=Logger()
    wrapper(logger.run_testcase)
 

I can use wrapper as a function, but can't use it as decorator. I have many functions decorated by this decorator and the simpliest way is to use multiprocessing inside decorator. But unfortunatly I can't solve this problem. Maybe someone has already solved a similar problem. I would be grateful for any hint.

I've trying to make run_testcase a top level function but still nothing.

[dwblas]

"pickle errors" generally means that multiprocessing can not pickle some method (you pass as class method as the target), which it uses to pass things around. The problem here is not the pickle error. The problem here is code that isn't tested as you go along. Start with smaller pieces and test each piece as you go along as you would then know that it is a multiprocessing problem and not a decorator problem, at least. And read the multiprocessing docs-->hint: you want to pass a function that is part of the class which will do the processing. Also this code

def capture(self,filename=time.time(),extension="png"):
     ImageGrab.grab().save("{f}.{e}".format(f=filename,e=extensiom)) 

will use the same file every time since time.time() will not change. I will leave it up to you to discover why as you should acquire some more knowledge about Python before you attempt a project like this.

[falekmarcin]

Yes. You are right about the timestamp (this is only example and doesn't affect the issue) but decorator works ok - decorator is working properly, is used for a long time and takes a lot of methods (including static methods). The problem occurred when using multiprocessing.

My Traceback:

    Traceback (most recent call last):
      File "C:\Interpreters\Python32\lib\pickle.py", line 679, in save_global
        klass = getattr(mod, name)
    AttributeError: 'module' object has no attribute 'run_testcase'
 
    During handling of the above exception, another exception occurred:
 
    Traceback (most recent call last):
      File "C:\EskyTests\w_Logger.py", line 19, in <module>
        logger.run_logger()
      File "C:\EskyTests\w_Logger.py", line 14, in run_logger
        self.run_testcase()
      File "C:\EskyTests\w_Decorators.py", line 14, in wrapper
        PythonHelper.run_in_parallel([function,args],[recorder.capture])
      File "C:\EskyTests\w_PythonHelper.py", line 25, in run_in_parallel
        process.start()
      File "C:\Interpreters\Python32\lib\multiprocessing\process.py", line 130, in start
        self._popen = Popen(self)
      File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 267, in __init__
        dump(process_obj, to_child, HIGHEST_PROTOCOL)
      File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 190, in dump
        ForkingPickler(file, protocol).dump(obj)
      File "C:\Interpreters\Python32\lib\pickle.py", line 237, in dump
        self.save(obj)
      File "C:\Interpreters\Python32\lib\pickle.py", line 344, in save
        self.save_reduce(obj=obj, *rv)
      File "C:\Interpreters\Python32\lib\pickle.py", line 432, in save_reduce
        save(state)
      File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
        f(self, obj) # Call unbound method with explicit self
      File "C:\Interpreters\Python32\lib\pickle.py", line 623, in save_dict
        self._batch_setitems(obj.items())
      File "C:\Interpreters\Python32\lib\pickle.py", line 656, in _batch_setitems
        save(v)
      File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
        f(self, obj) # Call unbound method with explicit self
      File "C:\Interpreters\Python32\lib\pickle.py", line 683, in save_global
        (obj, module, name))
    _pickle.PicklingError: Can't pickle <function run_testcase at 0x00000000027725C8>: it's not found as __main__.run_testcase