我正在尝试使用Python的multiprocessing
模块来处理一组大量的URL,我正在使用一个multiprocessing.Pool
对象创建工作进程,如下所示。
from multiprocessing import Pool, TimeoutError, cpu_count
class MyClass:
def square(self, x):
return x*x
@staticmethod
def getNumbers():
return range(10)
def calculate(self):
pool = Pool(processes=min(cpu_count(),8))
results = [pool.apply(self.square,(i,)) for i in self.getNumbers()]
pool.close()
pool.join()
for result in results:
print result
if __name__ == '__main__':
instance = MyClass()
instance.calculate()
然而,上述代码会导致以下拣选错误:
Traceback (most recent call last):
File "multi.py", line 24, in <module>
instance.calculate()
File "multi.py", line 15, in calculate
results = [pool.apply(self.square,(i,)) for i in self.getNumbers()]
File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
注意 之前有人在SO上提出了类似的问题,但是没有得到答案:cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
编辑 提供了更好的样例代码
pool.apply(self.processURL, injectionPoint, "GET")
应该改为pool.apply(self.processURL, (injectionPoint, "GET"))
吧? - Shaungmultiprocessing
中pickle<type 'instancemethod'>:
的问题,并正确处理以在类实例方法中使用。此链接提到的技巧:http://www.rueckstiess.net/research/snippets/show/ca1d7d90 - bawejakunal