从pickle文件加载multiprocessing.manager.dictionary时出现错误

5

我在加载一个multiprocessing.manager.dictionary对象时遇到了错误,我确定pickle文件存在于我运行Python的目录中。

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import multiprocessing
>>> a = open("test.pkl", "rb")
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__
    self._incref()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient
    s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1165, in load_put
    self.memo[self.readline()[:-1]] = self.stack[-1]
IndexError: list index out of range

这是保存为pickle格式的代码部分。
from multiprocessing import Manager, Pool
import pickle
from functools import partial

def output(dic, s):
    a = open(s + ".pkl", "wb")
    pickle.dump(dic, a)
    a.close()

data_list = [1,2,3,4]
pool = Pool(processes = 4)
m = Manager()
lock = m.Lock()
dic1 = m.dict()
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1
pool.map(func, data_list)
output(dic1, "test")
pool.close()
pool.join()

如果我在输出之前将字典打印出来,终端上的结果看起来很好。
1个回答

14

Manager.dict() 方法并不会返回一个真正的字典,而是返回一个代理对象,用于访问在不同进程中由管理器管理的实际字典。当您在该对象上调用方法时,调用将被转发到管理器,在那里实际字典才存在。

由于管理器和客户端进程之间的通信使用 pickle 协议进行,这些对象可进行 pickling,并且在取消 pickling 时会再次生成指向管理器进程内部实际对象的代理,这就是为什么在尝试取消 pickling 字典时会看到创建连接的尝试。

因此,如果您想要 pickle 管理器字典的内容,请先将其转换为实际字典:

...
pickle.dump(dict(dic), a)
...

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接