使用lambda函数对defaultdict进行pickle处理

Question

使用lambda函数对defaultdict进行pickle处理

3

我试图对 defaultdict() 进行序列化，但是失败了。正确的方法是什么？

pos = defaultdict(lambda: 0)
neg = defaultdict(lambda: 0)    

countdata = self.getCountdata(pos, neg, totals)
cPickle.dump(countdata, open(CDATA_FILE, 'w'))

这将会得到：

Traceback (most recent call last):
  File "sentiment_worker.py", line 146, in <module>
    MyDict().gearman_worker.work()
  File "sentiment_worker.py", line 28, in __init__
    self.train()
  File "sentiment_worker.py", line 91, in train
    cPickle.dump(countdata, open(CDATA_FILE, 'w'))
  File "/usr/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

当我尝试使用dill时：

import dill
dill.dumps(countdata, open(CDATA_FILE, 'w') )

这将会得到：

 File "sentiment_worker.py", line 151, in <module>
    if __name__ == '__main__':
  File "sentiment_worker.py", line 29, in __init__
    self.train()
  File "sentiment_worker.py", line 96, in train
    # cPickle.dump(countdata, open(CDATA_FILE, 'w'))
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 243, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 204, in dump
    pik = Pickler(file, protocol)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 387, in __init__
    StockPickler.__init__(self, *args, **kwds)
  File "/usr/lib/python2.7/pickle.py", line 202, in __init__
    raise ValueError("pickle protocol must be <= %d" % HIGHEST_PROTOCOL)
ValueError: pickle protocol must be <= 2

如何正确地对内容进行pickle和unpickle操作？

- user2129623

3

你的 Lambda 函数只是一个示例，还是你实际上只想要一个 defaultdict(int) 呢？ - timgeb

你好，我是 dill 的作者。你正在使用 dumps 而不是 dump。因此，你的文件句柄被视为 pickle 协议。 - Mike McKerns

你尝试过 dump 吗？ - iNikkz

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Mike McKerns · Accepted Answer

你可以使用 dill 来完成。你犯了一个拼写错误...如果你想将内容 dump 到文件中，你应该使用 dill.dump 而不是 dill.dumps。如果你想将内容 dump 到字符串中，请使用 dumps。

>>> import dill
>>> from collections import defaultdict
>>> pos = defaultdict(lambda: 0)
>>> neg = defaultdict(lambda: 0)
>>> countdata = (pos,neg)
>>> _countdata = dill.loads(dill.dumps(countdata))
>>> _countdata
(defaultdict(<function <lambda> at 0x10917f7d0>, {}), defaultdict(<function <lambda> at 0x10917f8c0>, {}))
>>>
>>> # now dump countdata to a file 
>>> with open('data.pkl', 'wb') as f:
...     dill.dump(countdata, f)
...
>>>