logo标签列表

内置函数与dill/pickle存在问题

pythonpickledill
3
当使用 dill 序列化一个类时：
import dill, pickle

class project(object):
    def __init__(self, name='', folder='', user_id='', version=-1 ):
        self.name, self.folder, self.user_id, self.version = name, folder, user_id, version
# Other code, not shown, since Big class .......

dill.dump(proj1, open('test_dill.pkl', 'w'), byref=False )

我收到了这个消息：

PicklingError: Can't pickle <built-in function raw_input>: it's not the same object as __builtin__.raw_input


  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 236, in dump
    pik.dump(obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 224, in dump
    self.save(obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 401, in save_reduce
    save(args)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 554, in save_tuple
    save(element)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 642, in _batch_appends
    save(tmp[0])

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 425, in save_reduce
    save(state)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 835, in save_module_dict
    StockPickler.save_dict(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 687, in _batch_setitems
    save(v)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

  File "D:\_devs\Python01\Anaconda27\lib\site-packages\dill\dill.py", line 993, in save_builtin_method
    StockPickler.save_global(pickler, obj)

  File "D:\_devs\Python01\Anaconda27\lib\pickle.py", line 759, in save_global
    (obj, module, name))

PicklingError: Can't pickle <built-in function raw_input>: it's not the same object as __builtin__.raw_input

我不确定内置函数是否可以被代码修改...听起来很神秘。
2
你的代码中没有raw_input...或许它被遗漏在...中了？ - Billy
无法复现。在去除'......'并创建一个名为'proj1'的对象后，这段代码运行正常（至少在Python 2上）。请发布一些可运行的内容，以证明问题。 - tdelaney
raw_input是内置类，没有在类代码中显示。raw_input的目的是什么？ - tensor
1个回答
4
我是dill的作者。不同的设置（例如byref）用于修改包含在dump中的对象类型。pickle和因此dill使用递归对象序列化，这意味着当一个对象引用另一个对象时，它可能（或可能不会）根据序列化设置对其进行序列化。pickle通常通过引用（byref=True）序列化函数和类，这意味着它们仅在导入时按名称序列化。dill有几个设置，例如byref=False（将函数和类对象序列化而不是引用）。还有recurse，它处理如何序列化全局引用--recurse=False将所有全局项序列化为dict，而recurse=True仅序列化全局引用的项目以及那些项目所引用的项目等。
简而言之，您正在序列化的任何内容都具有一些全局引用，这会触发递归序列化，直到它遇到dill无法pickle的内容。
以下是您可以尝试的一些方法：
1.将文件中的对象分开成单独的文件--这使您更多地依赖于引用pickling。
2.使用不同的设置，例如recurse=True，来更改dill尝试与目标对象一起序列化的对象。您可以使用dill.settings全局设置设置设置。
3.向您的类添加序列化助手方法，告诉pickle如何序列化您的类。
4.重构您的类（以某种方式，因为我无法看到其内容）以避免序列化问题。
由于您没有发布可演示错误的工作示例，因此很难深入回答。
错误信息显示：“PicklingError: 无法将<built-in function raw_input>进行封装：它与__builtin__.raw_input不是同一个对象”。这意味着在生成我的类的过程中，__builtin__方法已经被更改。我该如何更改内置类的定义？（看起来很奇怪）。 - tensor
1
@Tensor：不是这样的。看到这个错误最常见的原因是该对象（在本例中为builtin.raw_input）实际上并没有在该命名空间中定义，并且已经通过某种方式在那里进行了别名设置。这是一个非常常见的pickling错误，因为对象别名经常被使用。这就是Python 3引入qualname属性的原因之一——不幸的是，dill目前还没有利用它。 - Mike McKerns
除非看到你的代码，否则很难比我上面说的更多。如果你不提供一个简化的示例来演示你所看到的问题，那么你可能只会得到抽象的答案。 - Mike McKerns
dill 还包括一些 pickle 错误检测工具，位于 dill.detect 中。您也可以尝试使用它们来帮助诊断错误。 - Mike McKerns
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接