创建无法被pickle的对象

Question

如何轻松创建一个无法被pickle的对象，以便在我的RPC代码中测试边缘情况？

它需要具备以下特点：

1. 简单易懂
2. 可靠性高（不会在未来版本的Python或pickle中出现问题）
3. 跨平台

编辑：预期使用方式如下：

class TestRPCServer:
    def foo(self):
        return MagicalUnpicklableObject()

def test():
    with run_rpc_server_and_connect_to_it() as proxy:
        with nose.assert_raises(pickle.PickleError):
            proxy.foo()

- cube

---

你在构建什么？我们可以看一下相关对象的示例代码吗？ - GLaDOS

Answer 1

如果您只需要一个对象，在pickle时会抛出异常，用于测试的目的，您可以炸掉 __getstate__ 方法。

>>> class C:
...     def __getstate__(self):
...         raise Exception
... 
>>> pickle.dumps(C())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 723, in save_inst
    stuff = getstate()
  File "<stdin>", line 3, in __getstate__
Exception

---

如果你想要一个不那么人为的场景，可以考虑使用操作系统资源的对象，比如文件句柄、套接字、线程等。

>>> with open('spam.txt', 'w') as f:
...     pickle.dumps(f)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle file objects

- Benjamin Hodgson

---

第一种方法似乎正是我所需要的，谢谢。我卡住了，因为我试图让它抛出PickleError，但实际上这是不必要的。制作一个自定义异常类并抛出即可。 - cube

Answer 2

如果你想要一个明确的可以被pickle序列化的对象列表和不能被pickle序列化的对象列表，都使用pickle以及更高级的序列化工具如dill，这个文件包含了标准库对象的相对全面的列表。它提供了构建每个对象的简单方法（通常是一行代码），并显示了适用于不同Python版本的变体。

https://github.com/uqfoundation/dill/blob/cccbea9b715e16b742288e1e5a21a687a4d4081b/dill/_objects.py#L255

例如，pickle 对于以下对象会失败，而像 dill 这样的高级序列化器则不会：

>>> import dill
>>> dill.dumps(Ellipsis)
b'\x80\x03cdill.dill\n_eval_repr\nq\x00X\x08\x00\x00\x00Ellipsisq\x01\x85q\x02Rq\x03.'

高级序列化器也可以处理文件对象等，顺便说一下。

>>> dill.dumps(open('foo.pkl', 'w'))
b'\x80\x03cdill.dill\n_create_filehandle\nq\x00(X\x07\x00\x00\x00foo.pklq\x01X\x01\x00\x00\x00wq\x02K\x00\x89cdill.dill\n_get_attr\nq\x03cdill.dill\n_import_module\nq\x04X\x02\x00\x00\x00ioq\x05\x85q\x06Rq\x07X\x04\x00\x00\x00openq\x08\x86q\tRq\n\x89K\x00X\x00\x00\x00\x00q\x0btq\x0cRq\r.'

然而，pickle和dill（以及其他高级序列化工具）将在任何与python FrameType 直接相关的类型上失败，例如生成器：

>>> dill.dumps((i for i in []))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 243, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/mmckerns/lib/python3.4/site-packages/dill-0.2.6.dev0-py3.4.egg/dill/dill.py", line 236, in dump
    pik.dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 412, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pickle.py", line 499, in save
    rv = reduce(self.proto)
TypeError: can't pickle generator objects

- Mike McKerns