在Windows上进行多进程处理出现问题

Question

在Windows上进行多进程处理出现问题

10

我在Linux上使用Python进行开发，从未遇到过Windows上出现的这种问题。我正在使用multiprocessing库来加速计算，在Linux上运行得非常好。

但是在Windows上，情况并不如此顺利：

 * [INFO] Parsing 1 file using 2 threads

Traceback (most recent call last):
  File "main.py", line 170, in <module>
    master = ParsingMaster(parser, list(input_file), output_list, threads=num_threads)
Traceback (most recent call last):
  File "main.py", line 39, in __init__
  File "<string>", line 1, in <module>
    self.input_process.start()
  File "C:\Python26\lib\multiprocessing\forking.py", line 342, in main
  File "C:\Python26\lib\multiprocessing\process.py", line 104, in start
        self._popen = Popen(self)
self = load(from_parent)
  File "C:\Python26\lib\multiprocessing\forking.py", line 239, in __init__
  File "C:\Python26\lib\pickle.py", line 1370, in load
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python26\lib\multiprocessing\forking.py", line 162, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python26\lib\pickle.py", line 224, in dump
    return Unpickler(file).load()
    self.save(obj)
 File "C:\Python26\lib\pickle.py", line 858, in load
  File "C:\Python26\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 419, in save_reduce
    dispatch[key](self)
  File "C:\Python26\lib\pickle.py", line 880, in load_eof
    save(state)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    r aise EOFError
 File "C:\Python26\lib\pickle.py", line 649, in save_dict
EOFError
    self._batch_setitems(obj.iteritems())
  File "C:\Python26\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\multiprocessing\forking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python26\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python26\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python26\lib\pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "C:\Python26\lib\multiprocessing\managers.py", line 458, in __reduce__
    return type(self).from_address, \
AttributeError: type object 'SyncManager' has no attribute 'from_address'

我在Windows 7上同时测试Python 2.6和 2.7，但总是遇到同样的错误。有谁知道这是什么意思吗？

- Blender

1

你有阅读平台特定的文档吗？尤其是第一个限制吗？ - Sven Marnach

谢谢，我刚刚完成了。我只是将列表用作整个类的参数，因此不应该有任何腌制错误。我也没有子类化进程。 - Blender

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- user177800 · Accepted Answer

Windows有一些限制，以下是与您看到的错误相关的部分：

由于Windows缺少os.fork()，因此有一些额外的限制：

更多的可拾取性

确保所有传递给Process.__init__()的参数都是可拾取的。这意味着特别地，在Windows上不能直接使用绑定或未绑定的方法作为目标参数 - 只需定义一个函数并使用它。

此外，如果您子类化了Process，则请确保在调用Process.start()方法时实例将是可拾取的。

这意味着作为参数传递给 Process.__init__() 的某个东西无法被序列化或反序列化（在 Python 中进行的序列化）。它抱怨找不到该对象上的属性AttributeError: type object 'SyncManager' has no attribute 'from_address'，很可能是你的根本原因。那个SyncManager对象实际上是否可以被序列化符合pickle规则吗?

如果您是从Windows命令行运行此代码，您似乎也无法这样做。

不要这样做。将代码保存到文件中，并使用以下命令从文件中运行：

python myfile.py

那将解决你的问题。