Python：多进程未完成工作

Question

我正在使用Python 2.7和multiprocessing::Pool来并行运行作业。我已经简化了下面的示例，但这就是它的主要内容。

使用apply_async()函数为我的字典中的每个人创建一个文件。但是当我检查文件是否正确创建时，有时会发现文件未被创建。

现在我认为我在使用multiprocessing::Pool时做错了什么。请问您有任何建议吗？

import os
from multiprocessing import Pool

def outputFile(person):
    ofh=open(person+'.txt','w')
    ofh.write('test\n')
    ofh.close()

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputFile,args(person))
pool.close()
pool.join()
for person in person_dict:
    print os.path.isfile(person+'.txt')

---

True
True
False
True

- DolphinGenomePyramids

Answer 1

如果您没有在子进程中捕获并自己打印异常，那么您将看不到它们。以下程序没有任何输出：

import os
from multiprocessing import Pool

def outputFile(person):
    raise Exception("An exception")

pool = Pool(processes=4)
for person in range(100):
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

你需要捕获所有异常并手动打印回溯信息：

import os
from multiprocessing import Pool, Lock
import traceback

print_lock = Lock()

def outputFile(person):
    try:
        raise Exception("An exception")
    except:
        with print_lock:
            print "%s: An exception occurred" % person
            print traceback.format_exc()

pool = Pool(processes=4)
for person in range(100):
    args = (person, print_lock)
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

输出

0: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

1: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

...

99: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

注意：print_lock用于防止输出交错。

- user3657941

Answer 2

这是否与person_dict的内容有关？

我已经修改了您的代码并运行了多次。它们都产生了预期的结果。

这是我修改和测试过的代码：

import os
from multiprocessing import Pool

def outputfile(person):
    with open(person+'.txt','w') as ofh:
        ofh.write('test\n')

person_dict = {'a': 'a', 'b': 'b', 'c':'c', 'd':'d'}

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputfile, (person))
pool.close()
pool.join()

for person in person_dict:
    print(os.path.isfile(person+'.txt'))

- neurite

---

我不确定。通常情况下，我会说99%的时间我不会出错。但偶尔会出现进程中断且没有形成输出文件的情况。 - DolphinGenomePyramids

可能是子进程上的IO异常或一些运行时错误，这些错误在调用进程中不会显示出来。 - neurite