Python中多进程间的同步

Question

我有一个Python应用程序，它会生成一个单独的进程来完成一些工作（由于GIL（全局解释器锁）的性能问题，使用线程会导致问题）。那么，在Python中，如何同步跨进程共享资源呢？

我将数据移入一个队列，然后生成的进程会在从队列接收到数据时进行处理。但是，我需要确保数据以有序的方式输出，与复制时相同的顺序，因此我需要确保任何时候只有一个进程可以从/向队列读取/写入。最好的方法是什么？

谢谢， Ron

- stdcerr

---

http://en.wikipedia.org/wiki/Semaphore_(programming)? - Noelkd

Noelkd，谢谢 - 我找到了这个：http://docs.python.org/release/2.4.2/lib/semaphore-examples.html ，可以吗？ - stdcerr

没错，应该有一些更加更新的文档。如果没有其他人先写的话，我可能会写一个快速示例！ - Noelkd

你考虑过使用 multiprocessing.Pool.map 吗？它可以按顺序给出结果。 - Janne Karila

Answer 1

我认为你需要一个信号量，可以查看以下示例代码：

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        pool.acquire()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)
        pool.release()


pool = threading.BoundedSemaphore(value=1)


for i in range(10):
        t = ThreadClass()
        t.start()

输出结果如下：

Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000

相反：

import threading
import datetime


class ThreadClass(threading.Thread):
    def run(self):
        now = datetime.datetime.now()
        print "%s says hello, World! at time: %s"  % (self.getName(),now)




for i in range(10):
        t = ThreadClass()
        t.start()

输出结果如下：

Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000

 Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000

 Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000

使用Python中的BoundedSemaphore，您可以确保在任何人向队列写入之前，他们必须拥有该信号量。但是，这并不能确保将结果按正确顺序添加到队列中。

编辑：

如果要执行此操作并保持顺序，您需要类似于以下内容：

import multiprocessing
import datetime
import random
import time

def funfun(number):
    time.sleep(random.randint(0,10))
    now = datetime.datetime.now()
    return "%s says hello, World! at time: %s"  % (number,now)

if __name__ == "__main__":
    pool = multiprocessing.Pool(10)
    for item in pool.imap(funfun,[i for i in range(10)]):
        print item

这将会打印：

0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000

因此，您可以按正确顺序将作业附加到队列中，作业将等待它们的轮到添加到队列中。

- Noelkd