我有一个Python应用程序,它会生成一个单独的进程来完成一些工作(由于GIL(全局解释器锁)的性能问题,使用线程会导致问题)。那么,在Python中,如何同步跨进程共享资源呢?
我将数据移入一个队列,然后生成的进程会在从队列接收到数据时进行处理。但是,我需要确保数据以有序的方式输出,与复制时相同的顺序,因此我需要确保任何时候只有一个进程可以从/向队列读取/写入。最好的方法是什么?
谢谢, Ron
我有一个Python应用程序,它会生成一个单独的进程来完成一些工作(由于GIL(全局解释器锁)的性能问题,使用线程会导致问题)。那么,在Python中,如何同步跨进程共享资源呢?
我将数据移入一个队列,然后生成的进程会在从队列接收到数据时进行处理。但是,我需要确保数据以有序的方式输出,与复制时相同的顺序,因此我需要确保任何时候只有一个进程可以从/向队列读取/写入。最好的方法是什么?
谢谢, Ron
我认为你需要一个信号量,可以查看以下示例代码:
import threading
import datetime
class ThreadClass(threading.Thread):
def run(self):
now = datetime.datetime.now()
pool.acquire()
print "%s says hello, World! at time: %s" % (self.getName(),now)
pool.release()
pool = threading.BoundedSemaphore(value=1)
for i in range(10):
t = ThreadClass()
t.start()
输出结果如下:
Thread-1 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-2 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-3 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-4 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-5 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-6 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-7 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-8 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-9 says hello, World! at time: 2013-05-20 18:57:47.609000
Thread-10 says hello, World! at time: 2013-05-20 18:57:47.609000
相反:
import threading
import datetime
class ThreadClass(threading.Thread):
def run(self):
now = datetime.datetime.now()
print "%s says hello, World! at time: %s" % (self.getName(),now)
for i in range(10):
t = ThreadClass()
t.start()
输出结果如下:
Thread-1 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-2 says hello, World! at time: 2013-05-20 18:58:05.
531000
Thread-4 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-3 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-6 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-5 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-8 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-7 says hello, World! at time: 2013-05-20 18:58:05
.531000
Thread-10 says hello, World! at time: 2013-05-20 18:58:05.531000Thread-9 says hello, World! at time: 2013-05-20 18:58:0
5.531000
使用Python中的BoundedSemaphore,您可以确保在任何人向队列写入之前,他们必须拥有该信号量。但是,这并不能确保将结果按正确顺序添加到队列中。
编辑:
如果要执行此操作并保持顺序,您需要类似于以下内容:
import multiprocessing
import datetime
import random
import time
def funfun(number):
time.sleep(random.randint(0,10))
now = datetime.datetime.now()
return "%s says hello, World! at time: %s" % (number,now)
if __name__ == "__main__":
pool = multiprocessing.Pool(10)
for item in pool.imap(funfun,[i for i in range(10)]):
print item
这将会打印:
0 says hello, World! at time: 2013-05-21 00:38:48.546000
1 says hello, World! at time: 2013-05-21 00:38:55.562000
2 says hello, World! at time: 2013-05-21 00:38:47.562000
3 says hello, World! at time: 2013-05-21 00:38:51.578000
4 says hello, World! at time: 2013-05-21 00:38:50.578000
5 says hello, World! at time: 2013-05-21 00:38:48.593000
6 says hello, World! at time: 2013-05-21 00:38:52.593000
7 says hello, World! at time: 2013-05-21 00:38:48.593000
8 says hello, World! at time: 2013-05-21 00:38:50.593000
9 says hello, World! at time: 2013-05-21 00:38:51.609000
multiprocessing.Pool.map
吗?它可以按顺序给出结果。 - Janne Karila