在限制下,
shelve
模块的文档声明如下:(多个同时读取访问是安全的。) shelve模块不支持对存储对象进行并发读写访问。
据我所知,这意味着只要我不尝试让多个进程同时写入单个架子,我就应该没问题了。多个进程使用相同的架子作为只读缓存应该是安全的。对吗?
显然不是这样。经过一些挣扎,我最终得到了一个测试用例,似乎证明了异步从架子中读取时会出现一些非常糟糕的行为。以下脚本:
- Creates a
Shelf
and populates it with"i" : 2*i
fori
from 1 to 10. - Reads all those values back, to assure they got stored correctly.
Spawns processes to retrieve values for each key from the shelf file, and reports whether a value was retrieved or not.
import multiprocessing import shelve SHELF_FILE = 'test.shlf' def store(key, obj): db = shelve.open(SHELF_FILE, 'w') db[key] = obj db.close() def load(key): try: db = shelve.open(SHELF_FILE, 'r') n = db.get(key) if n is not None: print('Got result {} for key {}'.format(n, key)) else: print('NO RESULT for key {}'.format(key)) except Exception as e: print('ERROR on key {}: {}'.format(key, e)) finally: db.close() if __name__ == '__main__': db = shelve.open(SHELF_FILE, 'n') # Create brand-new shelf db.close() for i in range(1, 11): # populate the new shelf with keys from 1 to 10 store(str(i), i*2) db = shelve.open(SHELF_FILE, 'r') # Make sure everything got in there. print(', '.join(key for key in db)) # Should print 1-10 in some order db.close() # read each key's value from the shelf, asynchronously pool = multiprocessing.Pool() for i in range(1, 11): pool.apply_async(load, [str(i)]) pool.close() pool.join()
2, 4, 6, 8
,以此类推,一直到20(以某种顺序)。但实际上,无法从架子上检索任意值,并且有时请求会导致shelve
完全崩溃。实际输出如下(“NO RESULT”行表示返回None
的键):6, 7, 4, 5, 2, 3, 1, 10, 8, 9
ERROR on key 3: need 'c' or 'n' flag to open new db
ERROR on key 6: need 'c' or 'n' flag to open new db
Got result 14 for key 7
NO RESULT for key 10
Got result 2 for key 1
Got result 4 for key 2
NO RESULT for key 8
NO RESULT for key 4
NO RESULT for key 5
NO RESULT for key 9
根据错误信息,我的直觉是可能外部资源(比如
.dir
文件?)没有被适当地刷新到磁盘上(或者它们被其他进程删除了?)。即使这样,我也期望会出现进程等待磁盘资源的减速情况,而不是出现这些“噢,我想它不在这里了”或“你在说什么,这甚至不是一个架子文件”的结果。老实说,我根本不希望有任何写入这些文件的操作,因为工作进程只使用只读连接......我错过了什么,还是
shelve
在多进程环境中就是无法使用?这是在 Windows 7 上的 Python 3.3 x64,如果相关的话。