我的代码(遗传优化算法的一部分)并行运行几个进程,等待它们全部完成,然后读取输出,并使用不同的输入重复此过程。当我进行60次重复测试时,一切都很正常。既然如此,我决定使用更现实的重复次数200。但是我收到了以下错误:
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 302, in _handle_workers
pool._maintain_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 206, in _maintain_pool
self._repopulate_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 199, in _repopulate_pool
w.start()
File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 120, in __init__
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
这里是使用池的代码片段:
def RunMany(inputs):
from multiprocessing import cpu_count, Pool
proc=inputs[0]
pool=Pool(processes = proc)
results=[]
for arg1 in inputs[1]:
for arg2 in inputs[2]:
for arg3 in inputs[3]:
results.append(pool.apply_async(RunOne, args=(arg1, arg2, arg3)))
casenum=0
datadict=dict()
for p in results:
#get results of simulation once it has finished
datadict[casenum]=p.get()
casenum+=1
return datadict
RunOne函数会创建一个我所创造的类的对象,使用计算密集型的Python包来解决一个化学问题,大约需要30秒钟时间,并返回带有化学求解器输出的对象。我的代码以串行方式调用RunMany,并在其中并行调用RunOne。在我的测试中,我使用了10个处理器(计算机有16个)和20个对RunOne的调用池来调用RunOne。换句话说,len(arg1)* len(arg2)* len(arg3)= 20。当我的代码调用RunMany 60次时,一切都正常,但当我调用它200次时,我就没有内存了。
这是否意味着某个过程没有正确清理自己?我有内存泄漏吗?如何确定是否存在内存泄漏,如何找到泄漏的原因?我的200次循环中唯一增长的项是一个数字列表,它的大小从0增长到200。我有一个自定义类的对象字典,但其长度限制为50个条目 - 每次循环执行时,它会删除字典中的一个项目并用另一个项目替换它。
编辑:以下是调用RunMany代码的片段
for run in range(nruns):
#create inputs object for RunMany using genetic methods.
#Either use starting "population" or create "child" inputs from successful previous runs
datadict = RunMany(inputs)
sumsquare=0
for i in range(len(datadictsenk)): #input condition
sumsquare+=Compare(datadict[i],Target[i]) #compare result to target
with open(os.path.join(mainpath,'Outputs','output.txt'),'a') as f:
f.write('\t'.join([str(x) for x in [inputs.name, sumsquare]])+'\n')
Objective.append(sumsquare) #add sum of squares to list, to be plotted outside of loop
population[inputs]=sumsquare #add/update the model in the "population", using the inputs object as a key, and it's objective function as the value
if len(population)>initialpopulation:
population = PopulationReduction(population) #reduce the "population" by "killing" unfit "genes"
avgtime=(datetime.datetime.now()-starttime2)//(run+1)
remaining=(nruns-run-1)*avgtime
print(' Finished '+str(run+1)+' / ' +str(nruns)+'. Elapsed: '+str(datetime.datetime.now().replace(microsecond=0)-starttime)+' Remaining: '+str(remaining)+' Finish at '+str((datetime.datetime.now()+remaining).replace(microsecond=0))+'~~~', end="\r")
xrange
而不是range
,因为你只需要索引而不是列表。 - shuttle87pool.close()
和pool.join()
(在两个for
循环之间),一切似乎都很好。谢谢! - Jeff