我正在使用进化算法(CMAES)进行函数优化。为了更快地运行,我使用了多进程模块。我需要优化的函数在下面的代码中输入大矩阵
(input_A_Opt 和 input_B_Opt)
。
它们的大小达到几个GB。当我不使用多进程运行函数时,它可以正常工作。但是当我使用多进程时,似乎存在内存问题。如果我使用小的输入运行它,那么它可以正常工作,但是当我使用完整的输入运行时,就会出现以下错误:
File "<ipython-input-2-bdbae5b82d3c>", line 1, in <module>
opt.myFuncOptimization()
File "/home/joe/Desktop/optimization_folder/Python/Optimization.py", line 45, in myFuncOptimization
**f_values = pool.map_async(partial_function_to_optmize, solutions).get()**
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/usr/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
put(task)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
error: 'i' format requires -2147483648 <= number <= 2147483647
这是代码的简化版本(如果我使用输入的大小减小10倍运行它,一切正常):
import numpy as np
import cma
import multiprocessing as mp
import functools
import myFuncs
import hdf5storage
def myFuncOptimization ():
temp = hdf5storage.loadmat('/home/joe/Desktop/optimization_folder/matlab_workspace_for_optimization')
input_A_Opt = temp["input_A"]
input_B_Opt = temp["input_B"]
del temp
numCores = 20
# Inputs
#________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
P0 = np.array([ 4.66666667, 2.5, 2.66666667, 4.16666667, 0.96969697, 1.95959596, 0.44088176, 0.04040404, 6.05210421, 0.58585859, 0.46464646, 8.75751503, 0.16161616, 1.24248497, 1.61616162, 1.56312625, 5.85858586, 0.01400841, 1.0, 2.4137931, 0.38076152, 2.5, 1.99679872 ])
LBOpt = np.array([ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ])
UBOpt = np.array([ 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, ])
initialStdsOpt = np.array([2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, ])
minStdsOpt = np.array([ 0.030, 0.40, 0.030, 0.40, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.020, 0.050, 0.050, 0.020, 0.40, 0.020, ])
options = {'bounds':[LBOpt,UBOpt], 'CMA_stds':initialStdsOpt, 'minstd':minStdsOpt, 'popsize':numCores}
es = cma.CMAEvolutionStrategy(P0, 1, options)
pool = mp.Pool(numCores)
partial_function_to_optmize = functools.partial(myFuncs.func1, input_A=input_A_Opt, input_B=input_B_Opt)
while not es.stop():
solutions = es.ask(es.popsize)
f_values = pool.map_async(partial_function_to_optmize, solutions).get()
es.tell(solutions, f_values)
es.disp(1)
es.logger.add()
return es.result_pretty()
有什么建议可以解决这个问题吗?我是不是编码不当(对Python很新)或者应该使用其他多进程包,比如Scoop?