我一直在研究Q学习和神经网络。我相信我对此有正确的理解，但是我想听听有关我的神经网络和用Q值进行更新的代码的第二意见。
我已经创建了Mountain Car问题的MatLab实现和我的神经网络，我正在使用神经网络工具箱进行NN部分的操作。
这是一个具有2个输入、5-20个隐藏层（用于实验）和3个输出的网络（对应于Mountain Car中的动作）。
隐藏单元设置为tansig，输出为purelin，训练函数为traingdm。
这些步骤是否正确？
- 获取初始状态s -> [-0.5; 0.0]
- 使用Qs = net(s)运行网络... 这为我提供了一个1x3的矩阵，其中每个动作在初始状态s中对应着一个Q值。
- 使用epsilon-greedy算法选择一个动作a
- 模拟Mountain Car并获得s'（执行动作a的结果导致的新状态）
- 使用Qs_prime = net（s'）再次运行网络以获取s'的另一个矩阵中的Q值
现在我不确定这是否正确，因为我必须找出如何正确地更新NN的权重。
- 计算QTarget，即= reward+gamma * s'中的最大Q值？
- 创建一个目标矩阵（1x3），其中包含来自初始s的Q值，并将执行的动作a对应的Q值更改为QTarget
- 使用net = Train(net，s，Targets)更新NN中的权重
- s = s'
- 为新的s重复以上所有步骤
例如：
actions
1 2 3
Qs = 1.3346 -1.9000 0.2371
selected action 3(corresponding to move mountain car forward)
Qs' = 1.3328 -1.8997 0.2463
QTarget=reward+gamma*max(Qs') = -1+1.0*1.3328 = 0.3328
s= [-5.0; 0.0] and Targets = 1.3346 -1.9000 0.3328
Or I have this wrong and the Targets are 0 0 0.3328
since we don't know how good the other actions are..
这是我的Matlab代码（我使用R2011和神经网络工具箱）：
%create a neural network
num_hidden=5
num_actions=3
net= newff([-1.2 0.6; -0.07 0.07;], [num_hidden,num_actions], {'tansig', 'purelin'},'traingdm');
%network weight and bias initalization
net= init(net);
%turn off the training window
net.trainParam.showWindow = false;
%neural network training parameters
net.trainParam.lr=0.01;
net.trainParam.mc=0.1;
net.trainParam.epochs=100
%parameters for q learning
epsilon=0.9;
gamma=1.0;
%parameters for Mountain car task
maxEpisodes =10;
maxSteps=5000;
reset=false;
inital_pos=-0.5;
inital_vel=0.0;
%construct the inital state
s=[inital_pos;inital_vel];
Qs=zeros(1,3);
Qs_prime=zeros(1,3);
%training for maxEpisodes
for i=1:maxEpisodes
%each episode is maxSteps long
for j = 1:maxSteps
%run the network and get Q values for current state Qs-> vector of
%current Q values for state s at time t Q(s_t)
Qs=net(s);
%select an action
if (rand() <= epsilon)
%returns max Q value over all actions
[Qs_value a]=max(Qs);
else
%return a random number between 1 and 3(inclusive)
a = randint(1,1,3)+1;
end
%simulate a step of Mountain Car
[s_prime, action, reward, reset] = SimulateMC(s,a);
%get new Q values for S_prime -> Q(s_t+1)
Qs_prime=net(s_prime);
%Compute Qtarget for weight updates given by r+y*max Q(s_t+1) over all
%actions
Q_target = reward+gamma*max(Qs_prime);
%Create a Targets matrix with the orginal state s q-values
Targets=Qs;
%change q-value of the original action to the QTarget
Targets(a)=Q_target;
% update the network for input state s and targets
[net TR]=train(net,s,Targets);
%update the state for next step
s=s_prime;
%display exactly where the car is to user the NN learns if this output reaches -0.45
disp(s(1))
if reset==true
bestSteps=j
break
end
end
%reset for new episode
reset=false;
s=[inital_pos;inital_vel];
end
%test the network
%reset state
s=[inital_pos;inital_vel];
for i=1:maxEpisodes
for j=1:maxSteps
%run the network and get Q values for current state
Qs=net(s);
%select the max action always
[Qs_value a]=max(Qs);
%simulate a step of Mountain Car
[s_prime, action, reward, reset] = SimulateMC(s,a);
s=s_prime;
disp(s(1))
end
s=[inital_pos;inital_vel];
end
谢谢