以下是如何操作:
loss
变量将是输出相对于目标的交叉熵损失和L1惩罚的总和。下面是一个示例代码:
import torch
from torch.autograd import Variable
from torch.nn import functional as F
class MLP(torch.nn.Module):
def __init__(self):
super(MLP, self).__init__()
self.linear1 = torch.nn.Linear(128, 32)
self.linear2 = torch.nn.Linear(32, 16)
self.linear3 = torch.nn.Linear(16, 2)
def forward(self, x):
layer1_out = F.relu(self.linear1(x))
layer2_out = F.relu(self.linear2(layer1_out))
out = self.linear3(layer2_out)
return out, layer1_out, layer2_out
batchsize = 4
lambda1, lambda2 = 0.5, 0.01
model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
# usually following code is looped over all batches
# but let's just do a dummy batch for brevity
inputs = Variable(torch.rand(batchsize, 128))
targets = Variable(torch.ones(batchsize).long())
optimizer.zero_grad()
outputs, layer1_out, layer2_out = model(inputs)
cross_entropy_loss = F.cross_entropy(outputs, targets)
all_linear1_params = torch.cat([x.view(-1) for x in model.linear1.parameters()])
all_linear2_params = torch.cat([x.view(-1) for x in model.linear2.parameters()])
l1_regularization = lambda1 * torch.norm(all_linear1_params, 1)
l2_regularization = lambda2 * torch.norm(all_linear2_params, 2)
loss = cross_entropy_loss + l1_regularization + l2_regularization
loss.backward()
optimizer.step()
norm(all_linear2_params, 2)
,torch返回的是L2正则化的平方根。也就是说,表达式应该被求平方。 - Yuval Atzmonl1_regularization = lambda1 * torch.norm(layer1_out, 1)
l2_regularization = lambda2 * torch.norm(layer2_out, 2)
- אלימלך שרייבר其他所有(当前的)回答都在某种程度上是错误的,因为问题是关于将正则化添加到激活函数中的。 这个答案 是最接近的,因为它建议对输出的范数求和,这是正确的,但代码对权重的范数求和是不正确的。
正确的方法不是修改网络代码,而是通过前向钩子来捕获输出,就像 OutputHook
类中所示。从那里开始,对输出的范数求和很简单,但需要注意每次迭代清除已捕获的输出。
import torch
class OutputHook(list):
""" Hook to capture module outputs.
"""
def __call__(self, module, input, output):
self.append(output)
class MLP(torch.nn.Module):
def __init__(self):
super(MLP, self).__init__()
self.linear1 = torch.nn.Linear(128, 32)
self.linear2 = torch.nn.Linear(32, 16)
self.linear3 = torch.nn.Linear(16, 2)
# Instantiate ReLU, so a hook can be registered to capture its output.
self.relu = torch.nn.ReLU()
def forward(self, x):
layer1_out = self.relu(self.linear1(x))
layer2_out = self.relu(self.linear2(layer1_out))
out = self.linear3(layer2_out)
return out
batch_size = 4
l1_lambda = 0.01
model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
# Register hook to capture the ReLU outputs. Non-trivial networks will often
# require hooks to be applied more judiciously.
output_hook = OutputHook()
model.relu.register_forward_hook(output_hook)
inputs = torch.rand(batch_size, 128)
targets = torch.ones(batch_size).long()
optimizer.zero_grad()
outputs = model(inputs)
cross_entropy_loss = torch.nn.functional.cross_entropy(outputs, targets)
# Compute the L1 penalty over the ReLU outputs captured by the hook.
l1_penalty = 0.
for output in output_hook:
l1_penalty += torch.norm(output, 1)
l1_penalty *= l1_lambda
loss = cross_entropy_loss + l1_penalty
loss.backward()
optimizer.step()
output_hook.clear()
import torch
from torch.autograd import Variable
from torch.nn import functional as F
class MLP(torch.nn.Module):
def __init__(self):
super(MLP, self).__init__()
self.linear1 = torch.nn.Linear(128, 32)
self.linear2 = torch.nn.Linear(32, 16)
self.linear3 = torch.nn.Linear(16, 2)
def forward(self, x):
layer1_out = F.relu(self.linear1(x))
layer2_out = F.relu(self.linear2(layer1_out))
out = self.linear3(layer2_out)
return out
batchsize = 4
lambda1, lambda2 = 0.5, 0.01
model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
inputs = Variable(torch.rand(batchsize, 128))
targets = Variable(torch.ones(batchsize).long())
l1_regularization, l2_regularization = torch.tensor(0), torch.tensor(0)
optimizer.zero_grad()
outputs = model(inputs)
cross_entropy_loss = F.cross_entropy(outputs, targets)
for param in model.parameters():
l1_regularization += torch.norm(param, 1)**2
l2_regularization += torch.norm(param, 2)**2
loss = cross_entropy_loss + l1_regularization + l2_regularization
loss.backward()
optimizer.step()
norm(param, 2)
:torch 返回 L2 正则化的 平方根。也就是说,表达式应该被提高到 2 的幂次。 - Yuval Atzmonmy_layer
的权重应用L1正则化到损失函数中的单个层:def l1_penalty(params, l1_lambda=0.001):
"""Returns the L1 penalty of the params."""
l1_norm = sum(p.abs().sum() for p in params)
return l1_lambda*l1_norm
loss = loss_fn(outputs, labels) + l1_penalty(my_layer.parameters())