如何在PyTorch中进行完全连接的批量归一化？

Question

torch.nn有BatchNorm1d、BatchNorm2d和BatchNorm3d类，但它没有全连接的批量归一化类？在PyTorch中进行正常的批量归一化的标准方法是什么？

- patapouf_ai

---

您认为这些层为何不是完全连接的？ - Maximilian

Answer 1

好的，我明白了。 BatchNorm1d 还可以处理二阶张量，因此在正常的全连接情况下也可以使用 BatchNorm1d。

例如：

import torch.nn as nn


class Policy(nn.Module):
def __init__(self, num_inputs, action_space, hidden_size1=256, hidden_size2=128):
    super(Policy, self).__init__()
    self.action_space = action_space
    num_outputs = action_space

    self.linear1 = nn.Linear(num_inputs, hidden_size1)
    self.linear2 = nn.Linear(hidden_size1, hidden_size2)
    self.linear3 = nn.Linear(hidden_size2, num_outputs)
    self.bn1 = nn.BatchNorm1d(hidden_size1)
    self.bn2 = nn.BatchNorm1d(hidden_size2)

def forward(self, inputs):
    x = inputs
    x = self.bn1(F.relu(self.linear1(x)))
    x = self.bn2(F.relu(self.linear2(x)))
    out = self.linear3(x)


    return out

- patapouf_ai

---

这可能与机器学习无关，但是super调用应该像super(Policy, self).__init__()而不是super(Policy2, self).__init__()吧？在Python3中，甚至可以简化为super().__init__()。 - jneuendorf

应该是 F.relu(self.bn1(self.linear1(x))) 吧。 - Mohit Lamba

Answer 2

BatchNorm1d通常在ReLU之前使用，偏置是多余的，因此可以省略。

import torch.nn as nn

class Policy(nn.Module):
def __init__(self, num_inputs, action_space, hidden_size1=256, hidden_size2=128):
    super(Policy2, self).__init__()
    self.action_space = action_space
    num_outputs = action_space

    self.linear1 = nn.Linear(num_inputs, hidden_size1, bias=False)
    self.linear2 = nn.Linear(hidden_size1, hidden_size2, bias=False)
    self.linear3 = nn.Linear(hidden_size2, num_outputs)
    self.bn1 = nn.BatchNorm1d(hidden_size1)
    self.bn2 = nn.BatchNorm1d(hidden_size2)

def forward(self, inputs):
    x = inputs
    x = F.relu(self.bn1(self.linear1(x)))
    x = F.relu(self.bn2(self.linear2(x)))
    out = self.linear3(x)

    return out

- cdude

---

根据这项研究，批量归一化应该在ReLU之后进行：https://github.com/ducha-aiki/caffenet-benchmark/blob/master/batchnorm.md - Rahul Madhavan