使用Keras多个验证集进行验证

Question

我正在使用Keras的model.fit()方法训练模型。 我想使用多个验证集，在每个训练时期后分别进行验证，以便我可以获得每个验证集的一个损失值。 如果可能的话，它们应该在训练期间同时显示，并且也应该由keras.callbacks.History()回调返回。

我考虑的是这样的：

history = model.fit(train_data, train_targets,
                    epochs=epochs,
                    batch_size=batch_size,
                    validation_data=[
                        (validation_data1, validation_targets1), 
                        (validation_data2, validation_targets2)],
                    shuffle=True)

我目前不知道如何实现这个。是否可以通过编写自己的Callback来实现？或者您会如何解决这个问题？

- Eror

Answer 1

我最终基于历史回调编写了自己的Callback来解决问题。我不确定这是否是最好的方法，但以下Callback记录了训练和验证集的损失和指标，就像History回调一样，还记录了传递给构造函数的其他验证集的损失和指标。

class AdditionalValidationSets(Callback):
    def __init__(self, validation_sets, verbose=0, batch_size=None):
        """
        :param validation_sets:
        a list of 3-tuples (validation_data, validation_targets, validation_set_name)
        or 4-tuples (validation_data, validation_targets, sample_weights, validation_set_name)
        :param verbose:
        verbosity mode, 1 or 0
        :param batch_size:
        batch size to be used when evaluating on the additional datasets
        """
        super(AdditionalValidationSets, self).__init__()
        self.validation_sets = validation_sets
        for validation_set in self.validation_sets:
            if len(validation_set) not in [3, 4]:
                raise ValueError()
        self.epoch = []
        self.history = {}
        self.verbose = verbose
        self.batch_size = batch_size

    def on_train_begin(self, logs=None):
        self.epoch = []
        self.history = {}

    def on_epoch_end(self, epoch, logs=None):
        logs = logs or {}
        self.epoch.append(epoch)

        # record the same values as History() as well
        for k, v in logs.items():
            self.history.setdefault(k, []).append(v)

        # evaluate on the additional validation sets
        for validation_set in self.validation_sets:
            if len(validation_set) == 3:
                validation_data, validation_targets, validation_set_name = validation_set
                sample_weights = None
            elif len(validation_set) == 4:
                validation_data, validation_targets, sample_weights, validation_set_name = validation_set
            else:
                raise ValueError()

            results = self.model.evaluate(x=validation_data,
                                          y=validation_targets,
                                          verbose=self.verbose,
                                          sample_weight=sample_weights,
                                          batch_size=self.batch_size)

            for metric, result in zip(self.model.metrics_names,results):
                valuename = validation_set_name + '_' + metric
                self.history.setdefault(valuename, []).append(result)

然后我像这样使用它:

history = AdditionalValidationSets([(validation_data2, validation_targets2, 'val2')])
model.fit(train_data, train_targets,
          epochs=epochs,
          batch_size=batch_size,
          validation_data=(validation_data1, validation_targets1),
          callbacks=[history]
          shuffle=True)

- Eror

---

在最新版本的Keras中，self.model.metrics[i-1]是一个字符串。 __name__不再必要（并且会生成错误）。 - Waylon Flinn

你考虑把这个做成一个Python包吗？如果你没有时间，我可以帮你做。 - Luca Cappelletti

@LucaCappelletti 之所以我没有把它制作成一个包，是因为在几乎每个项目中我都会略微修改回调函数（例如支持数据生成器）。 从一个项目中复制粘贴代码并不是最好的工作流程，但这样做会更容易一些，正如你已经猜到的那样，我没有时间将其打包（除非你想等待一个月左右）。 如果你有时间，请随意将其打包。 我已经在我的账户中添加了我的GitHub用户名，所以如果你想的话可以将我添加到项目中。 - Eror

谢谢！您能否分享支持数据生成器的此代码变体？ - Jonas Sourlier

@cheesus 该软件包的正在进行版本可以在Github上找到，代码太大无法在此处共享。https://github.com/LucaCappelletti94/keras_validation_sets - Eror

Answer 2

我在 TensorFlow 2 上测试过，它可以正常工作。您可以在每个 epoch 结束时对尽可能多的验证集进行评估：

class MyCustomCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        res_eval_1 = self.model.evaluate(X_test_1, y_test_1, verbose = 0)
        res_eval_2 = self.model.evaluate(X_test_2, y_test_2, verbose = 0)
        print(res_eval_1)
        print(res_eval_2)

之后：

my_val_callback = MyCustomCallback()
# Your model creation code
model.fit(..., callbacks=[my_val_callback])

- Nima Aghli

Answer 3

考虑到当前的keras文档，您可以将回调函数传递给evaluate和evaluate_generator。因此，您可以使用不同的数据集多次调用evaluate。

我还没有测试过它，所以如果您在下面评论您的经验，我会很高兴。

- mss

---

我的具体用例是在每个训练时期后对多个数据集进行评估，而不必在循环中一次只训练一个时期。如果您显式调用evaluate，那么为每个数据集调用一次应该没有问题，因此您在那里不需要回调。 - Eror

Answer 4

假设您的数据和预处理如下：

# Model / data parameters
num_classes = 10
input_shape = (28, 28, 1)

# Load the data and split it between train and test sets
(x_train, y_train), (x_valid, y_valid) = keras.datasets.mnist.load_data()
y_train_cp = y_train
y_valid_cp = y_valid

# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_valid = x_valid.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_valid = np.expand_dims(x_valid, -1)
print("x_train shape:", x_train.shape)
print(x_train.shape[0], "train samples")
print(x_valid.shape[0], "validation samples")


# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_valid = keras.utils.to_categorical(y_valid, num_classes)

您可以创建多个验证，例如

_, x_train_subset, _, y_train_subset = train_test_split(x_train, y_train, test_size=0.25, random_state=42)
    validationMasks = {}
for i in range(0,10):
    validationMasks[i] = y_valid_cp==i

#Modified Train to Evaluate Over fitting
validationSets = {}

validationSets['train_sub'] = datagen_valid.flow(x_train_subset,y_train_subset)

# Specail Case Validation
#y_valid[validationMasks[randomDigit]]
validationSets['1'] = datagen_train.flow(x_valid[validationMasks[1]],y_valid[validationMasks[1]]) #Least confusing
validationSets['0'] = datagen_train.flow(x_valid[validationMasks[0]],y_valid[validationMasks[0]]) 
validationSets['6'] = datagen_train.flow(x_valid[validationMasks[6]],y_valid[validationMasks[6]])
validationSets['8'] = datagen_train.flow(x_valid[validationMasks[8]],y_valid[validationMasks[8]])

现在，您可以使用自定义回调

class MultipleValidationCallBack(keras.callbacks.Callback):

def on_epoch_end(self, epoch, logs=None):
    for thisValidationType in validationSets:
        #datagen = 
        thisLoss = self.model.evaluate(validationSets[thisValidationType])
        logs[thisValidationType+"_loss"] = thisLoss[0]
        logs[thisValidationType+"_acc"] = thisLoss[1]

模型拟合可以像这样,

modelLog = getModel().fit(
        datagenFlow_train,validation_data=datagenFlow_valid,
        callbacks=[MultipleValidationCallBack()], #
        epochs=5)
historyDf = pd.DataFrame(modelLog.history)
historyDf[['train_sub_loss','val_loss','0_loss','1_loss']].plot()

详情请参考可运行的端到端代码，解释请看Medium上的一篇文章

- user2458922