我正在尝试使用深度学习解决二元分类问题,目标类别之间存在高度的不平衡性(500k vs 31K)。我想编写一个自定义损失函数,应该是这样的:
最小化(100-((predicted_smallerclass)/(total_smallerclass))*100)
请给我一些指导,如何构建这个逻辑。
我正在尝试使用深度学习解决二元分类问题,目标类别之间存在高度的不平衡性(500k vs 31K)。我想编写一个自定义损失函数,应该是这样的:
最小化(100-((predicted_smallerclass)/(total_smallerclass))*100)
请给我一些指导,如何构建这个逻辑。
你可以通过将对数值乘以类别权重来向损失函数中添加类别权重。 常规的交叉熵损失函数如下:
loss(x, class) = -log(exp(x[class]) / (\sum_j exp(x[j])))
= -x[class] + log(\sum_j exp(x[j]))
对于加权情形:
loss(x, class) = weights[class] * -x[class] + log(\sum_j exp(weights[class] * x[j]))
因此,通过对logits进行乘法运算,您可以按其类别权重重新调整每个类别的预测。
例如:
ratio = 31.0 / (500.0 + 31.0)
class_weight = tf.constant([ratio, 1.0 - ratio])
logits = ... # shape [batch_size, 2]
weighted_logits = tf.mul(logits, class_weight) # shape [batch_size, 2]
xent = tf.nn.softmax_cross_entropy_with_logits(
weighted_logits, labels, name="xent_raw")
现在有一个支持每批次权重的标准损失函数:
tf.losses.sparse_softmax_cross_entropy(labels=label, logits=logits, weights=weights)
应该将权重从类别权重转换为每个样本的权重(形状为[batch_size])。请参见此处的文档。
您提出的代码似乎有误。我同意应该将损失乘以权重。
但是,如果您将逻辑输出乘以类别权重,最终会得到:
weights[class] * -x[class] + log( \sum_j exp(x[j] * weights[class]) )
第二项并不等于:
weights[class] * log(\sum_j exp(x[j]))
为了展示这一点,我们可以将后面的内容改写为:
log( (\sum_j exp(x[j]) ^ weights[class] )
所以这是我提出的代码:
ratio = 31.0 / (500.0 + 31.0)
class_weight = tf.constant([[ratio, 1.0 - ratio]])
logits = ... # shape [batch_size, 2]
weight_per_label = tf.transpose( tf.matmul(labels
, tf.transpose(class_weight)) ) #shape [1, batch_size]
# this is the weight for each datapoint, depending on its label
xent = tf.mul(weight_per_label
, tf.nn.softmax_cross_entropy_with_logits(logits, labels, name="xent_raw") #shape [1, batch_size]
loss = tf.reduce_mean(xent) #shape 1
使用tf.nn.weighted_cross_entropy_with_logits()
函数并将pos_weight
设置为1 /(预期正样本比例)。
inputs, labels = LoadData(batch_size=3)
logits = MyModelPredictions(inputs)
# Ensures that the loss for examples whose ground truth class is `3` is 5x
# higher than the loss for all other examples.
weight = tf.multiply(4, tf.cast(tf.equal(labels, 3), tf.float32)) + 1
onehot_labels = tf.one_hot(labels, num_classes=5)
tf.contrib.losses.softmax_cross_entropy(logits, onehot_labels, weight=weight)
我曾经需要处理一个多类别的、类别不平衡的数据集,以下是我的解决方案,希望能对正在寻找相似解决方案的人有所帮助:
以下内容应该放在您的训练模块中:
from sklearn.utils.class_weight import compute_sample_weight
#use class weights for handling unbalanced dataset
if mode == 'INFER' #test/dev mode, not weighing loss in test mode
sample_weights = np.ones(labels.shape)
else:
sample_weights = compute_sample_weight(class_weight='balanced', y=labels)
这段代码应放在您的模型类定义内:
#an extra placeholder for sample weights
#assuming you already have batch_size tensor
self.sample_weight = tf.placeholder(dtype=tf.float32, shape=[None],
name='sample_weights')
cross_entropy_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
labels=self.label, logits=logits,
name='cross_entropy_loss')
cross_entropy_loss = tf.reduce_sum(cross_entropy_loss*self.sample_weight) / batch_size
我使用tf.nn.weighted_cross_entropy_with_logits()函数处理了两个类别的操作:
classes_weights = tf.constant([0.1, 1.0])
cross_entropy = tf.nn.weighted_cross_entropy_with_logits(logits=logits, targets=labels, pos_weight=classes_weights)
""" Weighted binary crossentropy between an output tensor and a target tensor.
# Arguments
pos_weight: A coefficient to use on the positive examples.
# Returns
A loss function supposed to be used in model.compile().
"""
def weighted_binary_crossentropy(pos_weight=1):
def _to_tensor(x, dtype):
"""Convert the input `x` to a tensor of type `dtype`.
# Arguments
x: An object to be converted (numpy array, list, tensors).
dtype: The destination type.
# Returns
A tensor.
"""
return tf.convert_to_tensor(x, dtype=dtype)
def _calculate_weighted_binary_crossentropy(target, output, from_logits=False):
"""Calculate weighted binary crossentropy between an output tensor and a target tensor.
# Arguments
target: A tensor with the same shape as `output`.
output: A tensor.
from_logits: Whether `output` is expected to be a logits tensor.
By default, we consider that `output`
encodes a probability distribution.
# Returns
A tensor.
"""
# Note: tf.nn.sigmoid_cross_entropy_with_logits
# expects logits, Keras expects probabilities.
if not from_logits:
# transform back to logits
_epsilon = _to_tensor(K.epsilon(), output.dtype.base_dtype)
output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
output = log(output / (1 - output))
target = tf.dtypes.cast(target, tf.float32)
return tf.nn.weighted_cross_entropy_with_logits(labels=target, logits=output, pos_weight=pos_weight)
def _weighted_binary_crossentropy(y_true, y_pred):
return K.mean(_calculate_weighted_binary_crossentropy(y_true, y_pred), axis=-1)
return _weighted_binary_crossentropy
使用方法:
pos = #count of positive class
neg = #count of negative class
total = pos + neg
weight_for_0 = (1 / neg)*(total)/2.0
weight_for_1 = (1 / pos)*(total)/2.0
class_weight = {0: weight_for_0, 1: weight_for_1}
model = <your model>
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
loss=weighted_binary_crossentropy(weight_for_1),
metrics=tf.keras.metrics.Precision(name='precision')
)
\sum_
——你能解释一下吗?它似乎是LaTeX代码; 在Python中可以使用吗? - Ron Cohen