在测试数据上使用MultilabelBinarizer，其中标签不在训练集中。

Question

考虑下面这个多标签分类的简单示例（取自于这个问题：use scikit-learn to classify into multiple categories）

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn import preprocessing
from sklearn.metrics import accuracy_score

X_train = np.array(["new york is a hell of a town",
                "new york was originally dutch",
                "the big apple is great",
                "new york is also called the big apple",
                "nyc is nice",
                "people abbreviate new york city as nyc",
                "the capital of great britain is london",
                "london is in the uk",
                "london is in england",
                "london is in great britain",
                "it rains a lot in london",
                "london hosts the british museum",
                "new york is great and so is london",
                "i like london better than new york"])
y_train_text = [["new york"],["new york"],["new york"],["new york"],    ["new york"],
            ["new york"],["london"],["london"],["london"],["london"],
            ["london"],["london"],["new york","london"],["new york","london"]]

X_test = np.array(['nice day in nyc',
               'welcome to london',
               'london is rainy',
               'it is raining in britian',
               'it is raining in britian and the big apple',
               'it is raining in britian and nyc',
               'hello welcome to new york. enjoy it here and london too'])

y_test_text = [["new york"],["london"],["london"],["london"],["new york", "london"],["new york", "london"],["new york", "london"]]


lb = preprocessing.MultiLabelBinarizer()
Y = lb.fit_transform(y_train_text)
Y_test = lb.fit_transform(y_test_text)

classifier = Pipeline([
('vectorizer', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)


print "Accuracy Score: ",accuracy_score(Y_test, predicted)

代码运行良好，打印出了准确率分数，但是如果我将y_test_text更改为

y_test_text = [["new york"],["london"],["england"],["london"],["new york", "london"],["new york", "london"],["new york", "london"]]

我理解了

Traceback (most recent call last):
  File "/Users/scottstewart/Documents/scikittest/example.py", line 52, in <module>
     print "Accuracy Score: ",accuracy_score(Y_test, predicted)
  File "/Library/Python/2.7/site-packages/sklearn/metrics/classification.py", line 181, in accuracy_score
differing_labels = count_nonzero(y_true - y_pred, axis=1)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/compressed.py", line 393, in __sub__
raise ValueError("inconsistent shapes")
ValueError: inconsistent shapes

请注意，引入了“英格兰”标签，该标签不在训练集中。如何使用多标签分类，以便如果引入“测试”标签，则仍然可以运行某些指标？或者这是否可能？

编辑：谢谢大家的回答，我想我的问题更多地涉及scikit二进制转换器的工作方式或应该如何工作。鉴于我的简短示例代码，我还希望将y_test_text更改为

y_test_text = [["new york"],["new york"],["new york"],["new york"],["new york"],["new york"],["new york"]]

那应该可以运作——我的意思是我们已经为那个标签做好了准备，但在这种情况下我遇到了问题。

ValueError: Can't handle mix of binary and multilabel-indicator

- Scott Stewart

---

“some some of metrics”是什么意思？分类器无法预测它从未见过的标签。 - BrenBarn

请查看我编辑过的答案，我认为它涵盖了你所有的问题。 - Geeocode

谢谢Gyorgy！这正是我所需要的。应该能解决我的大问题。 - Scott Stewart

我很高兴，能够帮助你。 :) - Geeocode

Answer 1

如果您在训练集 y 中也“引入”了新标签，则可以这样做：

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn import preprocessing
from sklearn.metrics import accuracy_score

X_train = np.array(["new york is a hell of a town",
                "new york was originally dutch",
                "the big apple is great",
                "new york is also called the big apple",
                "nyc is nice",
                "people abbreviate new york city as nyc",
                "the capital of great britain is london",
                "london is in the uk",
                "london is in england",
                "london is in great britain",
                "it rains a lot in london",
                "london hosts the british museum",
                "new york is great and so is london",
                "i like london better than new york"])
y_train_text = [["new york"],["new york"],["new york"],["new york"],    
                ["new york"],["new york"],["london"],["london"],         
                ["london"],["london"],["london"],["london"],
                ["new york","England"],["new york","london"]]

X_test = np.array(['nice day in nyc',
               'welcome to london',
               'london is rainy',
               'it is raining in britian',
               'it is raining in britian and the big apple',
               'it is raining in britian and nyc',
               'hello welcome to new york. enjoy it here and london too'])

y_test_text = [["new york"],["new york"],["new york"],["new york"],["new york"],["new york"],["new york"]]


lb = preprocessing.MultiLabelBinarizer(classes=("new york","london","England"))
Y = lb.fit_transform(y_train_text)
Y_test = lb.fit_transform(y_test_text)

print Y_test

classifier = Pipeline([
('vectorizer', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', OneVsRestClassifier(LinearSVC()))])

classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)
print predicted

print "Accuracy Score: ",accuracy_score(Y_test, predicted)

输出：

Accuracy Score:  0.571428571429

关键部分是：

y_train_text = [["new york"],["new york"],["new york"],
                ["new york"],["new york"],["new york"],
                ["london"],["london"],["london"],["london"],
                ["london"],["london"],["new york","England"],
                ["new york","london"]]

我们也插入了"England"。这样做是有道理的，因为如果分类器以前没有看到标签，那么它怎么能预测出某个标签呢？所以我们通过这种方式创建了一个三个标签的分类问题。

lb = preprocessing.MultiLabelBinarizer(classes=("new york","london","England"))

您需要将类作为参数传递给MultiLabelBinarizer()，它将与任何y_test_text一起使用。

- Geeocode

---

很棒的答案。有几个建议。对于多标签分类，sklearn.metrics.accuracy_score() 计算子集准确率（即精确匹配）。然而，hamming_loss 则针对被预测的单个标签计算准确度。Consistent Multilabel Classification。 - Pramit

Answer 2

简而言之，这是一个不恰当的问题。分类假设所有标签都在一开始就已知，二值化也是一样。将其适配于所有标签，然后对任何你想要的子集进行训练。

- lejlot

---

我认为不方便的地方在于，人们可能更喜欢MultiLabelBinarizer忽略它没有看到的任何标签，而不是报错。与CountVectorizer的行为进行比较：如果在其transform()方法期间看到了它没有在fit()期间看到的标记，它将悄悄地忽略它们。当您使用相同的向量化器转换测试集时，这通常是您想要的情况之一。同样，当您使用MultiLabelBinarizer转换测试标签时，您可能希望它悄悄地忽略您在训练中没有看到的任何内容。 - Stephen

当你使用大量标签训练多标签分类器时，这个问题更容易出现。特别是在开发过程中使用数据集的子集时。为了解决这个问题，我只是提前手动清理标签。 - Stephen

我在这里遇到了类似的问题：https://stats.stackexchange.com/questions/298046/achieving-consistency-between-training-test-target-representations-in-multilabe - Stephen

Answer 3

如另一条评论所述，个人认为二元化处理器应该在“转换”时忽略未见过的类。 如果测试样本呈现的特征与训练时使用的不同，那么消耗二元化处理器结果的分类器可能会反应不佳。

我解决了这个问题，只需从样本中删除未见过的类。我认为这比动态更改拟合的二元化处理器或（另一种选择）扩展其以允许忽略更安全。

list(map(lambda names: np.intersect1d(lb.classes_, names), y_test_text))

没有运行您的实际代码

- javigzz