分类指标无法处理二进制和连续目标的混合。

Question

我尝试训练和测试几个scikit-learn模型，并尝试打印出它们的准确性。只有一些模型有效，其他模型会出现错误：

ValueError: Classification metrics can't handle a mix of binary and continuous targets. 

这个错误是什么意思？ 我该如何修改下面的代码以成功评估失败的模型？

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

from sklearn import linear_model
from sklearn import svm

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

from sklearn import preprocessing
from sklearn import utils

# Shuffle pandas rows randomly
from sklearn.utils import shuffle

# Disable annoying warnings
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)

# Performance
import time

# Import the data and assign the column names
colNames = []
for colName in range(0,3780):
    colNames.append("%s"%(colName))
colNames.append('class')
df = pd.read_csv("HoGTestData.csv", names=colNames)

# Randomly shuffle rows
df = shuffle(df)
df = df.head(20)

# Print some info on the dataset
print("Head of Data:")
print(df.head())
print("Shape of Data:")
print(df.shape)

# descriptions
print("Describe Data:")
#print(df.describe())

# class distribution
print(df.groupby('class').size())

# Split-out validation dataset
datasetData = df.values

# Determine shape and portion of data that is real data as opposed to labels
shape = datasetData.shape
thresh = int(shape[1]) - 1

# Extract labels and feature vectors
featureVectors = datasetData[:,0:thresh]
labels = datasetData[:,thresh:]

# Perform a standard scaler on the data
scaler = preprocessing.StandardScaler()
featureVectors = scaler.fit_transform(featureVectors)

# Encode labels to be acceptable
labelEncoder = preprocessing.LabelEncoder()
labels = labelEncoder.fit_transform(labels)

# Split data into training and testing data
test_size = 0.20
seed = 7
featureVectorTrain, featureVectorTest, labelsTrain, labelsTest = model_selection.train_test_split(featureVectors, labels, test_size=test_size, random_state=seed)

# Spot Check Algorithms
models = []
models.append(('SVM', svm.SVC()))
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
#models.append(('SGDRegressor', linear_model.SGDRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('BayesianRidge', linear_model.BayesianRidge())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('LassoLars', linear_model.LassoLars())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('ARDRegression', linear_model.ARDRegression())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('PassiveAggressiveRegressor', linear_model.PassiveAggressiveRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('TheilSenRegressor', linear_model.TheilSenRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('LinearRegression', linear_model.LinearRegression())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets

# Test options and evaluation metric
seed = 42
scoring = 'accuracy'

# evaluate each model in turn
results = []
names = []
print("---------------------------------------")
for name, model in models:
    start_time = time.time()
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
    elapsed_time = time.time() - start_time
    results.append(cv_results)
    names.append(name)
    msg = "{:3.2f} ({:3.2f})  Time elapsed: {:6.2f}".format(cv_results.mean(), cv_results.std(), elapsed_time)
    msg = "%s "%(name) + msg
    print(msg)
print("---------------------------------------")

print("Done")

这是脚本的输出结果：

Head of Data:
              0         1         2  ...        3778      3779  class
20573  0.124282  0.090376  0.088723  ...    0.148411  0.120542     -1
20461  0.154031  0.110177  0.087799  ...    0.100416  0.119484     -1
10416  0.340767  0.150863  0.025489  ...    0.047592  0.036171      1
52404  0.000000  0.000000  0.000000  ...    0.000000  0.000000     -1
42785  0.159105  0.118963  0.090405  ...    0.009996  0.027460     -1

[5 rows x 3781 columns]
Shape of Data:
(1024, 3781)
Describe Data:
class
-1    794
 1    230
dtype: int64
---------------------------------------
SVM 0.9878 (0.0123)  Time elapsed:  10.20
LR 0.9414 (0.0187)  Time elapsed:   7.09
LDA 0.9768 (0.0128)  Time elapsed:   6.60
KNN 0.8511 (0.0384)  Time elapsed:   3.06
CART 0.9047 (0.0358)  Time elapsed:   8.84
NB 0.9292 (0.0209)  Time elapsed:   0.36
---------------------------------------
Done

这里是labelsTrain变量：

print(labelsTrain)
[0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 0 0
 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 1 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 0 1 0 0 0 1 1 0 0 0
 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 1 1 1 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
 1 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 1
 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1 0 0
 0 0 0 0 1 1 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 1 0 1 0 0 0 0 0 0 1 0 1
 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0
 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0
 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0
 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0
 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0
 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0
 1 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 1
 0 0 0 0 1 0 1 0 0 1 1 1 1 0 1 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0
 0 0 1 0 0]

在 cross_val_score 函数期间出现错误：

# evaluate each model in turn
results = []
names = []
print("---------------------------------------")
for name, model in models:
    start_time = time.time()
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    print("start cross_val_score")
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
    print("done cross_val_score")
    elapsed_time = time.time() - start_time
    results.append(cv_results)
    #print(results)
    names.append(name)
    msg = "{:3.4f} ({:3.4f})  Time elapsed: {:6.2f}".format(cv_results.mean(), cv_results.std(), elapsed_time)
    msg = "%s "%(name) + msg
    print(msg)
print("---------------------------------------")

... 

---------------------------------------
start cross_val_score
done cross_val_score
SVM 0.9744 (0.0127)  Time elapsed:  10.46
start cross_val_score
done cross_val_score
LR 0.9194 (0.0390)  Time elapsed:   9.56
start cross_val_score
done cross_val_score
LDA 0.9780 (0.0106)  Time elapsed:   8.04
start cross_val_score
done cross_val_score
KNN 0.8657 (0.0319)  Time elapsed:   3.20
start cross_val_score
done cross_val_score
CART 0.9072 (0.0326)  Time elapsed:  10.20
start cross_val_score
done cross_val_score
NB 0.9182 (0.0327)  Time elapsed:   0.38
start cross_val_score
Traceback (most recent call last):
  File "/Users/me/Desktop/MachineLearning/Initial.py", line 112, in <module>
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 402, in cross_val_score
    error_score=error_score)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 917, in __call__
    if self.dispatch_one_batch(iterator):
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 182, in apply_async
    result = ImmediateResult(func)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 549, in __init__
    self.results = batch()
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 568, in _fit_and_score
    test_scores = _score(estimator, X_test, y_test, scorer, is_multimetric)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 605, in _score
    return _multimetric_score(estimator, X_test, y_test, scorer)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 635, in _multimetric_score
    score = scorer(estimator, X_test, y_test)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/scorer.py", line 98, in __call__
    **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/classification.py", line 176, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/classification.py", line 81, in _check_targets
    "and {1} targets".format(type_true, type_pred))
ValueError: Classification metrics can't handle a mix of binary and continuous targets

- PhilBot

Answer 1

您所有被注释掉的模型不是分类器，而是回归模型，对于回归模型，准确性是没有意义的。

您收到此错误是因为这些回归模型不会产生二进制输出，而是连续的浮点数（所有回归模型都是如此）。 因此，当scikit-learn尝试通过将二进制数字（真实标签）与浮点数（预测值）进行比较来计算准确性时，它不出所料地会出现错误。 这个原因在错误信息本身中也明确提示了:

Classification metrics can't handle a mix of binary and continuous target

- desertnaut

Answer 2

我使用了vecstack来进行堆叠模型，并设置needs_proba=True，然后出现了这个错误。通过在堆叠中更换度量标准，我解决了这个问题。因为默认情况下堆叠使用类别预测，所以如果您想要概率也必须更改度量标准。我定义了一个新的度量函数：

def get_classification_metric(testy, probs):
    from sklearn.metrics import precision_recall_curve
    precision, recall, thresholds = precision_recall_curve(testy, probs[:,1])
    # convert to f score
    fscore = (2 * precision * recall) / (precision + recall)
    # locate the index of the largest f score
    ix = np.argmax(fscore)
    return fscore[ix]

此函数可以在最佳阈值处找到最高的F1分数。因此，只需要在堆叠函数中设置 metric=get_classification_metric。

- RyanAbnavi