如何使用多输出分类器实现网格搜索交叉验证？

Question

我正在处理一个数据集，需要进行两个预测，即y的两列，每列也是多类别的。因此，我使用了带有MultiOutput分类器的XGBoost，并且想要使用Grid Search CV来调整它。

xgb_clf = xgb.XGBClassifier(learning_rate=0.1,
                n_estimators=3000,
                max_depth=3,
                min_child_weight=1,
                subsample=0.8,
                colsample_bytree=0.8,
                objective='multi:softmax',
                nthread=4,
                num_class=9,
                seed=27
                )
model = MultiOutputClassifier(estimator=xgb_clf)
    param_test1 = { 'estimator__max_depth':[3],'estimator__min_child_weight':[4]}
gsearch1 = GridSearchCV(estimator =model, 
 param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch1.fit(X_train_split,y_train_split)
gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_

但是当我这样做时，会出现错误。

_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py", line 431, in _process_worker
    r = call_item()
  File "/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py", line 285, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 253, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.6/dist-packages/joblib/parallel.py", line 253, in <listcomp>
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py", line 544, in _fit_and_score
    test_scores = _score(estimator, X_test, y_test, scorer)
  File "/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_validation.py", line 591, in _score
    scores = scorer(estimator, X_test, y_test)
  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_scorer.py", line 87, in __call__
    *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_scorer.py", line 300, in _score
    raise ValueError("{0} format is not supported".format(y_type))
ValueError: multiclass-multioutput format is not supported
"""

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-42-e53fdaaedf6b> in <module>()
      5 gsearch1 = GridSearchCV(estimator =model, 
      6  param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
----> 7 gsearch1.fit(X_train_split,y_train_split)
      8 gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_

7 frames
/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    708                 return results
    709 
--> 710             self._run_search(evaluate_candidates)
    711 
    712         # For multi-metric evaluation, store the best_index_, best_params_ and

/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_search.py in _run_search(self, evaluate_candidates)
   1149     def _run_search(self, evaluate_candidates):
   1150         """Search all candidates in param_grid"""
-> 1151         evaluate_candidates(ParameterGrid(self.param_grid))
   1152 
   1153 

/usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_search.py in evaluate_candidates(candidate_params)
    687                                for parameters, (train, test)
    688                                in product(candidate_params,
--> 689                                           cv.split(X, y, groups)))
    690 
    691                 if len(out) < 1:

/usr/local/lib/python3.6/dist-packages/joblib/parallel.py in __call__(self, iterable)
   1040 
   1041             with self._backend.retrieval_context():
-> 1042                 self.retrieve()
   1043             # Make sure that we get a last message telling us we are done
   1044             elapsed_time = time.time() - self._start_time

/usr/local/lib/python3.6/dist-packages/joblib/parallel.py in retrieve(self)
    919             try:
    920                 if getattr(self._backend, 'supports_timeout', False):
--> 921                     self._output.extend(job.get(timeout=self.timeout))
    922                 else:
    923                     self._output.extend(job.get())

/usr/local/lib/python3.6/dist-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    540         AsyncResults.get from multiprocessing."""
    541         try:
--> 542             return future.result(timeout=timeout)
    543         except CfTimeoutError as e:
    544             raise TimeoutError from e

/usr/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

/usr/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

ValueError: multiclass-multioutput format is not supported

我认为出现错误是因为我使用roc_auc作为我的评分方法，但我不知道如何修复它。我应该使用其他评分方法吗？

- SHIVANSHU SAHOO

---

请编辑并添加更多上下文到你的问题中：你正在使用哪种技术、平台和运行环境，你要解决的问题是什么。另外，请明确你想要实现什么目标。 - undefined

Answer 1

是的，你想得没错。问题出在ROC AUC分数仅适用于二元分类情况。相反，您可以使用所有类别的ROC AUC分数的平均值。

# from https://dev59.com/9VkS5IYBdhLWcg3wj3fP
from sklearn.metrics import roc_auc_score
import numpy as np

def roc_auc_score_multiclass(actual_class, pred_class, average = "macro"):

  #creating a set of all the unique classes using the actual class list
  unique_class = set(actual_class)
  roc_auc_dict = {}
  for per_class in unique_class:
    #creating a list of all the classes except the current class 
    other_class = [x for x in unique_class if x != per_class]

    #marking the current class as 1 and all other classes as 0
    new_actual_class = [0 if x in other_class else 1 for x in actual_class]
    new_pred_class = [0 if x in other_class else 1 for x in pred_class]

    #using the sklearn metrics method to calculate the roc_auc_score
    roc_auc = roc_auc_score(new_actual_class, new_pred_class, average = average)
    roc_auc_dict[per_class] = roc_auc

  return np.mean([x for x in roc_auc_dict.values()])

使用此函数，您可以获取每个类别与其他所有类别的ROC AUC分数。然后，您可以取这些值的平均值并将其用作评分器。您可能需要使用make_scorer函数将函数转换为评分器对象(https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html)。请注意保留HTML标签。

- Razmik Melikbekyan

---

我的问题不是多类别问题。我的问题是我有多个输出，即我的模型预测了两列y的值。因此，我需要对它们进行评估。 - undefined

是的，所提供的函数可以实现你想要的功能。它接收两个数组，并计算ROC AUC值，然后计算所有ROC AUC值的平均值。 - undefined