在caret中使用eml：类概率的错误

Question

我试图比较标准的神经网络方法和基于 ROC 度量的极限学习机分类器，使用 R 包中的 "nnet" 和 "elm" 方法来进行比较。对于 nnet，一切都工作正常，但是当使用 method = "elm" 时，会出现以下错误：

Error in evalSummaryFunction(y, wts = weights, ctrl = trControl, lev = classLevels,  : 
  train()'s use of ROC codes requires class probabilities. See the classProbs option of trainControl()
In addition: Warning messages:
1: In train.default(x, y, weights = w, ...) :
  At least one of the class levels are not valid R variables names; This may cause errors if class probabilities are generated because the variables names will be converted to: X1, X2
2: In train.default(x, y, weights = w, ...) :
  Class probabilities were requested for a model that does not implement them

当method = "nnet"时，我也遇到了第一个错误，但是我通过将分数设置为因子变量来解决了问题。因此，这不可能是问题所在。

我对R相对较新，也许错误很琐碎，但现在我卡住了...由于elmNN似乎是相对较新的实现，我在网上也找不到有关如何在caret中使用elm的任何信息。

gc <- read.table("germanCreditNum.txt")
colnames(gc)[25]<-"score"

gc_inTrain <- createDataPartition(y = gc$score,
    ## the outcome data are needed
    p = .8,
    ## The percentage of data in the
    ## training set
    list = FALSE)

str(gc_inTrain)
gc_training <- gc[ gc_inTrain,]
gc_testing <- gc[-gc_inTrain,]
nrow(gc_training) ## No of rows 
nrow(gc_testing)

gc_training$score <- as.factor(gc_training$score)

gc_ctrl <- trainControl(method = "boot",
    repeats = 1,
    classProbs = TRUE,
    summaryFunction = twoClassSummary)

neuralnetFit <- train(score ~ .,
    data = gc_training,
    method = "nnet",
    trControl = gc_ctrl,
    metric = "ROC",
    preProc = c("center", "scale"))

neuralnetFit
plot(neuralnetFit)
nnClasses <- predict(neuralnetFit, newdata = gc_testing)
str(nnClasses)

## start with ELM for German Credit

gc_ctrl2 <- trainControl(classProbs = TRUE, summaryFunction = twoClassSummary)
elmFit <- train(score ~ ., 
    data = gc_training, 
    method = "elm", 
    trControl = gc_ctrl2, 
    metric = "ROC", 
    preProc = c("center", "scale"))

elmFit
plot(elmFit)

elmClasses <- predict(elmFit, newdata = gc_testing)
str(elmClasses)
elmProbs <- predict(elmFit, newdata = gc_testing, type = "prob")
head(elmProbs) 

- martin_m

---

那个数据集是否容易找到？ - IRTFM

Answer 1

我不记得为什么没有为ELM包括概率模型（我可能有一个好理由）。您可以使用自定义方法来获取softmax值：

library(caret)

set.seed(1)
dat <- twoClassSim(100)

elm_fun <- getModelInfo("elm")[[1]]
elm_fun$prob <- function (modelFit, newdata, submodels = NULL)  {
  out <- exp(predict(modelFit, newdata))
  t(apply(out, 1, function(x) x/sum(x)))
}
mod <- train(Class ~ ., data = dat, 
             method = elm_fun,
             metric = "ROC",
             trControl = trainControl(classProbs = TRUE,
                                      summaryFunction = twoClassSummary))

马克斯

- topepo