我正在使用Caret软件包分析各种模型，并使用以下方法评估结果：
- print() [打印train()的结果]，
- predict()和
- resamples()。
为什么下面的例子中这些结果不同呢？
我对敏感性（真正例）感兴趣。为什么J48_fit的敏感性在评估中先是0.71，然后是0.81，最后又变成了0.71。
当我运行其他模型时，同样会出现这种情况——根据评估方式，敏感性会发生变化。
NB: 我在此处包含了两个模型，以说明resamples()函数必须输入两个模型，但我的主要问题是关于使用不同方法得到的结果之间的差异。
换句话说，train() (C5.0_fit/J48_fit)、predict()和resamples()的结果有何区别？背后发生了什么，哪个结果应该被信任？
例如：
library(C50)
data(churn)
Seed <- 10
# Set train options
set.seed(Seed)
Train_options <- trainControl(method = "cv", number = 10,
classProbs = TRUE,
summaryFunction = twoClassSummary)
# C5.0 model:
set.seed(Seed)
C5.0_fit <- train(churn~., data=churnTrain, method="C5.0", metric="ROC",
trControl=Train_options)
# J48 model:
set.seed(Seed)
J48_fit <- train(churn~., data=churnTrain, method="J48", metric="ROC",
trControl=Train_options)
# Get results by printing the outcome
print(J48_fit)
# ROC Sens Spec
# Best (sensitivity): 0.87 0.71 0.98
# Get results using predict()
set.seed(Seed)
J48_fit_predict <- predict(J48_fit, churnTrain)
confusionMatrix(J48_fit_predict, churnTrain$churn)
# Reference
# Prediction yes no
# yes 389 14
# no 94 2836
# Sens : 0.81
# Spec : 0.99
# Get results by comparing algorithms with resamples()
set.seed(Seed)
results <- resamples(list(C5.0_fit=C5.0_fit, J48_fit=J48_fit))
summary(results)
# ROC mean
# C5.0_fit 0.92
# J48_fit 0.87
# Sens mean
# C5.0_fit 0.76
# J48_fit 0.71
# Spec mean
# C5.0_fit 0.99
# J48_fit 0.98
顺便说一下，这里有一个可以同时获取三个结果的函数：
Get_results <- function(...){
Args <- list(...)
Model_names <- as.list(sapply(substitute({...})[-1], deparse))
message("Model names:")
print(Model_names)
# Function for getting max sensitivity
Max_sens <- function(df, colname = "results"){
df <- df[[colname]]
new_df <- df[which.max(df$Sens), ]
x <- sapply(new_df, is.numeric)
new_df[, x] <- round(new_df[, x], 2)
new_df
}
# Find max Sens for each model
message("Max sensitivity from model printout:")
Max_sens_out <- lapply(Args, Max_sens)
names(Max_sens_out) <- Model_names
print(Max_sens_out)
# Find predict() result for each model
message("Results using predict():")
set.seed(Seed)
Predict_out <- lapply(Args, function(x) predict(x, churnTrain))
Predict_results <- lapply(Predict_out, function(x) confusionMatrix(x, churnTrain$churn))
names(Predict_results) <- Model_names
print(Predict_results)
# Find resamples() results for each model
message("Results using resamples():")
set.seed(Seed)
results <- resamples(list(...),modelNames = Model_names)
# names(results) <- Model_names
summary(results)
}
# Test
Get_results(C5.0_fit, J48_fit)
非常感谢！