如何在R中使用CARET训练模型后计算95%置信区间？

Question

如何在R中使用CARET训练模型后计算95%置信区间？

3

我使用R包caret构建了不同的回归模型。我们如何计算预测值的95%置信区间？我已经参考了这里的讨论，但是它并没有奏效。

rm(list = ls())
library(caret)

data("mtcars")
Train_data = mtcars[1:26, -c(8,9)]
Test_data = mtcars[27:32, -c(8,9)]


set.seed(100)
model_pls <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 5, 
  method = "pls", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'), 
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3, 
    savePredictions = "final"
  )
)

model_rf <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 5, 
  method = "ranger", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'), 
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3, 
    savePredictions = "final"
  )
)

model_svmr <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 8, 
  method = "svmRadial", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'),
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3,
  )
)

# This does not generate confidence interval
PLS.pred = predict(model_pls, subset(Test_data, select = -hp))  
RF.pred = predict(model_rf, subset(Test_data, select = -hp)) 
RF.svm = predict(model_svmr , subset(Test_data, select = -hp)) 


# This is not working
predict(model_pls$finalModel, subset(Test_data, select = -hp), interval = "confidence")
predict(model_rf$finalModel, subset(Test_data, select = -hp), interval = "confidence")
predict(model_svmr$finalModel, subset(Test_data, select = -hp), interval = "confidence")

根据Michael Matta的建议，我尝试了以下代码，但是它并没有按照预期工作。

confint(model_pls, level = 0.95)
# Error in UseMethod("vcov"): no applicable method for 'vcov'

predict(model_pls, subset(Test_data, select = -hp), interval = "confidence")
# 64.47807  57.97479 151.59713 130.24356 183.20296  88.50035
# This does not show the CI.

- Yang Yang

1

我认为tidymodels会对你有很大帮助。它似乎不太适合caret，问题。 - Quinten

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- L D · Accepted Answer

置信区间来自已知分布和以下统计量，或者使用重新采样构建。RBF SVM，随机森林等没有已知分布的模型，即它们不能像线性模型（lm）一样对任何东西提供置信区间。

从这些模型中获取置信区间的方法是重新采样训练/测试数据集，重新训练，收集所需值（例如使用for循环）。然后，通过均值的已知分布估计所收集数据的期望值置信区间。

以下伪代码适用于几乎任何您想要的分数（准确度，RMSE等；有关注释，请参见下文）：

predictionsTrainAll <- c()
predictionsTestAll <- c() 
scoresTrain <- c()
scoresTest <- c()

for( i in 1:1000){
    d <- shuffle the original dataset,
    training <- draw training dataset from d,
    testing  <- draw testing datassetfrom d (such that training and testing do not have any intersection),
    
    model <- train a model on training data,
    predictionsTrain <- make predictions for training data,
    predictionsTest  <- make predictions for testing data,
    scoreTrain <- evaulate model and obtain any score you like on train,
    scoreTest  <- evaluate model and obtain any score you like on test,
    
    predictionsTrainAll <- append(predictionsTrainAll, predictionsTrain)
    predictionsTestAll <- append(predictionsTestAll, predictionsTest)
    scoresTrain <- append(scoresTrain, scoreTrain)
    scoresTest  <- append(scoresTest, scoreTest)
}

现在，我们可以估计scoresTrain和scoresTest的期望值。由于中心极限定理，我们可以假设期望值具有正态分布（或t分布，因为我们这里有有限的样本）。我们可以使用：

# scores should be /somehow/ normally distributed (symmetric by mean, meadian close to the mean)
hist(predictionsTrainAll)
hist(predictionsTestAll)
hist(scoresTrain)    
hist(scoresTest)     

# if the histogram are /somehow/ normal:
t.test(predictionsTrainAll)
t.test(predictionsTestAll)
t.test(scoresTrain)
t.test(scoresTest)

这将计算预测值和任何您想要的分数的期望值（真实均值）的95％置信区间。但是请注意，如果直方图呈偏斜状态，则均值的估计可能存在缺陷，并产生错误的置信区间。

二元分类器的示例案例：预测的真实均值为0，95％CI = [-0.32，0.32]，因为模型预测为零。然而，预测只能在[0; 1]之间，因此CI的负部分没有意义。这样的CI是正态/ t分布所暗示的对称性的结果。当所检查的分数/预测的直方图不服从正态分布时，就会发生这种情况。