我正在寻找创建通用函数来将
我已经了解并阅读了原始论文，从数学上理解该函数的输出不是概率，而是每个基本估计器构建孤立异常路径长度的平均值。
问题： 我想将该输出转换为以
当前方法：
sklearn's IsolationForest的decision_scores输出转换为真实概率[0.0, 1.0]。我已经了解并阅读了原始论文，从数学上理解该函数的输出不是概率，而是每个基本估计器构建孤立异常路径长度的平均值。
问题： 我想将该输出转换为以
元组(x,y)形式表示的概率，其中x=P(anomaly)，y=1-x。当前方法：
def convert_probabilities(predictions, scores):
from sklearn.preprocessing import MinMaxScaler
new_scores = [(1,1) for _ in range(len(scores))]
anomalous_idxs = [i for i in (range(len(predictions))) if predictions[i] == -1]
regular_idxs = [i for i in (range(len(predictions))) if predictions[i] == 1]
anomalous_scores = np.asarray(np.abs([scores[i] for i in anomalous_idxs]))
regular_scores = np.asarray(np.abs([scores[i] for i in regular_idxs]))
scaler = MinMaxScaler()
anomalous_scores_scaled = scaler.fit_transform(anomalous_scores.reshape(-1,1))
regular_scores_scaled = scaler.fit_transform(regular_scores.reshape(-1,1))
for i, j in zip(anomalous_idxs, range(len(anomalous_scores_scaled))):
new_scores[i] = (anomalous_scores_scaled[j][0], 1-anomalous_scores_scaled[j][0])
for i, j in zip(regular_idxs, range(len(regular_scores_scaled))):
new_scores[i] = (1-regular_scores_scaled[j][0], regular_scores_scaled[j][0])
return new_scores
modified_scores = convert_probabilities(model_predictions, model_decisions)
最小化、可重现的示例
import pandas as pd
from sklearn.datasets import make_classification, load_iris
from sklearn.ensemble import IsolationForest
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
# Get data
X, y = load_iris(return_X_y=True, as_frame=True)
anomalies, anomalies_classes = make_classification(n_samples=int(X.shape[0]*0.05), n_features=X.shape[1], hypercube=False, random_state=60, shuffle=True)
anomalies_df = pd.DataFrame(data=anomalies, columns=X.columns)
# Split into train/test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=60)
# Combine testing data
X_test['anomaly'] = 1
anomalies_df['anomaly'] = -1
X_test = X_test.append(anomalies_df, ignore_index=True)
y_test = X_test['anomaly']
X_test.drop('anomaly', inplace=True, axis=1)
# Build a model
model = IsolationForest(n_jobs=1, bootstrap=False, random_state=60)
# Fit it
model.fit(X_train)
# Test it
model_predictions = model.predict(X_test)
model_decisions = model.decision_function(X_test)
# Print results
for a,b,c in zip(y_test, model_predictions, model_decisions):
print_str = """
Class: {} | Model Prediction: {} | Model Decision Score: {}
""".format(a,b,c)
print(print_str)
问题
modified_scores = convert_probabilities(model_predictions, model_decisions)
# Print results
for a,b in zip(model_predictions, modified_scores):
ans = False
if a==-1:
if b[0] > b[1]:
ans = True
else:
ans = False
elif a==1:
if b[1] > b[0]:
ans=True
else:
ans=False
print_str = """
Model Prediction: {} | Model Decision Score: {} | Correct: {}
""".format(a,b, str(ans))
print(print_str)
展示一些奇怪的结果，例如：
Model Prediction: 1 | Model Decision Score: (0.17604259932311161, 0.8239574006768884) | Correct: True
Model Prediction: 1 | Model Decision Score: (0.7120367886017022, 0.28796321139829784) | Correct: False
Model Prediction: 1 | Model Decision Score: (0.7251531538304419, 0.27484684616955807) | Correct: False
Model Prediction: -1 | Model Decision Score: (0.16776449326185877, 0.8322355067381413) | Correct: False
Model Prediction: 1 | Model Decision Score: (0.8395087028516501, 0.1604912971483499) | Correct: False
模型预测: 1 | 模型决策分数: (0.0, 1.0) | 正确: 真
预测值为-1（异常）但概率仅为37%，或者预测值为1（正常）但概率为26%，这怎么可能呢？
请注意，这是一个玩具数据集，有标签，但无监督异常检测算法显然不会使用标签。