字典的平均值

Question

字典的平均值

3

使用sklearn中的鸢尾花数据集。我正在拆分数据应用感知器，记录得分在字典中，将样本大小（键）用于拟合模型到相应的分数（训练和测试分数作为元组）。

这会产生3个字典，因为我运行了3次循环。如何找到3次迭代的平均分数？我尝试将字典存储在列表中并进行平均，但它没有起作用。

例如：如果字典是

{21: (0.85, 0.82), 52: (0.80, 0.62), 73: (0.82, 0.45), 94: (0.81, 0.78)}
{21: (0.95, 0.91), 52: (0.80, 0.89), 73: (0.84, 0.87), 94: (0.79, 0.41)}
{21: (0.809, 0.83), 52: (0.841, 0.77), 73: (0.84, 0.44), 94: (0.79, 0.33)}

输出应该是{21:(0.869,0.853),52.....}，其中键21的值的第一个元素为(0.85+0.95+0.809)/3，第二个元素为(0.82+0.91+0.83)/3。

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.linear_model import Perceptron
from sklearn.model_selection import train_test_split

score_list=shape_list=[]
iris = load_iris()
props=[0.2,0.5,0.7,0.9]
df = pd.DataFrame(data= np.c_[iris['data'], iris['target']], columns= iris['feature_names'] + ['target'])
y=df[list(df.loc[:,df.columns.values =='target'])]
X=df[list(df.loc[:,df.columns.values !='target'])]

# number of trials
for i in range(3):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, train_size=0.7)
    results = {}
    for i in props:
        size = int(i*len(X_train))
        ix = np.random.choice(X_train.index, size=size, replace = False)
        sampleX = X_train.loc[ix]
        sampleY = y_train.loc[ix]
        #apply model
        modelP = Perceptron(tol=1e-3)
        modelP.fit(sampleX, sampleY)
        train_score = modelP.score(sampleX,sampleY)
        test_score = modelP.score(X_test,y_test)
        #store in dictionary
        results[size] = (train_score, test_score)

    print(results)

此外，如果有人了解统计学，是否有一种方法可以找到试验中的标准误差并打印出每个样本大小（字典键）的平均标准误差？

- freshman_2021

2个回答

1

假设所有结果都存储在列表rl中，以下程序将进行计算：

rl = [
    {21: (0.85, 0.82), 52: (0.80, 0.62), 73: (0.82, 0.45), 94: (0.81, 0.78)},
    {21: (0.95, 0.91), 52: (0.80, 0.89), 73: (0.84, 0.87), 94: (0.79, 0.41)},
    {21: (0.809, 0.83), 52: (0.841, 0.77), 73: (0.84, 0.44), 94: (0.79, 0.33)}
]

vd = {}
for k in rl[0].keys():
    vals = [[], []]
    for i in range(len(rl)):
        vals[0].append(rl[i][k][0])
        vals[1].append(rl[i][k][1])
    vd[k] = sum(vals[0])/len(vals[0]), sum(vals[1])/len(vals[1])
print(vd)

输出：

# {21: (0.8696666666666667, 0.8533333333333334),
#  52: (0.8136666666666666, 0.7600000000000001),
#  73: (0.8333333333333334, 0.5866666666666667),
#  94: (0.7966666666666667, 0.5066666666666667)}

或者，如果在同一个列表rl上同时使用zip和numpy。在这种情况下，我们也可以方便地按照与平均值相同的方式计算标准误差：

import numpy as np

rl2 = list(zip(rl[0].keys(), rl[0].values(), rl[1].values(), rl[2].values()))
vd2 = {rl2[i][0] : np.mean(list(zip(*rl2[i][1:])), axis=1) for i in range(len(rl2))}
print(vd2)
vd2_std = {rl2[i][0] : np.std(list(zip(*rl2[i][1:])), axis=1) for i in range(len(rl2))}
print(vd2)
print("Standard error\n", vd2_std)

输出：

# {21: array([0.86966667, 0.85333333]),
#  52: array([0.81366667, 0.76      ]),
#  73: array([0.83333333, 0.58666667]),
#  94: array([0.79666667, 0.50666667])}
# Standard error
# {21: array([0.05921899, 0.04027682]),
#  52: array([0.01932759, 0.11045361]),
#  73: array([0.00942809, 0.20038851]),
#  94: array([0.00942809, 0.19601587])}

- VirtualScooter

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Trenton McKinney · Accepted Answer

将现有的循环更新，将results保存到一个名为rl的列表中
由于已经使用了pandas，因此将rl加载到数据框中
将tuples的列扩展为单独的列
使用.agg获取指标
请使用python 3.8和pandas 1.3.1进行测试
- f-strings（例如f'TrS{c}', f'TeS{c}'）需要python >= 3.6

现有代码的更新

# select columns for X and y
y = df.loc[:, 'target']
X = df.loc[:, iris['feature_names']]

# number of trials
rl = list()  # add: save results to a list
for i in range(3):
    ...
    results = {}
    for i in props:
        ...
        ...
    rl.append(results)  # add: append results

获取指标的新代码

将metrics转换为tuples列表比转换为tuples元组更容易，因为一旦创建了tuples元组就是不可变的。这意味着可以将tuples添加到现有的list中，但不能添加到现有的tuple中。
- 因此，使用defaultdict创建tuples列表更加容易，然后使用map将每个值转换为tuples。
- k[3:]要求数字始终从index 3开始。

from collections import defaultdict

# convert rl to a dataframe
rl = [{21: (0.5714285714285714, 0.6888888888888889), 52: (0.6153846153846154, 0.7111111111111111), 73: (0.7123287671232876, 0.6222222222222222), 94: (0.7127659574468085, 0.6)}, {21: (0.6190476190476191, 0.6444444444444445), 52: (0.6923076923076923, 0.6444444444444445), 73: (0.3698630136986301, 0.35555555555555557), 94: (0.7978723404255319, 0.7777777777777778)}, {21: (0.8095238095238095, 0.5555555555555556), 52: (0.7307692307692307, 0.5555555555555556), 73: (0.7534246575342466, 0.5777777777777777), 94: (0.6170212765957447, 0.7555555555555555)}]
df = pd.DataFrame(rl)

# display(df)
                                         21                                        52                                         73                                        94
0  (0.5714285714285714, 0.6888888888888889)  (0.6153846153846154, 0.7111111111111111)   (0.7123287671232876, 0.6222222222222222)                 (0.7127659574468085, 0.6)
1  (0.6190476190476191, 0.6444444444444445)  (0.6923076923076923, 0.6444444444444445)  (0.3698630136986301, 0.35555555555555557)  (0.7978723404255319, 0.7777777777777778)
2  (0.8095238095238095, 0.5555555555555556)  (0.7307692307692307, 0.5555555555555556)   (0.7534246575342466, 0.5777777777777777)  (0.6170212765957447, 0.7555555555555555)

# expand the tuples
for c in df.columns:
    df[[f'TrS{c}', f'TeS{c}']] = pd.DataFrame(df[c].tolist(), index= df.index)
    df.drop(c, axis=1, inplace=True)

# get the mean and std
metrics = df.agg(['mean', 'std']).round(3)

# display(metrics)
      TrS21  TeS21  TrS52  TeS52  TrS73  TeS73  TrS94  TeS94
mean  0.667  0.630  0.679  0.637  0.612  0.519  0.709  0.711
std   0.126  0.068  0.059  0.078  0.211  0.143  0.090  0.097

# convert to dict
dd = defaultdict(list)

for k, v in metrics.to_dict().items(): 
    dd[int(k[3:])].append(tuple(v.values()))
    
dd = dict(zip(dd, map(tuple, dd.values())))
print(dd)

[out]:
{21: ((0.667, 0.126), (0.63, 0.068)),
 52: ((0.679, 0.059), (0.637, 0.078)),
 73: ((0.612, 0.211), (0.519, 0.143)),
 94: ((0.709, 0.09), (0.711, 0.097))}