Python求嵌套字典列表的平均值

Question

我有一个具有以下结构的列表：

data = [[
        {
            "id": 713,
            "prediction": 4.8,
            "confidence": [
                {"percentile": "75", "lower": 4.8, "upper": 5.7}
            ],
        },
        {
            "id": 714,
            "prediction": 4.93,
            "confidence": [
                {"percentile": "75", "lower": 4.9, "upper": 5.7}
            ],
        },
    ],
    [
        {
            "id": 713,
            "prediction": 5.8,
            "confidence": [
                {"percentile": "75", "lower": 4.2, "upper": 6.7}
            ],
        },
        {
            "id": 714,
            "prediction": 2.93,
            "confidence": [
                {"percentile": "75", "lower": 1.9, "upper": 3.7}
            ],
        },
    
    ]]

这里有一个包含两个列表的列表，但它也可能包含更多的列表。每个列表都包含一个带有id和置信区间的预测，这些置信区间在另一个带有字典的列表中。

我需要将这些列表合并，以便对于每个id只有一个字典，其中包含数值的平均值。

我尝试过搜索，但没有找到与此嵌套结构匹配的答案。

期望的输出应该如下所示：

merged_data = [
            {
                "id": 713,
                "prediction": 5.3,
                "confidence": [
                    {"percentile": "75", "lower": 4.5, "upper": 6.2}
                ],
            },
            {
                "id": 714,
                "prediction": 3.93,
                "confidence": [
                    {"percentile": "75", "lower": 3.4, "upper": 4.7}
                ],
            },
        ]

- Wessi

---

一个嵌套的for循环可以实现这个。请展示您尝试过的代码。 - roganjosh

请在结果中指定您希望看到的格式，因为问题描述不清楚。 - Vishal Singh

Answer 1

def merge_items(items):
    result = {}
    if len(items):
        result['id'] = items[0]['id']
        result['prediction'] = round(sum([item['prediction'] for item in items]) / len(items), 2)
        result['confidence'] = []
        result['confidence'].append({
            'percentile': items[0]['confidence'][0]['percentile'],
            'lower': round(sum(item['confidence'][0]['lower'] for item in items) / len(items), 2),
            'upper': round(sum(item['confidence'][0]['upper'] for item in items) / len(items), 2),
        })

    return result


result = []
ids = list(set([el['id'] for item in data for el in item]))
for id in ids:
    to_merge = [sub_item for item in data for sub_item in item if sub_item['id'] == id]
    result.append(merge_items(to_merge))

print(result)

- Alexander Lekontsev

Answer 2

dicc = {}

for e in l:
    for d in e:
        if d["id"] not in dicc:
            dicc[d["id"]] = {"prediction": [], "lower": [], "upper": []}

        dicc[d["id"]]["prediction"].append(d["prediction"])
        dicc[d["id"]]["lower"].append(d["confidence"][0]["lower"])
        dicc[d["id"]]["upper"].append(d["confidence"][0]["upper"])
        
        
for k in dicc:
    dicc[k]["average_prediction"] = sum(dicc[k]["prediction"])/len(dicc[k]["prediction"])
    dicc[k]["average_lower"] = sum(dicc[k]["lower"])/len(dicc[k]["lower"])
    dicc[k]["average_upper"] = sum(dicc[k]["upper"])/len(dicc[k]["upper"])

print(dicc)

{713: {'预测值': [4.8, 5.8], '下限': [4.8, 4.2], '上限': [5.7, 6.7], '平均预测值': 5.3, '平均下限': 4.5, '平均上限': 6.2}, 714: {'预测值': [4.936893921359024, 2.936893921359024], '下限': [4.9, 1.9], '上限': [5.7, 3.7], '平均预测值': 3.936893921359024, '平均下限': 3.4000000000000004, '平均上限': 4.7}}

- Manuel

---

这里缺少了“下限”和“上限”值的平均值。对于没有表述清楚，我深感抱歉。 - Wessi

其他两个变量的操作步骤相同。 - Manuel

Answer 3

您的问题实际上有三个部分。

1. 在准备进行某种汇总之前，如何解压列表并按ID进行分组？您有很多选择，但比较经典的方法是制作一个查找表并追加任何新值：

groups = {}

# `data` is the outer list in your nested structure
for d in (d for L in data for d in L):
    L = groups.get(d['id'], [])
    L.append(d)
    groups[d['id']] = L

2. 你如何聚合这些字典以获得所有数字值的平均值？有许多方法具有不同的数值稳定性。我将从一个简单的方法开始，该方法递归地遍历部分结果集和新条目。

请注意，这假设对象结构非常一致（就像您展示的那样）。如果您有时缺少键、长度不匹配或其他差异，您必须仔细考虑当这些结构合并时您想要发生的确切细节——没有一种通用的解决方案。

def walk(avgs, new, n):
    """
    Most of this algorithm is just walking the object structure.
    We keep any keys, lists, etc the same and only average the
    numeric elements.
    """
    if isinstance(avgs, dict):
        return {k:walk(avgs[k], new[k], n) for k in avgs}
    if isinstance(avgs, list):
        return [walk(x, y, n) for x,y in zip(avgs, new)]
    if isinstance(avgs, float):  # integers and whatnot also satisfy this
        """
        This is the only place that averaging actually happens.
        At the risk of some accumulated errors, this directly
        computes the total of the last n+1 items and divides
        by n+1.
        """
        return (avgs*n+new)/(n+1.)
    return avgs

def merge(L):
    if not L:
        # never happens using the above grouping code
        return None
    d = L[0]
    for n, new in enumerate(L[1:], 1):
        d = walks(d, new, n)
    return d

averaged = {k:merge(v) for k,v in groups.items()}

您可能只想要某些键（如预测结果）进行平均。您可以在分组对象上事先进行过滤，也可以事后进行过滤（事先进行过滤可能更有效）。

# before
groups = {
    # any transformation you'd like to apply to the dictionaries
    k:[{s:d[s] for s in ('prediction', 'confidence')} for d in L] for k,L in groups.items()
}

# after
averaged = {
    # basically the same code, except there's only one object per key
    k:{s:d[s] for s in ('prediction', 'confidence')} for k,d in averaged.items()
}

为了提高效率，我创建了很多中间列表，但这些并不是必要的。你可以完全应用滚动更新算法而不是分组然后聚合，从而节省一些内存。

averaged = {}

# `data` is the outer list in your nested structure
for d in (d for L in data for d in L):
    key = d['id']
    d = {s:d[s] for s in ('prediction', 'confidence')}  # any desired transforms

    if key not in averaged:
        averaged[key] = (d, 1)
    else:
        agg, n = groups[key]
        averaged[key] = (walk(agg, d, n), n+1)

averaged = {k:v[0] for k,v in averaged.items()}

我们仍然没有像你想要的那样格式化输出（我们有一个字典，而你想要一个列表，其中包含对象中的键）。不过这是一个很容易解决的问题：

def inline_key(d, key):
    # not a pure function, but we're lazy, and the original
    # values are never used
    d['id'] = key
    return d

final_result = [inline_key(d, k) for k,d in averaged.items()]

- Hans Musgrave

Answer 4

Try this :

from copy import deepcopy

input = [[
    {
        "id": 713,
        "prediction": 4.8,
        "confidence": [
            {"percentile": "75", "lower": 4.8, "upper": 5.7}
        ],
    },
    {
        "id": 714,
        "prediction": 4.936893921359024,
        "confidence": [
            {"percentile": "75", "lower": 4.9, "upper": 5.7}
        ],
    },
],
[
    {
        "id": 713,
        "prediction": 5.8,
        "confidence": [
            {"percentile": "75", "lower": 4.2, "upper": 6.7}
        ],
    },
    {
        "id": 714,
        "prediction": 2.936893921359024,
        "confidence": [
            {"percentile": "75", "lower": 1.9, "upper": 3.7}
        ],
    },

]]

final_dict_list = []

processed_id = []

for item in input:
    for dict_ele in item:
        if dict_ele["id"] in processed_id:
            for final_item in final_dict_list:
                if final_item['id'] == dict_ele["id"]:
                    final_item["prediction"] += dict_ele["prediction"]
                    final_item["confidence"][0]["lower"] += dict_ele["confidence"][0]["lower"]
                    final_item["confidence"][0]["upper"] += dict_ele["confidence"][0]["upper"]
        else:
            final_dict = deepcopy(dict_ele)
            final_dict_list.append(final_dict)
            processed_id.append(dict_ele["id"])


numer_of_items = len(input)
for item in final_dict_list:
    item["prediction"] /= numer_of_items
    item["confidence"][0]["lower"] /= numer_of_items
    item["confidence"][0]["upper"] /= numer_of_items

print(final_dict_list)

输出：

[
{'confidence': [{'upper': 6.2, 'lower': 4.5, 'percentile': '75'}], 'id': 713, 'prediction': 5.3},
{'confidence': [{'upper': 4.7, 'lower': 3.4000000000000004, 'percentile': '75'}], 'id': 714, 'prediction': 3.936893921359024}]

仅仅是指出，如果数据结构稍微不同的话，这个过程就会更加容易。

- Prashant Kumar