配置ruamel.yaml以允许重复键

Question

我将使用ruamel.yaml库来处理一个包含重复键的yaml文档。在这种情况下，重复键是一个合并键<<:。

以下是yaml文件dupe.yml的内容：

foo: &ref1
  a: 1

bar: &ref2
  b: 2

baz:
  <<: *ref1
  <<: *ref2
  c: 3

这是我的脚本：

import ruamel.yaml

yml = ruamel.yaml.YAML()
yml.allow_duplicate_keys = True
doc = yml.load(open('dupe.yml'))

assert doc['baz']['a'] == 1
assert doc['baz']['b'] == 2
assert doc['baz']['c'] == 3

运行时，它会抛出以下错误：

Traceback (most recent call last):
  File "rua.py", line 5, in <module>
    yml.load(open('dupe.yml'))
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/main.py", line 331, in load
    return constructor.get_single_data()
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
    return self.construct_document(node)
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 121, in construct_document
    for _dummy in generator:
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 1543, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 1448, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 174, in construct_object
    for _dummy in generator:
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 1543, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 1399, in construct_mapping
    merge_map = self.flatten_mapping(node)
  File "/usr/local/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 1350, in flatten_mapping
    raise DuplicateKeyError(*args)
ruamel.yaml.constructor.DuplicateKeyError: while constructing a mapping
  in "dupe.yml", line 8, column 3
found duplicate key "<<"
  in "dupe.yml", line 9, column 3

To suppress this check see:
   http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

Duplicate keys will become an error in future releases, and are errors
by default when using the new API.

如何让ruamel在不出错的情况下读取这个文件？文档中说allow_duplicate_keys = True可以使加载器容忍重复的键，但似乎并不起作用。

我正在使用Python 3.7和ruamel.yaml 0.15.90。

- mamacdon

---

在使用 Python 3 时，我建议您将 pathlib.Path 用作加载/转储的参数：yml.load(Path('dupe.yaml'))。此外请注意，自2006年9月以来，YAML 文件的建议扩展名是 .yaml。 - Anthon

Answer 1

That

yaml.allow_duplicate_keys = True

在0.15.91版本之前，此方法仅适用于非合并键。

从0.15.91版本开始，它可以使用，并且合并键会假定第一个键的值（就像非合并键一样），这意味着它的效果就像你写了：

Original Answer翻译成"最初的回答"

baz:
  <<: *ref1
  c: 3

而不是像你写的那样：原始答案

baz:
  <<: [*ref1, *ref2]
  c: 3

如果你需要这样做，你必须猴子补丁(flatten routine)处理合并键(merge keys)的例程，并影响所有后续带有双合并键的YAML文件的加载:

最初的回答：

import sys
import ruamel.yaml

yaml_str = """\
foo: &ref1
  a: 1

bar: &ref2
  b: 2

baz:
  <<: *ref1
  <<: *ref2
  c: 3

"""

def my_flatten_mapping(self, node):

    def constructed(value_node):
        # type: (Any) -> Any
        # If the contents of a merge are defined within the
        # merge marker, then they won't have been constructed
        # yet. But if they were already constructed, we need to use
        # the existing object.
        if value_node in self.constructed_objects:
            value = self.constructed_objects[value_node]
        else:
            value = self.construct_object(value_node, deep=False)
        return value

    merge_map_list = []
    index = 0
    while index < len(node.value):
        key_node, value_node = node.value[index]
        if key_node.tag == u'tag:yaml.org,2002:merge':
            if merge_map_list and not self.allow_duplicate_keys:  # double << key
                args = [
                    'while constructing a mapping',
                    node.start_mark,
                    'found duplicate key "{}"'.format(key_node.value),
                    key_node.start_mark,
                    """
                    To suppress this check see:
                       http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys
                    """,
                    """\
                    Duplicate keys will become an error in future releases, and are errors
                    by default when using the new API.
                    """,
                ]
                if self.allow_duplicate_keys is None:
                    warnings.warn(DuplicateKeyFutureWarning(*args))
                else:
                    raise DuplicateKeyError(*args)
            del node.value[index]
            # if key/values from later merge keys have preference you need
            # to insert value_node(s) at the beginning of merge_map_list
            # instead of appending
            if isinstance(value_node, ruamel.yaml.nodes.MappingNode):
                merge_map_list.append((index, constructed(value_node)))
            elif isinstance(value_node, ruamel.yaml.nodes.SequenceNode):
                for subnode in value_node.value:
                    if not isinstance(subnode, ruamel.yaml.nodes.MappingNode):
                        raise ruamel.yaml.constructor.ConstructorError(
                            'while constructing a mapping',
                            node.start_mark,
                            'expected a mapping for merging, but found %s' % subnode.id,
                            subnode.start_mark,
                        )
                    merge_map_list.append((index, constructed(subnode)))
            else:
                raise ConstructorError(
                    'while constructing a mapping',
                    node.start_mark,
                    'expected a mapping or list of mappings for merging, '
                    'but found %s' % value_node.id,
                    value_node.start_mark,
                )
        elif key_node.tag == u'tag:yaml.org,2002:value':
            key_node.tag = u'tag:yaml.org,2002:str'
            index += 1
        else:
            index += 1
    return merge_map_list

ruamel.yaml.constructor.RoundTripConstructor.flatten_mapping = my_flatten_mapping

yaml = ruamel.yaml.YAML()
yaml.allow_duplicate_keys = True
data = yaml.load(yaml_str)
for k in data['baz']:
    print(k, '>', data['baz'][k])

上面的内容给出了：

c > 3
a > 1
b > 2

- Anthon

---

谢谢。我意识到原问题有歧义，因此我已更新以澄清我寻求的行为就是第二个片段的行为：基本上将重复的合并键合并成一个。 - mamacdon

@mamacdon 我真的很想知道是哪个程序创建了这种糟糕的YAML。我可以更新我的答案，但从你的示例中不清楚哪个值具有优先权（[*ref1，*ref2]与[*ref2，*ref1]不同，如果锚定映射具有相同的键但具有不同的值，则会产生差异。你需要哪个版本（如果您不理解差异，请告诉我）？ - Anthon

YAML是手写的。它来自于我们团队Concourse CI管道中使用的配置文件。我不知道这是哪个版本的YAML。Concourse CI似乎正在使用库gopkg.in/yaml.v2作为其解析器。因此，我们将根据该库允许的任何输入进行开发。 - mamacdon

Answer 2

阅读库源代码后，我找到了一个解决方法。将选项设置为None可避免错误发生。

yml.allow_duplicate_keys = None

控制台仍会打印警告，但不会导致程序崩溃。

- mamacdon