如何解决循环引用检测问题

Question

我需要将 JSON 转换为 TOML 格式。

当我运行以下代码时，它会返回一个错误。

import toml

toml_config = toml.dumps(data)

data 是指：

{"general":{"log_level":{"value":"4","editable":"true","description":"debug=5, info=4, warning=3, error=2, fatal=1, panic=0"},"log_to_syslog":{"value":"false","editable":"true","description":"When set to true, log messages are being written to syslog."}},"postgresql":{"dsn":"true","postgres://localhost/chirpstack_ns?sslmode\n\n# Automatically apply database migrations.\n#\n# It is possible to apply the database-migrations by hand\n# (see https://github.com/brocaar/chirpstack-network-server/tree/master/migrations)\n# or let ChirpStack Application Server migrate to the latest state automatically, by using\n# this setting. Make sure that you always make a backup when upgrading ChirpStack\n# Application Server and / or applying migrations.\nautomigratemax_open_connections":0,"max_idle_connections":2},"redis":{"servers":["localhost:6379"],"password":"","database":0,"cluster":"false","master_name":"","pool_size":0},"network_server":{"net_id":"000000","deduplication_delay":"200ms","device_session_ttl":"744h0m0s","get_downlink_data_delay":{"value":"100ms","editable":"true"},"band":{"name":"EU868","uplink_dwell_time_400ms":"false","downlink_dwell_time_400ms":"false","uplink_max_eirp":-1,"repeater_compatible":"false"},"network_settings":{"installation_margin":10,"rx_window":0,"rx1_delay":1,"rx1_dr_offset":0,"rx2_dr":-1,"rx2_frequency":-1,"rx2_prefer_on_rx1_dr_lt":0,"rx2_prefer_on_link_budget":"false","gateway_prefer_min_margin":10,"downlink_tx_power":-1,"disable_mac_commands":"false","disable_adr":"false","max_mac_command_error_count":3,"enabled_uplink_channels":[],"class_b":{"ping_slot_dr":0,"ping_slot_frequency":0},"rejoin_request":{"enabled":"false","max_count_n":0,"max_time_n":0}},"scheduler":{"scheduler_interval":"1s","class_c":{"downlink_lock_duration":"2s","multicast_gateway_delay":"2s"}},"api":{"bind":"0.0.0.0:8000","ca_cert":"","tls_cert":"","tls_key":""},"gateway":{"ca_cert":"","ca_key":"","client_cert_lifetime":"8760h0m0s","backend":{"type":"mqtt","multi_downlink_feature":"hybrid","mqtt":{"event_topic":"gateway/+/event/+","command_topic_template":"gateway/{{ .GatewayID }}/command/{{ .CommandType }}","server":"tcp://localhost:1883","username":"","password":"","max_reconnect_interval":"1m0s","qos":0,"clean_session":"true","client_id":"","ca_cert":"","tls_cert":"","tls_key":""},"amqp":{"url":"amqp://guest:guest@localhost:5672","event_queue_name":"gateway-events","event_routing_key":"gateway.*.event.*","command_routing_key_template":"gateway.{{ .GatewayID }}.command.{{ .CommandType }}"},"gcp_pub_sub":{"credentials_file":"","project_id":"","uplink_topic_name":"","downlink_topic_name":"","uplink_retention_duration":"24h0m0s"},"azure_iot_hub":{"events_connection_string":{"value":"","editable":"true","description":"This connection string must point to the Service Bus Queue to which the IoT Hub is forwarding the (uplink) gateway events."},"commands_connection_string":{"value":"","editable":"true","description":"This connection string must point to the IoT Hub and is used by ChirpStack Network Server for sending commands to the gateways."}}}}},"monitoring":{"bind":"","prometheus_endpoint":"false","prometheus_api_timing_histogram":"false","healthcheck_endpoint":"false"},"join_server":{"resolve_join_eui":"false","resolve_domain_suffix":".joineuis.lora-alliance.org","default":{"server":"http://localhost:8003","ca_cert":"","tls_cert":"","tls_key":""}},"network_controller":{"server":"","ca_cert":"","tls_cert":"","tls_key":""}}

运行上面的代码时我遇到的错误是：

Traceback (most recent call last):
  File "<pyshell#38>", line 1, in <module>
    toml_string = toml.dumps(data)
  File "/usr/local/lib/python3.8/dist-packages/toml/encoder.py", line 67, in dumps
    raise ValueError("Circular reference detected")
ValueError: Circular reference detected

在深入研究后，我发现删除一个键-值对后，代码就能正常工作。这个键-值对是：

"type": "mqtt"

这个键值对的位置在 data["network_server"]["gateway"]["backend"]。

我无法理解这种情况，我尝试更改键值对字符串，但问题仍然存在。只有删除这个键值对才能解决问题，但我需要这个键值对。

非常感谢任何帮助。提前致谢。

- theashwanisingla

Answer 1

我尝试重现你的问题，因为我打算从json配置文件转换到toml配置文件。我已经安装了版本0.10.1的toml。

数据中有一个小问题（在"postgresql"关键字后的#注释），但是整个段落可以直接删除。

我将该示例精简为：

data = { 
        "network_server":{
                "downlink_data_delay":{},
                "gateway":{"key3":{"key4":{}}
                }   
        }   
}

import toml
print(toml.dumps(data)) # ValueError: Circular reference detected

但奇怪的是，当你保持结构并略微重命名键时，错误消失了！

只需从"network_server"中删除第一个 'n'，你就可以得到输出！

这一定是个漏洞。非常遗憾。

---

更新：查看源代码后，问题变得相当明显。

retval = ""
if encoder is None:
    encoder = TomlEncoder(o.__class__)
addtoretval, sections = encoder.dump_sections(o, "")
retval += addtoretval
outer_objs = [id(o)]
while sections:
    section_ids = [id(section) for section in sections]
    for outer_obj in outer_objs:
        if outer_obj in section_ids:
            raise ValueError("Circular reference detected")
    outer_objs += section_ids
    newsections = encoder.get_empty_table()
    for section in sections:
        addtoretval, addtosections = encoder.dump_sections(
            sections[section], section)

        if addtoretval or (not addtoretval and not addtosections):
            if retval and retval[-2:] != "\n\n":
                retval += "\n"
            retval += "[" + section + "]\n"
            if addtoretval:
                retval += addtoretval
        for s in addtosections:
            newsections[section + "." + s] = addtosections[s]
    sections = newsections
return retval

那么不是这样吗？好的，它跟踪章节的id，但是章节在循环迭代之间没有保留。当id被重复使用时，程序会出现混淆。

快速且粗糙的解决方法：修补toml/encoder.py文件中的dumps函数：

1. add allsections=[] before the while loop

2. add one line here:

        allsections += sections   # <---
        sections = newsections
    return retval
   

这是为了防止未使用的部分被垃圾回收，以保持它们的id在使用中。

欢迎在GitHub上向作者报告问题。

- VPfB