Python Json解析器

Question

Python Json解析器

3

我有一个具有以下结构的JSON文件：-

{
    "contributors": null, 
    "truncated": false, 
    "text": "@HomeShop18 #DreamJob to professional rafter", 
    "in_reply_to_status_id": null, 
    "id": 421584490452893696, 
    "favorite_count": 0, 
    "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Mobile Web (M2)</a>", 
    "retweeted": false, 
    "coordinates": null, 
    "entities": {
        "symbols": [], 
        "user_mentions": [
            {
                "id": 183093247, 
                "indices": [
                    0, 
                    11
                ], 
                "id_str": "183093247", 
                "screen_name": "HomeShop18", 
                "name": "HomeShop18"
            }
        ], 
        "hashtags": [
            {
                "indices": [
                    12, 
                    21
                ], 
                "text": "DreamJob"
            }
        ], 
        "urls": []
    }, 
    "in_reply_to_screen_name": "HomeShop18", 
    "id_str": "421584490452893696", 
    "retweet_count": 0, 
    "in_reply_to_user_id": 183093247, 
    "favorited": false, 
    "user": {
        "follow_request_sent": null, 
        "profile_use_background_image": true, 
        "default_profile_image": false, 
        "id": 2254546045, 
        "verified": false, 
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/413952088880594944/rcdr59OY_normal.jpeg", 
        "profile_sidebar_fill_color": "171106", 
        "profile_text_color": "8A7302", 
        "followers_count": 87, 
        "profile_sidebar_border_color": "BCB302", 
        "id_str": "2254546045", 
        "profile_background_color": "0F0A02", 
        "listed_count": 1, 
        "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png", 
        "utc_offset": null, 
        "statuses_count": 9793, 
        "description": "Rafter. Rafting is what I do. Me aur mera Tablet.  Technocrat of Future", 
        "friends_count": 231, 
        "location": "", 
        "profile_link_color": "473623", 
        "profile_image_url": "http://pbs.twimg.com/profile_images/413952088880594944/rcdr59OY_normal.jpeg", 
        "following": null, 
        "geo_enabled": false, 
        "profile_banner_url": "https://pbs.twimg.com/profile_banners/2254546045/1388065343", 
        "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png", 
        "name": "Jayy", 
        "lang": "en", 
        "profile_background_tile": false, 
        "favourites_count": 41, 
        "screen_name": "JzayyPsingh", 
        "notifications": null, 
        "url": null, 
        "created_at": "Fri Dec 20 05:46:00 +0000 2013", 
        "contributors_enabled": false, 
        "time_zone": null, 
        "protected": false, 
        "default_profile": false, 
        "is_translator": false
    }, 
    "geo": null, 
    "in_reply_to_user_id_str": "183093247", 
    "lang": "en", 
    "created_at": "Fri Jan 10 10:09:09 +0000 2014", 
    "filter_level": "medium", 
    "in_reply_to_status_id_str": null, 
    "place": null
}

该文件中几乎有1500个这样的字典。我想知道是否有任何现成的解析器，在Python中，用于懒惰解析此类文件。我希望该解析器每次仅返回一个字典，或在最坏的情况下，按行返回数据。我该怎么办？

- Apoorv Ashutosh

在这种情况下，一行是指一个字典对象，对吗？ - Aswin Murugesh

那么关于 http://docs.python.org/2/library/json.html 呢？ - greole

3

你知道你展示的 JSON 样例是无效的，对吧？ - thefourtheye

@thefourtheye，是的，那只是一个示例。实际上相当长。我应该发布它吗？ - Apoorv Ashutosh

1

@ApoorvAshutosh 不是关于大小的问题。 :) - thefourtheye

显示剩余2条评论

3个回答

2

您可以使用来自json.JSONDecoder的raw_decode而不必将整个文件读入内存：

raw_decode(s) 从s（以JSON文档开头的str）解码JSON文档，并返回Python表示和文档结束处的索引的2元组。

这可用于从可能在末尾具有无关数据的字符串中解码JSON文档。

import json

def iterparse(file_obj):
    decoder = json.JSONDecoder()
    buf = ""
    for line in file_obj:
        buf += line.strip()
        try:
            res = decoder.raw_decode(buf)
            buf = ""
            yield res[0]
        except ValueError:
            pass

with open("stuff.json") as f:
    for obj in iterparse(f):
        print obj

- root

不知怎么的，我并不真正理解这段代码是如何完美地生成第一个字典的。然而，其他字典并没有通过这种方式获得。 - Apoorv Ashutosh

@ApoorvAshutosh -- 当我在多个你的示例对象上测试时，它可以正确地工作... - root

0

尝试使用ijson解析器：

示例用法：

import ijson

parser = ijson.parse(urlopen('url/'))
stream.write('<geo>')
for prefix, event, value in parser:
    if (prefix, event) == ('earth', 'map_key'):
        stream.write('<%s>' % value)
        continent = value
    elif prefix.endswith('.name'):
        stream.write('<object name="%s"/>' % value)
    elif (prefix, event) == ('earth.%s' % continent, 'end_map'):
        stream.write('</%s>' % continent)
stream.write('</geo>')

- KiraLT

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- falsetru · Accepted Answer

下面是使用 json.JSONDecoder.raw_decode 的一个解决方法。我希望有人能提出更好的处理此问题的方法。

import json
import re

nonspace = re.compile(r'\S')
def iterparse(j):
    decoder = json.JSONDecoder()
    pos = 0
    while True:
        matched = nonspace.search(j, pos)
        if not matched:
            break
        pos = matched.start()
        decoded, pos = decoder.raw_decode(j, pos)
        yield decoded

使用示例：

>>> j = '''
... { "id": 1 }
... { "id": 2 }
... '''
>>> list(iterparse(j))
[{u'id': 1}, {u'id': 2}]