在Python中读取大型JSON文件（raw_decode）

Question

在Python中读取大型JSON文件（raw_decode）

7

我试图在Python中读取大型JSON文件（data.json）。由于该JSON文件具有多个JSON对象，并且在Python中将创建多个字典（字典数量未知），因此我使用了decoder.raw_decode()和generator。以下是代码：

import json
import pprint
import io
import pprint

def parse():

    with open('data.json',encoding='utf-8') as jfile:
        try:
            while True:
                decoder = json.JSONDecoder()
                obj, idx = decoder.raw_decode(jfile)
                yield obj
        except ValueError as e:
            print(e)
            pass
        else:
            print("aha")


def main():
    imputd=parse()
    if imputd: 
        while True:
            try:
                print(str(next(imputd)).readlines())
            except StopIteration as e:
                print(e)
                break

main()

I get the error:

Traceback (most recent call last):
  File "H:\Document\Python\j10.py", line 57, in <module>
    main()
  File "H:\Document\Python\j10.py", line 36, in main
    print(str(next(imputd)).readlines())
  File "H:\Document\Python\j10.py", line 21, in parse
    obj, idx = decoder.raw_decode(jfile)
  File "C:\Python34\lib\json\decoder.py", line 360, in raw_decode
    obj, end = self.scan_once(s, idx)
TypeError: first argument must be a string, not _io.TextIOWrapper

我根据Martijn的答案进行了代码编辑：

import json
import io



file=open('data.json.txt')
def readin():
    return file.read(2000)


def parse():
    decoder = json.JSONDecoder()
    buffer = ''    
    for chunk in iter(readin, ''):
        buffer += chunk
        while buffer:
            try:
                result, index = decoder.raw_decode(buffer)
                yield result
                buffer = buffer[index:]
            except ValueError:
                 # Not enough data to decode, read more
                break

def main():
    imputd=parse()
    if imputd: 
        while True:
            try:
                print(str(next(imputd)).readlines())
            except StopIteration as e:
                print(e)
                break

main()

当我遇到 UnicodeError 时：

Traceback (most recent call last):
  File "H:\Document\Python\j11.py", line 35, in <module>
    main()
  File "H:\Document\Python\j11.py", line 30, in main
    print(str(next(imputd)).readlines())
  File "H:\Document\Python\j11.py", line 14, in parse
    for chunk in iter(readin, ''):
  File "H:\Document\Python\j11.py", line 8, in readin
    return file.read(2000)
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 4217: character maps to <undefined>

- xccccx

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Martijn Pieters · Accepted Answer

你正在传递文件对象，但是decoder.raw_decode()仅接受文本数据。你需要自己进行读取：

obj, idx = decoder.raw_decode(jfile.read())

你需要生成从JSON数据创建的Python对象，因此在main()函数循环中，你的.readlines()调用也会失败。

然而，你没有正确使用raw_decode()。你需要负责为其提供文本块，它不会自动从文件中读取。如果你想以块处理文件，并且在JSON条目之间没有明确的分隔符，那么你将被迫按块读取文件：

decoder = json.JSONDecoder()
buffer = ''
for chunk in iter(partial(jfile.read, buffersize), ''):
     buffer += chunk
     while buffer:
         try:
             result, index = decoder.raw_decode(buffer)
             yield result
             buffer = buffer[index:]
         except ValueError:
             # Not enough data to decode, read more
             break

这仍会产生完全解码的对象；如果您的文件是一个长的JSON对象（如一个顶级列表或字典），那么这不会逐个产生该对象的内容；它仍将在产生之前读取整个对象。