在2013年的MongoNYC会议上,有一位演讲者提到他们使用维基百科的副本来测试全文搜索。我尝试复制这个操作,但由于文件大小和格式的原因,发现这并不容易。
以下是我的操作步骤:
当我尝试使用标准XML解析器解析文件时,Python在XML文件的大小上出现了错误。 有没有其他建议将9GB的XML文件转换为JSON格式以便加载到mongoDB中呢?
更新1:
根据Sean的建议,我也尝试了迭代元素树:
这里介绍一下
错误为:
以下是我的操作步骤:
$ wget http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
$ bunzip2 enwiki-latest-pages-articles.xml.bz2
$ python
>>> import xml.etree.ElementTree as ET
>>> tree = ET.parse('enwiki-latest-pages-articles.xml')
Killed
当我尝试使用标准XML解析器解析文件时,Python在XML文件的大小上出现了错误。 有没有其他建议将9GB的XML文件转换为JSON格式以便加载到mongoDB中呢?
更新1:
根据Sean的建议,我也尝试了迭代元素树:
>>> import xml.etree.ElementTree as ET
>>> context = ET.iterparse('enwiki-latest-pages-articles.xml', events=("start", "end"))
>>> context = iter(context)
>>> event, root = context.next()
>>> for i in context[0:10]:
... print(i)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '_IterParseIterator' object has no attribute '__getitem__'
>>> for event, elem in context[0:10]:
... if event == "end" and elem.tag == "record":
... print(elem)
... root.clear()
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '_IterParseIterator' object has no attribute '__getitem__'
同样,没有运气。
更新 2
根据Asya Kamsky下面的建议进行跟进。
尝试使用xml2json
:
$ git clone https://github.com/hay/xml2json.git
$ ./xml2json/xml2json.py -t xml2json -o enwiki-latest-pages-articles.json enwiki-latest-pages-articles.xml
Traceback (most recent call last):
File "./xml2json/xml2json.py", line 199, in <module>
main()
File "./xml2json/xml2json.py", line 181, in main
input = open(arguments[0]).read()
MemoryError
这里介绍一下
xmlutils
:$ pip install xmlutils
$ xml2json --input "enwiki-latest-pages-articles.xml" --output "enwiki-latest-pages-articles.json"
xml2sql by Kailash Nadh (http://nadh.in)
--help for help
Wrote to enwiki-latest-pages-articles.json
但是这些内容只有一条记录,没有迭代。
xmltodict
也很有前途,因为它使用迭代的Expat并且非常适合维基百科。但是在20分钟左右后,它也会耗尽内存:
>>> import xmltodict
>>> f = open('enwiki-latest-pages-articles.xml')
>>> doc = xmltodict.parse(f)
Killed
更新3
这是作为对Ross下面答案的回应,模仿他提到的链接来建立我的解析器:
from lxml import etree
file = 'enwiki-latest-pages-articles.xml'
def page_handler(page):
try:
print page.get('title','').encode('utf-8')
except:
print page
print "error"
class page_handler(object):
def __init__(self):
self.text = []
def start(self, tag, attrib):
self.is_title = True if tag == 'title' else False
def end(self, tag):
pass
def data(self, data):
if self.is_title:
self.text.append(data.encode('utf-8'))
def close(self):
return self.text
def fast_iter(context, func):
for event, elem in context:
print(elem)
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]
del context
process_element = etree.XMLParser(target = page_handler())
context = etree.iterparse( file, tag='item' )
fast_iter(context,process_element)
错误为:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in fast_iter
File "iterparse.pxi", line 484, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:112653)
File "iterparse.pxi", line 537, in lxml.etree.iterparse._read_more_events (src/lxml/lxml.etree.c:113223)
File "parser.pxi", line 596, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:83186)
lxml.etree.XMLSyntaxError: Extra content at the end of the document, line 22, column 1