使用Python3,Pandas 0.12
我试图将多个csv文件(总大小为7.9 GB)写入HDF5存储器以供以后处理。每个CSV文件包含大约一百万行,15列,数据类型大部分是字符串,但有些是浮点数。然而,当我尝试读取CSV文件时,会出现以下错误:
Traceback (most recent call last):
File "filter-1.py", line 38, in <module>
to_hdf()
File "filter-1.py", line 31, in to_hdf
for chunk in reader:
File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 578, in __iter__
yield self.read(self.chunksize)
File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 608, in read
ret = self._engine.read(nrows)
File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 1028, in read
data = self._reader.read(nrows)
File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandas\parser.c:6745)
File "parser.pyx", line 740, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:7146)
File "parser.pyx", line 781, in pandas.parser.TextReader._read_rows (pandas\parser.c:7568)
File "parser.pyx", line 768, in pandas.parser.TextReader._tokenize_rows (pandas\parser.c:7451)
File "parser.pyx", line 1661, in pandas.parser.raise_parser_error (pandas\parser.c:18744)
pandas.parser.CParserError: Error tokenizing data. C error: EOF inside string starting at line 754991
Closing remaining open files: ta_store.h5... done
编辑:
我找到了一个产生这个问题的文件。我认为它正在读取EOF字符。然而,我不知道如何解决这个问题。考虑到合并后的文件很大,检查每个字符串中的每个单个字符太麻烦了。(即使这样,我仍然不确定该怎么做。) 就我所检查的范围内,csv文件中没有奇怪的字符可以引起错误。我还尝试将error_bad_lines=False
传递给pd.read_csv()
,但错误仍然存在。
我的代码如下:
# -*- coding: utf-8 -*-
import pandas as pd
import os
from glob import glob
def list_files(path=os.getcwd()):
''' List all files in specified path '''
list_of_files = [f for f in glob('2013-06*.csv')]
return list_of_files
def to_hdf():
""" Function that reads multiple csv files to HDF5 Store """
# Defining path name
path = 'ta_store.h5'
# If path exists delete it such that a new instance can be created
if os.path.exists(path):
os.remove(path)
# Creating HDF5 Store
store = pd.HDFStore(path)
# Reading csv files from list_files function
for f in list_files():
# Creating reader in chunks -- reduces memory load
reader = pd.read_csv(f, chunksize=50000)
# Looping over chunks and storing them in store file, node name 'ta_data'
for chunk in reader:
chunk.to_hdf(store, 'ta_data', mode='w', table=True)
# Return store
return store.select('ta_data')
return 'Finished reading to HDF5 Store, continuing processing data.'
to_hdf()
编辑
如果我进入引发CParserError EOF的CSV文件,并手动删除问题行之后的所有行,那么CSV文件将被正确读取。但是我要删除的只是空白行。 奇怪的是,当我手动更正有误的csv文件时,它们可以单独加载到存储中。但是当我再次使用多个文件的列表时,“错误”的文件仍然会返回错误。
from pandas.io import parser; try: your read_csv look for file f except (parser.CParserError) as detail: print f, detail
- Jeff