我正在尝试处理包含在tar.gz文件中的csv文件,并且我遇到了传递正确数据/对象给csv模块的问题。
假设我有一个tar.gz文件,其中包含多个以下格式的csv文件。
1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30
1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26
1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31
1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38
我希望能够在不将tar文件中的每个csv文件解压并写入磁盘的情况下,直接在内存中访问每个csv文件。
例如:
import tarfile
import csv
tar = tarfile.open("tar-file.tar.gz")
for member in tar.getmembers():
f = tar.extractfile(member).read()
content = csv.reader(f)
for row in content:
print(row)
tar.close()
这将产生以下错误。
for row in content:
_csv.Error: iterator should return strings, not int (did you open the file in text mode?)
我也尝试了将f解析为字符串,就像csv模块文档中所述的那样。
content = csv.reader([f])
上述代码会产生相同的错误。
我尝试将文件对象f解析为ascii格式。
f = tar.extractfile(member).read().decode('ascii')
但这样会遍历每个CSV元素,而不是遍历包含元素列表的行。
['1']
['0']
['7']
['9']
['', '']
['S']
['A']
['M']
['P']
['L']
['E']
['_']
['A']
['', '']
['G']
['R']
snip...
['2']
['0']
['1']
['7']
['/']
['0']
['2']
['/']
['1']
['5']
[' ']
['2']
['2']
[':']
['5']
['7']
[':']
['3']
['8']
[]
[]
尝试将f作为 ASCII 解析并读取为字符串
f = tar.extractfile(member).read().decode('ascii')
content = csv.reader([f])
产生以下输出
for row in content:
_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
为了展示不同的输出,我使用了以下代码。
import tarfile
import csv
tar = tarfile.open("tar-file.tar.gz")
for member in tar.getmembers():
f = tar.extractfile(member).read()
print(member.name)
print('Raw :', type(f))
print(f)
print()
f = f.decode('ascii')
print('ASCII:', type(f))
print(f)
tar.close()
这将产生以下输出(对于此示例,每个csv包含相同的数据)。
./raw_data/csv-file1.csv
Raw : <class 'bytes'>
b'1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30\n1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26\n1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31\n1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38\n\n'
ASCII: <class 'str'>
1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30
1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26
1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31
1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38
./raw_data/csv-file2.csv
Raw : <class 'bytes'>
b'1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30\n1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26\n1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31\n1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38\n\n'
ASCII: <class 'str'>
1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30
1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26
1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31
1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38
./raw_data/csv-file3.csv
Raw : <class 'bytes'>
b'1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30\n1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26\n1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31\n1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38\n\n'
ASCII: <class 'str'>
1079,SAMPLE_A,GROUP,001,,2017/02/15 22:57:30
1041,SAMPLE_B,GROUP,023,,2017/02/15 22:57:26
1077,SAMPLE_C,GROUP,005,,2017/02/15 22:57:31
1079,SAMPLE_A,GROUP,128,,2017/02/15 22:57:38
我该如何使csv模块能够正确地读取tar模块提供的内存文件?谢谢。