在一个有100000列的文件中,一些计时结果显示count函数最快,但是结果却少了一列:
In [14]: %%timeit
with open("test.csv" ) as f:
r = csv.reader(f, delimiter="\t")
len(next(r))
....:
10 loops, best of 3: 88.7 ms per loop
In [15]: %%timeit
with open("test.csv" ) as f:
next(f).count("\t")
....:
100 loops, best of 3: 11.9 ms per loop
with io.open('test.csv', 'r') as fin:
num_columns = len(next(fin).split('\t'))
....:
10 loops, best of 3: 133 ms per loop
使用str.translate实际上似乎是最快的,尽管你需要再加1:
In [5]: %%timeit
with open("test.csv" ) as f:
n = next(f)
(len(n) - len(n.translate(None, "\t")))
...:
100 loops, best of 3: 9.9 ms per loop
使用pandas的解决方案给我报错:
in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7977)()
StopIteration:
使用readline会增加额外的开销:
In [19]: %%timeit
with open("test.csv" ) as f:
f.readline().count("\t")
....:
10 loops, best of 3: 28.9 ms per loop
In [30]: %%timeit
with io.open('test.csv', 'r') as fin:
num_columns = len(fin.readline().split('\t'))
....:
10 loops, best of 3: 136 ms per loop
Python 3.4 的结果不同:
In [7]: %%timeit
with io.open('test.csv', 'r') as fin:
num_columns = len(next(fin).split('\t'))
...:
10 loops, best of 3: 102 ms per loop
In [8]: %%timeit
with open("test.csv" ) as f:
f.readline().count("\t")
...:
100 loops, best of 3: 12.7 ms per loop
In [9]:
In [9]: %%timeit
with open("test.csv" ) as f:
next(f).count("\t")
...:
100 loops, best of 3: 11.5 ms per loop
In [10]: %%timeit
with io.open('test.csv', 'r') as fin:
num_columns = len(next(fin).split('\t'))
....:
10 loops, best of 3: 89.9 ms per loop
In [11]: %%timeit
with io.open('test.csv', 'r') as fin:
num_columns = len(fin.readline().split('\t'))
....:
10 loops, best of 3: 92.4 ms per loop
In [13]: %%timeit
with open("test.csv" ) as f:
r = csv.reader(f, delimiter="\t")
len(next(r))
....:
10 loops, best of 3: 176 ms per loop
StopIteration:
错误。 - Padraic Cunninghamawk '{print NF;quit}' file
- Mark Setchell