对于大量数据,被接受的解决方案将非常缓慢。得票最多的解决方案有些难以阅读,在处理数值数据时也很慢。如果每个新列可以独立于其他列进行计算,我会直接分配它们而不使用apply
。
使用虚假字符数据的示例
在DataFrame中创建100,000个字符串
df = pd.DataFrame(np.random.choice(['he jumped', 'she ran', 'they hiked'],
size=100000, replace=True),
columns=['words'])
df.head()
words
0 she ran
1 she ran
2 they hiked
3 they hiked
4 they hiked
假设我们想提取一些文本特征,就像原问题中所做的那样。例如,让我们提取第一个字符,计算字母'e'出现的次数并将短语大写。
df['first'] = df['words'].str[0]
df['count_e'] = df['words'].str.count('e')
df['cap'] = df['words'].str.capitalize()
df.head()
words first count_e cap
0 she ran s 1 She ran
1 she ran s 1 She ran
2 they hiked t 2 They hiked
3 they hiked t 2 They hiked
4 they hiked t 2 They hiked
时间
%%timeit
df['first'] = df['words'].str[0]
df['count_e'] = df['words'].str.count('e')
df['cap'] = df['words'].str.capitalize()
127 ms ± 585 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
def extract_text_features(x):
return x[0], x.count('e'), x.capitalize()
%timeit df['first'], df['count_e'], df['cap'] = zip(*df['words'].apply(extract_text_features))
101 ms ± 2.96 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
令人惊讶的是,通过遍历每个值,您可以获得更好的性能
%%timeit
a,b,c = [], [], []
for s in df['words']:
a.append(s[0]), b.append(s.count('e')), c.append(s.capitalize())
df['first'] = a
df['count_e'] = b
df['cap'] = c
79.1 ms ± 294 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
使用虚假数字数据的另一个示例
生成100万个随机数,并测试上述powers
函数。
df = pd.DataFrame(np.random.rand(1000000), columns=['num'])
def powers(x):
return x, x**2, x**3, x**4, x**5, x**6
%%timeit
df['p1'], df['p2'], df['p3'], df['p4'], df['p5'], df['p6'] = \
zip(*df['num'].map(powers))
1.35 s ± 83.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
给每列分配值的速度提高了25倍,并且非常易于阅读:
%%timeit
df['p1'] = df['num'] ** 1
df['p2'] = df['num'] ** 2
df['p3'] = df['num'] ** 3
df['p4'] = df['num'] ** 4
df['p5'] = df['num'] ** 5
df['p6'] = df['num'] ** 6
51.6 ms ± 1.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
我在这里提供了更详细的解释,说明为什么通常不建议使用apply
方法。
df.ix[:, 10:16]
。我觉得你需要将特征与数据集进行合并。 - Zelazny7