当我使用 Pandas 处理一个大型数据框(1500682 行)时,调用 groupby 与 rolling 和 apply 函数会导致非常缓慢的性能。我试图获得不同权重的滚动移动平均值。
代码中运行缓慢的部分是:
df['rolling'] = df.groupby('i2')['x'].rolling(3).apply(lambda x: x[-3]*0.1+x[-2]*0.9).reset_index(level=0, drop=True).reindex(df.index)
完整代码(包括数据)如下:
import pandas as pd
from random import randint
# data (it takes some time to create [less than 1 minute in my computer])
data1 = [[[[randint(0, 100) for i in range(randint(1, 2))] for i in range(randint(1, 3))] for i in range(5000)] for i in range(100)]
data2 = pd.DataFrame(
[
(i1, i2, i3, i4, x4)
for (i1, x1) in enumerate(data1)
for (i2, x2) in enumerate(x1)
for (i3, x3) in enumerate(x2)
for (i4, x4) in enumerate(x3)
],
columns = ['i1', 'i2', 'i3', 'i4', 'x']
)
data2.drop(['i3', 'i4'], axis=1, inplace = True)
df = data2.set_index(['i1', 'i2']).sort_index()
## conflicting part of the code ##
df['rolling'] = df.groupby('i2')['x'].rolling(3).apply(lambda x: x[-3]*0.1+x[-2]*0.9).reset_index(level=0, drop=True).reindex(df.index)
如果您能详细说明代码如何更加高效地执行,使其运行速度更快,我将不胜感激。