Pandas:按时间间隔滚动平均

118

我有一堆投票数据;我想计算 Pandas 滚动均值,以便根据三天窗口估计每天的结果。根据 这个问题rolling_* 函数基于指定数量的值来计算窗口,而不是特定的日期时间范围。

如何实现此功能?

示例输入数据:

polls_subset.tail(20)
Out[185]: 
            favorable  unfavorable  other

enddate                                  
2012-10-25       0.48         0.49   0.03
2012-10-25       0.51         0.48   0.02
2012-10-27       0.51         0.47   0.02
2012-10-26       0.56         0.40   0.04
2012-10-28       0.48         0.49   0.04
2012-10-28       0.46         0.46   0.09
2012-10-28       0.48         0.49   0.03
2012-10-28       0.49         0.48   0.03
2012-10-30       0.53         0.45   0.02
2012-11-01       0.49         0.49   0.03
2012-11-01       0.47         0.47   0.05
2012-11-01       0.51         0.45   0.04
2012-11-03       0.49         0.45   0.06
2012-11-04       0.53         0.39   0.00
2012-11-04       0.47         0.44   0.08
2012-11-04       0.49         0.48   0.03
2012-11-04       0.52         0.46   0.01
2012-11-04       0.50         0.47   0.03
2012-11-05       0.51         0.46   0.02
2012-11-07       0.51         0.41   0.00

每个日期只会有一行输出。


2
Pandas bug跟踪器中存在一个开放问题,请求此功能:https://github.com/pydata/pandas/issues/936。该功能尚不存在。回答此问题(https://dev59.com/MGYq5IYBdhLWcg3wrSWs)描述了一种获得所需效果的方法,但与内置的`rolling_*`函数相比,它通常会非常慢。 - BrenBarn
不可否认的是,doc 很糟糕,没有展示任何例子,甚至没有用简单的英语描述 *"you can pass rolling(..., window='7d')"*。 - smci
9个回答

127

同时,我们增加了时间窗口功能。查看此链接

In [1]: df = DataFrame({'B': range(5)})

In [2]: df.index = [Timestamp('20130101 09:00:00'),
   ...:             Timestamp('20130101 09:00:02'),
   ...:             Timestamp('20130101 09:00:03'),
   ...:             Timestamp('20130101 09:00:05'),
   ...:             Timestamp('20130101 09:00:06')]

In [3]: df
Out[3]: 
                     B
2013-01-01 09:00:00  0
2013-01-01 09:00:02  1
2013-01-01 09:00:03  2
2013-01-01 09:00:05  3
2013-01-01 09:00:06  4

In [4]: df.rolling(2, min_periods=1).sum()
Out[4]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  3.0
2013-01-01 09:00:05  5.0
2013-01-01 09:00:06  7.0

In [5]: df.rolling('2s', min_periods=1).sum()
Out[5]: 
                       B
2013-01-01 09:00:00  0.0
2013-01-01 09:00:02  1.0
2013-01-01 09:00:03  3.0
2013-01-01 09:00:05  3.0
2013-01-01 09:00:06  7.0

1
这应该是最佳答案。 - Ivan
14
rolling函数接受偏移量参数(如'2s'),其文档在此处:https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects - Guilherme Salomé
2
如果数据框中有多列,我们如何指定特定的列? - Brain_overflowed
1
@Brain_overflowed 设置为索引 - jamfie
min_period在这种方法中似乎不可靠。对于min_periods>1,由于时间戳精度/变量采样率,您可能会在意料之外的地方得到NaN。 - Albert James Teddy

53

这样怎么样:

首先将数据帧重新采样到1D间隔中。这会对所有重复日期的值进行平均。使用fill_method选项填充缺失的日期值。接下来,将重新采样的框架传递到窗口大小为3,最小周期为1的pd.rolling_mean函数中:

pd.rolling_mean(df.resample("1D", fill_method="ffill"), window=3, min_periods=1)

            favorable  unfavorable     other
enddate
2012-10-25   0.495000     0.485000  0.025000
2012-10-26   0.527500     0.442500  0.032500
2012-10-27   0.521667     0.451667  0.028333
2012-10-28   0.515833     0.450000  0.035833
2012-10-29   0.488333     0.476667  0.038333
2012-10-30   0.495000     0.470000  0.038333
2012-10-31   0.512500     0.460000  0.029167
2012-11-01   0.516667     0.456667  0.026667
2012-11-02   0.503333     0.463333  0.033333
2012-11-03   0.490000     0.463333  0.046667
2012-11-04   0.494000     0.456000  0.043333
2012-11-05   0.500667     0.452667  0.036667
2012-11-06   0.507333     0.456000  0.023333
2012-11-07   0.510000     0.443333  0.013333

更新: 正如Ben在评论中指出的那样,从pandas 0.18.0开始语法已经发生了变化。使用新的语法,代码应该为:

df.resample("1d").sum().fillna(0).rolling(window=3, min_periods=1).mean()

抱歉,作为Pandas新手,ffill使用什么规则来提供缺失值? - Anov
1
有几个填充选项。ffill代表向前填充,只需传递最近的非缺失值。同样,bfill代表向后填充,以相反的顺序执行相同的操作。 - Zelazny7
10
也许我理解有误,但是你是否忽略了同一天内的多次读数(在计算滚动平均值时,你应该赋予两个读数比一个读数更高的权重……) - Andy Hayden
4
很好的回答。只是需要注意,在pandas 0.18.0中,语法已经改变。新的语法是:df.resample("1D").ffill(limit=0).rolling(window=3, min_periods=1).mean() - Ben
1
为了在 pandas 版本 0.18.1 中复制原始答案的结果,我使用以下代码:df.resample("1d").mean().rolling(window=3, min_periods=1).mean() - JohnE

35

我刚刚有同样的问题,但数据点的时间间隔不规则。在这种情况下,重新采样不是一个很好的选择。所以我创建了自己的函数。也许对其他人也有用:

from pandas import Series, DataFrame
import pandas as pd
from datetime import datetime, timedelta
import numpy as np

def rolling_mean(data, window, min_periods=1, center=False):
    ''' Function that computes a rolling mean

    Parameters
    ----------
    data : DataFrame or Series
           If a DataFrame is passed, the rolling_mean is computed for all columns.
    window : int or string
             If int is passed, window is the number of observations used for calculating 
             the statistic, as defined by the function pd.rolling_mean()
             If a string is passed, it must be a frequency string, e.g. '90S'. This is
             internally converted into a DateOffset object, representing the window size.
    min_periods : int
                  Minimum number of observations in window required to have a value.

    Returns
    -------
    Series or DataFrame, if more than one column    
    '''
    def f(x):
        '''Function to apply that actually computes the rolling mean'''
        if center == False:
            dslice = col[x-pd.datetools.to_offset(window).delta+timedelta(0,0,1):x]
                # adding a microsecond because when slicing with labels start and endpoint
                # are inclusive
        else:
            dslice = col[x-pd.datetools.to_offset(window).delta/2+timedelta(0,0,1):
                         x+pd.datetools.to_offset(window).delta/2]
        if dslice.size < min_periods:
            return np.nan
        else:
            return dslice.mean()

    data = DataFrame(data.copy())
    dfout = DataFrame()
    if isinstance(window, int):
        dfout = pd.rolling_mean(data, window, min_periods=min_periods, center=center)
    elif isinstance(window, basestring):
        idx = Series(data.index.to_pydatetime(), index=data.index)
        for colname, col in data.iterkv():
            result = idx.apply(f)
            result.name = colname
            dfout = dfout.join(result, how='outer')
    if dfout.columns.size == 1:
        dfout = dfout.ix[:,0]
    return dfout


# Example
idx = [datetime(2011, 2, 7, 0, 0),
       datetime(2011, 2, 7, 0, 1),
       datetime(2011, 2, 7, 0, 1, 30),
       datetime(2011, 2, 7, 0, 2),
       datetime(2011, 2, 7, 0, 4),
       datetime(2011, 2, 7, 0, 5),
       datetime(2011, 2, 7, 0, 5, 10),
       datetime(2011, 2, 7, 0, 6),
       datetime(2011, 2, 7, 0, 8),
       datetime(2011, 2, 7, 0, 9)]
idx = pd.Index(idx)
vals = np.arange(len(idx)).astype(float)
s = Series(vals, index=idx)
rm = rolling_mean(s, window='2min')

你能包含相关的导入吗? - Bryce Drennan
请提供一个输入数据框的示例,如果计算时间间隔滑动窗口可以使用,谢谢。 - joshlk
在原帖中添加了一个示例。 - user2689410
6
现在可以使用 s.rolling('2min', min_periods=1).mean() 来完成相同的操作。 - kampta

9

user2689410的代码正是我所需要的。以下是我的版本(感谢user2689410),由于在DataFrame中一次性计算整行的平均值,因此更快。

希望我的后缀约定可读性强:_s:字符串,_i:整数,_b:布尔值,_ser:Series和_df:DataFrame。当您找到多个后缀时,类型可以是两者之一。

import pandas as pd
from datetime import datetime, timedelta
import numpy as np

def time_offset_rolling_mean_df_ser(data_df_ser, window_i_s, min_periods_i=1, center_b=False):
    """ Function that computes a rolling mean

    Credit goes to user2689410 at https://dev59.com/KmUo5IYBdhLWcg3wxB7T

    Parameters
    ----------
    data_df_ser : DataFrame or Series
         If a DataFrame is passed, the time_offset_rolling_mean_df_ser is computed for all columns.
    window_i_s : int or string
         If int is passed, window_i_s is the number of observations used for calculating
         the statistic, as defined by the function pd.time_offset_rolling_mean_df_ser()
         If a string is passed, it must be a frequency string, e.g. '90S'. This is
         internally converted into a DateOffset object, representing the window_i_s size.
    min_periods_i : int
         Minimum number of observations in window_i_s required to have a value.

    Returns
    -------
    Series or DataFrame, if more than one column

    >>> idx = [
    ...     datetime(2011, 2, 7, 0, 0),
    ...     datetime(2011, 2, 7, 0, 1),
    ...     datetime(2011, 2, 7, 0, 1, 30),
    ...     datetime(2011, 2, 7, 0, 2),
    ...     datetime(2011, 2, 7, 0, 4),
    ...     datetime(2011, 2, 7, 0, 5),
    ...     datetime(2011, 2, 7, 0, 5, 10),
    ...     datetime(2011, 2, 7, 0, 6),
    ...     datetime(2011, 2, 7, 0, 8),
    ...     datetime(2011, 2, 7, 0, 9)]
    >>> idx = pd.Index(idx)
    >>> vals = np.arange(len(idx)).astype(float)
    >>> ser = pd.Series(vals, index=idx)
    >>> df = pd.DataFrame({'s1':ser, 's2':ser+1})
    >>> time_offset_rolling_mean_df_ser(df, window_i_s='2min')
                          s1   s2
    2011-02-07 00:00:00  0.0  1.0
    2011-02-07 00:01:00  0.5  1.5
    2011-02-07 00:01:30  1.0  2.0
    2011-02-07 00:02:00  2.0  3.0
    2011-02-07 00:04:00  4.0  5.0
    2011-02-07 00:05:00  4.5  5.5
    2011-02-07 00:05:10  5.0  6.0
    2011-02-07 00:06:00  6.0  7.0
    2011-02-07 00:08:00  8.0  9.0
    2011-02-07 00:09:00  8.5  9.5
    """

    def calculate_mean_at_ts(ts):
        """Function (closure) to apply that actually computes the rolling mean"""
        if center_b == False:
            dslice_df_ser = data_df_ser[
                ts-pd.datetools.to_offset(window_i_s).delta+timedelta(0,0,1):
                ts
            ]
            # adding a microsecond because when slicing with labels start and endpoint
            # are inclusive
        else:
            dslice_df_ser = data_df_ser[
                ts-pd.datetools.to_offset(window_i_s).delta/2+timedelta(0,0,1):
                ts+pd.datetools.to_offset(window_i_s).delta/2
            ]
        if  (isinstance(dslice_df_ser, pd.DataFrame) and dslice_df_ser.shape[0] < min_periods_i) or \
            (isinstance(dslice_df_ser, pd.Series) and dslice_df_ser.size < min_periods_i):
            return dslice_df_ser.mean()*np.nan   # keeps number format and whether Series or DataFrame
        else:
            return dslice_df_ser.mean()

    if isinstance(window_i_s, int):
        mean_df_ser = pd.rolling_mean(data_df_ser, window=window_i_s, min_periods=min_periods_i, center=center_b)
    elif isinstance(window_i_s, basestring):
        idx_ser = pd.Series(data_df_ser.index.to_pydatetime(), index=data_df_ser.index)
        mean_df_ser = idx_ser.apply(calculate_mean_at_ts)

    return mean_df_ser

4
这个例子似乎需要按照@andyhayden评论中的建议使用加权平均值。例如,有两次调查在10月25日进行,分别在10月26日和10月27日各一次。如果你只是重新采样然后取平均值,这实际上会使得10月26日和10月27日的调查比10月25日的调查增加两倍的权重。
为了给每个调查相同的权重而不是给每天相同的权重,你可以尝试以下方法。
>>> wt = df.resample('D',limit=5).count()

            favorable  unfavorable  other
enddate                                  
2012-10-25          2            2      2
2012-10-26          1            1      1
2012-10-27          1            1      1

>>> df2 = df.resample('D').mean()

            favorable  unfavorable  other
enddate                                  
2012-10-25      0.495        0.485  0.025
2012-10-26      0.560        0.400  0.040
2012-10-27      0.510        0.470  0.020

这为您提供了进行基于投票而不是基于日期平均值的原材料。与之前一样,投票结果在10/25上平均,但是10/25的权重也被存储,并且是10/26或10/27的权重的两倍,以反映10/25进行了两次投票。

>>> df3 = df2 * wt
>>> df3 = df3.rolling(3,min_periods=1).sum()
>>> wt3 = wt.rolling(3,min_periods=1).sum()

>>> df3 = df3 / wt3  

            favorable  unfavorable     other
enddate                                     
2012-10-25   0.495000     0.485000  0.025000
2012-10-26   0.516667     0.456667  0.030000
2012-10-27   0.515000     0.460000  0.027500
2012-10-28   0.496667     0.465000  0.041667
2012-10-29   0.484000     0.478000  0.042000
2012-10-30   0.488000     0.474000  0.042000
2012-10-31   0.530000     0.450000  0.020000
2012-11-01   0.500000     0.465000  0.035000
2012-11-02   0.490000     0.470000  0.040000
2012-11-03   0.490000     0.465000  0.045000
2012-11-04   0.500000     0.448333  0.035000
2012-11-05   0.501429     0.450000  0.032857
2012-11-06   0.503333     0.450000  0.028333
2012-11-07   0.510000     0.435000  0.010000

请注意,10/27的滚动均值现在为0.51500(按投票权重计算),而不是52.1667(按天数加权)。
另外,请注意,从版本0.18.0开始,resamplerolling的API已经发生了变化。 pandas 0.18.0中的新功能:rolling pandas 0.18.0中的新功能:resample

3

我发现用户2689410的代码在我尝试以window ='1M'作为业务月份增量时出错,因为这个增量值引发了一个错误:

AttributeError: 'MonthEnd' object has no attribute 'delta'

我添加了一个选项,可以直接传递相对时间差,因此您可以为用户定义的时间段执行类似的操作。
感谢指导,这是我的尝试 - 希望对您有用。
def rolling_mean(data, window, min_periods=1, center=False):
""" Function that computes a rolling mean
Reference:
    https://dev59.com/KmUo5IYBdhLWcg3wxB7T

Parameters
----------
data : DataFrame or Series
       If a DataFrame is passed, the rolling_mean is computed for all columns.
window : int, string, Timedelta or Relativedelta
         int - number of observations used for calculating the statistic,
               as defined by the function pd.rolling_mean()
         string - must be a frequency string, e.g. '90S'. This is
                  internally converted into a DateOffset object, and then
                  Timedelta representing the window size.
         Timedelta / Relativedelta - Can directly pass a timedeltas.
min_periods : int
              Minimum number of observations in window required to have a value.
center : bool
         Point around which to 'center' the slicing.

Returns
-------
Series or DataFrame, if more than one column
"""
def f(x, time_increment):
    """Function to apply that actually computes the rolling mean
    :param x:
    :return:
    """
    if not center:
        # adding a microsecond because when slicing with labels start
        # and endpoint are inclusive
        start_date = x - time_increment + timedelta(0, 0, 1)
        end_date = x
    else:
        start_date = x - time_increment/2 + timedelta(0, 0, 1)
        end_date = x + time_increment/2
    # Select the date index from the
    dslice = col[start_date:end_date]

    if dslice.size < min_periods:
        return np.nan
    else:
        return dslice.mean()

data = DataFrame(data.copy())
dfout = DataFrame()
if isinstance(window, int):
    dfout = pd.rolling_mean(data, window, min_periods=min_periods, center=center)

elif isinstance(window, basestring):
    time_delta = pd.datetools.to_offset(window).delta
    idx = Series(data.index.to_pydatetime(), index=data.index)
    for colname, col in data.iteritems():
        result = idx.apply(lambda x: f(x, time_delta))
        result.name = colname
        dfout = dfout.join(result, how='outer')

elif isinstance(window, (timedelta, relativedelta)):
    time_delta = window
    idx = Series(data.index.to_pydatetime(), index=data.index)
    for colname, col in data.iteritems():
        result = idx.apply(lambda x: f(x, time_delta))
        result.name = colname
        dfout = dfout.join(result, how='outer')

if dfout.columns.size == 1:
    dfout = dfout.ix[:, 0]
return dfout

以下是一个3天时间窗口计算平均值的示例:

from pandas import Series, DataFrame
import pandas as pd
from datetime import datetime, timedelta
import numpy as np
from dateutil.relativedelta import relativedelta

idx = [datetime(2011, 2, 7, 0, 0),
           datetime(2011, 2, 7, 0, 1),
           datetime(2011, 2, 8, 0, 1, 30),
           datetime(2011, 2, 9, 0, 2),
           datetime(2011, 2, 10, 0, 4),
           datetime(2011, 2, 11, 0, 5),
           datetime(2011, 2, 12, 0, 5, 10),
           datetime(2011, 2, 12, 0, 6),
           datetime(2011, 2, 13, 0, 8),
           datetime(2011, 2, 14, 0, 9)]
idx = pd.Index(idx)
vals = np.arange(len(idx)).astype(float)
s = Series(vals, index=idx)
# Now try by passing the 3 days as a relative time delta directly.
rm = rolling_mean(s, window=relativedelta(days=3))
>>> rm
Out[2]: 
2011-02-07 00:00:00    0.0
2011-02-07 00:01:00    0.5
2011-02-08 00:01:30    1.0
2011-02-09 00:02:00    1.5
2011-02-10 00:04:00    3.0
2011-02-11 00:05:00    4.0
2011-02-12 00:05:10    5.0
2011-02-12 00:06:00    5.5
2011-02-13 00:08:00    6.5
2011-02-14 00:09:00    7.5
Name: 0, dtype: float64

3
为了让它简单明了,我使用了一个循环和类似这样的代码来帮助你开始(我的索引是日期时间):
import pandas as pd
import datetime as dt

#populate your dataframe: "df"
#...

df[df.index<(df.index[0]+dt.timedelta(hours=1))] #gives you a slice. you can then take .sum() .mean(), whatever

然后,您可以在该切片上运行函数。您可以看到添加一个迭代器以使窗口的开始不是数据帧索引中的第一个值将滚动窗口(例如,您也可以使用>规则来进行启动)。

请注意,对于超大型数据或非常小的增量,这可能效率较低,因为您的切片可能会变得更加费力(但对我而言,对于数十万行数据和几列数据,在几周内每小时执行一次窗口足够好)。


0

可视化滚动平均值以查看其是否有意义。我不明白为什么在请求滚动平均值时使用了总和。

  df=pd.read_csv('poll.csv',parse_dates=['enddate'],dtype={'favorable':np.float,'unfavorable':np.float,'other':np.float})

  df.set_index('enddate')
  df=df.fillna(0)

 fig, axs = plt.subplots(figsize=(5,10))
 df.plot(x='enddate', ax=axs)
 plt.show()


 df.rolling(window=3,min_periods=3).mean().plot()
 plt.show()
 print("The larger the window coefficient the smoother the line will appear")
 print('The min_periods is the minimum number of observations in the window required to have a value')

 df.rolling(window=6,min_periods=3).mean().plot()
 plt.show()

0

检查你的索引是真正的datetime,而不是str。可能会有所帮助:

data.index = pd.to_datetime(data['Index']).values

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接