我有一串带有键的数据,需要计算不同时间段(1分钟、5分钟、1天、1周)内的滚动计数。
是否可以在单个应用程序中计算所有四个窗口计数?
我有一串带有键的数据,需要计算不同时间段(1分钟、5分钟、1天、1周)内的滚动计数。
是否可以在单个应用程序中计算所有四个窗口计数?
是的,这是可能的。
如果您正在使用事件时间,您可以通过逐步增加时间间隔级联窗口来实现。所以您可以这样做:
DataStream<String> data = ...
// append a Long 1 to each record to count it.
DataStream<Tuple2<String, Long>> withOnes = data.map(new AppendOne);
DataStream<Tuple2<String, Long>> 1minCnts = withOnes
// key by String field
.keyBy(0)
// define time window
.timeWindow(Time.of(1, MINUTES))
// sum ones of the Long field
// in practice you want to use an incrementally aggregating ReduceFunction and
// a WindowFunction to extract the start/end timestamp of the window
.sum(1);
// emit 1-min counts to wherever you need it
1minCnts.addSink(new YourSink());
// compute 5-min counts based on 1-min counts
DataStream<Tuple2<String, Long>> 5minCnts = 1minCnts
// key by String field
.keyBy(0)
// define time window of 5 minutes
.timeWindow(Time.of(5, MINUTES))
// sum the 1-minute counts in the Long field
.sum(1);
// emit 5-min counts to wherever you need it
5minCnts.addSink(new YourSink());
// continue with 1 day window and 1 week window