我在理解水印和允许延迟的概念方面遇到了困难。
以下是摘自[邮件档案|https://www.mail-archive.com/user@flink.apache.org/msg08758.html]的内容,其中讨论了水印,但我仍有一些问题。以下是引用的示例:
Assume you have a
BoundedOutOfOrdernessTimestampExtractor
with a 2 minute bound and a 10 minute tumbling window that starts at 12:00 and ends at 12:10:If you have the following stream sequence:
12:01, A 12:04, B WM, 12:02 // 12:04 - 2 minutes 12:02, C 12:08, D 12:14, E WM, 12:12 12:16, F WM, 12:14 // 12:16 - 2 minutes 12:09, G
no allowed lateness
The window operator forwards the logical time to 12:12 when it receives
<WM, 12:12>
and evaluates the window which contains[A, B, C, D]
at this time and finally purges its state.<12:09, G>
is later ignored.allowed lateness of 3 minutes
The window operator evaluates the window when
<WM, 12:12>
is received, but its state is not purged yet. The state is purged when<WM, 12:14>
is received (window fire time 12:10 + 3mins allowed lateness).<12:09, G>
is again ignored.allowed lateness of 5 minutes
The window operator evaluates the window when
<WM, 12:12>
is received, but its state is not purged yet. When<12:09, G>
is received, the window is again evaluated but this time with[A, B, C, D, G]
and an update is sent out. The state is purged when a watermark of >= 12:15 is received.
据我所了解:
- 水印(Watermark)用于指示事件时间戳小于水印的任何元素将被丢弃。因此,水印 12:02 意味着 Flink 已经看到了所有它需要看到直到事件时间 12:02 的内容。任何具有小于此水印的事件时间戳(例如12:01)的元素都将被丢弃。
- 允许延迟(Allowed lateness)的概念仅适用于标记窗口结束的最后一个水印之后。
基于以上理解,我的问题如下:
- Flink 已经说过“我已经看到了时间戳在12:02之前的所有内容”,那么消息“12:02,C”是如何处理的?
- 我已经调整了流序列并在流序列中加入了另一条记录“12:01,CCC”,如下所示:
If you have the following stream sequence:
12:01, A 12:04, B WM, 12:02 // 12:04 - 2 minutes 12:02, C 12:01, CCC // Inserted by Sheel 12:08, D 12:14, E WM, 12:12 12:16, F WM, 12:14 // 12:16 - 2 minutes 12:09, G
这仍然在12:00-12:10窗口内,但水印WM在后面,显示时间为12:02。假设允许的迟到时间是5分钟。如果将允许的迟到时间考虑在内,这条记录是否会被“某种方式”接受,还是会被丢弃,因为水印12:02已经过去了?