Window Deduplication

Description

The Window Deduplication node (window-deduplication) splits the stream into fixed (tumbling) windows of event time and emits, for every window and every key, the record with the earliest record_time. The other records of that key in that window are dropped.

It differs from Deduplication: that node keeps one entry per key with an inactivity timeout that resets on every event, while this node works in fixed windows aligned to the clock (a one-hour window covers 10:00-11:00, 11:00-12:00, ...), and emits the kept record when the window closes, i.e. when the watermark passes its end.

Under the hood the node runs a Flink SQL window deduplication query over the incoming records.

• The input of the node is one record variable available before the node (by default #input) - all its fields, or a selected subset, become the fields of the output, plus record_time, window_start and window_end.
• The node clears all upstream variables - after the node only the output variable is available.

The node is available only in the Streaming processing mode, for unbounded streams.

Parameters and configuration

Name	Description
Input	The record variable and the fields to keep in the output.
Partition by	The fields making up the key; records with the same values of all of them are deduplicated together. With no fields, one record per window is kept.
Window size	The length of the window, e.g. 1 HOUR.
Output variable name	The name of the variable holding the kept record.

Additional considerations

• The node is stateful: open windows live in Flink state and are restored from a savepoint when the scenario is redeployed. Changing the key or the window size may make the state incompatible.
• Records are emitted when the window closes, not when they arrive. Late records (behind the watermark) are dropped.
• The node can't be mocked in scenario tests - it has to be fed real records.