Table (Source)

Description

Table source component is automatically generated based on the Flink Table API integration definition. The name of the component will be Integration-name Table. You can only ingest whole database tables or files using this component; check the Table API integration for more information.

Parameters and configuration

Name	Description
Table	Table from which records will be read and processed

Advanced parameters

Name	Description
Event time	Expression which evaluates to the time when the event was created, expressed as an Instant (UTC) datatype, not processing or ingestion time. Read 'Additional considerations' section for more information.
Max out-of-orderness	The maximum amount of time an event is allowed to be late before being ignored when computing the result for time-based stream transformations: aggregates in time windows and joins. Must not be negative. To read more about this mechanism see Flink documentation.
Idleness	The time period after which partition is marked as idle if no events are received from it. Must be a positive duration. To read more about this mechanism see Flink documentation.

Change Data Capture (CDC) tables

If the table definition produces a changelog (for example the postgres-cdc connector, or kafka with the debezium-json format), every insert, update and delete on the table enters the scenario as a separate record.

For such tables the source exposes an additional #cdcOperation variable:

Variable	Description
#input	The changed row: the new version for INSERT and UPDATE_AFTER, the previous one for UPDATE_BEFORE and DELETE.
#cdcOperation	The change operation: one of INSERT, UPDATE_BEFORE, UPDATE_AFTER, DELETE.

An update produces two records: UPDATE_BEFORE with the previous version of the row and UPDATE_AFTER with the new one.

For example, for a customers table with columns id and name:

Change in the source table	Emitted records
INSERT INTO customers VALUES (1, 'Alice');	{ "cdcOperation": "INSERT", "input": { "id": 1, "name": "Alice" } }
UPDATE customers SET name = 'Bob' WHERE id = 1;	{ "cdcOperation": "UPDATE_BEFORE", "input": { "id": 1, "name": "Alice" } } { "cdcOperation": "UPDATE_AFTER", "input": { "id": 1, "name": "Bob" } }
DELETE FROM customers WHERE id = 1;	{ "cdcOperation": "DELETE", "input": { "id": 1, "name": "Bob" } }

Additional considerations

Read here to learn more about Instant data type. If provided as a numeric value rather than Instant, it must be in epoch milliseconds (not seconds); a Long is the most natural data type for representing this UTC millisecond value.

caution

Getting event time wrong can lead to subtle errors in time-based logic that are difficult to diagnose - if such logic behaves unexpectedly, this should be the first thing to verify.