Topic (Source)
Description
Topic Source allows to read events from a Kafka topic managed by Nussknacker - one of the topics listed on the Topics page ("internal topics"). You can use internal Kafka topics to:
- route data between scenarios
- send events to a topic via public HTTPS or WebSocket endpoint
- receive results via HTTP callback, WebSocket subscription, or by pulling them from the HTTP API
Create and manage topics, with their schemas, on the Topics page (the Topics item of the Designer side menu). While editing the node, you can also click Create New Topic next to the Topic name parameter: it creates a topic with a random name and no schema, and puts its name into the parameter.
Parameters and configuration
| Name | Description |
|---|---|
| Topic name | Kafka topic to read from |
| Schema version | Version of the topic's schema (from the schema registry) used to deserialize events. Latest version resolves to the newest version every time the scenario is validated and deployed, so a new version is picked up on the next deployment. Shown only for topics with a schema; for a topic without a schema, the Content type parameter (JSON or PLAIN) is shown instead. |
Advanced parameters
| Name | Description |
|---|---|
| Event time | Expression which evaluates to the time when the event was created. For Kafka sources, creation timestamp is available in the Kafka event and can be accessed in SpEL as #inputMeta.timestamp. Check here |
| Max out-of-orderness | The maximum amount of time an element is allowed to be late before being ignored when computing the result for time-based stream transformations: aggregates in time windows and joins. To read more about this mechanism see Flink documentation |
| Idleness | The time period after which partition is marked as idle if no events are received from it. To read more about this mechanism see Flink documentation. |
Additional considerations
Event time
The Kafka event timestamp is the default value suggested by Designer for the Event time field. If you do not use this default, read here to learn more about Instant data type. If provided as a numeric value rather than Instant, it must be in epoch milliseconds (not seconds); a Long is the most natural data type for representing this UTC millisecond value.
Getting event time wrong can lead to subtle errors in time-based logic that are difficult to diagnose - if such logic behaves unexpectedly, this should be the first thing to verify.