Skip to main content
Version: 1.5

Data sources and sinks

Kafka topics as input and output to Nussknacker scenarios​

Prerequisites​

To fully understand how Nussknacker works with Kafka topics, it's best to read the following first:

If you want to use Streaming-Flink processing engine, this is also recommended:

Concepts​

Kafka topics are native streaming data input to Nussknacker and the native output where results of Nussknacker scenarios processing are placed. In Nussknacker terminology input topics are called sources, output topics are called sinks. This section discusses Kafka based source and sink components.

It is not uncommon that programs that write events to Kafka topics (“producers”) and programs that read events from Kafka topics (“consumers”) are implemented by different people. If consumers are to understand what producers put into the topic, they need to agree between themselves the data schema - the format and data types of the data transported over Kafka topics. This “contract” is kept in what is called a Schema Registry. Nussnacker is able to read it and use to determine what data is in the event and help with field names and data type related validation of SpEL expressions.

Kafka source​

Nussknacker will connect to Schema Registry to get the list of available schemas and will pre-populate the Topic name drop down of the kafka source component with the list of available topic names.

Nussknacker knows what data to expect in the Kafka event so it will hint field names and will perform additional validations on SpEL expressions involving data from the selected topic. You can select the schema version which should be used for that.

Kafka source

Kafka sink​

The kafka sink configuration form will show a list of fields defined in Schema Registry for the given topic. The result of the expression entered in the Key field will be used as a partitioning key when sending the record to Kafka.

Kafka sink

If you prefer to define the whole output event as one expression containing value to be written to the Kafka topic, instead of filling separete autogenerated fields, you can do that by switching Raw editor to true.

When using the Raw editor, you can choose the validation mode:

  • strict - requires providing all fields, including optional fields, without redundant fields;
  • lax - requires providing only required fields, you can skip optional fields and add redundant fields.