Skip to main content

Connecting to Apache Kafka® in Confluent® Cloud

In this tutorial​

Nussknacker allows creating decision scenarios on real-time data. As one way to ingest your data, you can connect directly to your Kafka and Schema Registry to read events from and write to your topics.

In this tutorial, we will explain how to connect to the Confluent Cloud from the Nussknacker Cloud and use a managed Apache Kafka as the input and output from Nussknacker.

Prerequisites​

We assume that:

  • You have already created the Confluent Cloud environment and a cluster with the Governance package enabled.
  • You have already created a Nussknacker Cloud account

Setting up Confluent Cloud​

note

You can skip this part if you already have your configuration snippet

  1. Go to Confluent Cloud
  2. Select the proper environment and choose the cluster you want to connect to
  3. Go to Clients
  4. Choose a Java client type
  5. Click Create Kafka cluster API key
    1. You will be offered to download the credentials file. You can save it but it won`t be necessary for the Nussknacker Cloud
  6. Click Create Schema Registry API key
    1. You will be offered to download the credentials file. You can save it but it won`t be necessary for the Nussknacker Cloud
  7. Your API keys should appear in the configuration snippet
  8. Copy the configuration snippet

alt_text

Setting up Nussknacker Cloud​

  1. Go to Nussknacker Cloud
  2. Click Create instance on the left panel
  3. Choose your instance name or use autogenerated value
  4. Choose Existing Kafka cluster
  5. Paste your configuration snippet into the Properties field. It will be automatically converted to a JSON representation.

nussknacker.png

  1. Click submit button
  2. Your Nussknacker instance connected to Confluent Cloud will be created within a couple of minutes

Creating a topic and a schema​

  1. In the Confluent Cloud cluster overview go to Topics
  2. Click the Add topic button
  3. Type your topic name
    1. Set the number of partitions
  4. Click Create with defaults
  5. Your topic is created, now let`s define a data contract
  6. Click the Create schema button
  7. Choose your schema type, for now, Avro and JSON Schema are supported in Nussknacker Cloud
  8. Enter your schema
  9. Click the Create button

confluentTopic.png

Creating a scenario​

  1. Go to Nussknacker Cloud management
  2. Open your running Nussknacker instance with the Open instance button
  3. Create your first scenario with the Create scenario button
    1. Choose the preferred scenario name
  4. Drag and drop the Kafka source from the left panel
  5. You should see your topic on the Topic drop-down list
  6. Add custom logic like filtering, sending to some other Kafka sink
  7. Press deploy button on the left
  8. Wait a while until scenario will be deployed

nussknackerScenario.png

Sending messages to and receiving messages from Nussknacker scenario​

You can read about messaging with Nussknacker in the documentation.