Skip to main content
Version: 1.5

Streaming-Lite components

caution

Streaming-Lite API should not be considered stable at the moment.

Common features of Lite components

Base API of Lite components is here This is a generic API that is used by Streaming-Lite engine, but can be used to implement own engines, e.g. running in Request-Response way, or embedded in your application. For that reason, this API has a few parameters, that are fixed in StreamingLite:

  • F - the effect monad, in Streaming-Lite it is Scala Future
  • Input - the type of data (e.g. events processed by the given engine). In Streaming-Lite it's ConsumerRecord[Array[Byte], Array[Byte]]
  • Result - the type of data produced by sinks of a given engine. In Streaming-Lite it's ProducerRecord[Array[Byte], Array[Byte]] You can see sample implementation of different Lite engine.

The data can be processed in microbatches, in Streaming-Lite the batch contains all the records that are read with single a consumer.poll.

Processing of a single batch ends with a list of results, each of the results may by either Result in case of success, or NuExceptionInfo in case of an error.

Sources and sinks

In Streaming-Lite sources and sinks are relatively simple, as all sources and sinks are based on Kafka:

  • A source transforms ConsumerRecord to Nussknacker Context.
  • A sink creates ProducerRecord from its parameters.
  • Both may return error.

Lite custom transformers

Generic Lite custom component is designed in continuation-passing style. Some helper traits are provided, you can also look at base components.

Capabilities transformer

Some transformers can work with arbitrary effect F, the examples are basic components like union or for-each. If custom component depends on the specific effect (e.g. in Streaming-Lite you want to invoke a custom service which returns Future[_]), you can use CapabilityTransformer provided by CustomComponentContext to return desired effect. If this component will be used with a different effect, error will be returned by scenario's compiler. See example for the details.