Skip to main content
Version: Next

Lite components

caution

Lite API should not be considered stable at the moment.

Common features of Lite components​

Base API of Lite components is here This is a generic API that is used by Lite engine, but can be used to implement own engines, e.g. running in Request-Response way, or embedded in your application. For that reason, this API has a few parameters, that are fixed in StreamingLite:

  • F - the effect monad, in Lite it is Scala Future
  • Input - the type of data (e.g. events processed by the given engine). In Lite, it's ConsumerRecord[Array[Byte], Array[Byte]]
  • Result - the type of data produced by sinks of a given engine. In Lite, it's ProducerRecord[Array[Byte], Array[Byte]] You can see sample implementation of different Lite engine.

The data can be processed in microbatches, in Lite engine in Streaming processing mode the batch contains all the records that are read with single a consumer.poll.

Processing of a single batch ends with a list of results, each of the results may by either Result in case of success, or NuExceptionInfo in case of an error.

Sources and sinks​

Streaming​

In Streaming processing mode sources and sinks are relatively simple, as all sources and sinks are based on Kafka:

  • A source transforms ConsumerRecord to Nussknacker Context.
  • A sink creates ProducerRecord from its parameters.
  • Both may return error.

Request-Response​

In Request-Response processing mode sources and sinks act slightly different. Both create a combination of a HTTP request (source) and a response (sink).

  • A source can handle GET and POST http requests
  • A sink is responsible for providing response payload

Custom transformers​

Generic Lite custom component is designed in continuation-passing style. Some helper traits are provided, you can also look at base components.

Capabilities transformer​

Some transformers can work with arbitrary effect F, the examples are basic components like union or for-each. If custom component depends on the specific effect (e.g. in Streaming you want to invoke a custom service which returns Future[_]), you can use CapabilityTransformer provided by CustomComponentContext to return desired effect. If this component will be used with a different effect, error will be returned by scenario's compiler. See example for the details.