Lite components

caution

Lite API should not be considered stable at the moment.

Common features of Lite components

Base API of Lite components is here This is a generic API that is used by Lite engine, but can be used to implement own engines, e.g. running in Request-Response way, or embedded in your application. For that reason, this API has a few parameters, that are fixed in StreamingLite:

• F - the effect monad, in Lite it is Scala Future
• Input - the type of data (e.g. events processed by the given engine). In Lite, it's ConsumerRecord[Array[Byte], Array[Byte]]
• Result - the type of data produced by sinks of a given engine. In Lite, it's ProducerRecord[Array[Byte], Array[Byte]] You can see sample implementation of different Lite engine.

The data can be processed in microbatches, in Lite engine in Streaming processing mode the batch contains all the records that are read with single a consumer.poll.

Processing of a single batch ends with a list of results, each of the results may by either Result in case of success, or NuExceptionInfo in case of an error.

Sources and sinks

Streaming

In Streaming processing mode sources and sinks are relatively simple, as all sources and sinks are based on Kafka:

• A source transforms ConsumerRecord to Nussknacker Context.
• A sink creates ProducerRecord from its parameters.
• Both may return error.

Request-Response

In Request-Response processing mode sources and sinks act slightly different. Both create a combination of a HTTP request (source) and a response (sink).

• A source can handle GET and POST http requests
• A sink is responsible for providing response payload

Custom transformers

Generic Lite custom component is designed in continuation-passing style. Some helper traits are provided, you can also look at base components.

Multiple outputs

A custom transformer declares its outputs with CustomStreamTransformer.outputs, the same way on every engine, see Flink components. Outputs that depend on the node's parameters work as on Flink too, see dynamic outputs.

On Lite, a component with only the main output implements LiteCustomComponent and gets that output's continuation. One with additional outputs implements LiteMultiOutputCustomComponent instead, whose createTransformation gets LiteOutputContinuations: continuations(output) is that output's continuation, and there is one for every declared output, connected or not. What is sent to an unconnected additional output is dropped, what is sent to an unconnected main output is counted as a dead end, as with a filter, and asking for an undeclared output throws IllegalArgumentException. Declaring additional outputs and returning a LiteCustomComponent fails at scenario compilation, as soon as a scenario connects one of them.

A context may be sent to any number of outputs, including none. Set the node's output variable on every output, to null where there is no value: the compiler types all outputs with the same context, so a missing variable fails only at runtime.

A component that routes contexts one by one can extend SingleElementRoutingComponent and implement only createRouting, which returns the outputs a context goes to, each with the context to send there. Results keep the order of the batch: consecutive contexts routed to the same output go to it together.

Capabilities transformer

Some transformers can work with arbitrary effect F, the examples are basic components like union or for-each. If custom component depends on the specific effect (e.g. in Streaming you want to invoke a custom service which returns Future[_]), you can use CapabilityTransformer provided by CustomComponentContext to return desired effect. If this component will be used with a different effect, error will be returned by scenario's compiler. See example for the details.