LLM Chat

Description

LLM Chat component allows prompting LLM chat models. It is available once at least one AI integration is configured in the Integrations view (Designer side menu).

Parameters and configuration

Name	Description
LLM integration	LLM integration to use
Chat model name	The chat model to use. The available models depend on the LLM integration selected
User message	Description of the concrete task to be performed.
System message	Optional; reusable rules that define how the model should behave. Sent before the prompt and evaluated for every event.
Output schema	Optional; a JSON Schema describing the desired response. When provided, the returned value becomes a typed record instead of plain text — see Structured output below.
Temperature	Temperature is a hyperparameter that controls the randomness of text generation. Lower values (e.g., 0.2) make the model’s output more focused and deterministic by favoring the most likely tokens, while higher values (e.g., 0.8 or above) produce more diverse and creative responses by flattening the probability distribution.
Max output tokens	Optional; upper limit on the length of the response, in tokens. A response that reaches it is cut off. Leave empty to use the default: 8192 for Anthropic, the provider's own for the others.
Output variable name	Variable name under which node result will be available in a subsequent nodes.

Returned value

By default the component returns the chat model's response as plain text (String).

When an Output schema is provided, the returned value becomes a typed record with raw and structured fields instead — see Structured output below.

Structured output

Provide a JSON Schema in the Output schema parameter to get a typed response instead of plain text. Nussknacker uses the schema in two ways:

• At runtime, the schema is sent to the model as a response-format constraint, steering it to answer in exactly that shape.
• At design time, the schema is converted into a Nussknacker type, so the rest of the scenario knows the fields and their types — with autocomplete and validation — before the scenario runs.

With a schema, the output variable becomes a record with two fields:

Field	Type	Description
raw	String	The model's response as text, exactly as it came back — parseable or not.
structured	the schema's type, or null	The response parsed and typed against the schema, or null when the response is not valid JSON or does not match the schema.

For example, this schema:

{
  "type": "object",
  "properties": {
    "brand":     { "type": "string" },
    "price":     { "type": "number", "minimum": 0 },
    "sentiment": { "type": "string", "enum": ["positive", "neutral", "negative"] }
  },
  "required": ["brand", "price", "sentiment"]
}

types the output (for an output variable named #output) as:

#output.raw                  : String
#output.structured           : Record{brand: String, price: BigDecimal, sentiment: String(negative) | String(neutral) | String(positive)}
#output.structured.price     : BigDecimal
#output.structured.sentiment : String(negative) | String(neutral) | String(positive)

Handling invalid responses

LLMs are probabilistic, so a model may occasionally return something that is not valid JSON or does not match the schema. When that happens, structured is null — a value you can branch on, rather than an error that stops the scenario:

• raw always holds the original text, so the response is never lost.
• Filter on #output.structured != null to separate successful extractions from failures and handle them separately (for example, route failures to a dead-letter sink), while the rest of the flow keeps running.

Additional considerations

• Without an Output schema, the response is plain text; if you expect JSON you can parse it manually using #CONV.toJson() or #CONV.toJsonOrNull(). With a schema, use the typed structured field instead of parsing by hand.
• Chat models may not follow the provided output structure - models can even not return a valid json in some cases.
• You can use string template during prompt creation to use data available in the scenario (e.g., documents retrieved from the vector store)
• The providers offer selected models. To use any other model, including one you run yourself, connect its server with an OpenAI-compatible integration.

Metrics

Every call to the chat model is reported in the scenario's metrics, tagged with nodeId, component, integration, provider and model (the model the node asks for):

Metric	What it counts
llm.tokens	Tokens, by tokenType. The types don't overlap, so they add up to all the tokens a call is billed for, and each has one price: input (input neither read from nor written to a prompt cache), cacheRead and cacheCreate (input read from / written to a prompt cache), output (generated tokens) and reasoning (reasoning tokens, where the provider reports them separately - Anthropic counts them in output). A provider console may show input with cached tokens included: that is input + cacheRead here.
llm.calls	Calls to the model, failed ones included.
llm.errors	Calls that failed, by error (the exception class).
llm.latency	Time of a single call to the model, failed ones included.

The scenario name and the instance are added the same way as for the other scenario metrics.