Skip to main content
Version: Current

Migration guide

To see the biggest differences please consult the changelog.

In version 2.3 (Not released yet)​

HTTP Endpoint and WebSocket source metadata​

!1204 removes Kafka record metadata from #inputMeta for HTTP Endpoint, WebSocket Client and WebSocket Server sources. The removed fields are headers, key, topic, partition, offset, timestamp, timestampType and leaderEpoch. Only #inputMeta.httpHeaders remains, containing forwarded HTTP headers with the _http_ prefix removed.

  • Replace HTTP header expressions such as #inputMeta.headers['_http_content-type'] with #inputMeta.httpHeaders['content-type']. Other Kafka record headers are no longer exposed.
  • Migration 504 clears the Event time parameter automatically when it contains exactly the former default, #inputMeta.timestamp. Custom Event time expressions remain unchanged. The new default is empty: the runtime uses the upstream record timestamp, falling back to the current time if none is available.

Ordinary Kafka sources continue to expose Kafka metadata through #inputMeta.

The Configurator requires Integration actions​

Reading it takes Integration.View, saving Integration.Create or Integration.Edit, deleting Integration.Delete; secrets are masked as ****** without Integration.Edit. The users/rules configuration cannot grant these actions, so there the Configurator is admin-only; grant them under RBAC.

The Topics tab requires Topic actions​

Listing and reading topics takes Topic.View, previewing their messages Topic.ViewMessages, creating one Topic.Create, changing retention or the schema Topic.Edit, deleting one Topic.Delete. Grant Topic.ViewMessages only to those who may see production records. The users/rules configuration cannot grant these actions, so there the Topics tab is admin-only; grant them under RBAC.

A webhook source's actions take: generating its endpoint Topic.Create (admin-only under users/rules), testing a WebSocket forwarder Scenario.Edit (Write), sending a data sample Scenario.Test (Deploy) in the scenario's category. The sample goes to the topic of the source as saved, so save the scenario before sending one.

The MCP tab requires MCP actions​

Listing and reading MCP definitions, the scenarios available as tools, a tools preview and a server's machine access take MCP.View; creating a definition MCP.Create, changing one MCP.Edit, deleting one MCP.Delete. Generating and revoking a server's tokens takes MCP.Edit and, as before, RBAC.Manage. The users/rules configuration cannot grant these actions, so there the MCP tab is admin-only; grant them under RBAC.

REST API changes​

  • !1469 categories of GET /api/user is an object instead of a list: the visible categories grouped by the processing modes of the scenario types configured in them, e.g. {"Unbounded-Stream": ["Category1", "Category2"]}. For the former list, take the distinct categories of all its values.

  • !970 Custom nodes with dynamic outputs, added fields:

    • outputs (list of output names, main output first, or null) in the node validation response and in each nodeResults entry of the scenario validation response,
    • dynamicOutputs (boolean) in the edgesForNodes entries of the component definitions.

  • !1264 The create-kv-namespace and delete-kv-namespace actions of POST /api/custom/nodes/{scenarioName}/actions and GET /api/custom/nodes/actions/kv-namespace/metadata are removed. Use /api/custom/kv-store-management/stores/{storeId}/namespaces instead (list, get, create, delete), with storeId from GET /api/custom/kv-store-management/stores; it needs an admin account. To keep managing namespaces, put the kv-store-management Designer extension (extensions/kv-store-management/*) on the Designer CLASSPATH and add the Key-Value Store tab to tabs: {id: "key-value-store", title: "Key-Value Store", type: "Remote", url: "nuEnterpriseKvStoreManagement/tab@/static/kv-store-management/remoteEntry.js", adminOnly: true}. examples/installation has both. The new API takes the value schema only as jsonSchema; it does not use typingResult, category or processingType.

  • !1403 allowedProcessingModes of the elements of GET /api/components lists the processing modes of the scenario types the component is available in, intersected with the modes it supports, instead of every mode it supports. A component available only in streaming scenario types returns ["Unbounded-Stream"] where it used to return all three modes.

Code API changes​

  • !970 Custom nodes with dynamic outputs:

    • DynamicComponent.FinalResults gained a fourth field, outputs, so a positional pattern match needs an extra _. The field defaults to None, and component jars that create FinalResults or call FinalResults.forValidation keep working without recompiling; only ones that pattern match on FinalResults or call its copy need recompiling.
    • CustomComponentSpecificData gained a fourth field, hasDynamicOutputs, and ValidationPerformed a fifth, outputs - positional pattern matches need an extra _.

  • !970 AdditionalInfoProvider.nodeAdditionalInfo gets the scenario's MetaData along with the node: nodeAdditionalInfo(config)(metaData, node). Add the parameter to your providers.

  • !1423 DeploymentManagerDependencies throws IllegalArgumentException when its executionContext ignores scala.concurrent.blocking.

Configuration changes​

  • !1447 The flinkSqlFunctions component provider (Flink SQL UDF / PTF nodes) is in components/flink-table/flinkTableFunctions.jar instead of components/flink-table/flinkTable.jar. A model classPath that lists the components/flink-table directory needs no change. One that lists flinkTable.jar and configures components.flinkSqlFunctions has to add components/flink-table/flinkTableFunctions.jar - without it the model fails to load. The same holds for deploymentConfig.modelArtifact.providedJarFiles, which must stay in sync with modelConfig.classPath.

Documentation and naming changes​

  • !1362 Installations run by the customer are called self-managed, not on-premise:
    • the documentation section moved from docs.nussknacker.io/onPremise/... to docs.nussknacker.io/selfManaged/...; update bookmarks and links in your own documentation
    • the assistant's embeddings file for these installations in the designer-assistant-embeddings image is embeddings-self-managed.store (was embeddings-on-premise.store); change assistantHttpService.assistant.embeddingsResourceUri if it points at the old name
    • nu-cli's get_instance_info MCP tool reports installation: "selfManaged" (was "onPremise"), and the assistant's chat request carries the same value

In version 2.3-20261014​

In version 2.3-20261013​

Configuration changes​

Configuration changes​

  • The default toolbar's Metrics button is { type: "scenario-metrics", disabled: { fragment: true } } instead of a custom-link to /metrics/$processName. A processToolbarConfig that overrides the metrics-panel with the old custom-link keeps working - /metrics/$processName now shows the scenario metrics screen - but always navigates away from the editor; replace it with scenario-metrics to honour scenarioMetrics.openAsOverlay.

  • The default processToolbarConfig no longer has the test-counts and test-hide buttons in the metrics-panel of bottomCenter. To keep them, override that panel in your configuration:

    processToolbarConfig.defaultConfig.bottomCenter: [
      { id: "view-panel", type: "buttons-panel", title: "view", buttons: [
        { type: "view-zoom-in" }, { type: "view-zoom-out" }, { type: "view-reset" }, { type: "edit-layout" }
      ] }
      { id: "metrics-panel", type: "buttons-panel", title: "Metrics", buttons: [
        { type: "custom-link", name: "metrics", icon: "/assets/buttons/metrics.svg", url: "/metrics/$processName", disabled: { fragment: true } }
        { type: "test-counts" }
        { type: "test-from-file", disabled: { archived: true }, hidden: { fragment: true } }
        { type: "test-generate", disabled: { archived: true }, hidden: { fragment: true } }
        { type: "test-hide" }
      ] }
    ]

REST API changes​

  • !1264 The create-kv-namespace and delete-kv-namespace actions of POST /api/custom/nodes/{scenarioName}/actions and GET /api/custom/nodes/actions/kv-namespace/metadata are removed. Use /api/custom/kv-store-management/stores/{storeId}/namespaces instead (list, get, create, delete), with storeId from GET /api/custom/kv-store-management/stores; it needs an admin account. To keep managing namespaces, put the kv-store-management Designer extension (extensions/kv-store-management/*) on the Designer CLASSPATH and add the Key-Value Store tab to tabs: {id: "key-value-store", title: "Key-Value Store", type: "Remote", url: "nuEnterpriseKvStoreManagement/tab@/static/kv-store-management/remoteEntry.js", adminOnly: true}. examples/installation has both. The new API takes the value schema only as jsonSchema; it does not use typingResult, category or processingType.

Configuration changes​

  • !1326 The default engine setup names changed: Lite Embedded is Embedded, Lite K8s is Kubernetes, and a Flink scenario type deployed to its MiniCluster (useMiniClusterForDeployment: true) is Flink (embedded). A scenario type naming its own engineSetupName keeps it. The names are shown in the Designer and in GET /api/scenarioParametersCombinations; a script, a nu-cli call (--engine) or an assistant prompt that passes the old name has to pass the new one.
  • !1453 Both scenario types of the installation bundle are in the Default category: the streaming one is no longer in Streaming, nor the request-response one in RequestResponse. A scenario keeps the category it was created in, and the Designer lists only the categories of its configuration, so an installation upgraded with its database moves its scenarios: UPDATE processes SET category = 'Default' WHERE category IN ('Streaming', 'RequestResponse'); (before the Designer starts, or restart it afterwards). RBAC roles and the users file granting Streaming or RequestResponse are to grant Default. A scenario type of your own added to the bundle's configuration in one of the old categories is moved to Default too, or the Designer keeps listing that category.

Other changes​

  • !1326 POST /api/processes given a processing mode without a category or an engine creates the scenario in the mode's default scenario type (the one named with deploymentConfig.defaultForCreation: true, else the one whose engine runs inside the Designer, else the first by name) instead of answering 400 when more than one matches. GET /api/scenarioParametersCombinations carries isDefault in every combination.

Parameter editor metadata​

In version 2.3-20261014​

In version 2.3-20261013​

Parameter editor configuration​

Parameter and UIParameter have a new optional uiMetadata JSON field. Recompile custom components against the updated components API. Existing JSON clients can omit this field. The new Decision Model component uses uiMetadata.capabilities on its Questions parameter so the editor can validate against the selected Decision AI integration's limits.

API tokens​

  • AuthenticatedUser and RealLoggedUser carry two more fields, identityKind (User or ServiceAccount) and authentication (Interactive or ApiToken(tokenId)), both defaulted, and AuthManager takes an optional ApiTokenAuthenticator. An authentication or authorization extension building these values needs no change.
  • Secured extension endpoints (CustomHttpServiceServerEndpointDefinition) answer a request authenticated with an API token with 403 unless they override acceptsApiTokens (default false); a PekkoCustomHttpServiceProvider does the same for its whole route. Opt in only where nothing behind it reads or sets a credential, since a token outlives a sign-in and must not be able to set up access surviving its revocation.
  • ScenarioUser (extensions-api) has a defaulted apiTokenId: Option[String], set when the activity was made with an API token. DbScenarioActivityRepository keeps it in the activity's additional_properties under apiTokenId.

Configuration changes​

  • A user none of whose roles has a rule in rules is now refused with a 403, under every authentication method. Add a rule for the roles such users should sign in with.

Code API changes​

  • AuthenticationConfiguration.rejectUsersWithoutRoles is gone. An AuthenticationProvider of your own drops its override.
  • ActionCatalog.validated(declaredActions) takes the configuration first: ActionCatalog.validated(config, declaredActions).

RBAC​

The designer/extensions/rbac extension keeps accounts, roles, groups and role bindings in PostgreSQL, under whichever authentication method is configured, in place of the users/rules lists of the authentication configuration. How to switch it on and its settings: Access control (RBAC) extension.

  • Requires PostgreSQL; with any other database the Designer does not start.
  • On first start the users/rules configuration is migrated into the database and not read afterwards; the designer warns on every startup while it is still there. A user with a password becomes a local account, every role a role plus a group matching its name, and an entry with a username and no password a machine client group recognising its subject. An entry with neither is not migrated.
  • A rule marked isAdmin becomes a group granting the built-in System Admin role to everyone arriving with that value. Review it.
  • The way back in for an administrator is rbac.bootstrapAdmin.enabled: true with rbac.bootstrapAdmin.login (required then) naming an account that does not exist: the start creates it holding System Admin, with rbac.bootstrapAdmin.password, or without one - only when no principal, local user or group, holds System Admin - with a generated password written to rbac.bootstrapAdmin.credentialFile (default rbac-bootstrap-admin-credential). An existing account is never changed. By default the bootstrap is off, and a start where System Admin was never granted to any principal fails; once it was, grants that expired or holders that were deactivated only log an error at startup. The last System Admin grant cannot be revoked.
  • The audit log needs RBAC.ViewAuditLog, which RBAC.Manage does not include.
  • The panel creates no local users and no groups until rbac.principals.localUserCreationEnabled and rbac.principals.groupCreationEnabled (both default false) are switched on; the latter covers editing a group too. What the users/rules migration creates does not depend on them. With Basic Auth, set rbac.principals.localUserCreationEnabled: true, or nobody new can be given an account.
  • rbac.principals.groupsEnabled and rbac.principals.machineClientsEnabled (both default true) switch off groups and machine clients altogether: existing ones of a kind switched off admit and grant nobody, including those the migration created.
  • rbac.grants.requireExpiry: true refuses a role grant without an expiry; rbac.grants.defaultExpiry (default 90d) is what the panel proposes.
  • Logins are case-insensitive. A users file with two accounts differing only by case migrates the first of them.
  • A local account's identity is a generated id, not its login. Scenario activities and actions record that id from now on, while earlier entries keep the login, and an impersonation names its target by that id. The name shown as a scenario's author is the entry's username or, without one, its login in lower case. A scenario migration to another environment that impersonates the migrating user there sends this id, which means nothing to that environment's own database, so leave impersonation off for remote environments.
  • Every instance re-reads roles, bindings and local credentials every rbac.refreshInterval (default 10s), so a change made on another instance, a deactivated account or a reset password can take that long to apply.

The Integrations and MCP menu entries need Integration.View and MCP.View​

The users/rules configuration cannot grant these actions, so outside RBAC only admins see the two entries. The endpoints behind them are not affected. Make the users who need them admins, or grant the actions under RBAC.

Scenario.Demo and Core.Impersonate are off by default​

Under every authentication method. Set authorization.scenarioDemoActionEnabled: true and authorization.impersonateActionEnabled: true where they are used. Off means off under the users/rules configuration too: neither an admin nor a role with the Demo permission or with globalPermissions: ["Impersonate"] has the action. A scenario migration to a remote environment that impersonates the migrating user there needs impersonation on in that environment.

In version 2.3-20261014​

In version 2.3-20261013​

Resource-scoped actions​

  • ScopeKind has a third value, Resource: an action granted per resource. ScenarioAction.Invoke and McpAction.Use are the first; the users file cannot name resources, so under file-based authorization they are admin-only (LegacyPermissionMapping maps Invoke to no permission). AccessScope gains resource: Option[ResourceId]; check on a Resource-kind action is called with AccessScope.onResource.
  • AuthorizationRules has a new defaulted method, resourceGrants, and UserGrants / UserGrantsSummary a new defaulted field of the same name; /api/user reports it as resourceGrants. An implementation that grants Resource-kind actions must override it, or the summary misreports what check allows.
  • RBAC: rbac_role_actions gains a scope column (migration V1_004, this module's own Flyway); a role's actionScopes are read and written through the roles API (RoleDto.actionScopes, UpsertRoleRequestDto.actionScopes), and GET /api/custom/rbac/resources/{resource} lists what a role can be granted on. Extensions owning a resource register a GrantableResourceListingFactory (custom-http-service-api) through META-INF/services to appear there.
  • MCP: definitions get a stable id (McpDefinitionDto.id, read-only). The module now versions its schema with its own Flyway instance (history table mcp_schema_history, migrations under db/migration/mcp): the baseline leaves the tables Slick created alone, and V1_002 adds the id column with a UUID for every existing row.

Configuration changes​

  • !1267 Historical Monitoring requires monitoring.metrics to point to the installation's existing InfluxDB database and environment. For Flink collected through Prometheus/Telegraf (metric_version = 1), set monitoring.metrics.flinkReporter = prometheus and provision Grafana overview panels generated with that profile. Native Influx reporters keep the default influxdb profile. Standalone Docker and distribution launches must explicitly set MONITORING_INFLUX_URL to an endpoint ending in /query; INFLUXDB_URL alone does not enable the overview. The shipped Compose bundle supplies the endpoint in HOCON and does not need this variable. Telegraf metric_version = 2 is not supported by the prometheus profile. Update the installation configuration and Grafana dashboards together with Designer; replacing the image alone does not add the connection configuration or dashboards. countsSettings and stored measurements are unchanged.

  • Docker images (nussknacker, nussknacker-lite-runtime-app, designer-assistant-embeddings) moved from docker-gitlab.touk.pl/nussknacker-cloud/dummy-for-docker-registry/ to registry.nussknacker.io: released versions are in the nussknacker project, snapshots in nussknacker-snapshot. Both are private, so log in with docker login registry.nussknacker.io and give Kubernetes an image pull secret for that registry. The default dockerImageName of the lite-k8s deployment manager follows: it is registry.nussknacker.io/nussknacker/nussknacker-lite-runtime-app for a release and registry.nussknacker.io/nussknacker-snapshot/nussknacker-lite-runtime-app for a snapshot.

  • To enable Key-Value Store namespace management in an existing Designer installation, include extensions/kv-store-management/* in the Designer's CLASSPATH and add this entry to tabs:

    {
      id: "key-value-store"
      url: "nuEnterpriseKvStoreManagement/tab@/static/kv-store-management/remoteEntry.js"
      title: "Key-Value Store"
      type: "Remote"
      adminOnly: true
    }

    The extension is shipped in the Designer image. Its management API is /api/custom/kv-store-management/stores; all operations require administrator access. The store listing includes storeId and processingTypes alongside componentId, label, and backendType. Clients must use the returned storeId in /stores/{storeId}/namespaces, rather than assuming that a component ID identifies one backend across all scenario types. Identical backend configurations are grouped; different configurations with the same component ID remain separate. Scenario nodes continue to use componentId. Existing namespaces and entries require no migration.

  • !1240 scenarioStateVerification is no longer a setting of the flink-k8s-operator deployment manager. It never had an effect there - the state is verified before a redeploy only by flinkStreaming - and a configuration still setting it keeps loading; the key is ignored and can be removed.

  • !1239 With podForJobJars, the flink-k8s-operator deployment manager now writes the job JAR to <jobJarStorageMountPoint>/jars/ in that pod, so the pod has to mount the job JAR storage at jobJarStorageMountPoint, the path the Flink pods mount it at.

Other changes​

  • !1238 The flink-k8s-operator deployment manager had the checkpoint and savepoint directories swapped: checkpoints went to <jobStateStorageMountPoint>/savepoints and savepoints to <jobStateStorageMountPoint>/checkpoints. A scenario deployed after the upgrade writes each to the directory named after it. Checkpoints and savepoints taken before it stay where they were - Flink keeps their full paths, so a job still restores from them; only a person or a script looking for them by directory has to look in the other one.

Configuration changes​

  • !1267 Historical Monitoring requires monitoring.metrics to point to the installation's existing InfluxDB database and environment. For Flink collected through Prometheus/Telegraf (metric_version = 1), set monitoring.metrics.flinkReporter = prometheus and provision Grafana overview panels generated with that profile. Native Influx reporters keep the default influxdb profile. Telegraf metric_version = 2 is not supported by the prometheus profile. Update the installation configuration and Grafana dashboards together with Designer; replacing the image alone does not add the connection configuration or dashboards. countsSettings and stored measurements are unchanged.

  • !1240 scenarioStateVerification is no longer a setting of the flink-k8s-operator deployment manager. It never had an effect there - the state is verified before a redeploy only by flinkStreaming - and a configuration still setting it keeps loading; the key is ignored and can be removed.

  • !1239 With podForJobJars, the flink-k8s-operator deployment manager now writes the job JAR to <jobJarStorageMountPoint>/jars/ in that pod, so the pod has to mount the job JAR storage at jobJarStorageMountPoint, the path the Flink pods mount it at.

  • !1227 The Script Processor reads its limits (runtimeTimeout, validationTimeout, runtimeStatementLimit, validationStatementLimit, resultLimits.*) right from its entry, e.g. components.scriptProcessor.runtimeTimeout. Since it got a component provider of its own they had to be nested one level deeper, as components.scriptProcessor.scriptProcessor.runtimeTimeout; that is still read, and a field set in both places is taken from the entry. Settings left under components.base.scriptProcessor.*, where the component lived at first, have not been read since the move - set them in the component's entry.

REST API changes​

  • !1314 ScenarioWithDetails, returned by /api/processes and /api/processes/{scenarioName}, no longer has lastAction, lastStateAction and lastDeployedAction, the fields described in the #7563 entry of version 2.3-20260915-1. The deployed version and the time of the deployment are in currentDeployment, e.g. "currentDeployment": {"versionId": 3, "deployedAt": "2026-10-13T10:00:00Z"}, which is left out when the scenario is not deployed.

Code API changes​

  • !1314 ProcessAction, ProcessActionId and ScenarioStatusActionDetails were removed from nussknacker-extensions-api, and ProcessActionState is no longer public. Scenario actions are ScenarioActivity records:

    • DeploymentResult has two new cases, InProgress and ExecutionFinished, and dateFinished moved to the completed results (DeploymentResult.Completed). An exhaustive match on DeploymentResult has to handle them.
    • DeploymentId.fromActionId and toActionIdOpt are renamed to fromActivityId and toActivityIdOpt and take and return a ScenarioActivityId.
    • ScenarioActionName.ScenarioStatusActions was removed, and ScenarioActionName no longer reads the legacy run now name as RUN_OFF_SCHEDULE.

  • !1314 In the listener API, ListenerScenarioWithDetails no longer has lastAction, lastStateAction and lastDeployedAction, and OnActionExecutionFinished.actionId is a ScenarioActivityId.

  • !1314 In the custom HTTP service API, ScenarioService.ScenarioWithDetails has deployedVersionId: Option[VersionId] in place of lastAction, lastStateAction and lastDeployedAction.

  • !1197 NussknackerServicesForCustomHttpService has a new required constructor argument, processingTypes: IO[Set[ProcessingType]], after getModelConfig. Code constructing this service bundle directly, including extension tests, must supply the configured processing type names and rebuild against the updated API. Extensions receiving the bundle from the Designer need no construction changes.

  • !1080 RequestResponseSource.extractTraceId is now final. A request-response source reading its correlation ID from another header overrides traceIdFromHeaders(headers: Map[String, String]): Option[TraceId] instead, so that the HTTP endpoint's error events use the same ID as processing, also for a request that could not be parsed.

  • !1080 pl.touk.nussknacker.util.MdcUtils was removed from nussknacker-common-components. Replace MdcUtils.withMdc(key, value)(action) with SLF4J's Using.resource(MDC.putCloseable(key, value))(_ => action), which is what it did.

Other changes​

Other changes​

  • !1236 Monitoring Scenario, User activity and Insights are now available through Activity. Existing URLs redirect with their filters and history range preserved; no configuration change is needed. The new /api/monitoring/activity endpoint remains admin-only.

  • !1235 Home (/home) is the default landing page for authenticated users. Existing scenario deep links and custom local start pages continue to work. Home applies scenario permissions on the server; it does not grant access to the administrator-only Monitoring pages. Recently opened scenarios use the existing view-session history, so this section stays empty when tracking is disabled. Completion times are the times at which the Designer observed the deployment state; only retained deployment and scheduled-execution rows are listed. Home metrics require live data tagged with the current deployment ID. Name-only data kept in the Designer JVM is shown as missing coverage because a previous execution can still update that storage during redeployment.

  • !1238 The flink-k8s-operator deployment manager had the checkpoint and savepoint directories swapped: checkpoints went to <jobStateStorageMountPoint>/savepoints and savepoints to <jobStateStorageMountPoint>/checkpoints. A scenario deployed after the upgrade writes each to the directory named after it. Checkpoints and savepoints taken before it stay where they were - Flink keeps their full paths, so a job still restores from them; only a person or a script looking for them by directory has to look in the other one.

  • !1277 The jars Nussknacker ships leave out the libraries that the runtime loading them already has. The Docker images and the distribution need no change. If you put the jars together yourself:

    • The ververica and awsManagedFlink deployment managers load only when the flinkStreaming manager's jar, nussknacker-flink-manager.jar, is in managersDirs too, as flink-k8s-operator already needed. All the jars in managersDirs share one classloader, and these managers use the Flink and Nussknacker classes of that jar instead of bundling them.
    • Deployment managers and Designer extensions use the libraries in the Designer's lib/ directory, so use them only with the Designer of the same version.
    • Flink component jars need flinkExecutor.jar of the same version on the model classpath, and lite component jars the lite runtime of the same version.
    • Some libraries in the Designer's lib/ are newer: gson 2.13.1, joda-time 2.12.7, snakeyaml 2.6 and geny 1.1.1. The lite runtime has lz4-java 1.10.3, commons-codec 1.22.0, joda-time 2.12.7, snakeyaml 2.6 and geny 1.1.1. Custom components and extensions that use these libraries run with these versions.

Monitoring Health​

!1237 Monitoring Health combines the former Status and Targets pages. Their URLs redirect to /monitoring/health, preserving query parameters and fragments. The existing application and live overview remains available in an expandable section. Monitoring retains administrator access. GET /api/monitoring/health reads at most 200 current problem deployment summaries from persisted status observations; its response marks truncation. It does not poll engine diagnostics per row. An observed problem time is not the exact start of the failure. Health shows current unresolved problems, not a recent incident history: there is no age cutoff. A persisted problem remains until its status stops reporting a problem, it is replaced or recovery is established as described below, or the scenario is archived. Time alone does not establish recovery. Older, replaced deployments are excluded. Concurrent batch deployments retain their own problem rows while a newer deployment is running; a deployment created after the problem was observed that finishes successfully on the same target clears the older problem from this view. Two absent targets match the default of an engine without target support. Overlapping runs and runs on a different target cannot establish recovery. Periodic run history is not included in this endpoint. A periodic scenario is represented only when its current scenario status reports a problem; scheduling the next run can hide a previous failed run here. Use Home or scenario activity to inspect periodic run history. An empty Health table does not establish that all periodic runs succeeded.

Expanded deployment details poll the existing opt-in diagnostic API while refresh is enabled. Health reads persisted problem summaries every 10 seconds, scenario statuses and runtime samples every 30 seconds, and engine diagnostics and expanded deployment details every 60 seconds. The engine reading age continues advancing while refresh is paused; its exact reception time is available in the tooltip. The expanded application overview refreshes live activity every 30 seconds (previously 10 seconds), keeps its other requests at 30 seconds and shares status/sample polling with Health. A green engine card reflects the last successful reading; its displayed age also reveals delayed refreshes or a paused view. All automatic refreshes can be paused. Record errors use the latest collection-window samples; these are not cumulative counters since deployment. Diagnostic jobs now include an optional deployment identity, and failures include their kind, optional node id and at most 40 stack frames (512 characters each). Embedded managers report history startup, capacity and lifetime loss counters. Manager-wide counters and target coverage are returned only to administrators; ordinary scenario readers retain their own job evidence. Multi-target coverage remains separate for each target, including after all its job history is lost. Collection is bounded by 100 deployments and 10 samples per deployment by default; capacity eviction and runtime restart create gaps. No time-based retention or complete exception delivery is promised. Flink's retention and loss counts remain unknown when its engine does not expose them. AI policy strips stack frames along with messages when runtime data access is disabled, including on older-server client fallbacks. Custom code constructing or matching diagnostic Scala case classes should be recompiled against the extended signatures.

Other changes​

  • !1236 Monitoring Scenario, User activity and Insights are now available through Activity. Existing URLs redirect with their filters and history range preserved; no configuration change is needed. The new /api/monitoring/activity endpoint remains admin-only.

API tokens​

  • AuthenticatedUser and RealLoggedUser carry two more fields, identityKind (User or ServiceAccount) and authentication (Interactive or ApiToken(tokenId)), both defaulted, and AuthManager takes an optional ApiTokenAuthenticator. An authentication or authorization extension building these values needs no change.
  • Secured extension endpoints (CustomHttpServiceServerEndpointDefinition) answer a request authenticated with an API token with 403 unless they override acceptsApiTokens (default false); a PekkoCustomHttpServiceProvider does the same for its whole route. Opt in only where nothing behind it reads or sets a credential, since a token outlives a sign-in and must not be able to set up access surviving its revocation.
  • ScenarioUser (extensions-api) has a defaulted apiTokenId: Option[String], set when the activity was made with an API token. DbScenarioActivityRepository keeps it in the activity's additional_properties under apiTokenId.

Request-response access modes​

  • A request-response scenario with no accessMode behaves as before: the configured request-response.security.basicAuth user when there is one, otherwise anyone, with /definition and /swagger-ui public. Nothing changes until a scenario chooses open or token.
  • Under token, /definition and /swagger-ui need the same credentials as the invocation; set request-response.security.publicDefinition: true to keep them public.
  • With accessMode: open the definition no longer lists the Basic security scheme, since the scenario takes no credentials. A request-response.security block holding no basicAuth no longer marks paths as Basic-secured.
  • ApiToken.Prefix and ApiToken.HeaderName (security) now refer to ApiTokenFormat in extensions-api (pl.touk.nussknacker.security), where the request-response runtime reads them; their values are unchanged.
  • extensions-api has ScenarioInvocationAccess and ScenarioInvocationAccessHolder, which the Designer sets when it starts serving. ScenarioRoute (request-response runtime) takes the scenario's ProcessVersion and an InvocationGuard; its former constructor stays and keeps the legacy behaviour. EmbeddedRequestResponseScenarioValidator is a class built from the engine configuration (EmbeddedRequestResponseScenarioValidator(engineConfig)).
  • ApiTokenAuthenticator (security) is implemented through decide(token): Future[ApiTokenDecision] (Authenticated(user, tokenReadAt, tokenExpiresAt, snapshotConfirmedAt), Rejected(reason) or Unavailable(reason)); authenticate is now final and derived from it. An implementation overriding authenticate has to implement decide instead.
  • AuthorizationRules has decide(action, scope)(user): AccessDecision (allowed, grantedUntil), defaulting to check with no expiry, so existing implementations keep compiling; override it where grants expire. AuthorizationService.decide is its counterpart of check.
  • ScenarioInvocationDecision (extensions-api) carries how long an answer may be reused: Allowed(principal, apiTokenId, validFor), Unauthenticated(validFor) and Forbidden(validFor) are case classes now, and Unavailable(reason) is new - the request-response runtime answers it with 503. InvocationValidity.deadline / validFor compute validFor from the data a decision was made on.
  • RequestResponseInterpreter.apply, RequestResponseOpenApiGenerator.generateOpenApiDefinition and RequestResponseOpenApiGenerator.generateScenarioDefinition take the scenario's accessMode: RequestResponseAccessMode next to securityConfig; pass RequestResponseAccessMode.Legacy for the former output. Under token the definition declares the bearerAuth and apiKeyAuth security schemes.

Token introspection through the Designer​

  • DeploymentManager has tokenIntrospectionSupport: TokenIntrospectionSupport (default NoTokenIntrospection) and tokenIntrospectionSupportFor(deploymentTarget), defaulting to it. A manager whose request-response runtime runs outside the Designer and checks API tokens through it returns TokenIntrospectionThroughDesignerApi(designerUrl). A manager wrapping another one has to forward both, as CachingProcessStateDeploymentManager does.
  • DeploymentData has a defaulted tokenIntrospection: Option[TokenIntrospection] (extensions-api): the Designer's address, the deployment id and the deployment's DeploymentCredential, set for a manager declaring TokenIntrospectionThroughDesignerApi. It is part of the JSON of DeploymentData, so it reaches the runtime.
  • DeploymentManager has stillServing(deploymentId: newdeployment.DeploymentId): Future[Boolean] (default false) and stillServingFor(deploymentId, deploymentTarget), defaulting to it: whether the runtime of a deployment replaced by a redeploy still serves, which keeps its credential valid. A manager whose runtime may outlive its replacement (a rolling update) overrides it; a manager wrapping another one has to forward both, as CachingProcessStateDeploymentManager does.
  • Designer migration V1_080 adds the nullable introspection_token_hash column to deployments, holding the SHA-256 of that credential, and the nullable superseded_by and superseded_at columns, set on a deployment replaced by a redeploy. DeploymentEntityData and DeploymentRepository.saveNewDuringDeployDeployment take the hash as a defaulted introspectionTokenHash. DeploymentRepository.cancelNonFinalDeployments takes a required supersededBy: Option[DeploymentId]: the replacing deployment for a redeploy, None otherwise.
  • LiveDataToken and DeploymentCredential generate and hash their value through DeploymentSecret (extensions-api); the values and hashes are unchanged.

MCP servers accept API tokens only​

  • The MCP JSON-RPC endpoint (POST /api/custom/mcp/<name>) authenticates with the Designer's API tokens (Authorization: Bearer nuk_...) and serves a call only when the token's owner holds MCP.Use on that definition (granted in Access control on the definition, or by a role granting everything). A signed-in session is refused with 401, a token without the grant with 403; both carry a JSON-RPC error body. The X-Mcp-Authorization header is no longer read; a server may instead name a header of its own (McpDefinitionDto.tokenHeader, column token_header, MCP migration V1_004) in which the token is accepted. NussknackerServicesForCustomHttpService carries apiTokens: Option[ApiTokenAuthenticator] for an extension endpoint that reads a token from a header of its own choosing.
  • A server may be set to open access (McpDefinitionDto.access: TOKEN or OPEN, column access, MCP migration V1_005, existing servers stay TOKEN); mcp.allowOpenAccess: false forbids it. The old NONE mode maps to this by hand: a server that used to run without authentication has to be opened again deliberately.
  • The per-definition modes NONE, BASIC and BEARER are removed: McpDefinitionDto has no authType, basicUsername, basicPassword or bearerToken, and MCP migration V1_003 drops those columns from nu_mcp_definitions - the plaintext credentials they held are deleted. Every MCP client configured with a static secret must be given an API token of a service account holding MCP.Use on its server before the upgrade (the MCP tab's "Issue access" does this in one step); until then its calls are refused.
  • McpHttpServiceProvider takes the AuthorizationService; McpDefinition has no auth.
  • custom-http-service-api gains the MachineAccessProvisioning SPI (MachineAccessProvisioningFactory through META-INF/services): an extension owning a resource sets up a machine client's access in one call - a service account, a role granting one action on that resource, the binding and an API token. The RBAC extension provides it; the MCP tab's "Generate token" uses it through GET/POST /api/custom/mcp/definitions/{name}/access....

Resource-scoped actions​

  • ScopeKind has a third value, Resource: an action granted per resource. ScenarioAction.Invoke and McpAction.Use are the first; the users file cannot name resources, so under file-based authorization they are admin-only (LegacyPermissionMapping maps Invoke to no permission). AccessScope gains resource: Option[ResourceId]; check on a Resource-kind action is called with AccessScope.onResource.
  • AuthorizationRules has a new defaulted method, resourceGrants, and UserGrants / UserGrantsSummary a new defaulted field of the same name; /api/user reports it as resourceGrants. An implementation that grants Resource-kind actions must override it, or the summary misreports what check allows.
  • RBAC: rbac_role_actions gains a scope column (migration V1_004, this module's own Flyway); a role's actionScopes are read and written through the roles API (RoleDto.actionScopes, UpsertRoleRequestDto.actionScopes), and GET /api/custom/rbac/resources/{resource} lists what a role can be granted on. Extensions owning a resource register a GrantableResourceListingFactory (custom-http-service-api) through META-INF/services to appear there.
  • MCP: definitions get a stable id (McpDefinitionDto.id, read-only). The module now versions its schema with its own Flyway instance (history table mcp_schema_history, migrations under db/migration/mcp): the baseline leaves the tables Slick created alone, and V1_002 adds the id column with a UUID for every existing row.

Other changes​

  • !1235 Home (/home) is the default landing page for authenticated users. Existing scenario deep links and custom local start pages continue to work. Home applies scenario permissions on the server; it does not grant access to the administrator-only Monitoring pages. Recently opened scenarios use the existing view-session history, so this section stays empty when tracking is disabled. Completion times are the times at which the Designer observed the deployment state; only retained deployment and scheduled-execution rows are listed. Home metrics require live data tagged with the current deployment ID. Name-only data kept in the Designer JVM is shown as missing coverage because a previous execution can still update that storage during redeployment.

Parameter editor configuration​

Parameter and UIParameter have a new optional editorConfig JSON field. Recompile custom components against the updated components API. Existing JSON clients can omit this field. The new Decision Model component uses editorConfig.capabilities on its Questions parameter so the editor can validate against the selected Decision AI integration's limits.

Runtime data access for AI and live error details​

aiDataAccess.allowRuntimeData defaults to false. Set it to true, or set AI_DATA_ACCESS_ALLOW_RUNTIME_DATA=true, and restart Designer to allow automatic runtime evidence in Nu Assistant, WebMCP and nu-cli MCP. Reload the browser after changing settings. This policy is independent of deploymentDiagnostics.enabled, live data collection, and ordinary human CLI access. It does not govern the LLM/Decision Model components inside scenarios.

Explain passes the selected record's variables, expression results, captured service responses and error details as a separate message.runtimeData attachment. The assistant backend checks the policy before adding it to a model question. When disabled, the UI explains that record data is withheld; node definitions and manually supplied text remain available. When enabled, recognizable credential keys, authorization headers and URL credentials/query values are masked; business data is preserved. Attachments exceeding 65536 characters are explicitly omitted; nesting is limited to 32 levels. This is best-effort credential masking, not complete anonymization of arbitrary business text.

AI diagnostics clients use GET /api/processes/{name}/diagnostics?forAi=true. The server omits exception messages when runtime access is disabled and reports runtimeDataIncluded=false. Browser and CLI MCP clients also omit messages if an older server does not explicitly return runtimeDataIncluded=true. The ordinary diagnostics endpoint remains available to authorized human callers when diagnostics is enabled. This is a policy for the application's automatic AI paths, not an agent identity mechanism: an external agent running ordinary human CLI commands cannot be distinguished from a human by this flag. Disabling access does not revoke data already sent to a model or retained in an existing conversation.

Live data and scenario test errors now include optional exceptionDetails: a snapshot of original exception classes, messages and stack frames, displayed separately in node details regardless of AI access. The existing throwable string remains. Snapshots contain at most 16 causes, 64 stack frames per cause and 2048 message characters per cause, with truncation indicated explicitly. They are captured only as part of existing live-data sampling; they do not add a background collector. An empty JVM exception message falls back to the actual class where available, not null or a guessed cause. Empty stack traces stay empty. Legacy live-data uploads remain readable but cannot recover classes or causes that were never stored. Runtime and Designer must both be upgraded to preserve the new details across remote uploads. Re-encoding an old upload does not invent original exception types.

Diagnostic activity and historical graphs​

The diagnostics response includes up to 20 recent activity entries, including failed deployment attempts and their version IDs (activity, activityTruncated). Comments and attachment contents are omitted. Nu Assistant/WebMCP and nu-cli MCP expose get_scenario_version for a single historical graph and get_scenario_runtime_metrics for bounded JVM history. These tools follow the diagnostics feature flag; metrics additionally require administrator access and scenario read permission. AI clients must filter activity error messages and worker event messages when runtimeDataIncluded is not explicitly true, and withhold metric samples under the same policy.

Worker failure evidence​

Diagnostics can optionally receive canonical worker failure events through a separately authenticated, cluster-scoped collector. No collector is enabled by default. See the installation guide for token setup, bounded stdout/stderr capture and retention. workerEvidence is keyed by engine job ID and reports source coverage explicitly, with readable worker IDs and time bounds separate from collector liveness. Runtime mappings include optional workerAssignments for verified assignment intervals; historical metrics and event correlation do not assume a job occupied its latest worker throughout its lifetime. A shared worker OOM proves a process failure, not which job caused it. Engine jobs now optionally include endedAt, used with startedAt to exclude evidence outside their lifetime.

Scenario JVM metrics and process identity​

Flink deployment diagnostics optionally include a runtime mapping (scope, cluster ID, worker IDs, mapping status and whether the JVM is shared). Configure deploymentConfig.metricsClusterId to match the reporter's cluster_id tag, retaining tm_id, job_id and task_id. Existing host-only history cannot be reliably assigned to a deployment. GET /api/processes/{name}/diagnostics/metrics requires administrator access and scenario READ; AI clients must pass forAi=true. It returns bounded series, explicit unavailable/ambiguous states and gaps, rather than assigning shared JVM memory to one job. With monitoring.metrics.flinkReporter = prometheus, JVM diagnostics return unsupported and an unavailableReason: Cloud-normalized metrics do not retain the native fields and verified worker/job identity needed here. General Metrics overview support does not imply JVM-history support. JVM history uses at most four concurrent queries through the executor shared with Metrics overview. If shared capacity is exhausted, the response has status = busy and an unavailableReason, retaining already fetched series. This is temporary query pressure, not evidence of a runtime failure or absent metrics; retry later instead of repeatedly requesting the whole history. If a query also fails for another reason, unavailable takes precedence over busy, while available series remain in the response. Neither status by itself establishes a failure of the scenario engine. See JVM metrics setup for reporter configuration, external RR pod scraping and the Grafana dashboard.

Deployment diagnostics are opt-in​

Deployment diagnostics are experimental and disabled by default. To enable them, set deploymentDiagnostics.enabled = true in Designer configuration, or set the environment variable DEPLOYMENT_DIAGNOSTICS_ENABLED=true, and restart Designer. Reload browser clients and restart nu-cli MCP after changing the setting so that their tool lists are refreshed.

When disabled, the diagnostics API route is not registered (404), no diagnostic reads of engine exception history or deployment activities take place, and Nu Assistant/WebMCP and nu-cli MCP do not register get_scenario_diagnostics. Missing settings on older servers also mean disabled. Diagnostics are read on request, with no background polling or durable persistence of engine history in Designer. This flag applies globally, not per scenario or deployment. It does not disable standard engine logs, Flink's own exception history, deployment activity recording, status polling or live data preview.

Embedded deployment managers now keep a bounded in-memory history independently of live data preview: the last 100 deployments per manager, with up to 10 failures per deployment. Record failures fill the available sample slots; later record failures increase loss counters without rendering messages or stacks. Deployment failures can replace a retained sample, so the latest deployment failure cannot be displaced by a record-error storm. Message and frame redaction runs outside the history lock. Diagnostics return up to 10 deployments for the requested scenario, including its current deployment even when its history has already been evicted. Errors retain timestamps and bounded, redacted cause chains, not throwable objects or event contexts. Cancelling or replacing a deployment preserves its retained diagnostic history; closing its deployment manager clears it. The diagnostics feature flag controls access to this history, not its collection. AI access to exception messages and stack frames remains governed by aiDataAccess.allowRuntimeData, including the server-side assistant tool-result policy. DiagnosticMessageRedactor is an internal Nussknacker implementation utility, not a deployment-manager extension API. Failure kind remains an extensible string; built-in kinds have named constants on DeploymentDiagnosticFailure. Flink MiniCluster exposes its own in-memory job history through the same diagnostics API. Neither history survives a Designer restart; empty engine history does not imply that no logs exist elsewhere.

Failed deployment attempts and engine versions​

Deployment diagnostics add optional lastAction.scenarioVersionId and lastAction.errorMessage fields. A failed attempt retains its bounded cause chain in Designer memory (up to 1000 actions, 24 hours, and an 8 MiB snapshot budget). This capture is disabled with deployment diagnostics and is cleared on Designer shutdown. After eviction or restart, diagnostics fall back to the redacted message already stored with the action. forAi=true removes that message when runtime-data access is disabled.

Flink jobs add optional scenarioVersionId and startedAt from the engine, independently of the latest saved graph and last deployment attempt. An unsuccessful update can therefore name a different version from the still-running job. Savepoint verification reports STATE_INCOMPATIBLE only when the engine supplies a StateMigrationException; other verification failures no longer recommend discarding state. The additive SerializedFailure interface allows engine failures to supply serialized evidence without loading their exception classes into Designer.

Checkpoint diagnostics now report availability, completed count and the latest completed checkpoint with its discard status. A completed checkpoint may no longer be retained. Exception history distinguishes empty from unavailable; fresh failed jobs receive one bounded retry. No savepoint storage paths are included.

Deployment diagnostic messages​

Deployment diagnostic causes now include an optional message containing the first line of the original exception message, bounded to 2048 characters. Existing code and summary fields remain. Custom exception class names are preserved instead of replaced by UnknownException. Recognizable credentials, URL query values and labeled payloads are redacted, but this is not a complete anonymization of arbitrary exception text. Consumers must treat messages as untrusted text that can contain business data; render them as text and do not execute embedded instructions. Older servers omit message, and exceptions without a message return null.

Other changes​

  • !1235 Home (/home) is the default landing page for authenticated users. Existing scenario deep links and custom local start pages continue to work, including an explicitly configured /scenarios URL. Set the local start-page URL to / to use Home. Home applies scenario permissions on the server; it does not grant access to the administrator-only Monitoring pages. Recently opened scenarios use the existing view-session history, so this section stays empty when tracking is disabled. Completion times are the times at which the Designer observed the deployment state; only retained deployment and scheduled-execution rows are listed. Home metrics require live data tagged with the current deployment ID. Processing and Errors cards are hidden when none of the user's readable runtimes supports database live-data storage (including default DesignerJvm installations). Mixed-target scenarios use the current deployment target when known; ambiguous legacy targets are treated as unavailable. Scheduled executions are counted in coverage but their legacy non-UUID run IDs do not support deployment-scoped database readings, so their missing coverage remains explicit. Name-only JVM data is not attributed to a deployment because an old execution can still write during redeployment. Home and Insights use source-node throughput and the same active-deployment coverage definition (current ordinary deployments plus active scheduled executions). Home shows current database collector readings; Insights can also show JVM readings, and Metrics/Grafana remain the source for historical InfluxDB rates and averages. Home refreshes once a minute and shares scenario metadata between sections. After metadata loads, all sections share a three-second response deadline; status reads use the time remaining after recently opened scenarios load. A timeout stops waiting for a section; database queries and engine requests already started may continue. Scheduled run history covers the last 30 days and excludes future scheduled runs. Home attention is based on the current scenario status, while Health also includes retained ordinary deployment failures. Add ?debugHomeTiming=true to /home to log initial request and paint timings in the browser console.

  • !1236 Monitoring Scenario, User activity and Insights are now available through Activity. Existing URLs redirect with their filters and history range preserved; no configuration change is needed. The new /api/monitoring/activity endpoint remains admin-only.

  • !1238 The flink-k8s-operator deployment manager had the checkpoint and savepoint directories swapped: checkpoints went to <jobStateStorageMountPoint>/savepoints and savepoints to <jobStateStorageMountPoint>/checkpoints. A scenario deployed after the upgrade writes each to the directory named after it. Checkpoints and savepoints taken before it stay where they were - Flink keeps their full paths, so a job still restores from them; only a person or a script looking for them by directory has to look in the other one.

MCP tool execution timeout​

Entries in the LLM component provider's mcpIntegrations configuration now accept toolExecutionTimeout, which defaults to 30 seconds. Previously the client used LangChain4j's independent 60-second tool execution limit, equal to the default Request-Response HTTP timeout. A slow tool could therefore leave the caller with an HTTP 503 before the agent had time to handle the tool error.

The existing timeout setting still controls the HTTP transport and defaults to 60 seconds. To retain the previous tool execution limit, set toolExecutionTimeout: 60 seconds on the integration. For Request-Response scenarios, leave enough time within the HTTP request timeout for LLM calls and any additional tool executions; this setting limits each tool call, not the whole agent conversation.

In version 2.3-20261009​

Other changes​

  • !838 DB migration V1_078 adds the scenario_view_sessions and designer_view_sessions tables and indexes on scenario_activities ("user_name", "created_at") and live_data ("updated_at"). The scenario_activities index is built without CONCURRENTLY, so on a large table the migration blocks writes to it while it runs.
  • !838 The Designer now writes view sessions to the database: at most once per monitoring.scenarioViewTracking.minimumTimeBetweenReports (15 seconds) for each user and open scenario, and for each user with the Designer open. With many concurrent users, consider raising db.maximumPoolSize. Set monitoring.scenarioViewTracking.enabled: false to turn it off.
  • !838 DeploymentManager has a new optional engineDiagnosticsSupport, NoEngineDiagnosticsSupport by default. See the scaladoc of EngineDiagnosticsSupported.diagnostics before implementing it.

In version 2.3-20261008​

Component descriptions and AI usage hints​

!856 adds optional description and aiUsageHint fields to component definitions and UI configuration. Existing configuration and saved scenarios need no changes. Recompile custom component/provider code against the new components API: the added Scala case-class parameters change binary signatures and extractor arity, even though constructor calls can omit the defaulted fields.

Use description for the component's business purpose and aiUsageHint for short, specific semantics such as side effects, idempotency or missing-data behavior. General scenario-authoring procedures belong in skills. Both fields are metadata exposed through the definition API; aiUsageHint is not a confidential field or a permission boundary. Defaults supplied by the component can be overridden by componentsUiConfig, then by AdditionalUIConfigProvider.

Configuration changes​

  • authentication.enableBasicAuthForTechnicalAccounts (OIDC) is renamed to authentication.enableBasicAuthAccounts. The old name is still read where the new one is not set, but is deprecated.
  • Anonymous access was removed. The authentication.anonymousUserRole setting is gone and is ignored where it is still configured: every request now needs credentials, and a request without them is answered with 401. With the OAuth2 and Oidc methods, opening the Designer with ?signin is no longer needed to sign in: every visit without a session is sent to the identity provider. An instance that offered looking around without an account should configure authentication.autoLogin, which signs every visitor in as a shared account - see Designer configuration.
  • The Dummy authentication method was removed. An installation that set authentication.method: "Dummy" has to choose another method.
  • With BasicAuth and BasicAuthForm, a users file with no entry setting password or encryptedPassword now fails the start. Add one, or register a UserLookupFactory where the accounts come from elsewhere.
  • !1079 The LLM components jar was renamed from components/common/enterprise-components.jar to components/common/llm.jar. A modelConfig.classPath that lists the components/common directory needs no change. One that lists jars one by one must replace the old name - an entry pointing at the missing old jar fails the deployment. The same holds for deploymentConfig.modelArtifact.providedJarFiles, which must stay in sync with modelConfig.classPath.

Live data preview uploaded to the Designer API​

  • New DESIGNER_API live data storage, for runtimes that cannot open a connection to the Designer database. It is configured per deployment target, in the deployment configuration of a flinkStreaming target deploying to a standalone cluster - for the targets created at runtime, in the shared settings of their dynamicTargets.deploymentManagers entry. What is collected (enabled, maxNumberOfRecords, ...) stays in modelConfig.liveDataPreview: 
    deploymentConfig {
      targets: [ ... ]
      dynamicTargets {
        deploymentManagers: [
          {
            type: "flinkStreaming"
            liveDataPreview.storage {
              type: "DESIGNER_API"
              url: "https://designer.example.com" # the Designer as reachable from the runtime
              uploadIntervalInSeconds: 1          # default
              idleIntervalInSeconds: 2            # default, how often the counts go up while nobody watches live data
              uploaderInactivityTimeoutInSeconds: 300 # default
              allowInsecureUrl: false             # default, set to true only to allow a plain http url
            }
          }
        ]
      }
    }
    A target without its own liveDataPreview.storage uses modelConfig.liveDataPreview.storage, so existing DESIGNER_DB setups keep working; only DESIGNER_API can be configured per target. No database credentials are shipped to the runtime: the Designer mints a token for every deployment, sends it with the upload settings in the deploy request and stores only its hash; the token stops being accepted once the deployment is not active anymore.
  • The runtime calls PUT /api/liveData/deployments/{deploymentId}/collectors/{collectorId} with Authorization: Bearer <token>. If the Designer sits behind an authenticating proxy (e.g. oauth2-proxy), that path has to be let through without user login.
  • Every uploading storage - DESIGNER_DB as well as DESIGNER_API - now splits what it sends. The node counts and the throughput go up every idleIntervalInSeconds (new for DESIGNER_DB, default 2), whether or not anybody has the scenario open, so they are available to any view that reads them. The live data samples - the records, expression results, service invocations and exceptions - are sent every uploadIntervalInSeconds, but only while somebody watches the scenario. Reading a scenario's live data is what marks it as watched, for both storages.
  • Uploading the samples on demand is on by default for both uploading storages and can be turned off with modelConfig.liveDataPreview.samplesOnDemand: false - the samples are then uploaded every uploadIntervalInSeconds all the time, as before, and a DESIGNER_DB runtime doesn't read deployments. The DESIGNER_JVM storage is not affected - its samples are always collected. The Lite runtime ignores the setting for now and always uploads the samples.
  • How the runtime learns that somebody watches a scenario differs by storage: DESIGNER_API is told in the response to each upload, while DESIGNER_DB reads live_data_watched_until from the deployments table. A DESIGNER_DB runtime therefore needs SELECT on deployments besides its rights on live_data; where it cannot read that column it keeps uploading the samples, as it did before, and says so in the log.
  • Switching an existing scenario type to DESIGNER_API requires redeploying its scenarios - jobs deployed earlier have no token and don't upload live data. Scheduled (periodic) scenarios and the lite-k8s deployment manager don't support this storage. The flink-k8s-operator and ververica deployment managers read it from modelConfig.liveDataPreview.storage only.
  • API changes:
    • DeploymentData has a new liveDataUpload: Option[LiveDataUpload] field (defaults to None).
    • LiveDataPreviewSupport has a new LiveDataPreviewUploadedToDesignerApi case - exhaustive matches in custom deployment managers need to handle it.
    • DeploymentManager has a new liveDataPreviewSupportFor(deploymentTarget) method, returning liveDataPreviewSupport by default.
    • LiveDataCollectingListener.createListenerFor has a new liveDataUploadOpt parameter.
    • ModelConfig.LiveDataPreviewMode.Enabled has a new samplesOnDemand: Boolean field (defaults to true) - positional pattern matches on it need one more placeholder.
    • GET /api/liveData/{scenarioName} responds with its own LiveDataDto instead of ResultsWithCountsDto: it no longer has the always empty assertionsResults field and has a new countsOnly field, true when the live data holds the counts without the samples.

Code API changes​

  • AnonymousAccessSupport, NoAnonymousAccessSupport and AuthenticatedUser.createAnonymousUser were removed, along with anonymousUserRole in AuthenticationConfiguration and in the basic auth, OAuth2, OIDC and dummy configurations. ImpersonationConsideringInputEndpoint.withPossibleImpersonation no longer takes the anonymousAccessEnabled flag. AuthCredentials.NoCredentialsProvided was removed: a request without credentials never reaches authentication. A custom AuthenticationProvider that implemented getAnonymousRole should drop it.
  • AuthenticationProvider.createAuthenticationResources and OAuth2ServiceFactory.create take a further userLookup: UserLookup; AuthenticationResources.impersonationSupport becomes impersonationTargets: Option[UserLookup]. ImpersonationSupport, ImpersonatedUserData, UsersFileConfiguration, AuthenticationConfiguration.users and the configurations' usersFile, getUserRoles and findUserById are gone. Register a UserLookupFactory to provide a store of your own. A custom AuthenticationProvider also has to answer signsInAgainstConfiguredAccounts - true where it checks passwords against the users file.
  • A users file entry setting both password and encryptedPassword now fails the start under every authentication method. Keep one.

Other changes​

  • !1194 A database migration drops tables the Designer no longer reads. Back up any of them you still need before upgrading:
    • process_actions and process_comments - their data was copied to scenario_activities in version 1.18
    • sticky_notes - sticky notes are stored in the scenario graph now, and notes from this table were never moved there
    • scheduled_scenarios_orphaned_for_manual_review and scheduled_scenario_deployments_orphaned_for_manual_review (PostgreSQL only) - copies of scheduling data that pointed at deleted scenarios, kept for manual review
  • modelConfig.liveDataPreview.retentionTime can be set to null to turn off the age limit of the live data preview - only maxNumberOfRecords then limits what it keeps, for the records, expression results, service invocations and exceptions alike, however old they are. A missing retentionTime still means 1 hour. It has to be an explicit HOCON null, and it applies only when retentionTimeInMinutes is not set. LiveDataPreviewMode.Enabled.retentionTime is now an Option[Duration].
  • GET /authentication/<method>/settings no longer returns the frontend strategy at its top level. It returns options, the ways of signing in - SingleSignOn carrying the former strategy under strategy, Password and AutoLogin - and default, either {"type": "Use", "option": <the option's type>} or {"type": "Chooser"}. The OAuth2 strategy no longer carries anonymousAccessAllowed. Clients reading the strategy must look for the SingleSignOn option; nu login from an older CLI falls back to its browser snippet instead of driving the sign-in itself.

Designer deployment action metrics​

!834 Designer deployment action timings are now reported by a single tagged timer, deployments.action.duration, instead of the separate deployment, cancel and singleExecution timers:

Removed timerPreviously exported asReplaced by
deploymentmetrics_deployment_*operation="deploy" or operation="redeploy", stage="request"
cancelmetrics_cancel_*operation="cancel", stage="request"
singleExecutionmetrics_singleexecution_*operation="run_off_schedule", stage="request"

All three are now series of the same timer, for example designer_deployments_action_duration_count, told apart by the operation and stage labels. stage="request" measures the command as a whole, as the removed timers did. The new timer measures further stages - a cancel, for instance, is also measured per deployment target at stage="execution" - and carries context tags such as scenario_name and deployment_target - see Deployment action metrics for the full list.

The Designer JMX reporter now publishes in the designer domain instead of the default metrics one, so every Designer JMX metric changes its exported prefix from metrics_ to designer_. The repository gauges, for example, move from metrics_scenarios_count*, metrics_fragments_count* and metrics_deployedscenarios_count* to designer_scenarios_count*, designer_fragments_count* and designer_deployedscenarios_count*. Lite JMX metrics retain the metrics_ prefix.

The reporter also keeps the metric name as the first JMX key property. The Prometheus JMX exporter names a metric after the first key property of its object name, which for a tagged metric would otherwise be an arbitrary tag - the deployment action timer would be exported as designer_none_*, after its deployment_target="none" tag, with the metric name demoted to a name label. The configured exporter lowercases label names; the new snake_case tag keys are preserved as deployment_manager_type, deployment_target, scenario_id and scenario_name in PromQL queries.

JMX Prometheus Java agent​

!834 The bundled jmx_prometheus_javaagent was upgraded from 0.20.0 to 1.6.0. Some automatically exported JVM metric names changed, including jvm_memory_bytes_{committed,used,max,init}, which became jvm_memory_{committed,used,max,init}_bytes.

The agent now serves metrics at /metrics instead of /. Update clients that explicitly scrape the root path; Prometheus configurations using its default metrics path do not need this change.

  • !1030 The ai-agent component has a new parameter When an MCP server is unavailable, set to Fail the event for existing nodes. When a tool call cannot reach its MCP server or times out, the event now goes to error handling; before, the failure was given to the model as the tool's result. To keep the previous behaviour, set the parameter to Continue without its tools. An error the tool itself returns is still given to the model.

In version 2.3-20260929-1​

Configuration changes​

  • !1045 The component additional info provider jar was renamed from components/common/cloud-additional-info-provider.jar to components/common/additional-info.jar. A modelConfig.classPath that lists the components/common directory needs no change. One that lists jars one by one must replace the old name - an entry pointing at the missing old jar fails the deployment. The same holds for deploymentConfig.modelArtifact.providedJarFiles, which must stay in sync with modelConfig.classPath.
  • !1047 The KV store component moved out of components/common/enterprise-components.jar into components/common/kv-store.jar. A modelConfig.classPath that lists the components/common directory needs no change. One that lists jars one by one must add components/common/kv-store.jar - without it a model configured with the kvStore provider fails to load. The same holds for deploymentConfig.modelArtifact.providedJarFiles, which must stay in sync with modelConfig.classPath.
  • !1015 The HTTP enricher moved out of components/common/enterprise-components.jar into components/common/http.jar. A modelConfig.classPath that lists the components/common directory needs no change. One that lists jars one by one must add components/common/http.jar - without it a model configured with the http provider fails to load. The same holds for deploymentConfig.modelArtifact.providedJarFiles, which must stay in sync with modelConfig.classPath.
  • cloud.websocket.url and cloud.websocket.accessToken moved to cloud.websocket.read.url and cloud.websocket.read.accessToken.
  • !990 The Designer no longer serves resources under /submodules: a request for /submodules/... now returns the Designer page instead of the file. Move a custom Module Federation remote from web/submodules/... to web/static/... and change its URL in tabs from /submodules/... to /static/.... The Configurator and MCP tabs are now nuEnterpriseIntegrations/tab@/static/integrations/remoteEntry.js and nuEnterpriseMcp/tab@/static/mcp/remoteEntry.js. The Designer rewrites the old nuConfigurator/* tab URLs to these automatically.
  • !979 The scenario migrations (DefaultModelMigrations) moved out of model/defaultModel.jar into model/migrations.jar, so that a model built on a different ProcessConfigCreator picks up the same migrations. Nothing lists model/ as a directory, so every modelConfig.classPath that lists model/defaultModel.jar must add model/migrations.jar explicitly - without it no migration is applied and the model version drops to 0. The same holds for deploymentConfig.modelArtifact.providedJarFiles, which must stay in sync with modelConfig.classPath.

Code API changes​

  • !1067 The TestProcess.TestResults methods updateNodeResult, updateNodeOutputResult, updateExpressionResult, updateExternalInvocationResult and updateExceptionResult are removed. Use ResultsCollectingListenerHolder.withListener for raw values or ResultsCollectingListenerHolder.withTestEngineListener for JSON values, pass events to the listener's callbacks, and read listener.results after collection. Use TestServiceInvocationCollector with that listener for service and sink invocations. These collectors are in pl.touk.nussknacker.engine.testmode in nussknacker-scenario-compiler.

Lite metric reporter lifecycle​

!1065 MetricsReporter.createAndRunReporter now returns AutoCloseable instead of Unit. Recompile custom Lite metric reporters and return a handle that stops the reporter and releases its scheduler, connections and metric listeners. The runtime owns that handle and closes it on shutdown or if a later startup step fails. If the reporter throws before returning its handle, it must clean up any resources it already acquired itself.

LiteMetricRegistryFactory.prepareRegistry now takes the classloader to discover and run the reporters with, and returns LiteMetricRegistry instead of MetricRegistry - an AutoCloseable exposing the Dropwizard registry as .registry, whose close() stops the reporters started for it.

Other changes​

  • !973 Fix scenarios failing on a null aggregateBy, and return null from a numeric aggregate that aggregated nothing, in place of 0 or Double.NaN:
    • State incompatibility: aggregate-tumbling and aggregate-session nodes have to be redeployed with their state reset. aggregate-sliding, the aggregate-tumbling "After window closes, also when no event for key" emit mode, single-side-join and full-outer-join are unaffected.
    • #AGG.average, #AGG.median, #AGG.stddevPop, #AGG.stddevSamp, #AGG.varPop and #AGG.varSamp return null instead of Double.NaN; #AGG.sum returns null instead of a typed 0. Replace isNaN() checks with null checks, and use #aggregate == null ? 0 : #aggregate where a number is required.
    • Arithmetic on a null aggregate fails at runtime, but a comparison does not: #aggregate > 100 is false and #aggregate < 100 is true. Guard both, for example #aggregate != null && #aggregate > 100.
    • Counting with #AGG.sum over 1 now answers null for a window that counted nothing. Switch it to #AGG.countWhen over true.
    • aggregate-sliding stores the first 0 of a slice, which it skipped before. A scenario whose aggregateBy is mostly 0 grows its state, and with "emit when event leaves the window" it now also emits for a key that only ever saw zeros.
    • pl.touk.nussknacker.engine.api.typed.NumberTypeUtils is removed.
  • !1050 In flink-components-utils, the key parameter of the keyed mappers (KeyedValueMapper, GenericKeyedValueMapper, StringKeyedValueMapper, StringKeyOnlyMapper, GenericKeyOnlyMapper) and of the richflink groupBy, groupByKey and groupByWithValue is a ParameterNameWithLabel instead of a ParameterName. Pass paramNameWithLabel of the key parameter's declaration, or ParameterNameWithLabel(name, label).

Parameter descriptors extracted into named constants​

No parameter name changed, so stored scenarios need no migration.

  • Where the text of a validation error names a parameter, it now uses the parameter's label instead of its technical name, e.g. in the '<label>' - prefix of the plain-text errors returned by the testing API. Labels changed across most components, so update clients that match on this text.
  • Component parameter constants are now UpperCamelCase with a ParamName suffix and hold a ParameterName rather than a String. Some moved or are gone: the Kafka names live in KafkaParameters (kafka-components-utils) and SchemedKafkaParameters (schemed-kafka-components-utils), KafkaFactory and WatermarkStrategyValidationHandler.splitName were removed, and the label constants are private. Take a label from the matching paramNameWithLabel.
  • KafkaUniversalComponentTransformer now matches the topic parameter by name. A subclass that renames that parameter must override topicParamName alongside toTopicParamDeclaration; the name was previously ignored.

In version 2.3-20260921-1​

REST API changes​

  • !894 Validation errors changed:
    • message was reworded and now carries the remedy that used to sit in description. Clients that match on the exact text of a message need updating.
    • description of NodeValidationError, EnricherMockValidationError and AssertionValidationError is optional: it is not required in the OpenAPI document and is null when absent. Clients that read it must handle null, clients that build it can omit it. message stays required. description is now null where it used to be:
      • an empty string, e.g. for MultiSelectInvalidFormat or a validator that gave "",
      • a copy of message, for CustomNodeError and EagerExpressionEvaluationError,
      • Enter a value in this field, for EmptyMandatoryField and for an empty scenario name, fragment name, node name or node id.

Code API changes​

  • !894 Validation errors changed:
    • description is now Option[String] in ParameterValidationError and every case class in ProcessCompilationError that extends it (e.g. CustomParameterValidationError), in CompileTimeAndRuntimeValidator and RegExpValidator, and in the REST model classes NodeValidationError, EnricherMockValidationError and AssertionValidationError. Use Some(text), or None where you used "". message stays a String, and ScenarioNameValidationError and ScenarioLabelValidationError keep a String description. Existing JSON and HOCON configuration of RegExpValidator still works, and description in it may now be omitted or null.
    • NonNegativeDurationValidation.Description and PositiveDurationValidation.Description were removed, with no replacement - drop the reference. Both Message vals stay, with reworded text.
    • The first field of ProcessCompilationError.NonUniqueEdgeType and of ProcessCompilationError.UnsupportedEdgeNextToCustomNodeOutputs was renamed from edgeType to outputName. It now carries the output label the graph shows instead of the EdgeType rendered through toString. Update named arguments and named pattern bindings; positional construction and pattern matches are unaffected.
    • A node other than split with more than one unnamed outgoing edge is now rejected with NonUniqueEdgeType("main", ...) instead of NonUniqueEdgeType("main output", ...).

In version 2.3-20260915-2​

In version 2.3-20260915-1​

Artifacts are published to internal repositories only​

  • Maven artifacts are published exclusively to nexus.touk.pl. Snapshots are no longer published to Maven Central (central.sonatype.com).
  • The ML modules changed their Maven organization from pl.touk.nussknacker.ml to pl.touk.nussknacker.
  • Docker images (nussknacker, nussknacker-lite-runtime-app) are published to docker-gitlab.touk.pl/nussknacker-cloud/dummy-for-docker-registry/ instead of DockerHub (touk/...).
  • The default dockerImageName of the lite-k8s deployment manager changed from touk/nussknacker-lite-runtime-app to docker-gitlab.touk.pl/nussknacker-cloud/dummy-for-docker-registry/nussknacker-lite-runtime-app. The registry is private, so the runtime pods need image pull secrets (configured via k8sDeploymentConfig). Set dockerImageName explicitly if your cluster should pull the runtime image from somewhere else.

License key is now required​

The Designer now reads a signed license key from the licenseKey configuration entry (or the NU_LICENSE_KEY environment variable). See License key for details.

  • An installation without a valid key loses access to every component delivered by a component provider. Such components stay visible in the creator panel but are locked: they cannot be dragged onto the canvas, scenarios using them fail validation and cannot be deployed. Built-in components (filter, choice, variable, split, ...) and fragments keep working. Scenarios already running on the cluster are not touched.
  • Before upgrading, obtain the license key issued for your installation and add it to the Designer configuration: licenseKey: "<the key>" (or set NU_LICENSE_KEY). A malformed key stops the Designer at startup with an error in the log; a missing or expired key keeps the Designer running with all provider components locked and a banner explaining why.
  • Component and provider names use separate allowlists. Entries in components match only component names (either unqualified or as providerName/componentName); entries in the optional providers list match only provider names. When replacing a key created with an earlier preview of this feature, move provider-level entries such as flink* from components to providers. The special components: ["*"] entry continues to unlock every component.
  • Code constructing ModelDependencies must now pass its component licensing policy explicitly. Use ComponentLicensing.Unrestricted only where license enforcement is intentionally not applicable; the Designer passes a ComponentLicensing.Restricted policy derived from the verified license key.

Deployment status mechanism​

The scenario status is now resolved from the deployments table (kept fresh by the deployments statuses synchronizer) instead of being derived by merging the action log with the engine state. A deploy/redeploy now stores a deployment row with a stable id (periodic scenarios are the exception - they keep their own scheduling mechanism and are not represented in the deployments table). On Flink this id is used as the job's JobID, so the synchronizer can correlate the running job with the stored status; lite-k8s stores it as a k8s Deployment annotation.

  • Existing running jobs have to be redeployed or canceled once. Jobs deployed before this version were started with the old deployment id scheme, so they are not correlated with the new deployments table. After the upgrade such scenarios are shown with a PROBLEM status with the message "The scenario was deployed before an upgrade and its status is no longer tracked. Redeploy it to restore tracking or cancel it to stop the old job. Redeploy will replace the running job, preserving its state." — this is intentional so that a still-running job is not mistaken for a stopped one (which could lead to losing its state). Such scenarios are also counted as active for deployment limits until they are resolved. Redeploy each scenario that should keep running; the redeploy replaces the orphaned job, preserving its state via savepoint. Cancel scenarios that are no longer needed. After redeploy or cancel the status is tracked normally and the message disappears.
  • For lite-k8s, the deployment id is stored as a nussknacker.io/deploymentId annotation on the scenario's k8s Deployment (metadata only, so it does not trigger a pod rollout).
  • The deployments table is periodically cleaned up: final-status rows (CANCELED/FINISHED/PROBLEM) older than deploymentStatusesSynchronization.keepFinalDeploymentsFor (default 24 hours) are deleted, except the newest final row of each scenario. Use the scenario activities for deployment history. The cleanup runs every deploymentStatusesSynchronization.cleanupInterval (default 1 hour).

Deployment targets​

  • A scenario type can be served by several deployment managers at once, one per deployment target. Such a scenario type has no deploymentConfig.type; it lists its targets in deploymentConfig.targets instead:

    deploymentConfig {
      targets: [
        {
          # The first target is the default one - it is where the commands that don't point at any target go.
          name: "Standalone Flink"
          type: "flinkStreaming"
          restUrl: "http://jobmanager:8081"
          useMiniClusterForDeployment: false
        }
        {
          name: "Flink MiniCluster"
          type: "flinkStreaming"
          useMiniClusterForDeployment: true
        }
      ]
    }

    A targets entry is the deployment configuration of the manager handling that target, type included, plus a name - the deployment target name shown in the designer, which has to be unique. The targets are presented in the configured order, and the first one is the default. targets and type are mutually exclusive, and every entry of targets configures a single Deployment Manager, so targets cannot be nested. A scenario type configured with type keeps behaving exactly as before: one Deployment Manager, no deployment targets.

  • All the targets of a scenario type share its model and the resources bound to it - in particular, Flink targets share a single MiniCluster, the same one used for scenario testing and state verification. Periodic scheduling wraps a whole Deployment Manager, which has no notion of per-deployment targets, so a scenario type split into targets reports no scheduling support. A target that runs scenarios outside the designer's JVM still needs modelConfig.liveDataPreview.storage to be DESIGNER_DB for live data preview; that requirement is now reported per target.

  • Deployment targets can also be created, changed, reordered and removed at runtime - see the configuration docs. New DB migration V1_076: the deployment_targets and deployment_target_order tables, and a nullable deployment_target column in deployments.

  • A scenario can only change its deployment target while it has no deployment that may still exist on an engine (anything but CANCELED/FINISHED) - each target is backed by its own engine and only replaces its own deployments, so cancel the scenario first. For the same reason, a target cannot be removed while a scenario may still be running on it, and a scenario whose recorded target is no longer configured cannot be deployed at all.

  • A scenario that is running when you upgrade cannot change its deployment target until it is cancelled. Deployments created before this version have no target recorded, so it is not known which engine runs them. The designer fills the target in where it can: on startup, scenario types that recover jobs ask their engines. Cancelling such a scenario is sent to every configured target (engines treat cancelling a scenario they do not run as a no-op), but the admin-only POST /api/adminProcessManagement/stop/{name} and .../snapshot/{name} cannot fan out that way and are rejected with an explicit error.

  • Source-breaking changes for code compiled against Nussknacker's API modules:

    • The DeploymentManager SPI gained a deploymentTargetsSupport method. It defaults to NoDeploymentTargetsSupport, the previous single-target behavior, so existing implementations need no change.
    • DeploymentManagerProvider gained a deploymentTargetType(config) method declaring the one kind of target the provider serves when configured with the given config, and which can therefore be created at runtime. It defaults to None, so existing implementations need no change.
    • DeploymentSynchronisationSupported.getCurrentDeploymentStatuses now returns DeploymentStatusesFromEngine instead of a bare Map[DeploymentId, DeploymentStatus]. Wrap the map you returned before - the companion's single-argument apply reports no unreachable targets, which is right for a single-engine manager.
    • The LiveDataPreviewSupport hierarchy gained LiveDataPreviewStoredInDesignerJvmOrDb, reported when some targets store live data in the designer's JVM and the others in its database. Matches over it have to handle the new case.
    • Every scenario command that acts on a running job carries the deployment target it is meant for: deploymentTarget: Option[DeploymentTarget] was added to DMCancelScenarioCommand, DMStopScenarioCommand, DMMakeScenarioSavepointCommand, DMCancelDeploymentCommand and DMStopDeploymentCommand, next to the DMRunDeploymentCommand/DMValidateScenarioCommand fields that had it already. A Deployment Manager serving several targets routes on that field in processCommand, so DeploymentTargetsSupported.processCommandForTarget is gone - send the command itself instead.
    • New fields breaking exhaustive pattern matches over the case classes (construction stays source-compatible where the field defaults to None): DMRunDeploymentCommand.deploymentTarget, DMValidateScenarioCommand.deploymentTarget, ProcessStateDefinitionManager.ScenarioStatusWithScenarioContext.deploymentTarget and DeploymentStatusDetails.deploymentTarget. ScenarioActivity.ScenarioDeployed/ScenarioRedeployed, DeployRequest (restmodel) and UIDefinitions (restmodel) gained required fields, which breaks construction too, as do the constructors of CachingProcessStateDeploymentManager and EmbeddedDeploymentManager.
    • FlinkDeploymentManager takes a SharedFlinkMiniCluster where it took a FlinkMiniClusterWithServices, and FlinkMiniClusterScenarioJobRunner gained a required constructor parameter. Acquire the cluster with SharedFlinkMiniClusters.acquire(modelClassLoader, miniClusterConfig) and close the handle it returns when the manager is closed.
    • The embedded engine treats cancelling a scenario it does not run as a no-op instead of failing. This applies to plain lite-embedded too, and a caller that relied on the failure to detect an unknown scenario no longer gets it.
    • ScenarioStatusDetailsDto (restmodel) gained a DuringDeploy subtype. Every scenario status JSON gained a nullable deploymentTarget field - not only the ones of a scenario that is being deployed or is running.

Authentication​

  • !519 OidcAuthenticationConfiguration gained an enableBasicAuthForTechnicalAccounts option, defaulting to false. When enabled, users in the OIDC usersFile that have a password/encryptedPassword set can additionally authenticate via HTTP Basic - intended for technical/machine accounts that cannot go through the OIDC flow (for example the RemoteEnvironment scenario-migration client). OIDC-only users, which have no password, are unaffected. Existing configurations keep OIDC-only authentication.

Authorization decisions go through AuthorizationService​

Authorization moved from the Read/Write/Deploy permission bundles to per-action grants. The users file is unchanged - its permissions still expand to the same actions - but what the API reports, and what a few endpoints require, did change.

Behaviour

  • Deleting a scenario (the API-only DELETE /api/processes/{name}, available for archived scenarios) now requires admin - previously any role with the Write permission in the scenario's category could delete it. It is also out of reach of an impersonated session, even one impersonating an admin.
  • Endpoints returning records read off a scenario's sources now require Scenario.ViewLiveData and Scenario.Test: POST /api/nodes/{scenarioName}/records, which had no authorization of its own at all, POST /api/scenarioTesting/{scenarioName}/generatedTestData, and performTest when the request carries WITH_LIVE_DATA. The source to read from is named by the request rather than taken from the stored scenario, so the grant on the scenario in the URL does not bound which records come back. Scenario.Test comes from the Deploy permission, so roles holding only Read or Write lose these endpoints.
  • Impersonating an admin (possible only with isAdminImpersonationPossible) no longer grants admin-only endpoints: the impersonated session keeps the admin's category-wide data access, but AdminAccess now reflects the real session's identity. Chained impersonation (impersonating from an already impersonated session) remains impossible.

HTTP API

  • /api/user replaces categoryPermissions/globalPermissions with categoryActions/globalActions, listing full "<Resource>.<Name>" action identifiers (Scenario.Deploy, Core.Impersonate, and config-authored permissions as Custom.<name>), and adds a roles field naming the roles the identity carries. isAdmin stays, mirroring Core.AdminAccess in globalActions. Both lists are filled for an admin as well, where the old fields came back empty for one. External API clients must update to the new field names and identifier format.
  • Global permission names are matched case-insensitively throughout: they appear in /api/user canonicalized to lowercase, and /api/settings normalizes tabs' requiredPermission the same way. A client comparing these against the config-authored casing must switch to a case-insensitive comparison (the Designer frontend already does).
  • ScenarioTestCapabilities (POST /api/scenarioTesting/{scenarioName}/capabilities) gained testWithProvidedData, for running a test on records the caller supplies - it needs neither a parameter form nor the right to read a source, which neither existing entry expressed. testWithLiveData now reports the full Test + ViewLiveData pair, so a caller holding one but not the other is told beforehand instead of being refused on the following request.

Code compiled against the API

  • AdminUser, CommonUser and RulesSet are removed. LoggedUser is now a pure identity carrying no permission data: RealLoggedUser(id, username, roles), ImpersonatedUser, or the internal SystemUser. LoggedUser.can(category, permission), CommonUser.categories/categoryPermissions and RulesSet.permissions are gone with them - authorization decisions go entirely through the new AuthorizationService: check(action, scope) / checkGlobal / checkInCategory for a single decision, and visibleCategories for enumeration.
  • AuthenticationProvider.createAuthenticationResources gained a third parameter, infrastructure: SecurityInfrastructure - any custom AuthenticationProvider implementation must update its signature accordingly.
  • AuthenticationConfiguration no longer declares isAdminImpersonationPossible nor rules - the authentication.isAdminImpersonationPossible config key and the users file's rules section keep working (the Designer reads them itself), but a custom AuthenticationConfiguration overriding those members must drop the overrides.
  • NussknackerServicesForCustomHttpService now carries authorizationService: AuthorizationService, actionCatalog: ActionCatalog and clock: java.time.Clock - custom HTTP service providers should use authorizationService for authorization decisions instead of interpreting LoggedUser themselves. Code constructing this class must supply all three new fields.
  • CustomHttpServiceProviderFactory declares providedActions: List[ResourceAction], the actions its endpoints check - every factory has to implement it, with List.empty where there are none. Resource names may not collide across extensions, and the Designer refuses to start if they do.
  • An authorization backend other than the users file is supplied by implementing AuthorizationRules and registering an AuthorizationRulesFactory for the ServiceLoader. Its create receives the installation's ActionCatalog, assembled from those declarations before the authorization service exists and fixed for the life of the process. Every action the backend evaluates has to be in actionUniverse, which it builds from that catalog: one missing from it is denied under impersonation and absent from /api/user, whatever check() answers.

New CanonicalNode subtype for per-node error-recovery edges​

  • !359 The per-node error-recovery output edge adds a new canonicalnode.FlatNodeWithError(data, nextError) case to the sealed CanonicalNode hierarchy. Any code that pattern-matches exhaustively over CanonicalNode without a default case (for example a custom ProcessMigration that rewrites the canonical graph) must add a branch for FlatNodeWithError; otherwise it will fail to compile with match may not be exhaustive once built against this version.

Module Federation scope of the components frontend renamed​

  • The container scope of the components frontend was renamed from nu_components to nuComponents, so that it is derived from the package name like every other remote. The default componentsAppUrl / scenariosAppUrl were updated accordingly. No action is required for existing configs: a tab url still using the old scope is mapped to the current one by the Designer frontend. If you override these urls (directly in your config or through the NU_FE_COMPONENTS_URL / NU_FE_SCENARIOS_URL environment variables), update the scope - nu_components/list@... to nuComponents/list@... and nu_components/scenarios@... to nuComponents/scenarios@... - because that mapping rewrites the whole url to the standard /static/components/remoteEntry.js mount, dropping a custom location.

Flink state will be incompatible because of multiple changes that involve state serialization, main of which are:

  • updated Kryo serialization library in Flink 2.0
  • changes in typing for lists, maps and sets in Flink 2.0
  • various code changes in Nussknacker

Refer to Flink release notes for detailed Flink API changes.

Request-Response interaction (pause/resume) support​

  • Request-Response scenarios can now pause at an interaction node and be resumed later via a new /resume endpoint, secured the same way as the main invocation endpoint (if request-response.security.basicAuth is configured, /resume requires it too - a valid continuationId alone is not enough). A scenario with one or more interaction nodes now compiles into several interpreters internally (one for the main flow, one per resume point); this is transparent to scenario authors but is worth knowing when reasoning about deployment/compilation time or per-node metrics for such scenarios.
  • New request-response.continuationMaxCacheSize configuration entry (default 100000) bounds the number of paused continuations kept in memory at once, independent of continuationTtl - a burst of interactions can no longer grow the cache unboundedly before entries expire. Tune it down for memory-constrained deployments, or up if continuations legitimately spike higher.
  • Metrics for a scenario using interaction nodes are not scenario-version-scoped (a redeploy continues the same metric series rather than starting a new one), but a resume interpreter's metrics carry an additional resumeInteractionNodeId tag so they don't collide with the main flow's metrics for the same node id.
  • A metric registered by more than one live interpreter of the same scenario - which happens while a redeployed version is still draining - is now kept until every one of them has been closed, and a gauge registered by the newer interpreter takes over from the draining one's. Registering the same gauge name twice from a single interpreter is still an error, as before.
  • When the paused-continuation cache is full, the dropped continuation is logged at WARN by the runtime (naming the scenario, the interaction node and the continuationId), and a later /resume call for it responds 400 with a message listing why a continuationId may not be resumable (never issued, already resumed, expired, or dropped because the cache was full). The same 400 replaces the previous, terser "Unknown or invalid continuationId" message for every unrecognised continuationId.
  • A continuationId is consumed atomically, so two /resume calls with the same one - including two arriving concurrently - resume it at most once: exactly one of them runs the scenario, the other gets a 400 saying the continuation is no longer resumable. Treat /resume as non-idempotent: retrying it after a timeout may find the continuation already consumed rather than re-running it.
  • An environment that does not want pause/resume at all can hide the component instead of relying on scenario authors, by disabling it in that environment's model configuration: componentsUiConfig.interaction.disabled: true.
  • Resume support differs by engine - it is not equally reliable everywhere:
    • Embedded engine (EmbeddedDeploymentManager / RequestResponseDeploymentStrategy) - fully supported. New request-response.continuationTtl configuration entry (default 5 minutes) bounds both how long a continuationId stays resumable and how long a redeployed or stopped scenario version keeps its /resume endpoint alive for already-paused continuations ("draining") before being fully shut down. The main invocation endpoint (/) stops accepting new requests as soon as a scenario is redeployed or stopped - only /resume is kept alive, for up to continuationTtl. Draining applies only to scenarios that actually contain an interaction node: one without any resume point is torn down immediately on cancel/redeploy, exactly as before, rather than being held for continuationTtl. A redeploy drains correctly whether or not it changes the scenario version - every live version of a scenario serves the same /resume path, and a continuationId is matched against every live interpreter for the scenario (it is an unguessable, single-use UUID, so the caller does not need to say which version issued it).
    • Kubernetes (lite-k8s) - no draining, and resume is reliable only with a single replica. There is no equivalent of the embedded engine's draining window: cancelling a scenario tears down its Deployment/Service immediately (K8sDeploymentManager.cancelScenario), with no grace period for in-flight continuations, so any /resume call after that fails once the pods terminate. Redeploying instead relies on plain Kubernetes rolling-update behavior: the old pod keeps serving both / and /resume until Kubernetes sends it SIGTERM (governed by terminationGracePeriodSeconds, 30s by default; there is no PreStop hook delaying shutdown for pending resumes), so a continuation not resumed within that window is lost regardless of continuationTtl. Separately, the paused-continuation store is per-pod and in-memory and the Service has no session affinity, so with more than one replica /resume calls routed to a different pod than the one that issued the continuationId fail unpredictably - deploy scenarios using interaction nodes with a fixed replica count of 1 (e.g. via FixedReplicasCountK8sScalingOptionsDeterminer; not enforced automatically, but the designer logs a warning when deployed with more than one replica). Treat /resume on k8s as best-effort around deploys and cancels until graceful draining is implemented there.

Request-Response collect is now a join and union was removed​

  • !386 In the Request-Response processing mode, collect changed from a single-input component into a join, and union is no longer available there. An automatic migration replaces the union + collect shape with a single collect join. If a scenario still fails to compile after the upgrade, replace union + collect with a single collect join manually.
  • union itself is unchanged in the Lite streaming (Kafka) and Flink processing modes; only its availability in Request-Response was removed.

Request-Response sources support HTTP methods​

  • !500 The request source has a new method parameter (POST, GET, PUT, DELETE, PATCH), defaulting to POST. Existing scenarios keep POST and the slug stays a scenario property, so no scenario migration is needed.
  • !640 Request-response: queryParameters now works with every method - for body methods the values are merged into #input (names must not collide with inputSchema properties); new scenarios start with an empty-object inputSchema. Code API: slug helpers moved to the new RequestResponseSlug (common-api), the body/query-params source traits are replaced by RequestResponseSource.parse(IncomingRequest) (a former query-params source declares method = Get itself), OpenApiSourceDefinition carries bodySchema/queryParametersSchema.
  • GET and DELETE take their input from query parameters; POST, PUT and PATCH read a JSON body. RFC 9110 leaves content in GET and DELETE requests without defined semantics and intermediaries are free to drop it, so a body is not read for them.
  • For GET and DELETE the input is described by a new queryParameters parameter on the request source instead of the inputSchema scenario property, because only the source knows the method. It accepts a flat object of scalars and lists of scalars - nested objects, lists of objects and nullable or union-typed fields are rejected when the scenario is saved, since a query string cannot express them.
  • Custom Request-Response sources: RequestResponsePostSource[T] was renamed to RequestResponseBodySource[T] and RequestResponseGetSource[T] to RequestResponseQueryParamsSource[T] (whose parse now returns Request[T] instead of Request[Any]). Sources now expose def method: RequestResponseHttpMethod (defaults to POST).
  • A custom source serving a method other than POST must also expose it as a method node parameter holding a string literal (as the built-in request source does), because everything that documents the endpoint - the OpenAPI definition and the Designer's Integration section - reads the method off the scenario graph. A source whose def method disagrees with the graph declaration fails at scenario start with an explicit error, since the generated documentation would otherwise describe a method the endpoint does not serve.
  • A Request-Response scenario's effective slug (the explicit slug, or the one derived from the scenario name) must now be unique among non-archived scenarios of the same processing type. Saving or migrating a scenario whose slug is already taken is rejected; existing scenarios with duplicate slugs stay loadable, but their next save requires a unique slug. Previously two scenarios could hold the same slug as long as they were not deployed at the same time.
  • !112 !135 !435 !600 #7553 The evictable state API was reworked and moved from pl.touk.nussknacker.engine.flink.api.state to pl.touk.nussknacker.engine.flink.api, so every import of it has to be updated. Custom Flink components extending it need the following adaptations:
    • LatelyEvictableStateFunction is no longer a class but a trait, and it no longer implements onTimer. The abstract classes of the family - EvictableStateFunction, LatelyEvictableStateCoFunction and the previous LatelyEvictableStateFunction - are gone, together with StateHolder and LatelyEvictableStateFunctionMixin, whose members live on across the trait ladder below. What is left is a ladder of traits: LatelyEvictableFunction (the eviction deadline and its single timer per key), LatelyEvictableStateFunction[StateType] (the above plus one ValueState), LatelyEvictableKeyedProcessFunction[Key, In, Out] (the eviction machinery on a KeyedProcessFunction base, with onTimer wired for you) and LatelyEvictableStateKeyedProcessFunction[Key, In, Out, StateType] (all of the above combined). They live in LatelyEvictableFunction.scala, previously EvictableState.scala. A component extending EvictableStateFunction also moves to the combining trait - mind that this is not behaviour-preserving: the old moveEvictionTime registered a timer on every call and overwrote the recorded eviction time unconditionally, moving it backwards included, while moveEvictionDeadlineTo only moves the deadline forward and registers a timer only when none is pending. With out-of-order events the state now lives until the latest deadline rather than the last-written one.
    • A component that used to extend LatelyEvictableStateFunction[In, Out, StateType] (the key was hardcoded to String) extends the combining trait instead, which is the closest equivalent: 
      class MyFunction extends LatelyEvictableStateKeyedProcessFunction[Key, In, Out, StateType] {
        // stateDescriptor and processElement unchanged
      }
    • A component whose Flink base is not a KeyedProcessFunction - a CoProcessFunction, say - mixes the granular traits into that base and wires the timer by hand: 
      class MyCoFunction
          extends CoProcessFunction[In1, In2, Out]
          with LatelyEvictableStateFunction[StateType] {

        override def onTimer(timestamp: Long, ctx: CoProcessFunction[In1, In2, Out]#OnTimerContext, out: Collector[Out]): Unit =
          handleOnTimer(timestamp, ctx.timerService())

        // stateDescriptor and processElement1 / processElement2 unchanged
      }
      Omitting that onTimer compiles and fails silently - CoProcessFunction.onTimer (like KeyedProcessFunction.onTimer) is a concrete no-op, so eviction timers fire and evict nothing, and the state grows without bound. This is why the KeyedProcessFunction case has a trait that wires it for you.
    • Timer registration and current-time resolution live in the new WithTimeMode and KeyedProcessFunctionWithTimeMode[Key, In, Out] traits, which the ladder mixes in. KeyedProcessFunctionWithTimeMode declares protected type FlinkCtx and protected type FlinkTimerCtx; a class declaring its own alias under either of those names needs the override modifier. LatelyEvictableStateKeyedProcessFunction carries those aliases too, since it mixes that trait in.
    • moveEvictionTime(offset, ctx) is gone; call moveEvictionDeadlineTo(currentTime(ctx) + offset, ctx.timerService()), which takes an absolute deadline rather than an offset. In a function without KeyedProcessFunctionWithTimeMode (the CoProcessFunction case above) there is no currentTime - use ctx.timestamp() + offset, which is the same thing in the default event-time mode. doMoveEvictionTime(time, timeService) became moveEvictionDeadlineTo(deadlineTimestamp, timerService); only the name changed.
    • Eviction timers are registered according to def timeMode (EventTime / ProcessingTime), which defaults to EventTime, so a component that does not override it keeps its previous event-time eviction. Override it to evict in processing time - and then compute deadlines from currentTime(ctx) rather than ctx.timestamp(), so the deadline and the timer live in the same time domain.
    • evictStates() was replaced by clearAllState(), and clearing the eviction deadline moved to the caller. LatelyEvictableStateFunction implements it as a final method - state.clear() followed by the new clearAdditionalState() hook - so a subclass that used to override evictStates() to drop extra state on top of the base state (like a counter of its own) overrides clearAdditionalState() instead, with no super call to remember. On the bare LatelyEvictableFunction the method is abstract: a component built directly on that rung is forced by the compiler to say what eviction drops.
    • clearStateOlderThan(timestamp) - dropping the part of the state that a fired timer made unreachable - defaults to a no-op, which is what a state that lives and dies with the deadline needs. A state holding per-timestamp entries (the aggregator's buckets, say) overrides it and derives its own cutoff.
    • readState() was removed - read the state directly with state.value().
    • latestEvictionTimeForKey is no longer protected. Read the current deadline with protected def evictionDeadline: Option[Long]; move it with moveEvictionDeadlineTo.
    • AggregatorFunctionMixin[MapT] became BucketedAggregatorFunction: no type parameter, no StateHolder self-type, and it no longer evicts - it manages the aggregate buckets and nothing else, on a bare RichFunction. The buckets live behind the protected bucketsState (a BucketsState wrapping a MapState with the buckets and a ValueState with the sorted list of bucket timestamps), set up by BucketedAggregatorFunction.open, so an extending class only chains super.open(openContext). This state layout change is savepoint-incompatible for aggregates, as its Changelog entry notes. Its members changed with it:
      • the implicit rangeMap: FlinkRangeMap[MapT] member is gone, and FlinkRangeMap itself was deleted;
      • addElementToState returns Unit instead of the new map and gained a [K <: AnyRef] type parameter;
      • computeFinalValue(newState, timestamp) became computeFinalValue(timestamp, keys);
      • minimalResolutionMs is a val derived from the window length instead of a fixed def = 60000;
      • AggregatorFunction itself lost [MapT] and the implicit parameter list, gained nodeName, and takes KeyedValue[AnyRef, AnyRef] instead of StringKeyedValue.
    • Eviction of the buckets comes in three shapes:
      • LatelyEvictableKeyedProcessFunction combined with LatelyEvictableBucketedAggregatorFunction - deadline-driven eviction with onTimer wired for you (AggregatorFunction, FullOuterJoinAggregatorFunction);
      • LatelyEvictableBucketedAggregatorFunction on a hand-picked base with a hand-written onTimer calling handleOnTimer - when the timer also does other work or the base is not a KeyedProcessFunction (EmitWhenEventLeftAggregatorFunction, SingleSideJoinAggregatorFunction);
      • the bare BucketedAggregatorFunction - when the function drives eviction entirely from its own onTimer via clearBucketsOlderThan / bucketsState.clear() (EmitExtraWindowWhenNoDataTumblingAggregatorFunction).
  • !600 The traits that hook into a Flink function's lifecycle - LazyParameterInterpreterFunction, SingleLazyParameterFunction, LatelyEvictableFunction, LatelyEvictableStateFunction, BucketedAggregatorFunction, StatefulFunction - declare open (and close, where they have one) as abstract override and chain to super. LazyParameterInterpreterFunction takes RichFunction as a supertype rather than a self-type, which is what makes that chaining possible. The open/close of LazyParameterInterpreterFunction and of the evictable-state family used to be terminating overrides, so a component mixing both families had to invoke each of them by name.
    • A component that combined them like this: 
      override def open(openContext: OpenContext): Unit = {
        super[LatelyEvictableStateFunction].open(openContext)
        super[LazyParameterInterpreterFunction].open(openContext)
      }
      now writes super.open(openContext) once, and an open/close override that did nothing else can be dropped entirely. The qualified form keeps compiling, but it now runs the rest of the chain twice - replace it rather than leave it.
    • A component that overrides open or close and does not call super no longer initialises (or releases) what these traits own. That was already true, but it is now the single thing to get right.
    • abstract override requires a concrete open/close from a base class in the linearization. Mixing one of these traits into a class extending AbstractRichFunction (so KeyedProcessFunction, CoProcessFunction, RichFlatMapFunction, ...) satisfies that - Flink 2.x leaves RichFunction.open/close abstract and implements them in AbstractRichFunction. Intermediate traits still compile; a concrete class that mixes one of these traits without such a base fails with class X needs to be a mixin. ... is marked 'abstract' and 'override', but no concrete implementation could be found in a base class.
  • OneParamLazyParameterFunction was renamed to SingleLazyParameterFunction, and AbstractOneParamLazyParameterFunction to AbstractSingleLazyParameterFunction. Apart from the open/close changes described above, signatures are unchanged.

Named outputs for custom components; deduplication rejected output​

!625 Custom components declare their outputs as one ordered list (head = main output); deduplication declares passed and rejected. A custom node of such a component is stored in canonical JSON with an outputs field and connects only through named CustomNodeOutput edges. Migration 502 (CustomNodeOutputsMigration, parameterized so other components can reuse it) rewrites stored deduplication scenarios automatically; a setup replacing DefaultModelMigrations, or a component gaining an additional output after release, needs an equivalent migration - without it the stored scenarios stop validating. An OLDER Nussknacker importing canonical JSON in the new shape silently drops everything downstream of such a node (ScenarioGraph payloads fail loudly instead) - only exchange such scenarios with versions supporting this feature. Every custom node now registers a dead_end meter, counting records that leave through an unwired main output.

Configuration changes​

  • !645 OpenAPI enricher: allowedMethods changed from a list of strings defaulting to ["GET"] to a typed setting that defaults to all methods.

    • An installation that does not set allowedMethods now exposes every operation the definition declares, not only GET. Add allowedMethods: ["GET"] to keep the previous behaviour.
    • This removes the idempotency guard the old default provided: a definition containing DELETE or POST operations now produces such enrichers by default, and an at-least-once engine can re-invoke them when a scenario is replayed.
    • Method names are now validated and read case-insensitively. An unknown name (for example ["GTE"]) or an empty list now fails at startup; both were silently accepted before, and both reduced the component to zero services.

  • !413 OpenAPI enricher: the codesToInterpretAsEmpty configuration option was removed. 404 responses yield an empty (null) result under the default Error Strategy (Accept 2xx and 404); other status codes previously configured there are no longer interpreted as empty responses.

  • #7181 #7620 Added designer configuration: stickyNotesSettings

    • maxContentLength - max length of a sticky notes content (characters)
    • maxNotesCount - optional, max count of sticky notes inside one scenario/fragment (if not set the amount of sticky notes in scenario is not limited)
    • enabled - if set to false stickyNotes feature is disabled, stickyNotes cant be created, they are also not loaded to graph

  • #7534 shouldVerifyBeforeDeploy configuration entry available for Flink deployment was renamed to scenarioStateVerification.enabled

  • #6860 Configuration entry kafkaEspProperties.forceLatestRead is replaced with kafkaEspProperties.defaultOffsetResetStrategy:

    • forceLatestRead is missing -> keep defaultOffsetResetStrategy missing or set to "None"
    • forceLatestRead: false -> defaultOffsetResetStrategy: "None"
    • forceLatestRead: true -> defaultOffsetResetStrategy: "ToLatest"

  • #7568 The process-json button in processToolbarConfig was renamed to process-export

  • #7590 All akka.* configuration options and AKKA_* and Akka environment variables were replaced with their pekko.* and PEKKO_* counterparts - it's enough to do a quick search and replace

  • #7693 Configuration entry secondaryEnvironment.remoteConfig.uri is moved to secondaryEnvironment.uri.

  • #8042 When adding multiple OpenAPI components, ComponentPrefix should be added to distinguish the components. In the new version of the component, OpenAPI Service should be selected. For this reason, the existing scenario may not compile. The automatic migration for this parameter is not easy and has to be done manually or through updates in the DB if you can read the swagger service name.

  • #8116 Improved Kafka metadata caching

    • Dependencies for TopicSelectionStrategy (SchemaRegistryClient, and KafkaConfig) should be passed via constructor.
    • kafkaConfig.topicsExistenceValidationConfig.validatorConfig and kafkaConfig.topicsWithoutSchemaFetchTimeout were moved to common Kafka admin configuration kafkaConfig.kafkaAdminConfig - see Kafka integration for more details. The properties were moved as follows:
      • topicsExistenceValidationConfig.validatorConfig.autoCreateFlagFetchCacheTtl -> kafkaAdminConfig.cacheConfig.autoCreateTopicSettingExpirationTime
      • topicsExistenceValidationConfig.validatorConfig.topicsFetchCacheTtl -> kafkaAdminConfig.cacheConfig.topicsExpirationTime
      • topicsExistenceValidationConfig.validatorConfig.adminClientTimeout -> kafkaAdminConfig.clientTimeout
      • topicsWithoutSchemaFetchTimeout -> kafkaAdminConfig.clientTimeout

  • #8523 Kafka components configuration changes:

    • Configuration of Kafka components via legacy modelConfig.kafka entry is not supported for Flink engine. Use modelConfig.components.kafka.config instead.
    • Configuration of Kafka bootstrap servers via legacy kafkaAddress is not supported anymore. Use kafkaProperties."bootstrap.servers" instead

  • #8788 globalParameters.explicitUidInStatefulOperators option was removed - now we always set uid to node id in stateful operators

  • #8842!13 avroKryoGenericRecordSchemaIdSerialization Avro optimization option in Kafka source configuration was replaced with a more capable optimizedGenericRecordSerialization, which can be used with multiple Schema Registry instances:

    • optimizedGenericRecordSerialization.enabled - same as previous avroKryoGenericRecordSchemaIdSerialization boolean flag
    • optimizedGenericRecordSerialization.schemaRegistryId - Schema Registry instance identifier - each distinct SR environment should get an explicit ID number; defaults to a hash of SR URL, which will make Flink state incompatible if the address changes

  • #9037 Changes around Table-API based Component Providers:

    • Component Provider for Table Source and Table Sink - rename from flinkTableDataSource to flinkTableIO
    • Component Provider for Join and Aggregate components
      • renamed from flinkTableOps to flinkBatchOps
      • is no longer loaded by default - no need to disable it for streaming configurations, and for batch configurations, it can be enabled by components.flinkBatchOps.disabled: false config under modelConfig

  • #9231 OpenAPI component supports new optional cache configuration:

    • cacheFileLocation - path to persisted cache with last valid OpenAPI definitions.
    • Discovery refresh now uses stale-on-error fallback. If refresh fails, cached definitions are used from memory and, when configured, from file cache.
    • Configuration key openApiServicesDiscoveryCacheTtl was renamed to discoveryCacheTtl.

  • !824 The built-in component groups now have default display names, applied without any componentsGroupMapping configuration. Previously the raw group names were used, so an installation which did not configure componentsGroupMapping will see the toolbox panels renamed accordingly.

  • Some built-in components have been moved to a different component group, so they appear in a different toolbox panel (and are reported under a different componentGroupName by /api/components) than before:

    • for-each, union, decision-table and collect were moved from custom / optionalEndingCustom to base,
    • raise-error and logger were moved from services to miscellaneous,
    • http was moved from enrichers to services,
    • delay, previousValue, deduplication, state-transformer, union-memo, aggregate-sliding, aggregate-tumbling, aggregate-session, single-side-join, full-outer-join, the Table API aggregate and join, flink-sql-query, debounce and throttle were moved from custom / optionalEndingCustom to the new statefulTransformers group.

    If you referred to these components' groups - for example in componentsGroupMapping, or in a componentGroup override in componentsUiConfig - adjust the configuration to the new groups.

  • The Components tab filters by the displayed group name, so its GROUP query parameter changed accordingly (/components?GROUP=base is now /components?GROUP=Basic+Components). Bookmarked links using the old names stop matching.

REST API changes​

  • #7563 ProcessAction.buildInfo field was renamed to ProcessAction.modelInfo and is optional now. ProcessAction type is used in ScenarioWithDetails.lastDeployedAction, ScenarioWithDetails.lastStateAction and ScenarioWithDetails.lastAction which are returned by /processes, /processesDetails endpoints. It is also used by /components/$id/usages endpoint
  • #6860 #7562
    • Deploy and cancel http request requires valid json in request body (see DeployRequest and CancelRequest) instead of plain text, e.g. {"comment": "example text"}.
    • For KafkaFlinkSource it is possible to provide optional deployment parameter, e.g. {"comment": "example text", "nodesDeploymentData": {"my_source_node_id": {"offsetResetStrategy": "ToLatest"}}}.
  • #7658 /app/processingtype/reload endpoint doesn't reload Deployment Managers
  • #7693 Scenario migration uses MigrateScenarioRequestDtoV3 where remoteUserName is removed. To provide username use impersonation mechanism.
  • #7871 CustomHttpServiceProvider providing Pekko route was renamed to PekkoCustomHttpServiceProvider.
  • #7959#9012 Scenario testing API changes:
    • Scenario testing API (on path prefix /scenarioTesting) is refactored, with modified endpoint paths and request/response format
    • endpoints moved from /processManagement API to /scenarioTesting API
      • /test (test with form)
      • /generateAndTest (test with generated data) and merged into a single endpoints /performTest
      • /test to /performTest
      • /testCase to /performTestCase * introduced new representation of test results, grouped per transition between nodes (toggleable in on API request level)
  • #8997 Node IDs were changed to technical UUIDs and separated from user-facing node names:
    • Existing scenarios are automatically migrated: previous human-readable node id is moved to node name, and a new UUID is generated as node id.
    • In scenario JSON and HTTP DTOs, node id is now technical and stable (UUID), while node name is user-editable and should be used for UI-facing text.
    • ValidationResult response contains nodeNames map (nodeId -> nodeName) to allow clients to render readable node labels.
    • KafkaExceptionInfo payload now contains nodeName, and existing nodeId field now contains technical UUID values (not user labels).
    • Links/query params that point to nodes by id (e.g. visualization URLs with nodeId=...) now require technical UUIDs. Old links based on former human-readable ids stop matching nodes.
    • Join nodes' branchParameters[].branchId in the scenario graph (designer/REST DTO) now also uses the predecessor node UUID, consistent with edges (edge.from); the persisted CanonicalProcess keeps the node name there, so no scenario migration is required.
  • !625 New EdgeType.CustomNodeOutput(name: String) discriminator (JSON "type": "CustomNodeOutput"): every outgoing edge of a node whose component declares additional outputs is persisted with it, and UINodeEdges lists the declared outputs (main first) for such components. Clients decoding EdgeType exhaustively must handle the new case; an unrecognized discriminator in a ScenarioGraph payload fails decoding loudly.
  • !878 The whole /api/deployments API was removed and now returns 404. Use the scenario management endpoints instead

Code API changes​

  • !680 ModelClassLoader now exposes classPath: ModelClassPath, which keeps, for every configured modelConfig.classPath entry, the URL it resolved to and the URLs it expanded to; urls is unchanged and still returns the flattened list. Consequently FlinkModelJarProvider takes a ModelClassPath and an Option[ModelArtifactConfig] instead of List[URL] - use ModelClassPath.fromUrls(urls) to keep the previous behaviour, or pass modelClassLoader.classPath to be able to skip cluster-provided entries, as configured via deploymentConfig.modelArtifact (see the Changelog). FlinkClient.runProgram takes an additional flinkConfiguration: Map[String, String] parameter - pass Map.empty to keep the previous behaviour. The same two values are now constructor parameters of FlinkDeploymentManager and RemoteFlinkScenarioJobRunner, and arguments of FlinkScheduledExecutionPerformer.create - pass new FlinkModelJarProvider(modelDataProvider.modelClassLoader.classPath) and Map.empty to keep the previous behaviour.
  • !689 Category is now typed as CategoryId (pl.touk.nussknacker.engine.api.CategoryId, wraps the id String) instead of a raw String, in LoggedUser/CommonUser (security), WithCategories/ProcessingTypeConfig/ProcessMigration.migrateProcess (extensions-api), ScenarioService (custom-http-service-api), and ScenarioWithDetailsForMigrations (restmodel). Wrap with CategoryId(...) / unwrap with .value at call sites.
  • request-response-runtime: the routes serving one scenario were reorganised for the interaction feature, which is source-breaking for code building request-response routes itself (ScenarioDeploymentExtension implementations and similar). ScenarioRouteHelpers.resumeRoute takes an extra continuationTtl argument and now answers an unknown continuationId itself instead of rejecting, so the separate ScenarioRouteHelpers.unknownContinuationIdRoute is gone. On ScenarioRoute, resumeRoutes, unknownContinuationIdRoute, routeWithoutUnknownContinuationIdFallback, drainingRoute and combinedDrainingRoute were replaced by combinedRoute (a whole scenario version), invocationAndDefinitionRoutes (everything but /resume) and ScenarioRoute.resumeRoute (one /resume route shared by every interpreter mounted under a scenario slug, across all its live versions - a single route is required, because the request entity can only be read once). ScenarioRoute no longer takes a scenarioVersionId: the resume endpoint is not version-scoped.
  • request-response-components-api: new ContinuationNotResumableException, thrown when a /resume reaches the interpreter but the continuation has been consumed or has expired in the meantime; the runtime answers it with 400 rather than 500. Code matching on the previous IllegalArgumentException from that path needs updating.
  • scenario-api: CustomNode and Enricher gained a new optional field, mockExpression: Option[Expression] = None, used to inline a fixed value in place of the node's real (side-effecting) evaluation - currently used to mock an interaction node's resumed input during ad-hoc scenario testing. The field defaults to None, so existing scenario JSON and code constructing these case classes without it keep working unchanged.
  • !625 Named outputs for custom components: CustomStreamTransformer gained def outputs: NonEmptyList[ComponentOutput] (head = main output; the default keeps subclasses source-compatible), and ComponentOutput wraps a String refined as NonEmpty And Trimmed (ComponentOutput.Name), so a name may be neither empty nor padded with whitespace. CustomComponentSpecificData gained a third field (outputs), so a positional pattern match needs an extra _. Sealed CanonicalNode, SubsequentNode and EdgeType each gained a case for the multi-output shape (CustomNodeWithOutputs / CustomNodeOutput), so exhaustive matches - e.g. a custom ProcessMigration - must be extended. New save-blocking ProcessCompilationError cases: UnknownCustomNodeOutput, MissingCustomNodeOutputName, UnsupportedEdgeNextToCustomNodeOutputs, DuplicateCustomNodeOutputNames and MultipleOutputsToSameJoin; additionally a node other than split may now have at most one unnamed outgoing edge (previously such an API-submitted graph saved and got spliced into one chain). It is now rejected with NonUniqueEdgeType("main output", ...).
    • splittedgraph.part.CustomNodePart was reshaped into a sealed trait with one case per shape: MultiOutputCustomNodePart(node, outputs: NonEmptyList[SplittedOutput]) for a node connected through named edges - every wired continuation, the main one included, lives in outputs (each SplittedOutput carries its own name and the nextParts/ends of its output; an output the scenario leaves unwired has no entry) and node is a bare EndingNode - and SingleOutputCustomNodePart(node, nextParts, ends) for the released single-output form, where the continuation stays embedded in node. The multi-output case computes the inherited nextParts/ends members as the concatenation of every output's, so generic consumers still see every downstream part, but construction and pattern matches against the old single class break. Its counterpart compiledgraph.part.CustomNodePart folds every output into one type - CustomNodePart(transformer, contextBefore, validationContext, outputs: NonEmptyList[CompiledOutput]) - where a CompiledOutput carries the output's identity (output: ComponentOutput) plus its own node, nextParts and ends. The list is built from the component declaration, so the head is always the main output (with a DeadEnd standing in when the scenario leaves it unwired) and the connected additional outputs follow in declaration order. node is no longer a field of the part: it is outputs.head.node, so it also leaves the extractor. nextParts/ends concatenate every output's, so positional construction and pattern matches break there too (scenario-compiler/.../splittedgraph/part.scala, scenario-compiler/.../compiledgraph/part.scala).
    • FlinkCustomStreamTransformation is unchanged for a component with only the main output: transform still returns a single DataStream[ValueWithContext[AnyRef]] and the FlinkCustomStreamTransformation(...) factories still take the same lambdas, so no such component needs migrating. A component with additional outputs implements the new, unrelated FlinkMultiOutputStreamTransformation, whose transform returns NonEmptyList[(ComponentOutput, DataStream[ValueWithContext[AnyRef]])] instead: the returned list restates the component's outputs declaration with a stream attached to each entry, keyed by the output it belongs to. Job registration looks the streams up by key, so their order does not matter. The main output always needs exactly one stream, whether or not the scenario wires it; an additional output needs one only when the scenario connects it, and the streams of the unconnected ones are never read. Omitting a stream that is needed, or returning two under the same key, fails job registration loudly. Its lambda form is FlinkMultiOutputStreamTransformation { (start, ctx) => ... }. A FlinkCustomStreamTransformation returns no key at all: registration wires its single stream to the main output whatever the component named it, so renaming the main output alone is no reason to reach for the multi-output trait. Registration fails there rather than letting an output the scenario wired silently emit nothing. Additional outputs are usually routed as Flink side outputs: FlinkCustomNodeContext.createOutputTag(output, typeInfo) gives a ready tag, the component emits under it and reads the stream back with getSideOutput on the emitting operator. typeInfo describes what that output emits and becomes the type of the stream getSideOutput returns, so pass the one the emitting operator uses for its own output, normally from FlinkCustomNodeContext.valueWithContextInfo; nothing verifies it, because OutputTag compares by name alone. That is one pass over the stream with each element going to exactly one output, and taking the tag from the same place on both ends keeps them identical. Nothing forces that shape though - any stream will do, so an output may have its own chain of operators. Every output has the same shape: the Context carried must be consistent with the node's declared output context (for typical components, the unchanged input context), and the accompanying value becomes the node's output variable on that output. A component with nothing to report on an output emits ValueWithContext(null, ctx), which means the output variable is in scope downstream of that output holding null rather than being absent.
    • Joins (canHaveManyInputs = true) cannot declare additional outputs - the combination is rejected when the component definition is loaded; duplicate declared output names are rejected the same way. Like other component definition errors, the rejection is an IllegalArgumentException thrown at model load, so one invalid declaration prevents the whole processing type from starting.
    • A component declaring any additional outputs (a non-default outputs tail) has to return an implementation carrying the new component.SupportsMultipleOutputs marker, which on Flink only FlinkMultiOutputStreamTransformation carries. If the returned implementation does not carry it - another runtime's, or a Flink FlinkCustomStreamTransformation, which structurally cannot route more than one output - a scenario that connects one of the declared additional outputs fails compilation with CustomNodeError, so the designer reports it on save instead of letting the deployment fail. Only connected additional outputs are checked: a scenario that wires none of them - even one that wires only the main output - keeps compiling, because an unconnected output costs it nothing at runtime. This is also what rejects such a scenario on Lite: additional outputs target the Flink streaming runtime for now, so no lite implementation carries the marker. Adding them to another engine means marking that runtime's own transformation type.
  • !405 schemed-kafka-components-utils: the parameter handling of UniversalKafkaSinkFactory was extracted to the new UniversalKafkaWriterFactory trait (shared with the request-response kafka service). Behavior is unchanged. KafkaUniversalComponentTransformer stays an abstract class (subclasses are unaffected), but its shared logic now lives in the KafkaUniversalComponentTransformerTrait it extends. The only source-breaking change: TransformationState moved from the UniversalKafkaSinkFactory companion object to the UniversalKafkaWriterFactory companion object.
  • !417 Request-Response: sinks can declare the HTTP status code and encoding of the scenario response.
    • The outputSchema scenario property can now be a map from HTTP status code to JSON Schema with a required default entry (e.g. {"default": {...}, "404": {...}}). The previous single-JSON-Schema format is still read as the default schema, so existing scenarios keep working without any migration.
    • New request-response components-api traits: SinkWithHttpStatusCode (its code typed as the new HttpStatusCode enum) and SinkWithResponseEncoder; the built-in response sink implements both, picking the encoder for its declared code and falling back to the default schema. ResponseRequestSinkImplFactory.createSink takes an extra statusCode: HttpStatusCode.
    • Lite components-api: ScenarioInterpreter has a new member sinks: Map[NodeId, Sink] that custom implementations must provide.
    • In the request-response runtime, invokeToOutput now returns RequestResponseOutput(results, statusCode, responseEncoder) instead of List[Any], getSchemaOutputProperty returns OutputSchemaMap instead of Json, and RequestResponseHttpHandler.invoke returns ScenarioResponse(body, statusCode) instead of Json.
  • #9399 pl.touk.nussknacker.ui.process.migrate.RemoteEnvironment changes, used to support comparing scenario versions against a remote (secondary) environment:
    • One new abstract method, versionsWithDifferences. Any external implementation of this trait must implement it.
    • versionsWithDifferences is backed by a new endpoint, POST /processes/{scenarioName}/versions-with-differences, which takes a scenario graph in the request body and answers with the subset of that environment's own versions that differ from it. The comparison deliberately runs on the environment that already stores those versions, so one graph is sent and a summary comes back, rather than full scenario graphs being pulled across the network to be compared locally. One request answers for as many of the scenario's most recent versions as the limit query parameter asks for (default 50, maximum 500) - the answering environment applies that limit before loading any graph, so it never walks its whole history, and reports oldestComparedVersionId when it stopped short. Version comments are returned separately, in versionComments, covering every version rather than only the compared ones - the answering environment reads them from its own activity list, so no second request is made for them. Comments recorded against scheduled or ad-hoc executions, and against deployments that failed, do not describe a version and are not included. The endpoint requires the Read permission on the scenario. Comparing scenario versions against a remote environment therefore requires that environment to run Nussknacker 1.19.0 or newer; against an older one the designer reports that the differences could not be determined.
    • processVersions now returns Future[RemoteScenarioVersions] instead of Future[List[ScenarioVersion]]. The new wrapper carries a remoteUnavailable flag, so implementations report "the remote environment could not be reached" without failing the Future - previously indistinguishable from "the remote has no such scenario". Implementations return RemoteScenarioVersions(versions, remoteUnavailable = false) for a successful fetch.
    • GET /remoteEnvironment/{scenarioName}/versions now responds 502 Bad Gateway when the remote environment is unreachable or rejects our credentials, instead of returning an empty list. A remote environment that simply does not have the scenario yet still returns an empty list.
    • GET /remoteEnvironment/{scenarioName}/versions now also requires the Read permission on the scenario and responds 403 Forbidden without it. Previously the endpoint had no permission check, so any authenticated user could read the remote environment's version history for any scenario; a client that relied on getting an empty list for an inaccessible scenario has to handle 403 now.
    • Comparing against a remote environment now fetches the remote scenario graph without validation and dictionary resolution, so both sides of the comparison are the raw stored graph. Scenarios using dictionaries previously reported differences on every node whose parameters referenced a dictionary.
    • GET /processes/{scenarioName} accepts a new optional skipScenarioGraph query parameter (default false). With it the response omits scenarioGraph, which is what processVersions uses when it needs only the version history. The parameter is additive - a client that does not send it, or an environment that does not recognise it, is unaffected.
  • !338 Flink components-api: keyed end-of-input flushing and restore re-pacing operators (package pl.touk.nussknacker.engine.flink.api.operator).
    • EndInputFlushableFunction[OUT] is now KeyedFlushFunction[K, OUT], and its flush callback gains the current key and a shorter name: flushPendingEventsForCurrentKey(output) → flushForCurrentKey(key: K, output). Implementations (a KeyedProcessFunction wired through OneInputFlushingKeyedOperator) must add the K type parameter and update the method signature.
    • New restore-able operators: KeyedRestoreFunction[K, OUT] plus RestorableKeyedOperator[K, IN, OUT] (restore-only) and RestorableOneInputFlushingKeyedOperator[K, IN, OUT] (end-of-input flushing + restore). A KeyedProcessFunction mixes in KeyedRestoreFunction to re-pace its per-key buffered events once after a state restore via restoreForCurrentKey(key, timerService); implementations that need no rescheduling (e.g. event time) make it a no-op.
  • !155 TransformStateTransformer was rewritten as a dynamic component (and registered as the new state-transformer component):
    • TransformStateFunction is now private[transformer] - it is an implementation detail and should not be referenced directly
    • the #previous variable was replaced by #current, the stateTimeoutSeconds: Long parameter was replaced by ttl: Duration, and a new mandatory initialValue parameter was added
  • !247 Script Processor added Expression.Language.Python as a new Expression.Language.ScriptLanguage. Code that pattern-matches exhaustively on Expression.Language or ScriptLanguage should handle this new case or use the existing _: ScriptLanguage branch.
  • !242 Built-in parameter validators: runtime validation, @NotNull/@NotEmpty support and naming cleanup.
    • Affected validators: NotNullValidator, NotBlankValidator, RegExpValidator, MinimalNumberValidator, MaximalNumberValidator, and the new NotEmptyCollectionValidator. Each now also implements RuntimeValidator; when attached to a parameter, the evaluated value is validated at runtime after expression evaluation, and a violation throws ParameterRuntimeValidationException (a NonTransientException). Compile-time behavior is unchanged.
    • Behavior change: validators auto-attached via annotations (@NotBlank to NotBlankValidator, @Min to MinimalNumberValidator, @Max to MaximalNumberValidator) now also reject invalid values at runtime. A scenario that previously passed a dynamic blank or out-of-range value through such a parameter at runtime will now fail with ParameterRuntimeValidationException.
    • New annotation mappings: @NotNull now attaches NotNullValidator and @NotEmpty attaches NotEmptyCollectionValidator. Both annotations were previously ignored, so a parameter carrying them gains new validation at both compile time and runtime, and a scenario that compiled before may now fail compilation or at runtime. @NotNull is attached in addition to the MandatoryExpressionValidator that non-optional parameters already receive (the two checks are intentionally distinct). NotEmptyCollectionValidator only checks java.util.Collection and java.util.Map (by design), so @NotEmpty on a String or array is not validated; use @NotBlank for string parameters.
    • NotBlankValidator, RegExpValidator, and NotEmptyCollectionValidator pass null through to allow validator chaining; combine them with NotNullValidator to also reject null.
    • Other built-in validators (MandatoryExpressionValidator, JsonExpressionValidator, FixedValuesExpressionValidator, LiteralIntegerExpressionValidator, MultiSelectFixedValuesExpressionValidator) remain compile-time only.
    • Built-in validators were renamed to reflect when they run: *ExpressionValidator runs at compile time on the expression, *ValueValidator at runtime on the evaluated value, *Validator at both. Renames (Code API change): MandatoryParameterValidator → MandatoryExpressionValidator, NotNullParameterValidator → NotNullValidator, NotBlankParameterValidator → NotBlankValidator, NotEmptyCollectionParameterValidator → NotEmptyCollectionValidator, RegExpParameterValidator → RegExpValidator, FixedValuesValidator → FixedValuesExpressionValidator, LiteralIntegerValidator → LiteralIntegerExpressionValidator, JsonValidator → JsonExpressionValidator, MultiSelectFixedValuesValidator → MultiSelectFixedValuesExpressionValidator. MinimalNumberValidator, MaximalNumberValidator, and CompileTimeEvaluableValueValidator keep their names. The JSON serialization format (the "type" field) is unchanged, so saved scenarios and configurations are not affected.
    • New CompileTimeAndRuntimeValidator trait (extends CompileTimeParameterValidator with RuntimeParameterValidator) is the base for validators that run at both stages: an implementation provides message, description, validate, and compileTimeError, while both isValid methods and a default runtimeError (which uses message) are inherited. All built-in dual validators extend it.
  • Runtime parameter validation: Validator, ParameterValidator, and CustomParameterValidator trait hierarchies refactored.
    • Validator is now a sealed trait with no isValid method. Two new sub-traits carry the validation logic:
      • CompileTimeValidator — the existing isValid(paramName, expression, value, label)(nodeId) returning Validated[PartSubGraphCompilationError, Unit]. All built-in validators implement this.
      • RuntimeValidator — new; isValid(paramName, expression, value: Any)(nodeId) returning Validated[ParameterRuntimeValidationError, Unit]. Called after expression evaluation at runtime.
    • CompileTimeParameterValidator extends ParameterValidator with CompileTimeValidator and RuntimeParameterValidator extends ParameterValidator with RuntimeValidator are the new intermediate traits.
    • CustomParameterValidator is now a sealed trait with only def name: String. It no longer extends Validator. Replace usages of the old CustomParameterValidator (which extended Validator) with:
      • CustomCompileTimeParameterValidator extends CustomParameterValidator with CompileTimeValidator — compile-time custom validation.
      • CustomRuntimeParameterValidator extends CustomParameterValidator with RuntimeValidator — runtime custom validation.
      • A class may extend both if it needs to validate at both stages.
    • CustomParameterValidatorDelegate is renamed to CustomParameterValidatorByNameLoader. JSON serialization format is unchanged — it still encodes/decodes as "type": "CustomParameterValidatorDelegate" with a "name" key.
    • CustomParameterValidatorLoader.resolved now returns ParameterValidator with WithUnderlyingCustomParameterValidator instead of CustomParameterValidator. Code that previously assigned resolved to a CustomParameterValidator variable must be updated.
    • New CustomParameterValidatorByClassLoader loads a CustomParameterValidator by fully-qualified class name (used by the @CustomValidator annotation). It serializes as "type": "CustomParameterValidatorDelegate" with a "className" key.
    • New @CustomValidator(classOf[MyValidator]) Java annotation in pl.touk.nussknacker.engine.api.validation attaches a CustomParameterValidator implementation to a parameter by class (alternative to the existing name-based CustomParameterValidatorByNameLoader).
    • ParameterValidator JSON codec rewritten manually (was @ConfiguredJsonCodec). ScenarioPropertyConfig also receives an explicit Decoder in place of @JsonCodec.
  • !316 Parameter labels are now represented by the ParameterLabel value object (pl.touk.nussknacker.engine.api.parameter.ParameterLabel) instead of a raw String.
    • Parameter.labelOpt is now Option[ParameterLabel] instead of Option[String]. Wrap raw strings with ParameterLabel(...) at construction sites. Parameter.label itself is removed, in favor of paramName.displayName.
    • There is no serialization/wire change - the designer-facing UIParameter.label stays a String.
  • !772 Compilation errors and the validation APIs a component calls now carry ParameterNameWithLabel (name plus labelOpt, rendered by displayName) instead of ParameterName, so a message can show the label the user sees. Pass a label wherever the component knows one - parameter.paramNameWithLabel if a Parameter is at hand.
    • Fields and parameters holding it were renamed paramName -> paramNameWithLabel. Named arguments and member reads must be updated; positional patterns still work.
    • ValidationContext.withVariableOverriden was renamed to withVariableOverridden.
    • ParameterNameWithLabel.apply(String) is gone - write ParameterNameWithLabel(ParameterName("myParam")).
    • withBranchId suffixes the name only - compare paramNameWithLabel.name, not a pre-built ParameterName("key for branch b1").
    • Validation message texts changed: identifiers are wrapped in single quotes, and the plain-text node endpoints name the parameter. Code matching on exact message text must be updated.
    • ParameterRuntimeValidationException gained a paramLabel: Option[ParameterLabel] field next to paramName, so code constructing it positionally must be updated. It can now be built from the whole identity with ParameterRuntimeValidationException(paramNameWithLabel, input, message).
    • ProcessCompilationError.UnknownProperty was removed. It had no producer left after #7757 dropped the unknown scenario property validation. A test source parameter with no value in the submitted test record is now reported as MissingParameters. Code matching on UnknownProperty must be updated.
  • #8719 Feature: Add possibility to pass trace id to Context
    • added optional traceId field at Context, default None
    • Fro now RequestResponseSource transforms Record[T] instead T as a record
  • #8723 Change: Passing Context instead of ContextId at EagerServiceWithStaticParametersAndReturnType.invoke
  • #7165
    • pl.touk.nussknacker.engine.api.deployment.DeploymentManager:
      • new command DMPerformSingleExecutionCommand, which must be handled in DeploymentManager.processCommand method (handle it the same as DMCustomActionCommand with actionName=run now)
      • added new arguments to def resolve and getProcessState methods (latestVersionId: VersionId, deployedVersionId: Option[VersionId]), which will be provided by Nu when invoking this method
    • pl.touk.nussknacker.engine.api.deployment.ProcessStateDefinitionManager:
      • added new arguments to def processState method (latestVersionId: VersionId, deployedVersionId: Option[VersionId])
      • added new methods with default implementations:
        • def visibleActions: List[ScenarioActionName] - allows to specify, which actions are applicable to scenario (and consequently should be visible in Designer), by default all previously available actions
        • def actionTooltips(processStatus: ProcessStatus): Map[ScenarioActionName, String] - allows to define custom tooltips for actions, if not defined the default is still used
      • modified method:
        • def statusActions(processStatus: ProcessStatus): List[ScenarioActionName] - changed argument, to include information about latest and deployed versions
  • #7379 Removed CustomAction mechanism. If there were any custom actions defined in some custom DeploymentManager implementation, they should be modified to use the predefined set of actions or otherwise replaced by custom links and handled outside Nussknacker.
  • #7368 #7502 Renamed PeriodicSourceFactory to EventGeneratorSourceFactory
  • #7364 The DeploymentManager must implement def schedulingSupport: SchedulingSupport. If support not added, then NoSchedulingSupport should be used.
  • #7511 Changes around flink-based scenario testing. As an entry point to all migration steps, assume that FlinkMiniClusterWithServices is a new FlinkMiniClusterHolder
    • From perspective of testkit (TestScenarioRunner.flinkBased) module usage
      • flink-tests module doesn't depend on flink-test-utils module. To create FlinkMiniClusterWithServices follow steps below. Example migration process is also available in PR with the related change.
        • FlinkSpec inheritance should be removed from test class
        • Test class should extend BeforeAndAfterAll
        • FlinkMiniClusterWithServices should be created using val flinkMiniClusterWithServices = FlinkMiniClusterFactory.createUnitTestsMiniClusterWithServices()
        • FlinkMiniClusterWithServices should be closed in afterAll block
    • From perspective of flink-test-utils module usage follow steps below. Example migration process is also available in PR with the related change. Caution: this module is deprecated; to avoid further migrations issues, tests should be rewritten to testkit stack
      • Instead of using FlinkSpec.flinkMiniCluster.createExecutionEnvironment method, should be used FlinkSpec.flinkMiniCluster.withDetachedStreamExecutionEnvironment which properly closes created environment
      • MiniClusterExecutionEnvironment class was removed, plain StreamExecutionEnvironment is returned instead
        • To access methods such as withJobRunning, import ScalatestMiniClusterJobStatusCheckingOps._ and then invoke these methods on flinkMiniCluster
        • Method withJobRunning was renamed to withRunningJob and it doesn't invoke StreamExecutionEnvironment.execute. It should be called before this method
        • Method executeAndWaitForFinished was renamed to waitForJobIsFinished and it doesn't invoke StreamExecutionEnvironment.execute. It should be called before this method; also, this method cancel job if check ended up with error now
        • Method assertJobNotFailing was renamed to checkJobIsNotFailing
        • Some methods are not available in ScalatestMiniClusterJobStatusCheckingOps:
          • executeAndWaitForStart, waitForStart, stopJob - should be used withRunningJob/waitForJobIsFinished or FlinkMiniClusterWithServices.miniCluster methods directly instead
          • Other methods were considered too much low-level and were removed
      • Instead of using ResultsCollectingListenerHolder.registerListener or ResultsCollectingListenerHolder.registerTestEngineListener should be used withListener/withTestEngineListener methods which properly cleanup allocated resources.
  • #7540 FlinkStreamingDeploymentManagerProvider was renamed to FlinkDeploymentManagerProvider, FlinkStreamingRestManager and FlinkRestManager abstraction layers were removed - only FlinkDeploymentManager exists
  • #7563 ProcessConfigCreator.buildInfo and NkGlobalParameters.buildInfo were renamed to modelInfo Also, they return a ModelInfo value class now. To create it from Map, use ModelInfo.fromMap. To access underlying map, use ModelInfo.parameters.
  • #7566 Scenario status and deployment statuses are decoupled now
    • Changes in DeploymentManager interface
      • DeploymentManager has only getScenarioDeploymentsStatuses method (previous getProcessStates returning List[StatusDetails]).
      • Method DeploymentManager.resolve should be removed - this work is done by Designer itself
      • DeploymentManagerInconsistentStateHandlerMixIn mixin should be also removed
      • stateQueryForAllScenariosSupport was renamed to deploymentsStatusesQueryForAllScenariosSupport
    • Other changes:
      • StatusDetails was renamed to DeploymentStatusDetails
      • Fields: externalDeploymentId, externalDeploymentId, attributes, attributes were removed from StatusDetails
  • #7658 DeploymentManager is not reloaded during processing type reload
    • DeploymentManagerProvider.createDeploymentManager() takes BaseModelDataProvider instead of BaseModelData
      • To access modelClassLoader use BaseModelDataProvider.modelClassLoader
      • To access other fields from ModelData, use BaseModelDataProvider.getCurrentModelData()
      • To create BaseModelDataProvider from ModelData use BaseModelData.toModelDataProvider extension method
  • #7598 Classes from pl.touk.nussknacker.engine.sttp package were moved to pl.touk.nussknacker.http.backend
  • #7586 Renamed ComponentUseCase to ComponentUseContext.
    • Moved NodeDeploymentData from FlinkCustomNodeContext to ComponentUseContext.LiveRuntime.
    • Introduced ComponentUseContextProvider in ScenarioCompiler, which retains ComponentUseCase functionality but is now intended for internal use only.
  • #7618 NonTransientException is a normal class instead of case class. We recommend to create explicit classes extending this class, but still it can be created directly but with new NonTransientException(...)
  • #7590 Akka and Akka HTTP were replaced with Apache Pekko 1.0.3 / Apache Pekko HTTP 1.0.1. These versions are API-compatible with previously used Akka versions, so after updating dependencies it should be enough to replace all akka.* imports with org.apache.pekko.* and change the akka prefix in configuration entries to pekko. More details can be found in:
  • #7632 Simplified parameter editors API.
    • Removed the single editor from the UIParameter and now only the editors property is available.
    • Replaced the editor parameter with editors in the engine API Parameter.
    • Similarly, ParameterConfig in engine API now accepts an editors list instead of a single editor.
    • Removed a DualEditor from the components API.
      • Renamed a SimpleEditor to the Editor in components API.
      • For now on, you can add multiple editors on a single param. The first editor annotation is treated as a default editor.
  • #7711 #7984 TypingResult API changes:
    • TypedNull.wihoutValue returns Unknown type instead of TypedNull
    • canBeConvertedTo renamed to canBeLooselyAssignedTo
    • canBeStrictlyConvertedTo renamed to canBeStrictlyAssignedTo
  • #7768
    • ModelData.withThisAsContextClassLoader was renamed to withModelClassloaderAsContextClassLoader
    • New, DeploymentManager.scenarioCompilationDependenciesResource method was added. For flink-based DMs it should be implemented as miniClusterWithServices.createDetachedStreamExecutionEnvironment[SyncIO].map(new FlinkScenarioCompilationDependencies(_))
  • #7824 Introduced Json display for Unknown type.
    • Unknown is a case class now and it has DisplayStrategy field. It affects how the type is displayed.
    • There is Unknown object which extends the case class with DefaultDisplayStrategy for compatibility.
    • When introducing new pattern matches regarding types Unknown(_) should be used so all display strategies are targeted.
  • #8054 ProcessObjectDependencies class was replaced by ModelConfig
  • #8256 The support for kafka source generating custom classes (KafkaSourceFactory) was removed We recommend a migration to more generic approach: UniversalKafkaSourceFactory
  • #8308 ToJsonEncoder refactor:
    • ToJsonEncoder was moved to pl.touk.nussknacker.engine.api.json.encoders package
    • ToJsonEncoder is a sealed trait, if you want to extend it, use either StrictToJsonEncoder (old failOnUnknown = true) or LooseToJsonEncoder (old failOnUnknown = false)
    • highPriority become a method instead of parameter
    • ToJsonEncoder.encode method now returns Validated and is available only for StrictToJsonEncoder
    • ToJsonEncoder.encodeUnsafe method behave as the previous ToJsonEncoder.encode (errors are thrown as Exceptions)
  • #8523 Kafka components configuration changes:
    • KafkaConfig was renamed to KafkaComponentsConfig
    • KafkaComponentsConfig.kafkaProperties is mandatory now
    • KafkaComponentsConfig.kafkaAddress was removed
  • #8576 NodeId value class become used more frequently in the API: NodeComponentInfo, ContextId, ProcessCompilationError, ProcessListener and FlinkCustomNodeContext
  • #8997 Node identity semantics changed in API modules:
    • NodeId should be treated as internal technical identity (UUID) and not shown to end users as a readable label.
    • NodeName is now the user-facing label; for user-visible errors/logs/messages use NodeName, optionally with NodeId only as technical details.
    • If your custom code (including integrations based on *-api modules) assumed nodeId == displayed node name, it must be updated to use both fields explicitly.
    • In places where you serialize/deserialize scenario-related HTTP payloads, ensure mappings preserve both id and name fields.
  • #8632 DynamicComponent's [T] generic parameter was replaced with Implementation type We recommend to not invoke DynamicComponent.implemenation() method directly. To check how component behave during compilation, use TestNodeCompiler instead. Read Testing for more info.
  • #8781 WithExceptionHandler helper trait was removed because it made it easy to unintentionally hide original class' close method
  • #8778 #8777 Legacy Flink 1.x APIs removal
    • Custom, Nussknacker's TimestampWatermarkHandler abstraction was replaced by Flink's WatermarkStrategy
      • FlinkSourceTestSupport.timestampAssignerForTest was renamed to watermarkStrategyForTest
      • CustomizableTimestampWatermarkHandlerSource was renamed to CustomizableWatermarkStrategySource[Raw]
      • StandardTimestampWatermarkHandler was renamed to WatermarkStrategyUtils
      • Nussknacker's SimpleSerializableTimestampAssigner was removed - use Flink's SimpleSerializableTimestampAssigner instead
    • AssignerWithPunctuatedWatermarks / AssignerWithPeriodicWatermarks interfaces were removed from Nussknacker API
      • BlockingQueueSource now takes extractTimestampFun: T => Long, maxOutOfOrderness: Duration instead of AssignerWithPunctuatedWatermarks[T]
      • EmitWatermarkAfterEachElementCollectionSource was removed. Use CollectionSource combined with WatermarkStrategyUtils.afterEachEvent
      • BoundedOutOfOrdernessPunctuatedExtractor were removed - use WatermarkStrategyUtils.afterEachEvent instead
      • BoundedOutOfOrderPreviousElementAssigner was removed - use custom WatermarkStrategy instead
    • BasicFlinkSink.toFlinkFunction was replaced by toFlinkSink; the new method returns sink2.Sink instead of SinkFunction
  • #8788 ExplicitUidInOperatorsSupport trait was removed. Use setUidAndNameToNodeId extension method available in DataStreamImplicits instead
  • #8786 StandardFlinkSource, BaseFlinkSource and CustomizableWatermarkStrategySource traits were removed. Use setUidAndNameToNodeId, assignTimestampsAndWatermarks and context initialization (FlinkContextInitializingFunction) directly
  • #9327 FileSystemBind and associated bind mounting helpers in Docker tests were removed. Use ContainerVolume instead.
  • Removed pl.touk.nussknacker.engine.process.registrar.StreamExecutionEnvPreparer.sideOutputGetter. Custom StreamExecutionEnvPreparer implementations should remove this method.

Other changes​

  • !860 Kafka client library was updated to 4.2.1, the lowest supported version of Kafka broker is 2.5.
  • !863 #AGG.list aggregate now returns the aggregated elements oldest first
  • !485 Behavior change: duration parameters of several base components are now validated at scenario compile time. A scenario that previously compiled becomes invalid when a constant (compile-time evaluable) duration violates the new constraint - fix the reported parameter value:
    • must be positive (> 0): deduplication ttl, state-transformer ttl, union-memo stateTimeout, full-outer-join windowLength, aggregate-sliding / aggregate-tumbling windowLength, aggregate-session sessionTimeout, SQL enrichers (query / lookup) Cache TTL (only when set), kafka / table Idleness (only when set)
    • must not be negative (>= 0): delay delay (constant expressions only; a non-positive value computed per event still releases the event immediately), single-side-join windowLengthBeforeEvent / windowLengthAfterEvent, event-generator schedule (zero still means the maximum generation rate), kafka / table / event-generator Max out-of-orderness
  • !248 The delay component was reworked:
    • The key parameter was renamed to keyBy. Existing scenarios using the delay component must update this parameter (rename key to keyBy), either manually or via a scenario/DB migration.
    • A new advanced parameter timeMode (EventTime / ProcessingTime) was added. Its default can be configured via the optional components.base.delay.timeMode setting; when not configured it defaults to ProcessingTime (keyBy and delay have fixed defaults "" and 100ms).
    • Behavior change: the previous default was event time. Delay nodes that do not set timeMode now run in processing time. To keep the previous behavior, either set the node's timeMode to EventTime, or configure components.base.delay.timeMode: "EventTime".
  • #7116 Improve missing Flink Kafka Source / Sink TypeInformation
    • We lost support for old ConsumerRecord constructor supported by Flink 1.14 / 1.15
    • If you used Kafka source/sink components in your scenarios then state of these scenarios won't be restored
  • #8997 Operational impact of random/stable technical node IDs:
    • During first migration of existing scenarios, node ids change to UUIDs, so external tools comparing scenario JSON by node ids can observe large one-time diffs.
    • Migration NodeIdToUuidMigration must be applied in model migration list, so legacy scenarios are migrated to UUID-based node ids.
    • Metrics dimensions based on nodeId changed to UUID values. If dashboards/alerts used former human-readable ids, they require updates. Where available, prefer nodeName metric tag for human-readable presentation.
    • Kafka consumer groups using naming strategy containing nodeId (e.g. processId-nodeId) will change identifiers after migration. This can require offsets migration / planned cutover.
    • Flink operator UIDs are now based on technical node IDs. For scenarios migrated from old ids, savepoints/checkpoints created before migration can become incompatible.
  • #7257 #7259 components-api module doesn't depend on async-http-client-backend-future, http-utils module is delivered by flink-executor and lite-runtime modules. If your component had compile-time dependency to http-utils, it should be replaced by provided scope If your component relied on the fact that components-api depends on async-http-client-backend-future, async-http-client-backend-future should be added as a provided dependency
  • #7347 All calls to org.apache.flink.api.common.functions.RichFunction.open(Configuration), which is deprecated, were replaced with calls to org.apache.flink.api.common.functions.RichFunction.open(OpenContext)
  • #7364
    • additional, necessary, db schema changes concerning the periodic/scheduling mechanism introduced in #7519
    • additionally modified in #7552
    • the PeriodicDeploymentManager is no longer a separate DM type
    • in scenarioTypes config section, the deploymentConfig of a periodic scenario type (only Flink was supported so far) may have looked like that:
      deploymentConfig: {
        type: "flinkPeriodic"
        restUrl: "http://jobmanager:8081"
        deploymentManager {
          db: { <config of the custom db data source> },
          processingType: streaming,
          jarsDir: ./storage/jars
        }
      }
    • changes:
      • the type: "flinkPeriodic" is no longer supported, instead type: "flinkStreaming" with additional setting supportsPeriodicExecution: true should be used
      • the db config is now optional - the periodic DM may still use its custom datasource defined here in legacyDb section
      • when custom db section not defined here, then main Nussknacker db will be used
    • config after changes may look like that:
      deploymentConfig: {
        type: "flinkStreaming"
        scheduling {
          enabled: true
          processingType: streaming
          jarsDir: ./storage/jars
        }
        restUrl: "http://jobmanager:8081"
      }
  • #7335 Deployment managers are loaded using separate class loader (not the Application ClassLoader - /opt/nussknacker/managers/* should be removed from CLASSPATH definition). The default location for deployment managers jars is the managers folder inside the working directory.
  • #7458 #7534 Flink scenario testing mechanism and scenario state verification mechanism changes
    • Scenario testing and scenario state verification is now limited by a timeout to ensure proper resources cleaning. In some cases it might be needed to change the timeout value. To do that, set deploymentConfig.scenarioTesting.timeout or/and deploymentConfig.scenarioStateVerification.timeout to desired values. Notice that this properties should be configured along with akka.http.server.request-timeout
  • #7468 When a namespace is configured, Kafka consumer groups are also namespaced. This change should have been introduced as of starting from Nussknacker 1.15 when a feature flag useNamingStrategyForConsumerGroupId was removed to temporarily disable consumer group namespacing.
  • #7578 Component labels will be auto-generated for all components that don't have label defined in ComponentDefinition. Labels will be visible in components palette and in components list. Auto-generated label is created by formating component's name into Title Case, e.g. for component with name aggregate-session, label will be Aggregate Session. If component provider has componentPrefix set, it'll be included in auto-generated label.
  • #7553 #8011 #key variable created in window components won't be a list in a string, but it'll have provided types, e.g. instead of "[12.0, ab]" it'll now be [12.0, "ab"]. To keep #key value as before, you'll need to embrace previous values inside curly braces and add .toString, e.g. #input.a #input.b -> {#input.a, #input.b}.toString. Also, if you provide one element in groupBy expression, it will be presented as one element list instead of scalar.
  • #8011 During scenario compilation, redundant parameters used in node are treated only as warning now. They are skipped and compilation passes.
  • #8198 Support for test data without sourceId specified is removed - now, you always have to specify it
  • #7137#8317 Updated Flink 1.19.2 -> 1.20.2.
  • #8209 Nussknacker now requires flink to be run with replaced lib/flink-scala_2.12-x.x.x.jar by pl.touk:flink-scala lib for the same scala version as used Nussknacker distribution. We provide prebuild flink docker images on Docker Hub
  • #8478 The behavior of enum to json encoding has been changed - now it uses .name() instead of .toString()
  • #8842!13 pl.touk.nussknacker.engine.flink.api.serialization.SerializersRegistrar SPI was moved to pl.touk.nussknacker.engine.flink.api.typeinformation.SerializersRegistrar
  • #8943 spring-jcl was replaced with commons-logging - projects depending on Nu should update their exclusion rules
  • #9336 WithKafkaContainer and WithSchemaRegistryContainer test traits were moved from Flink-specific nussknacker-flink-test-utils to nussknacker-kafka-test-utils, methods for accessing service addresses were moved into proper Container objects
  • !13 SerializersRegistrar SPI was restored in a new namespace, as pl.touk.nussknacker.engine.flink.api.typeinformation.SerializersRegistrar

The nussknacker-ml jar-carrier image is removed​

  • The nussknacker-ml jar-carrier image is removed. The mlflow and pmml enricher jars now ship inside the designer image and the lite runtime image at /opt/nussknacker/components/common/, where the default model classpaths already look. Deployments that copied these jars via an init container must remove that init container in the same deploy that adopts this version: removing it earlier silently drops the jars, and keeping it pins an image tag that no longer exists.

The nussknacker-enterprise-extensions jar-carrier image is removed​

  • The nussknacker-enterprise-extensions jar-carrier image is removed; the enterprise jars ship inside the designer and lite runtime images. Extension selection moves to the deployment CLASSPATH, with one directory per extension under extensions/<name>/* (e.g. extensions/config-loader/*, extensions/configurator-loader/*, extensions/mcp/*, extensions/assistant/*, extensions/nodes-actions/*). Deployments that copied jars from the carrier image via an init container must switch to the baked images in the same deploy as the version bump: removing the init container early drops the jars before the baked image is in place, and bumping the version early pulls a nussknacker-enterprise-extensions tag that no longer exists.

In version 1.18.0​

Configuration changes​

  • #6944

    • Button name for 'test adhoc' was renamed from test-with-form to adhoc-testing If you are using custom button config remember to update button type to type: "adhoc-testing" in processToolbarConfig

  • #7039

    • Scenario Activity audit log is available
    • logger name, scenario-activity-audit, it is optional, does not have to be configured
    • it uses MDC context, example of configuration in logback.xml:
        <logger name="scenario-activity-audit" level="INFO" additivity="false">
            <appender-ref ref="STDOUT_FOR_SCENARIO_ACTIVITY_AUDIT"/>
        </logger>

        <appender name="STDOUT_FOR_SCENARIO_ACTIVITY_AUDIT" class="ch.qos.logback.core.ConsoleAppender">
          <encoder>
              <Pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - [scenarioId=%X{scenarioId}][version=%X{scenarioVersionId}][user=%X{username}] %msg%n</Pattern>
          </encoder>
        </appender>

  • #6979 Add type: "activities-panel" to the processToolbarConfig which replaces removed { type: "versions-panel" } { type: "comments-panel" } and { type: "attachments-panel" }

  • #6958 Added message size limit in the "Kafka" exceptionHandler: maxMessageBytes. Its default value reflects Kafka's default size limit of 1 MB (max.message.bytes), you need to increase it if your error topic allows for larger messages. Remember to add some margin for Kafka protocol overhead (100 bytes should be enough).

Code API changes​

  • #6971 DeploymentManagerDependencies API changes:
    • Added field scenarioActivityManager: ScenarioActivityManager
    • scenarioActivityManager can be used by any DeploymentManager to save scenario activities in the Nu database
    • there is NoOpScenarioActivityManager implementation available (if needed for tests etc.)
  • #6971 DeploymentManager compatible API changes:
    • DeploymentManager may now optionally extend ManagerSpecificScenarioActivitiesStoredByManager
    • managers extending that trait may internally handle some manager-specific ScenarioActivities and may be queried about those custom activities
  • #6695 SingleTypingResult API changes:
    • Renamed objType to runtimeObjType which indicates a current object in a runtime.
  • #6766
    • Process API changes:
      • Field ScenarioWithDetails.labels was added
      • Field ScenarioWithDetails.tags was removed (it had the same value as labels and was not used)
  • #6988 Removed unused API classes: MultiMap, TimestampedEvictableStateFunction. MultiMap was incorrectly handled by Flink's default Kryo serializer, so if you want to copy it to your code you should write and register a proper serializer.
  • #7162 When component declares that requires parameter with either SpelTemplateParameterEditor or SqlParameterEditor editor, in the runtime, for the expression evaluation result, will be used the new TemplateEvaluationResult class instead of String class. To access the previous String use TemplateEvaluationResult.renderedTemplate method.
  • #7246
    • Typing api changes:
      • CanBeSubclassDeterminer.canBeSubclassOf changed to AssignabilityDeterminer.isAssignableLoose.
      • TypingResult.canBeSubclassOf changed to TypingResult.canBeConvertedTo

REST API changes​

  • #6944
    • New endpoint /api/scenarioTesting/{scenarioName}/adhoc/validate
  • #6766
    • Process API changes:
      • PUT /api/processes/{processName} - optional scenarioLabels field added
    • Migration API changes:
      • POST /api/migrate supports v2 request format (with scenarioLabels field)
  • #7021
    • Definitions API changes:
      • GET /api/processDefinitionData/*}
        • added optional query param enrichedWithUiConfig
        • added requiredParam property to the response for parameter config at components['component-id'].parameters[*]
  • #7246 Changes in DictApiEndpoints:
    • DictListRequestDto expectedType: TypingResultInJson -> Json

Other changes​

  • #6692 Kryo serializers for UnmodifiableCollection, scala.Product etc. are registered based on class of Serializer instead of instance of Serializer. If you have values that were serialized by these Serializers in some state, the state won't be restored after upgrade.

  • #6952 Improvement: TypeInformation support for scala.Option: If you used CaseClassTypeInfoFactory with case classes that contain the Option type, the state won't be restored after the upgrade.

  • #6805 Updated Flink 1.18.1 -> 1.19.1. Due to backwards incompatible changes in this Flink version update, Nussknacker 1.18 will not work with Flink versions pre-1.19 right away. If you want to keep using Flink pre-1.19 with current Nussknacker, please refer to compatibility providing plugins in https://github.com/TouK/nussknacker-flink-compatibility.

  • #6912 Improvement: Make TimeMeasuringService usable with other Lifecycle traits

    • Services that use TimeMeasuringService must be rebuilt

  • Performance optimization:

    • #7058 Add missing Flink TypeInformation for better serialization
      • In case of using base (bounded and unbounded) Flink components state will be probably not compatible
      • FlinkCustomNodeContext.typeInformationDetection has been removed, please use TypeInformationDetection.instance instead
      • FlinkCustomNodeContext.forCustomContext has been removed, please use TypeInformationDetection.instance.forValueWithContext instead
    • #7097 Flink base types registration mechanism
      • In case of using types: java.time.LocalDate, java.time.LocalTime, java.time.LocalDateTime with CaseClassTypeInfo mechanism, state probably will be lost

  • #7113 Scala 2.13 was updated to 2.13.15, you should update your flink-scala-2.13 to 1.1.2

  • #7187 JSON decoding in request source (request-response processing mode) and in kafka source (streaming processing mode): For small decimal numbers is used either Integer or Long (depending on number size) instead of BigDecimal. This change should be transparent in most cases as this value was mostly used after #CONV.toNumber() invocation which still will return a Number.

In version 1.17.0​

Code API changes​

  • #6248 Removed implicit conversion from string to SpeL expression (pl.touk.nussknacker.engine.spel.Implicits). The conversion should be replaced by pl.touk.nussknacker.engine.spel.SpelExtension.SpelExpresion.spel.
  • #6282 If you relied on the default value of the topicsExistenceValidationConfig.enabled setting, you must now be aware that topics will be validated by default (Kafka's auto.create.topics.enable setting is only considered in case of Sinks). Create proper topics manually if needed.
  • Component's API changes
    • #6711 SingleComponentConfig changed to ComponentConfig for better domain naming. Associated functions and objects also changed to ...ComponentConfig....
    • #6418 Improvement: Pass implicit nodeId to EagerServiceWithStaticParameters.returnType Now method returnType from EagerServiceWithStaticParameters requires implicit nodeId param
    • #6462 CustomStreamTransformer.canHaveManyInputs field was removed. You don't need to implement any other method in replacement, just remove this method.
    • #6418 Improvement: Pass implicit nodeId to EagerServiceWithStaticParameters.returnType
      • Now method returnType from EagerServiceWithStaticParameters requires implicit nodeId param
    • #6340 TestRecordParser trait used in SourceTestSupport trait changed to work on lists instead of single records - its parse method now takes List[TestRecord] instead of a single TestRecord and returns a list of results instead of a single result.
    • #6520 ExplicitTypeInformationSource trait was removed - now TypeInformation produced by SourceFunction passed to StreamExecutionEnvironment.addSource is detected based on TypingResult (thanks to TypeInformationDetection)
      • BlockingQueueSource.create takes ClassTag implicit parameter instead of TypeInformation
      • EmitWatermarkAfterEachElementCollectionSource.create takes ClassTag implicit parameter instead of TypeInformation
      • CollectionSource's TypeInformation implicit parameter was removed
      • EmptySource's TypeInformation implicit parameter was removed
    • #6545 FlinkSink.prepareTestValue was replaced by prepareTestValueFunction - a non-parameter method returning a function. Thanks to that, FlinkSink is not serialized during test data preparation.
  • TypingResult API changes
    • #6436 Changes to TypingResult of SpEL expressions that are maps or lists:
      • TypedObjectTypingResult.valueOpt now returns a java.util.Map instead of scala.collection.immutable.Map
        • NOTE: selection (.?) or operations from the #COLLECTIONS helper cause the map to lose track of its keys/values, reverting its fields to an empty Map
      • SpEL list expression are now typed as TypedObjectWithValue, with the underlying TypedClass equal to the TypedClass before this change, and with value equal to a java.util.List of the elements' values.
        • NOTE: selection (.?), projection (.!) or operations from the #COLLECTIONS helper cause the list to lose track of its values, reverting it to a value-less TypedClass like before the change
    • #6566 TypedObjectTypingResult.fields are backed by ListMap for correct RowTypeInfo's fields order purpose. If #5457 migrations were applied, it should be a transparent change
      • Removed deprecated TypedObjectTypingResult.apply methods - should be used Typed.record factory method
      • Typed.record factory method takes Iterable instead of Map
    • #6570 TypingResult.canBeSubclassOf generic parameter checking related changes. Generic parameters of Typed[java.util.Map[X, Y]], Typed[java.util.List[X]], Typed[Array[X]] were checked as they were either covariant or contravariant. Now they are checked more strictly - depending on collection characteristic.
      • Key parameters of Typed[java.util.Map[Key, Value]] is treated as invariant
      • Value parameters of Typed[java.util.Map[Key, Value]] is treated as covariant
      • Element parameters of Typed[java.util.List[Element]] is treated as covariant
      • Element parameters of Typed[Array[Element]] is treated as covariant
  • #6503 FlinkTestScenarioRunner cleanups
    • runWithDataAndTimestampAssigner method was removed. Instead, timestampAssigner was added as an optional parameter into runWithData
    • new runWithDataWithType was added allowing to test using other types than classes e.g. records
  • #6567 Removed ability to set Flink's execution mode in sources: TableSource, CollectionSource and in FlinkTestScenarioRunner.runWithData method. Now you can configure it under modelConfig.executionMode or for test purposes through FlinkTestScenarioRunnerBuilder.withExecutionMode method.
  • #6610 Add flink node context as parameter to BasicFlinkSink. Now one can use FlinkCustomNodeContext in order to build sink in BasicFlinkSink#toFlinkFunction method.
  • #6635 #6643 TypingResultTypeInformation related changes
    • TypingResultAwareTypeInformationCustomisation API was removed
    • FlinkCustomNodeContext.typeInformationDetection is deprecated - use TypeInformationDetection.instance instead
    • FlinkCustomNodeContext.valueWithContextInfo.forCustomContext is is deprecated - use TypeInformationDetection.instance.forValueWithContext instead
  • #6640 BestEffort*Encoder naming changes:
    • All BestEffort*Encoder classes renamed to fit To<TargetFormat>(SchemaBased)Encoder naming schema
    • JsonToNuStruct renamed to FromJsonDecoder (to fit From<SourceFormat>Decoder naming schema)
    • ToJsonEncoder renamed to ToJsonEncoderCustomisation
    • ToJsonBasedOnSchemaEncoder renamed to ToJsonSchemaBasedEncoderCustomisation
  • #6586 For now on, the SQL enricher automatically converts types as shown below:
    • java.sql.Array -> java.util.List
    • java.sql.Time -> java.time.LocalTime
    • java.sql.Date -> java.time.LocalDate
    • java.sql.Timestamp -> java.time.Instant
    • java.sql.Clob -> java.lang.String

REST API changes​

  • #6437 Removed deprecated operation to create a scenario: POST /api/processes/{name}/{category}. POST /api/processes should be used instead.
  • #6213 Improvement: Load resource config only in test context
    • WithConfig from test-utils modules behaviour changes: now it only parses given config, without resolving reference configs, system env variables etc.

Configuration changes​

  • #6635 globalParameters.useTypingResultTypeInformation parameter was removed. Now we always use TypingResultTypeInformation
  • #6797 AVRO_USE_STRING_FOR_STRING_TYPE environment variable is not supported anymore - we always use String for String type in Avro. If you didn't set up this environment variable, no action is needed

In version 1.16.3​

Code API changes​

  • #6527 Changes to TypingResult of SpEL expressions that are maps or lists:
    • TypedObjectTypingResult.valueOpt now returns a java.util.Map instead of scala.collection.immutable.Map
      • NOTE: selection (.?) or operations from the #COLLECTIONS helper cause the map to lose track of its keys/values, reverting its fields to an empty Map
    • SpEL list expression are now typed as TypedObjectWithValue, with the underlying TypedClass equal to the TypedClass before this change, and with value equal to a java.util.List of the elements' values.
      • NOTE: selection (.?), projection (.!) or operations from the #COLLECTIONS helper cause the list to lose track of its values, reverting it to a value-less TypedClass like before the change

In version 1.16.0​

Code API changes​

  • #6184 Removed Remote[] string part from forwarded username for scenario creation and updates. processes and process_versions tables won't store username with this part anymore in createdBy and modifiedBy columns.
  • #6053 Added impersonation mechanism:
    • OverrideUsername permission was renamed as Impersonate and is now used as a global permission.
    • AuthManager is now responsible for authentication and authorization. AuthenticationResources handles only plugin specific authentication now. This leads to following changes in AuthenticationResources API:
      • authenticate() returns AuthenticationDirective[AuthenticatedUser] and not Directive1[AuthenticatedUser]
      • authenticate(authCredentials) receives PassedAuthCredentials parameter type instead of AuthCredentials as anonymous access is no longer part of AuthenticationResources logic
      • authenticationMethod() returns EndpointInput[Option[PassedAuthCredentials]] instead of EndpointInput[AuthCredentials]. The Option[PassedAuthCredentials] should hold the value that will be passed to the mentioned authenticate(authCredentials).
      • AuthenticationResources extends AnonymousAccessSupport trait:
        • AnonymousAccessSupport has one method getAnonymousRole() which returns anonymous role name. If you do not want to have an anonymous access mechanism for your authentication method you can extend your AuthenticationResources implementation with NoAnonymousAccessSupport trait.
      • AuthenticationResources has a field impersonationSupport of type ImpersonationSupport:
        • ImpersonationSupport is a trait stating whether authentication method supports impersonation. If you don't want impersonation support you can assign NoImpersonationSupport object to it. If you wish to have it - assign ImpersonationSupported abstract class to it and implement getImpersonatedUserData(impersonatedUserIdentity) method which returns required user's data for the impersonation by user's identity.
    • AnonymousAccess extending AuthCredentials was renamed to NoCredentialsProvided. It does not represent anonymous access to the designer anymore but simply represents passing no credentials.
    • AuthenticationConfiguration has one additional Boolean property isAdminImpersonationPossible which defines whether admin users can be impersonated by users with the Impersonate permission. The property is set to false by default for BasicAuthenticationConfiguration, OAuth2Configuration and DummyAuthenticationConfiguration.
  • #6087 #6155 DeploymentManager API changes:
    • DMRunDeploymentCommand.savepointPath was replaced by updateStrategy: DeploymentUpdateStrategy
      • In places where savepointPath = None was passed, the DeploymentUpdateStrategy.ReplaceDeploymentWithSameScenarioName(StateRestoringStrategy.RestoreStateFromReplacedJobSavepoint) should be passed
      • In places where savepointPath = Some(path) was passed, the DeploymentUpdateStrategy.ReplaceDeploymentWithSameScenarioName(StateRestoringStrategy.RestoreStateFromCustomSavepoint(path)) should be passed
    • DMValidateScenarioCommand.updateStrategy was added
      • In every place should the DeploymentUpdateStrategy.ReplaceDeploymentWithSameScenarioName(StateRestoringStrategy.RestoreStateFromReplacedJobSavepoint) should be passed
    • deploymentSynchronisationSupport field was added for purpose of synchronisation of statuses. If synchronisation mechanism is not used in context of given DM, you should return NoDeploymentSynchronisationSupport object. The synchronisation mechanism is used by /api/deployments/{deploymentId}/status endpoint. Other endpoints don't use it.
  • #6249 TopicName trait was introduced and is used in context of specialized topic name (for kafka sources and sinks). Moreover, UnspecializedTopicName case class was added and is used in places when the specialization is unknown/not needed.

Configuration changes​

  • #6082 Default Influx database was changed from esp to nussknacker_metrics

In version 1.15.0​

Code API changes​

  • #5609 #5795 #5837 #5798 Refactoring around DeploymentManager's actions:
    • Custom Actions
      • CustomAction, CustomActionParameter and CustomActionResult moved from extension-api to deployment-manager-api module
      • CustomActionResult.req was removed
      • CustomAction was renamed to CustomActionDefinition
      • CustomActionRequest (from the extension-api) was renamed to CustomActionCommand
      • CustomActionRequest has additional comment parameter (like deploy and cancel actions)
    • Other "action" methods - all methods operating on a scenario (or its deployment) were replaced by case classes and one method handling them all: processCommand(command):
      • validate - DMValidateScenarioCommand
      • deploy - DMRunDeploymentCommand
      • cancel with deploymentId argument - DMCancelDeploymentCommand
      • cancel without deploymentId argument - DMCancelScenarioCommand
      • stop with deploymentId argument - DMStopDeploymentCommand
      • stop without deploymentId argument - DMStopScenarioCommand
      • savepoint - DMMakeScenarioSavepointCommand
      • test - DMTestScenarioCommand
    • "Action type" was renamed to "action name". Loosened the restriction on the name of the action:
      • ProcessActionType (enum with fixed values) is replaced with ScenarioActionName,
      • in ProcessAction attribute actionType renamed to actionName
      • in table process_actions column action_type is renamed to action_name
    • DeploymentManagerDependencies.deploymentService was splitted into deployedScenariosProvider and actionService
    • Events renamed:
      • OnDeployActionSuccess renamed to OnActionSuccess
      • OnDeployActionFailed renamed to OnActionFailed
  • #5762 for the Flink-based TestRunner scenario builder you should replace the last component that was testResultService with testResultSink
  • #5783 Return type of allowedProcessingMode method in Component trait has been changed to AllowedProcessingModes type which is one of:
    • AllowedProcessingModes.All in case of all processing modes allowed
    • AllowedProcessingModes.SetOf(nonEmptySetOfAllowedProcessingModes) in case only set of processing modes is allowed
  • #5757 Refactored API around FlinkSource
    • Added StandardFlinkSource with more granular additional traits replacing the need for FlinkIntermediateRawSource
    • Removed BasicFlinkSource and FlinkIntermediateRawSource. Sources extending these traits should now extend StandardFlinkSource. For reference on how to migrate, see changes in FlinkKafkaSource or CollectionSource
    • Renamed FlinkSource's sourceStream method to contextStream
    • Removed EmptySourceFunction
  • #5757 Added support for bounded sources and Flink runtime mode in Flink tests
    • CollectionSource now takes Flink's Boundedness with default Unbounded and RuntimeExecutionMode with default None as a parameters. It's encouraged to set the Boundedness to bounded if applicable
    • Boundedness and RuntimeExecutionMode is also possible to set in FlinkTestScenarioRunner in new overloading runWithData method

Configuration changes​

  • #5744 Extracted unbounded stream specific components into separate module:
    • Components periodic, union-memo, previousValue, aggregates, joins and delay from base were moved into base-unbounded module. They are now built as flinkBaseUnbounded.jar under work/components/flink/flinkBaseUnbounded.jar.
    • Configuration of tumbling windows aggregate offset is changed at the ComponentProvider level: components.base.aggregateWindowsConfig.tumblingWindowsOffset should now be set as components.baseUnbounded.aggregateWindowsConfig.tumblingWindowsOffset
    • If you previously specified base component jar explicitly in modelConfig.classPath as components/flink/flinkBase.jar and want to retain the unbounded specific components you need to add components/flink/flinkBaseUnbounded.jar explicitly.
    • #5887 When using a custom DesignerConfig, ensure that long text elements like 'generate file' are positioned in the last row to prevent excessive spacing between elements.

Other changes​

  • #5574 Removed the support for the pluggable expression languages: ExpressionConfig.languages removed
  • #5724 Improvements: Run Designer locally
    • Introduce JAVA_DEBUG_PORT to run the Designer locally with remote debugging capability
    • Removed SCALA_VERSION, please use NUSSKNACKER_SCALA_VERSION instead of it
  • #5824 Decision Table parameters rename:
    • "Basic Decision Table" -> "Decision Table"
    • "Expression" -> "Match condition"
  • #5881 nussknacker-interpreter module was renamed to nussknacker-scenario-compiler
  • #5875 Added configurable idle timeout to Flink Kafka source with the default value of 3 minutes. You can configure this timeout in Kafka component config at idleTimeout.duration or disable it at idleTimeout.enabled. You can learn about idleness in Flink general docs and Kafka connector-specific docs
  • #5875 Removed useNamingStrategyForConsumerGroupId feature flag allowing for disabling namespaced Kafka consumer groups
  • #5848: Introduced a new method for handling colors, aimed at simplifying customization. Now, all colors are centrally stored in a single location. Refer to README.md for details on theme colors customization.
  • #5914 Removed dev-specific configuration files dev-application.conf and dev-tables-definition.sql from public distribution artifacts

In version 1.14.0​

Code API changes​

  • #5271 Changed AdditionalUIConfigProvider.getAllForProcessingType API to be more in line with FragmentParameter
    • SingleComponentConfigWithoutId renamed to ComponentAdditionalConfig
    • field params: Map[String, ParameterConfig] changed to parameterConfigs: Map[String, ParameterAdditionalUIConfig]
    • ParameterAdditionalUIConfig is handled analogously to FragmentParameter (expect for valueCompileTimeValidation, which isn't yet handled)
      • ParameterConfig.defaultValue -> ParameterAdditionalUIConfig.initialValue
      • ParameterConfig.hintText -> ParameterAdditionalUIConfig.hintText
      • most of the capabilities of ParameterConfig.editor and ParameterConfig.validators are covered by ParameterAdditionalUIConfig.valueEditor and ParameterAdditionalUIConfig.valueCompileTimeValidation
  • #5285 Changes around scenario id/name fields:
    • CanonicalProcess.id of type String was replaced by name field of type ProcessName
    • CanonicalProcess.withProcessId was renamed to withProcessName
    • ScenarioWithDetails.id was removed (it had the same value as name)
    • ScenarioWithDetails.processId changed the type to Option[ProcessId] and will have always None value
    • ComponentUsagesInScenario.id was removed (it had the same value as name)
    • ComponentUsagesInScenario.processId was removed
    • ListenerScenarioWithDetails.id was removed (it had the same value as name)
    • ValidatedDisplayableProcess.id of type String was replaced by name field of type ProcessName
    • DisplayableProcess.id of type String was replaced by name field of type ProcessName, processName field is removed
    • deprecated AsyncExecutionContextPreparer.prepareExecutionContext was removed
    • AsyncExecutionContextPreparer.prepare now takes ProcessName instead of String
  • #5288 #5474 RemoteEnvironment / ModelMigration changes:
    • ProcessMigration.failOnNewValidationError was removed - it wasn't used anywhere anymore
    • RemoteEnvironment.testMigration result types changes
      • shouldFailOnNewErrors field was removed - it wasn't used anywhere anymore
      • converted field was replaced by the processName field which was the only information that was used
    • RemoteEnvironment.migrate takes ScenarioParameters instead of category
  • #5361 Parameter has new, optional labelOpt field which allows to specify label presented to the user without changing identifier used in scenario graph json (Parameteter.name)
  • #5356 Changes in AdditionalUIConfigProvider.getAllForProcessingType now require model reload to take effect.
  • #5393 #5444
    • Changes around metadata removal from the REST API requests and responses:
      • DisplayableProcess was renamed to ScenarioGraph
      • ScenarioGraph fields that were removed: name, processingType, category - all these fields already were in ScenarioWithDetails
      • ProcessProperties field removed: isFragment - this field already was in ScenarioWithDetails
      • ScenarioWithDetails field json.validationResult was moved into the top level of ScenarioWithDetails
      • ScenarioWithDetails field json was renamed into scenarioGraph and changed the type into ScenarioGraph
      • ValidatedDisplayableProcess was renamed to ScenarioGraphWithValidationResult
      • ScenarioGraphWithValidationResult all scenario graph fields were replaced by one scenarioGraph: DisplayableProcess field
    • Migration mechanisms (RemoteEnvironment and TestModelMigrations) use ScenarioWithDetailsForMigrations instead of ScenarioWithDetails
  • #5424 Naming cleanup around ComponentId/ComponentInfo
    • ComponentInfo was renamed to ComponentId
    • ComponentId was renamed to DesignerWideComponentId
    • new ComponentId is serialized in json to string in format $componentType-$componentName instead of separate fields (name and type)
    • NodeComponentInfo.componentInfo was renamed to componentId
  • #5438 Removed sealed trait CustomActionError, now CustomActionResult is always used
  • #5465 #5457 Typed related changes
    • CommonSupertypeFinder shouldn't be created directly anymore - CommonSupertypeFinder.* predefined variables should be used instead, in most cases just (CommonSupertypeFinder.Default)
    • TypedObjectTypingResult.apply removed legacy factory method taking List[(String, TypingResult)] - should be used variant with Map
    • TypedObjectTypingResult.apply removed legacy factory method taking TypedObjectDefinition - should be used variant with Map
    • TypedObjectTypingResult.apply is deprecated - should be used Typed.record(...) instead. It will be removed in further releases
    • TypedObjectDefinition was removed
    • Typed.empty was removed, TypedUnion now handles only >= 2 types
      • Typed.apply(vararg...) was replaced by Typed.apply(NonEmptyList) and Typed.apply(firstType, secondType, restOfTypesVaraarg...) If you have a list of types and you are not sure how to translate it to TypingResult you can try to use Typed.fromIterableOrUnknownIfEmpty but it is not recommended - see docs next to it.
      • TypedUnionis not a case class anymore, but is still serializable - If it was used in a Flink state, state will be probably not compatible
    • #5517 Legacy OnFinished listener-api event was removed
    • #5474 Component class now need to specify allowedProcessingModes. Most of the implementations (CustomStreamTransformer, Service, SinkFactory) has default wildcard (None). For SourceFactory you need to specify which ProcessingMode this source support. You have predefined traits: UnboundedStreamComponent, BoundedStreamComponent, RequestResponseComponent, AllProcessingModesComponent that can be mixed into the component
    • #5474 Changes around new scenario metadata (aka "parameters"):
      • ScenarioWithDetails: added processingMode and engineSetupName fields
    • #5522, #5521, #5519 DeploymentManager API related changes:
      • In the DeploymentManager:
        • DeploymentManager.getProcessState(ProcessIdWithName, Option[ProcessAction]) become final. You should implement resolve method instead. It does the same, only List[StatusDetails] are already determined.
        • Method DeploymentManager.getProcessStates signature was changed and now requires an implicit freshnessPolicy: DataFreshnessPolicy
        • Trait AlwaysFreshProcessState and method getFreshProcessStates were removed, instead of it please use getProcessStates with DataFreshnessPolicy.Fresh policy
        • Managers FlinkStreamingRestManager and FlinkRestManager require new parameter: scenarioStateCacheTTL: Option[FiniteDuration]
      • In the DeploymentManagerProvider:
        • New methods were added: defaultEngineSetupName and engineSetupIdentity. They have default implementations, you should consider to replace them by your own
        • New, overloaded createDeploymentManager was added. In the new one most of the parameters were bundled into DeploymentManagerDependencies class which allows to easier pass these dependencies to delegates. Also, this method returns ValidateNel[String, DeploymentManager]. You can return errors that will be visible to users e.g. invalid configuration etc. The old one is deleted.
        • Method createDeploymentManager signature was changed and now requires new parameter: scenarioStateCacheTTL: Option[FiniteDuration]
    • #5526 Refactored namespaces:
      • Removed ObjectNaming SPI
      • Removed logging when using naming strategy
      • Replaced ObjectNaming with single NamingStrategy which prepares a name with a prefix from namespace key from ModelConfig or returns the original name if the value is not configured
    • #5535 ProcessingTypeConfig.classpath contains now raw, String entries instead of URL. The String to URL converting logic is now inside ModelClassLoader.apply
  • #5505 anonymous access functionality for Tapir-based API
    • AuthenticationResources & AnonymousAccess traits were changed to be able to introduce anonymous access feature
    • AuthCredentials class was changed too
  • #5373#5694 changes related to Components and LazyParameters:
    • LazyParameter can be evaluated on request thanks to its evaluate method
    • Params data class was introduced as a replacement for runtime parameters values defined as Map[String, Any]. Params data class, in its extraction methods, assumes that a parameter with the given name exists in the underlying Map.
    • TypedExpression was removed from BaseDefinedParameter hierarchy in favour of TypingResult
    • TypedExpression doesn't depend on ExpressionTypingInfo anymore
    • ServiceInvoker refactoring (parameters map was removed, a context is passed to its method)
    • ProcessListener interface changed slightly
    • ParameterWithExtractor util was replaced with ParameterDeclaration.
    • classes renaming:
      • LazyParameterInterpreter to LazyParameterInterpreter
      • GenericNodeTransformation to DynamicComponent
      • SingleInputGenericNodeTransformation to SingleInputDynamicComponent
      • JoinGenericNodeTransformation to JoinDynamicComponent
      • JavaGenericTransformation to JavaDynamicComponent
      • JavaGenericSingleTransformation to JavaSingleInputDynamicComponent
      • JavaGenericJoinTransformation to JavaJoinDynamicComponent
      • JavaSourceFactoryGenericTransformation to JavaSourceFactoryDynamicComponent
      • GenericContextTransformationWrapper to DynamicComponentWrapper
      • SingleGenericContextTransformationWrapper to SingleInputDynamicComponentWrapper
      • SourceFactoryGenericContextTransformationWrapper to SourceFactoryDynamicComponentWrapper
      • JoinGenericContextTransformationWrapper to JoinDynamicComponentWrapper
    • type NodeTransformationDefinition (inside DynamicComponent) renamed to ContextTransformationDefinition
  • #5641 PeriodicProcessDeployment/DeploymentWithJarData/PeriodicProcess now takes type parameter CanonicalProcess or Unit to point out whether it contains scenario json.
  • #5656 pl.touk.nussknacker.engine.api.expression.Expression#language method returns Language trait instead of String
  • #5707 ParameterName data class was introduced. It replaces String in whole places where it's used as a parameter name
  • #5754 Fix for broken encoding mechanism in tests from file with Avro format, revert [0d9b600][https://github.com/TouK/nussknacker/commit/0d9b600]
    • Classes ResultsCollectingListener, TestResults, ExpressionInvocationResult, ExternalInvocationResult depend on T
    • Classes TestResults.nodeResults uses ResultContext instead of Context
    • Classes TestResults.exceptions uses ExceptionResult instead of NuExceptionInfo
    • Added variableEncoder to ResultsCollectingListenerHolder.registerRun

REST API changes​

  • #5280#5368 Changes in the definition API:
    • /api/processDefinitionData/componentIds endpoint is removed
    • /api/processDefinitionData/* response changes:
      • services, sourceFactories, sinkFactories, customStreamTransformers and fragmentInputs maps fields were replaced by one components map with key in format $componentType-$componentName and moved into top level of response
      • typesInformation field was renamed into classes, moved into top level of response and nested clazzName inside each element was extracted
      • componentsConfig field was removed - now all information about components are available in the components field
      • nodeId field inside edgesForNodes was renamed into componentId in the flat $componentType-$componentName format
      • defaultAsyncInterpretation field was removed
  • #5285 Changes around scenario id/name fields:
    • /api/process(Details)/** endpoints:
      • id fields was removed (it had the same value as name)
      • processId fields return always null
      • .json.id fields was renamed to .json.name
    • /api/components/*/usages endpoint:
      • id fields was removed (it had the same value as name)
      • processId fields was removed
    • /api/processes/**/activity/attachments - processId fields was removed
    • /api/processes/**/activity/comments - processId fields was removed
    • GET processes/$name/$version/activity/attachments - $version segment is removed now
  • #5393 Changes around metadata removal from the REST API requests and responses:
    • /api/processValidation was changed to /api/processValidation/$scenarioName and changed request type
    • /api/testInfo/* was changed to /api/testInfo/$scenarioName/* and changed request format regarding code API changes
    • /api/processManagement/generateAndTest/$samples was changed to /api/processManagement/generateAndTest/$scenarioName/$samples
    • /api/processesExport/* was changed to /api/processesExport/$scenarioName/* and changed response format regarding code API changes
    • /api/processes/import/$scenarioName was changed response into {"scenarioGraph": {...}, "validationResult": {...}
    • GET /api/processes/* and /api/processesDetails/* changed response format regarding code API changes
    • PUT /api/processes/$scenarioName was changed request field from process to scenarioGraph
    • /api/adminProcessManagement/testWithParameters/$scenarioName was changed request field from displayableProcess to scenarioGraph
  • #5424 Naming cleanup around ComponentId/ComponentInfo
    • Endpoints returning test results (/api/processManagement/test*) return nodeId instead of nodeComponentInfo now
    • /processDefinitionData/* response: field type was replaced by componentId inside the path .componentGroups[].components[]
  • #5462 /api/processes/category/* endpoint was removed
  • #5474 POST /api/processes/$scenarioName/$category?isFragment=$isFragment resource become deprecated. It will be replaced by POST /processes with fields: name, isFragment, forwardedUserName, category, processingMode, engineSetupName. Three last fields are optional. Please switch to the new API because in version 1.5, old API will be removed.
  • POST /api/nodes/$scenarioName/validation response for object in validationErrors array can have details of the error

Configuration changes​

  • #5297 componentsUiConfig key handling change:
    • $processingType-$componentType-$componentName format was replaced by $componentType-$componentName format
  • #5323 Support for the legacy categories configuration format was removed. In the new format, you should specify category field inside each scenario type.
  • #5419 Support for system properties was removed from model configuration (they aren't resolved and added to merged configuration)
  • #5474 You have to ensure that in every scenarioType model's classPath, in every jar are only components with not colliding processing modes. Also at least one component has defined processing mode other than wildcard. On the other hand starting from this version, you can use the same category for many scenarioTypes. You only have to ensure that they have components with other processing modes or other deployment configuration.
  • #5558 The processToolbarConfig toolbar with type: "process-info-panel" no longer accepts the buttons property. It only display scenario information now. However, a new toolbar with type: "process-actions-panel" has been introduced, which does accept the buttons property and renders actions similar to the old type: "process-info-panel".

Helm chart changes​

  • #5515 #5474 Helm chart now has two preconfigured scenario types (streaming and request-response) instead of one (default). Because of that, scenario created using previous version of helm chart will have invalid configuration in the database. To fix that, you have to manually connect to the database and execute sql statement: 
      UPDATE processes SET processing_type = 'given-scenario-type' where processing_type = 'default';

Other changes​

  • #4287 Cats Effect 3 bump Be careful with IO monad mode, we provide an experimental way to create IORuntime for the cat's engine.
  • #5432 Kafka client, Confluent Schema Registry Client and Avro bump
  • #5447 JDK downgraded from 17 to 11 in lite runner image for scala 2.13
  • #5465 Removed strictTypeChecking option and SupertypeClassResolutionStrategy.Union used behind it
  • #5517 Removed legacy mechanism marking scenario finished based on the fact that the last action was deploy and job was finished. The new mechanism leverage deployment id which was introduced in #4462 in 1.11 version.
  • #5474 The mechanism allowing migration between two environments uses by default the new, scenario creating API. In case when the secondary environment is in the version < 1.14, you should switch secondaryEnvironment.useLegacyCreateScenarioApi flag to on.
  • #5526 Added namespacing of Kafka consumer group id in both engines. If you have namespaces configured, the consumer group id will be prefixed with namespace key from model config - in that case a consumer group migration may be necessary for example to retain consumer offsets. For gradual migration, this behaviour can be disabled by setting useNamingStrategyInConsumerGroups = false in KafkaConfig. Note that the useNamingStrategyInConsumerGroups flag is intended to be removed in the future.

In version 1.13.1​

Code API changes​

  • #5447 JDK downgraded from 17 to 11 in lite runner image for scala 2.13

In version 1.13.0​

Code API changes​

  • #4988 Method definition def authenticationMethod(): Auth[AuthCredentials, _] was changed to def authenticationMethod(): EndpointInput[AuthCredentials]
  • #4860 DeploymentManagerProvider implementations have to implement the method def scenarioPropertiesConfig(config: Config): Map[String, ScenarioPropertyConfig] instead of def additionalPropertiesConfig(config: Config): Map[String, AdditionalPropertyConfig]
  • #4919 Improvement: Support for handling runtime exceptions at FlinkTestScenarioRunner:
    • TestProcess.exceptions type changed from List[ExceptionResult[T]] to List[NuExceptionInfo[_ <: Throwable]]
  • #4912 Changes in scenario details:
    • pl.touk.nussknacker.restmodel.processdetails.BaseProcessDetails[_] and pl.touk.nussknacker.restmodel.processdetails.BasicProcess used in rest resources were merged into pl.touk.nussknacker.restmodel.scenariodetails.ScenarioWithDetails
    • pl.touk.nussknacker.restmodel.processdetails.BaseProcessDetails[_] used in pl.touk.nussknacker.ui.listener.services.PullProcessRepository were moved into listener-api and renamed into pl.touk.nussknacker.ui.listener.ListenerScenarioWithDetails
    • pl.touk.nussknacker.restmodel.processdetails.ProcessDetails and pl.touk.nussknacker.restmodel.processdetails.ValidatedProcessDetails type aliases are not available anymore - you should probably use ScenarioWithDetails in these places
    • pl.touk.nussknacker.restmodel.processdetails.ProcessVersion was moved into pl.touk.nussknacker.engine.api.process.ScenarioVersion
    • pl.touk.nussknacker.restmodel.processdetails.StateActionsTypes was moved into ProcessActionType.StateActionsTypes
  • #4959 listener-api module become decoupled from restmodel module. Some classes were moved to extensions-api module to make it possible:
    • pl.touk.nussknacker.restmodel.displayedgraph package was renamed to pl.touk.nussknacker.engine.api.displayedgraph
    • pl.touk.nussknacker.restmodel.displayedgraph.ValidatedDisplayableProcess was moved to pl.touk.nussknacker.restmodel.validation package
    • pl.touk.nussknacker.restmodel.process.ProcessingType was moved to pl.touk.nussknacker.engine.api.process package
    • pl.touk.nussknacker.restmodel.scenariodetails.ScenarioVersion was moved to pl.touk.nussknacker.engine.api.process package
  • #4745 Added method ScenarioBuilder to create fragments with specified input node id instead of taking a default from fragment id
  • #4745 Add more errors for scenario and node id validation and change names, messages of existing ones
  • #4928 #5028 Validator.isValid method now takes expression: Expression, value: Option[Any] instead of value: String which was not really value, but expression. Straight-forward migration is to change method definition and now use expression.expression instead of value if your validator depends on raw expression. If validator was doing quasi-evaluation, for example trimming ' to get string, you can just take value and cast it to desired class.
    • LiteralNumberValidator is removed, to achieve same result use CompileTimeEvaluableValueValidator with parameter of Number type,
    • LiteralIntegerValidator is considered deprecated and will be removed in the future, to achieve same result use CompileTimeEvaluableValueValidator with parameter of Integer type,
    • LiteralRegExpParameterValidator is renamed to RegExpParameterValidator
    • annotation pl.touk.nussknacker.engine.api.validation.Literal was renamed to pl.touk.nussknacker.engine.api.validation.CompileTimeEvaluableValue
  • #5079 AuthCredentials is moved to pl.touk.nussknacker.security in extensions-api
  • #5103
    • Values of ExpressionConfig.globalImports and ExpressionConfig.dictionaries aren't wrapped with WithCategories anymore
    • WithCategories.apply with categories varrag variant is replaced by version with head category and tail categories varrag Previous version was commonly wrongly used as an "object without categories specified" but in fact it was "object with empty categories list" which means that object should be never visible. To create "object without categories specified" use, WithCategories.anyCategory. If you want to pass just a list of categories, use WithCategories(value, Some(list), SingleComponentConfig.zero)
  • #5171 Changes around ComponentType values changes:
    • In ComponentType values:
      • Built-in component's artificial component types (Filter, Split, Switch, Variable, MapVariable) were replaced by BuiltIn type
      • Processor and Enricher component types were replaced by Service
      • Fragments was replaced by Fragment
      • CustomNode was replaced by CustomComponent
    • In ComponentInfo: Order of parameters swapped + names of them changed componentType -> type, componentName -> name
  • #5209 Now TestScenarioRunner doesn't load components from ComponentProvider automatically. Instead, it loads some predefined set of components. Rest of them you need to pass components using withExtraComponents method. Components loaded automatically:
    • TestScenarioRunner.liteBased - from base provider
    • TestScenarioRunner.kafkaLiteBased - from base and kafka providers
    • TestScenarioRunner.requestResponseBased - from base and requestResponse providers
    • TestScenarioRunner.flinkBased - from base provider TestScenarioRunner now also uses global variables from default model
  • #4956 Refactor: Cleanup TestResults
    • Changed signature DeploymentManager.test method, and removed variableEncoder param
    • Classes TestResults, ExpressionInvocationResult, ExternalInvocationResult don't depend on T
    • Classes NodeResult is removed. Instead, Context is used directly
    • Removed variableEncoder from ResultsCollectingListenerHolder.registerRun
    • Removed ResultContext, please use Context instead of it
  • #5240 Simpler result types in TestScenarioRunner
    • RunResult and RunUnitResult has no generic parameter anymore
    • RunResult and its descendants has no success method anymore - for RunListResult should be used successes instead

REST API changes​

  • #4745 Change api/properties/*/validation endpoint request type
    • Replace processProperties with additionalFields
    • Add id field for scenario or fragment id
  • #5039#5052 Changes in endpoints
    • api/parameters/*/suggestions request
      • variables is renamed to variableTypes and it should have only local variables now
    • api/processes/** response
      • .json.validationResult.nodeResults.variableTypes doesn't contain global variables types anymore
    • api/processDefinitionData/* response
      • .processDefinition.globalVariables is removed
    • api/parameters/*/validate request
      • scenarioName is removed
      • processProperties is removed

Configuration changes​

  • #4860 In file-based configuration, the field scenarioTypes.<scenarioType>.additionalPropertiesConfig is renamed to scenarioTypes.<scenarioType>.scenarioPropertiesConfig
  • #5077 In SQL enricher configuration, connectionProperties was changed to dataSourceProperties

Other changes​

  • #4901 Improvements TestScenarioRunner:
    • Changes at FlinkProcessRegistrar.register passing resultCollector instead of testRunId
  • #5033 Scala 2.13 was updated to 2.13.12, you may update your flink-scala-2.13 to 1.1.1 (it's not required, new version is binary-compatible)
  • #5059 #5100 Categories configuration doesn't allow configuring multiple categories for the same scenario type. If you have such a case, you have to extract another scenario types and assign each category to each scenario type. Because of this change configuration of categories was also removed from Components configuration
  • #4953 Stricter validation in base components:
    • Boolean expressions in Switch and Filter nodes are required not null values
    • Variable values in MapVariable, FragmentOutput and Variable are mandatory
    • Field names in MapVariable, FragmentOutput are required to be unique
  • #4698 Due to change in program argument encoding all scheduled batch scenarios handled by periodic DM must be cancelled before upgrade

In version 1.12.6​

Other changes​

  • #5447 JDK downgraded from 17 to 11 in lite runner image for scala 2.13

In version 1.12.x​

Code API changes​

  • #4574 Improvements: at KafkaClient and RichKafkaConsumer in kafka-test-utils
    • RichKafkaConsumer.consumeWithJson needs json decoder
    • removed RichKafkaConsumer.consumeWithConsumerRecord, use RichKafkaConsumer.consumeWithJson instead of it
    • RichKafkaConsumer.defaultSecondsToWait renamed to RichKafkaConsumer.DefaultSecondsToWait
    • KafkaClient.sendMessage accepts generic content with json encoder
  • #4583 DeploymentManager has new variants of method cancel and stop taking DeployomentId next to ProcessName. They will be used with batch processing mechanism (periodic DM) so it is necessary to implement it only if your DM will be wrapped by PeriodicDeploymentManager
  • #4685 In AuthenticationResources trait it was added two new methods that have to be implemented in the child classes: def authenticationMethod(): Auth[AuthCredentials, _] and def authenticate(authCredentials: AuthCredentials): Future[Option[AuthenticatedUser]]. The first one tells what authentication method will be used (it's for Tapir-based API purposes) and the latter one is the authentication action itself. The def authenticate(): Directive1[AuthenticatedUser] should be treated as deprecated. It's used in the NU APIs which are still Akka HTTP-based. When we get rid of Akka HTTP, it will be removed.
  • #4762 Rename RegExpParameterValidator to LiteralRegExpParameterValidator

REST API changes​

  • #4697 Change api/parameters/*/validate and api/parameters/*/suggestions endpoints.
    • Use processingType instead of processName
    • Add scenarioName parameter to ParametersValidationRequest used in api/parameters/*/validate
  • #4602 Cleaning subprocess usages after NU 1.11 release
    • Removed isSubprocess endpoint param, use isFragment from now on.
    • Removed backward compatibility for subprocess fields.

Other changes​

  • #4492 Allow testing fragments using ad-hoc testing method. By default, NU enables that feature but if you have some custom processToolbarConfig settings then You would like to remove hidden: { fragment: true } flag for type: "test-with-form", type: "test-counts" and type: "test-hide" inside processToolbarConfig -> "test-panel".

In version 1.11.0​

Code API changes​

  • #4295 DeploymentManager.getProcessState(ProcessName) method now returns List[StatusDetails] instead of Option[StatusDetails]. If you are a consumer of this API and want to have the same behavior as previously, you can use InconsistentStateDetector.extractAtMostOneStatus method for that. Notice, that in the future visibility of this method may be changed into private.
  • #4462 StatusDetails.deploymentId field changed type intoOption[DeploymentId]. It contains, internal designer's deployment id. It is used to assign deployment on runtime side with deployment action on designer side in periodic DM for purpose of correct status synchronization. If you want to make it filled, you should pass the correct value in DeploymentData.deploymentId. Current value of deploymentId: Option[ExternalDeploymentId] was moved into externalDeploymentId field. ProcessAction has a new field - id. ProcessStateDefinitionManager.processState variant of methods with multiple parameters was removed - you should use ProcessStateDefinitionManager.processState(StatusDetails) variant instead.
  • #4353 Removed isCancelled/isDeployed flags based on ProcessAction, ProcessAction.action renamed to actionType. Trait Process is removed.
  • #4484 ProcessIdWithName moved from package pl.touk.nussknacker.restmodel.process to pl.touk.nussknacker.engine.api.process DeploymentManager.getProcessState(ProcessName, Option[ProcessAction]) method now takes ProcessIdWithName as an argument instead of ProcessName. The same with PostprocessingProcessStatus.postprocess.

REST API changes​

  • #4454 Rename 'subprocess' to 'fragment' along with all endpoints (with backward compatibility).
    • isSubprocess query parameter is renamed to isFragment. isSubprocess will be removed in 1.12.0
  • #4462 Process state API returns externalDeploymentId instead of deploymentId.

Other changes​

  • #4514 AkkaHttpBackend in designer is replaced by AsyncHttpClientFutureBackend. To use custom http client configuration use ahc.properties file and make sure it is available in the classpath.

In version 1.10.0​

Code API changes​

  • #4352 TypedObjectTypingResult#fields are no longer ordered, fields will be sorted for presentation (see TypedObjectTypingResult#display)
  • #4294 HttpRemoteEnvironmentConfig allows you to pass flag passUsernameInMigration - (default true). When set to true, migration attaches username in the form of Remote[userName] while migrating to secondary environment. To use the old migration endpoint, set to false.
  • #4278 Now expression compiler and code suggestions mechanism are reusing the same types extracted based on model. Before the change types in compiler were lazily extracted. Because of this change, some expressions can stop to compile. You may need to add WithExplicitTypesToExtract to some of yours SourceFactory implementations. See extending classes for examples on how to implement it.
  • #4290 Renamed predicates used in ClassExtractionSettings:
    • ClassMemberPatternPredicate renamed to MemberNamePatternPredicate
    • AllMethodNamesPredicate renamed to AllMembersPredicate
  • #4299, #4300 StateStatus is identified by its name. ProcessState serialization uses this name as serialized state value.
    Sealed trait StateStatus is unsealed, all members are replaced by corresponding SimpleStateStatus state definitions, custom statuses are defined within each ProcessStateDefinitionManager. ProcessAction is moved from restmodel to extensions-api, package engine.api.deployment.
  • #4339 Improvements: Don't fetch state for archived/unarchived scenario, return computed based on last state action At BaseProcessDetails we provide lastStateAction field which can have an influence on the presented state of the scenario. We currently use it to distinguish between cancel / not_deployed and to detect inconsistent states between the designer and engine
  • #4302 State inconsistency detection was moved from designer to DeploymentManager. DeploymentManager.getProcessState for internal purposes returns Option[StatusDetails] which is based on job status from deployment manager (instead of Option[ProcessState] which contains UI info). There is separate getProcessState that returns ProcessState which is a status from engine resolved via InconsistentStateDetector and formatted with UI-related details. PeriodicProcessEvent uses StatusDetails instead of ProcessState. Constants defined in ProblemStateStatus are renamed to match UpperCamelCase formatting.
  • #4350 StateStatus.isDuringDeploy, StateStatus.isFinished, StateStatus.isFailed, StateStatus.isRunning, ProcessState.isDeployed methods were removed. Instead, you should compare status with specific status.
  • #4357 Changed structure of MetaData in CanonicalProcess - TypeSpecificData automatically migrated to ProcessAdditionalFields
    • Example MetaData structure before migration:
    {
      "id": "scenarioName",
      "typeSpecificData": {
        "parallelism": 1,
        "spillStateToDisk": true,
        "checkpointIntervalInSeconds": null,
        "type": "StreamMetaData"
      },
      "additionalFields": {
        "description": null,
        "properties": {
          "someCustomProperty": "someCustomValue"
        }
      }
    }
    • Example MetaData structure after migration:
    {
      "id": "scenarioName",
      "additionalFields": {
        "description": null,
        "properties": {
          "parallelism" : "1",
          "spillStateToDisk" : "true",
          "useAsyncInterpretation" : "",
          "checkpointIntervalInSeconds" : "",
          "someCustomProperty": "someCustomValue"
        },
        "metaDataType": "StreamMetaData"
      }
    }

Configuration changes​

  • #4283 For OIDC provider, accessTokenIsJwt config property is introduced, with default values false. Please mind, that previous Nussknacker versions assumed its value is true if authentication.audience was defined.
  • #4357 TypeSpecificData properties are now be configured in DeploymentManagerProvider:
    • Main configuration is done through additionalPropertiesConfig like other additional properties
    • Initial values overriding defaults from the main configuration can be set in metaDataInitializer

Other changes​

  • #4305 scala-compiler and scala-reflect are now included in flink-scala, so you can simplify your deployment by removing them and updating to new (flink-scala JAR) (this doesn't introduce any functional changes)

REST API changes​

  • #4350 delete action is available only for archived scenarios. Before the change it was checked that scenario is not running

In version 1.9.0​

Code API changes​

  • #4030 Changes for purpose of local testing of designer with other urls than on engine side
    • ProcessingTypeConfig.modelConfig now contains ConfigWithUnresolvedVersion instead of Config. Old Config value is in ConfigWithUnresolvedVersion.resolved
    • ModelConfigLoader.resolveInputConfigDuringExecution takes ConfigWithUnresolvedVersion instead of Config. Use ConfigWithUnresolvedVersion.apply for easy transition between those classes
  • #3997 Removal of obsolete subprocessVersions. It affects MetaData, ProcessMetaDataBuilder and DisplayableProcess properties.
  • #4122, #4132, #4179, #4189
    • Use ProcessStateDefinitionManager.stateDefinitions to describe states: 1) their default properties 2) how the states are presented in filter-by-status options.
      (see an example of basic definitions in SimpleProcessStateDefinitionManager and SimpleStateStatus).
    • State defaults and allowed actions are moved to SimpleStateStatus, FlinkStateStatus, PeriodicStateStatus, EmbeddedStateStatus and K8sStateStatus from corresponding state-definition-managers (see example FlinkProcessStateDefinitionManager).
    • Type CustomStateStatus.name renamed to StatusName
    • ProcessResources exposes new endpoint /api/procecesses/statusDefinitions
    • Within the base set of statuses used in Embedded, Flink, K8 and Periodic mode (SimpleStateStatus), statuses Failing, Failed, Error, Warning, FailedToGet and MulipleJobsRunning are replaced by one ProblemStateStatus which is parametrized by specific message. ProblemStateStatus provides several builder methods, one for each corresponding removed state. Those builders allow to preserve the exact moments when each state appears in the scenario lifecycle.
    • Displayed tooltip and description of ProblemStateStatus have the same value.
    • Removed SimpleStateStatus.Unknown
    • Removed status FailedStateStatus. Use ProblemStateStatus instead.
    • Status configuration for icon, tooltip and description is obligatory.
  • #4104 DeploymentManager.findJobStatus was renamed to getProcessState. New DataFreshnessPolicy parameter was added. Returned type was changed to WithDataFreshnessStatus[T] where T is the previous value and cached: Boolean is additional information that should be provided. If you provide DeploymentManager which communicate remotely with some service, and you want to use standard build-in caching for ProcessState, wrap your DeploymentManager using CachingProcessStateDeploymentManager.wrapWithCachingIfNeeded in your DeploymentManagerProvider. Thanks to that, caching will be handled as expected, and your DeploymentManager just should extend AlwaysFreshProcessState which provide the same interface as the previous one, with only method name changed. Especially, when you use 'PeriodicDeploymentManagerProvider', delegate should already return DeploymentManager wrapped by caching mechanism.
  • #4131 Parameter.defaultValue now holds Option[Expression] instead of Option[String]. You have to wrap a String with Expression.spel()
  • #4224 If you're using Flink with Nussknacker built with Scala 2.13, add this jar in your Flink installation to lib dir. It's our implementation of org.apache.flink.runtime.types.FlinkScalaKryoInstantiator (sources are here) which is needed to properly (de)serialize Flink state when using scala 2.13. Hopefully, it's temporary solution, until Flink becomes really scala-free and gets rid of this FlinkScalaKryoInstantiator class or allows to have it in the job code (not Flink libs).
  • #4190 - introduced possibility to configure offset in FlinkComponentsProvider (components.base.aggregateWindowsConfig.tumblingWindowsOffset, by default 0) for aggregates with tumbling windows. You might want to set it up, especially when you want your daily windows to be aligned according to your timezone if it's not UTC. See example in Flink docs

Other changes​

  • #4122, #4132 Changes in state definitions:
    • In ProcessStateDefinitionManager default behaviour of methods statusTooltip, statusDescription and statusIcon is to return default properties defined via stateDefinitions. It is not necessary to override those methods when all definitions have fixed default properties.
    • To introduce custom status properties, extensions to basic definitions, use OverridingProcessStateDefinitionManager.
    • OverridingProcessStateDefinitionManager allows to specify delegate (previously only SimpleProcessStateDefinitionManager was available) and custom state definitions.
    • Parameter delegate in OverridingProcessStateDefinitionManager has no default value, it should be provided explicitly.
    • There is additional validation when all processing types are reloaded from configuration: check if all processing types state definitions configuration is correct. (see comment in ProcessStateDefinitionService)
  • #3997 Due to removal of deprecated field subprocessVersions migration of scenarios from and to remote environment, for Nussknacker version 1.9.0+ and older will not be possible. Use export and import as a workaround while working between older and newer version.

Other changes​

  • #3675 Improvements: Normalize kafka components params name
    • Renamed kafka topic param name from topic to Topic
    • Renamed kafka value param name from value to Value

In version 1.8.0​

Scenario authoring changes​

  • #3924
    • Fixup: {} is now interpreted as "allow everything schema", not as "object schema". Objects schemas have to have declared "type": "object".
    • Unknown is now allowed on sinks in both validation modes if output schema is "everything allowed schema".

Code API changes​

  • #3924 - changes to SwaggerTyped hierarchy
    • SwaggerMap(valuesType) -> SwaggerObject(Map.empty, additionalProperties = AdditionalPropertiesEnabled(valuesType))
    • AdditionalPropertiesSwaggerTyped -> AdditionalPropertiesEnabled
    • AdditionalPropertiesWithoutType -> AdditionalPropertiesEnabled(SwaggerAny)
    • SwaggerRecursiveSchema/SwaggerUnknownFallback -> SwaggerAny

Other changes​

  • #3835 Removed Signals and QueryableState. This change affects:

    • Configuration
    • Components and DeploymentManager API
    • REST API

  • #3823, #3836, #3843 - scenarios with multiple sources can be tested from file

    • TestDataGenerator#generateTestData returns JSON test records instead of raw bytes. Test records are serialized to a file by designer Test record can optionally contain timestamp which is used to sort records generated by many sources
    • TestDataParser was replaced with TestRecordParser that turns a single JSON test record into a source record
    • TestData.newLineSeparated helper was removed. Scenario test records have to be created explicitly. Each scenario test record has assigned source
    • DeploymentManager#test takes ScenarioTestData instead of TestData
    • Designer configuration testDataSettings.testDataMaxBytes renamed to testDataMaxLength

  • #3916 Designer configuration environmentAlert.cssClass renamed to environmentAlert.color

  • #3922 Bumps: jwks: 0.19.0 -> 0.21.3, jackson: 2.11.3 -> 2.13.4

  • #3929 From now, SchemaId value class is used in every place where schema id was represented as an Int. For conversion between SchemaId and Int use SchemaId.fromInt and SchemaId.asInt. Use ConfluentUtils.toSchemaWithMetadata instead of SchemaWithMetadata.apply for conversion between Confluent's SchemaMetadata and ours SchemaWithMetadata.

  • #3948 Now, we are less dependent from Confluent schema registry. To make it possible, some kafka universal/avro components refactors were done. Most important changes in public API:

    • ConfluentSchemaBasedSerdeProvider.universal was replaced by UniversalSchemaBasedSerdeProvider.create

    Some other, internal changes:

    • Non-confluent classes renamed and moved to desired packages
    • Extracted new class: SchemaIdFromMessageExtractor to make Confluent logic explicit and moved to top level
    • Extracted SchemaValidator to make Confluent logic explicit and be able to compose
    • Some renames: ConsumerRecordUtils -> KafkaRecordUtils
    • RecordDeserializer -> AvroRecordDeserializer (also inheritance replaced by composition)
    • (De)SerializerFactory - easier abstractions
    • ConfluentSchemaRegistryFactory is not necessary now - removed

In version 1.7.0​

Scenario authoring changes​

  • #3701 Right now access in SpEL to not existing field on TypedMap won't throw exception, just will return null
  • #3727 Improvements: Change RR Sink validation way:
    • Added param Value validation mode at RR response component
    • We no longer support nullable param from Everit schema. Nullable schema are supported by union with null e.g. `["null", "string"]

Configuration changes​

  • #3768 request-response-embedded and streaming-lite-embedded DeploymentManager types where replaced by one lite-embedded DeploymentManager type with two modes: streaming and request-response like it is done in lite-k8s case

Code API changes​

  • #3560, #3595 Remove dependency on flink-scala. In particular:
    • Switched from using scala.DataStream to datastream.DataStream. Some tools exclusive to scala datastreams are available in engine.flink.api.datastream
    • Scala based TypeInformation derivation is no longer used, for remaining cases flink-scala-utils module is provided (probably will be removed in the future)
  • #3680 SubprocessRef::outputVariableNames type is changed from Option[Map[String,String]] with default None, to Map[String,String] with default Map.empty
  • #3692 Rename mockedResult to externalInvocation in test results collectors.
  • #3606 Removed nussknacker-request-response-app. As a replacement you can use:
    • nussknacker-request-response-app in version <= 1.6
    • Lite K8s engine with request-response processing mode
    • lite-embedded Deployment Manager with request-response processing mode
  • #3610 Removed deprecated code. For details see changes in pull request.
  • #3607 Request-response jsonSchema based encoder:
    • ValidationMode moved to package pl.touk.nussknacker.engine.api.validation in nussknacker-components-api
    • BestEffortJsonSchemaEncoder moved to package pl.touk.nussknacker.engine.json.encode in nussknacker-json-utils
  • #3738 Kafka client libraries upgraded to 3.2.3. If using older Flink version, make sure to use 2.8.x client libraries. For Flink versions 1.15.0-1.15.2 include also fixed KafkaMetricWrapper
  • #3668 Method runWithRequests of RequestResponseTestScenarioRunner (returned by TestScenarioRunner.requestResponseBased()) now returns ValidatedNel with scenario compilation errors instead of throwing exception in that case

REST API changes​

  • #3576 /processes endpoint without query parameters returns all scenarios - the previous behaviour was to return only unarchived ones. To fetch only unarchived scenarios isArchived=false query parameter has to be passed.

Other changes​

  • #3824 Due to data serialization fix, Flink scenarios using Kafka sources with schemas may be incompatible and may need to be restarted with clean state.

In version 1.6.0​

  • #3440 Feature: allow to define fragment's outputs
    • Right now using fragments in scenario is changed. We have to provide each outputName for outputs defined in fragment.

Scenario authoring changes​

  • #3370 Feature: scenario node category verification on validation From now import scenario with nodes from other categories than scenario category will be not allowed.
  • #3436 Division by zero will cause validation error. Tests that rely on 1/0 to generate exceptions should have it changed to code like 1/{0, 1}[0]
  • #3473 JsonRequestResponseSinkFactory provides also 'raw editor', to turn on 'raw editor' add SinkRawEditorParamName -> "true"
  • #3608 Use ZonedDateTime for date-time JsonSchema format, OffsetTime for time format.

Code API changes​

  • #3406 Migration from Scalatest 3.0.8 to Scalatest 3.2.10 - if necessary, see the Scalatest migration guides, https://www.scalatest.org/release_notes/3.1.0 and https://www.scalatest.org/release_notes/3.2.0
  • #3431 Renamed helper-utils to default-helpers, separated MathUtils from components-utils to math-utils, removed dependencies from helper-utils
  • #3420 DeploymentManagerProvider.typeSpecificInitialData takes deploymentConfig Config now
  • #3493, #3582 Added methods DeploymentManagerProvider.additionalPropertiesConfig, DeploymentManagerProvider.additionalValidators
  • #3506 Changed LocalDateTime to Instant in OnDeployActionSuccess in listener-api
  • #3513 Replace EspProcess with CanonicalProcess in all parts of the API except for the compiler.
  • #3545 TestScenarioRunner.flinkBased should be used instead of NuTestScenarioRunner.flinkBased. Before this, you need to import pl.touk.nussknacker.engine.flink.util.test.FlinkTestScenarioRunner._
  • #3386 Changed CustomProcessValidator validate method. It now receives CanonicalProcess instead of DisplayableProcess and returns ValidatedNel[ProcessCompilationError, Unit] instead of ValidationResult. Moved CustomProcessValidator from module nussknacker-restmodel in package validation to nussknacker-extensions-api.
  • #3586 Module nussknacker-ui was renamed to nussknacker-designer, ui.conf was renamed to designer.conf, defaultUiConfing.conf renamed to defaultDesignerConfig.conf

REST API changes​

  • #3506 Dates returned by REST API (createdAt, modifiedAt, createDate) are now returned in Zulu time, with timezone indication. This affects e.g. /api/procecesses, /api/processes/{scenarioId}, /api/processes/{scenarioId}/activity
  • #3542 Node additional info path renamed from nodes/{scenarioId}/additionalData to nodes/{scenarioId}/additionalInfo

Scenario API changes​

  • #3471, #3553 RequestResponseMetaData(path) is changed to RequestResponseMetaData(slug). V1_033__RequestResponseUrlToSlug migration is ready for that, the change also applies to Scenario DSL.
  • #3513 Scenario DSL returns CanonicalProcess instead of EspProcess.
  • #3630 SubprocessOutput changed to SubprocessUsageOutput, changes in OutputVar definition

Configuration changes​

  • #3425 Deployment Manager for request-response-embedded configuration parameters changed:
    • interface -> http.interface
    • port -> http.port
    • definitionMetadata -> request-response.definitionMetadata
  • #3502 Refactor of KafkaProperties: kafkaAddress property has been deprecated. Please provide kafkaProperties."bootstrap.servers" instead

Other changes​

  • #3441 Updated Flink 1.14.5 -> 1.15.2. Some Flink artefacts no longer have Scala version. Test using Flink may need to disable checkpointing or reduce time between checkpoints to prevent timeouts or long waits for tasks to finish.

In version 1.5.0​

Configuration changes​

  • #2992 deploySettings changed to deploymentCommentSettings, now when specified require you to also specify field validationPattern, specifying exampleComment is optional.
  • commentSettings fields modified. matchExpression changed to substitutionPattern, link changed to substitutionLink.
  • #3165 Config is not exposed over http (GET /api/app/config/) by default. To enable it set configuration enableConfigEndpoint to true.
  • #3223 OAuth2 configuration defaultTokenExpirationTime changed to defaultTokenExpirationDuration
  • #3263 Batch periodic scenarios carry processing type to distinguish scenarios with different categories. For existing scenarios processing type is migrated to default. Set deploymentManager.processingType to default or update periodic scenarios table with actual processing type value - ideally it should be same value as the periodic engine key in scenarioTypes.

Code API changes​

  • #2992 OnDeployActionSuccess in ProcessChangeEvent now requires instance of Option[Comment] instead of Option[String] as parameter with deploymentComment information. Added abstract class Comment in listener-api.
  • #3136 Improvements: Lite Kafka testkit
    • ConfluentUtils.serializeRecordToBytesArray replaced by ConfluentUtils.serializeDataToBytesArray
    • ConfluentUtils.deserializeSchemaIdAndRecord replaced by ConfluentUtils.deserializeSchemaIdAndData
  • #3178 Improvements: more complex test scenario runner result:
    • Right now each method from TestScenarioRunner should return ValidatedNel[ProcessCompilationError, RunResult[R]] where:
      • Invalid is representation of process compilation errors
      • Valid is representation of positive and negative scenario running result
  • #3255 TestReporter util class is safer to use in parallel tests, methods require passing scenario name
  • #3265 #3288 #3297 #3299#3309 #3316 #3322 #3328 #3330 Changes related with UniversalKafkaSource/Sink:
    • RuntimeSchemaData is generic - parametrized by ParsedSchema (AvroSchema and JsonSchema is supported).
    • NkSerializableAvroSchema renamed to NkSerializableParsedSchema
    • SchemaWithMetadata wraps ParsedSchema instead of Avro Schema.
    • SchemaRegistryProvider refactoring:
      • rename SchemaRegistryProvider to SchemaBasedSerdeProvider
      • decouple SchemaRegistryClientFactory from SchemaBasedSerdeProvider
    • KafkaAvroKeyValueDeserializationSchemaFactory renamed to KafkaSchemaBasedKeyValueDeserializationSchemaFactory
    • KafkaAvroValueSerializationSchemaFactory renamed to KafkaSchemaBasedValueSerializationSchemaFactory
    • KafkaAvroKeyValueSerializationSchemaFactory renamed to KafkaSchemaBasedKeyValueSerializationSchemaFactory
  • #3253 DeploymentManager has separate validate method, which should perform initial scenario validation and return reasonably quickly (while deploy can e.g. make Flink savepoint etc.)
  • #3313 Generic types handling changes:
    • Typed.typedClass(Class[_], List[TypingResult]) is not available anymore. You should use more explicit Typed.genericTypeClass instead
    • We check count of generic parameters in Typed.genericTypeClass - wrong number will cause throwing exception now
    • We populate generic parameters by correct number of Unknown in non-generic aware versions of Typed factory methods like Typed.apply or Typed.typedClass
  • #3071 More strict Avro schema validation:
    • ValidationMode.allowOptional was removed, instead of it please use ValidationMode.lax
    • ValidationMode.allowRedundantAndOptional was removed, instead of it please use ValidationMode.lax
    • Changes of ValidationMode, fields: acceptUnfilledOptional and acceptRedundant were removed
  • #3376 FlinkKafkaSource.flinkSourceFunction, FlinkKafkaSource.createFlinkSource and DelayedFlinkKafkaConsumer.apply takes additional argument, FlinkCustomNodeContext now
  • #3272 KafkaZookeeperServer renamed to EmbeddedKafkaServer, zooKeeperServer field changed type to Option and is hidden now.
  • #3365 Numerous renames:
    • module nussknacker-avro-components-utils -> nussknacker-schemed-kafka-components-utils
    • module nussknacker-flink-avro-components-utils -> nussknacker-flink-schemed-kafka-components-utils
    • package pl.touk.nussknacker.engine.avro -> pl.touk.nussknacker.engine.schemedkafka
    • object KafkaAvroBaseComponentTransformer -> KafkaUniversalComponentTransformer
  • #3412 More strict filtering method types. Methods with parameters or result like Collection[IllegalType] are no longer available in SpEl.
  • #3542 Numerous renames:
    • trait NodeAdditionalInfo -> AdditionalInfo,
    • class MarkdownNodeAdditionalInfo -> MarkdownAdditionalInfo
    • trait NodeAdditionalInfoProvider -> AdditionalInfoProvider - the SPI provider's configuration files must be renamed from pl.touk.nussknacker.engine.additionalInfo.NodeAdditionalInfoProvider to pl.touk.nussknacker.engine.additionalInfo.AdditionalInfoProvider
    • method AdditionalInfoProvider.additionalInfo renamed to nodeAdditionalInfo and new method added propertiesAdditionalInfo

REST API changes​

  • #3169 API endpoint /api/app/healthCheck returning short JSON answer with "OK" status is now not secured - before change it required to be an authenticated user with "read" permission.

Scenario authoring changes​

  • #3187 #3224 Choice component replaces Switch component. "Default" choice edge type, exprVal and expression are now deprecated. For existing usages, you don't need to change anything. For new usages, if you want extract value e.g. to simplify choice conditions, you need to define new local variable before choice using variable component. "Default" choice edge type can be replaced by adding "true" condition at the end of list of conditions

Breaking changes​

  • #3328 Due to addition of support for different schema type (AvroSchema and JsonSchema for now) serialization format of NkSerializableParsedSchema has changed. Flink state compatibility of scenarios which use Avro sources or sinks has been broken.
  • #3365 Due to renames (see section Code API changes) Flink state compatibility of scenarios which use Avro sources or sinks has been broken.

Other changes​

  • #3249#3250 Some kafka related libraries were bumped: Confluent 5.5->7.2, avro 1.9->1.11, kafka 2.4 -> 3.2. It may have influence on your custom components if you depend on kafka-components-utils or avro-components-utils module
  • #3376 Behavior of Flink's Kafka deserialization errors handling was changed - now instead of job failure, invalid message is omitted and configured exceptionHandler mechanism is used.

In version 1.4.0​

Configuration changes​

  • security.rolesClaim changed to security.rolesClaims, type changed to list of strings
  • kafka.schemaRegistryCacheConfig configuration entry was added - it was hardcoded before. Default value of kafka.schemaRegistryCacheConfig.availableSchemasExpirationTime was changed from 1 minute to 10 seconds which will cause more often schema cache invalidation
  • #3031 Attachments are now stored in database (see more in section Other changes). attachmentsPath was removed. Optional config attachments.maxSizeInBytes was introduced with default value of 10mb

Code API changes​

  • #2983 Extract Permission to extensions-api
    • Moved pl.touk.nussknacker.ui.security.api.Permission (security module) to pl.touk.nussknacker.security.Permission (extension-api module)
  • #3040 Deprecated pl.touk.nussknacker.engine.api.ProcessListener.sinkInvoked method. Switch to more general endEncountered method.
  • #3076 new implicit parameter componentUseCase: ComponentUseCase was added to invoke() method of all services extending EagerServiceWithStaticParameters

Other changes​

  • #3031 Attachments are now stored in database. As this feature was rarely used, automatic migration of attachments from disk to db is not provided. To stay consistent db table process_attachments had to be truncated.

Breaking changes​

  • #3029 KafkaConfig has new field schemaRegistryCacheConfig: SchemaRegistryCacheConfig. Flink state compatibility has been broken.
  • #3116 Refactor SchemaRegistryClientFactory so it takes dedicated config object instead of KafkaConfig. This change minimizes chance of future Flink state compatibility break. SchemaIdBasedAvroGenericRecordSerializer is serialized in Flink state, so we provide it now with as little dependencies as necessary. Flink state compatibility has been broken again.
  • #3363 Kafka consumer no longer set auto.offset.reset to earliest by default. For default configuration files, you can use KAFKA_AUTO_OFFSET_RESET env variable to easily change this setting.

In version 1.3.0​

Code API changes​

  • #2741 #2841 Remove custom scenario provides some changes on API:
    • Replace ProcessDeploymentData by CanonicalProcess (as VO)
    • Replace scenario jsonString by CanonicalProcess at DeploymentManager, ProcessConfigEnricherInputData
  • #2773 Using VersionId / ProcessId / ProcessName instead of Long or String:
    • PullProcessRepository API was changed, right now we use VersionId instead of Long
  • #2830 RunMode is renamed to ComponanteUseCase and Normal value is split into: EngineRuntime, Validation, ServiceQuery, TestDataGeneration. RunMode.Test becomes ComponanteUseCase.TestRuntime
  • #2825, #2868 #2912 API modules changes:
    • Extracted new modules:
      • nussknacker-scenario-api with all scenario API parts from api and interpreter
      • nussknacker-components-api (and nussknacker-lite-components-api, nussknacker-flink-components-api etc.), which contain API for creating components
      • nussknacker-common-api - base value classes shared between scenario-api and components-api like NodeId, Metadata etc.
      • nussknacker-extensions-api - API of extensions other than components
    • Because of that, some changes in code were also introduced:
      • NodeId moved from pl.touk.nussknacker.engine.api.context.ProcessCompilationError to pl.touk.nussknacker.engine.api
      • NodeExpressionId, DefaultExpressionId and branchParameterExpressionId moved from pl.touk.nussknacker.engine.api.context.ProcessCompilationError to pl.touk.nussknacker.engine.graph.expression
      • JobData no longer contains DeploymentData, which is not accessible for components anymore
      • DisplayJson, WithJobData, MultiMap moved to utils
      • Some methods from API classes (e.g. Parameter.validate) and classes (InterpretationResult) moved to interpreter
      • DeploymentManagerProvider.createDeploymentManager takes now BaseModelData as an argument instead of ModelData. If you want to use this data to invoke scenario, you should cast it to invokable representation via: import ModelData._; modelData.asInvokableModelData
  • #2878 #2898 #2924 Cleaning up of -utils modules
    • Extracted internal classes, not intended to be used in extensions to nussknacker-internal-utils module
    • Extracted component classes, not used directly by runtime/designer to nussknacker-components-utils module
    • Extracted kafka component classes, not used directly by lite-kafka-runtime/kafka-test-utils to nussknacker-kafka-components-utils
    • Moved some classes that are in fact part of API to -api modules (e.g. ToJsonEncoder)
    • Module renames:
      • nussknacker-avro-util to nussknacker-avro-components-utils
      • nussknacker-flink-avro-util to nussknacker-flink-avro-components-utils
      • nussknacker-flink-kafka-util to nussknacker-flink-kafka-components-utils
      • nussknacker-flink-util to nussknacker-flink-components-utils
      • nussknacker-request-response-util to nussknacker-request-response-components-utils
      • nussknacker-model-util to nussknacker-helpers-utils
    • Minor changes in code:
      • Use val docsConfig = new DocsConfig(config); import docsConfig._ instead of implicit val docsConfig = (...); import DocsConfig._
      • Some components specific methods are not available from KafkaUtils. Instead, they are available from KafkaComponentsUtils
      • ToJsonEncoder.encoder takes Any => Json function instead of BestEffortJsonEncoder as a parameter
  • #2907 Hide some details of metrics to utils-internal (InstantRateMeter, InstantRateMeterWithCount), use method added to MetricsProviderForScenario
  • #2916 Changes in ProcessState API.
    • Six similar methods creating ProcessState based on StateStatus and some other details merged to one.
      • Methods removed:
        • Two variants of ProcessState.apply taking ProcessStateDefinitionManager as a parameter
        • SimpleProcessState.apply
        • Two variants of ProcessStatus.simple
        • ProcessStatus.createState taking ProcessStateDefinitionManager as a parameter
      • Method added: ProcessStateDefinitionManager.processState with some default parameter values
    • ProcessStatus class is removed at all. All methods returning ProcessState by it moved to SimpleProcessStateDefinitionManager and removed previousState: Option[ProcessState] from it. If you want to keep previous state's deployment details and only change "status details" just use processState.withStatusDetails method
    • ProcessState, CustomAction and its dependencies moved from nussknacker-deployment-manager-api to nussknacker-scenario-deployment-api, restmodel module not depend on deployment-manager-api anymore
    • #2969 Action ProcessActionType.Deploy is now available by default for scenarios in SimpleStateStatus.DuringDeploy state. Mind this if you depend on OverridingProcessStateDefinitionManager or SimpleProcessStateDefinitionManager, and specifically on theirs statusActions method. As an exception, implementation for Flink FlinkProcessStateDefinitionManager stays the same as before (only ProcessActionType.Cancel is possible in this state), but this may be unified in the future.

Other changes​

  • #2886 This change can break previous flink snapshot compatibility. Restoring state from previous snapshot asserts that restored serializer UID matches current serializer UID. This change ensures that in further release deployments UIDs persisted within snapshots are not re-generated in runtime.
  • #2950 Remove MATH helper, use NUMERIC methods (they work better with some number types conversions)

In version 1.2.0​

Configuration changes​

  • #2483 COUNTS_URL environment variable is not INFLUXDB_URL, without query path part.
  • #2493 kafka configuration should be moved to components provider configuration - look at components.kafka in dev-application.conf for example
  • #2624 Default name for process tag is now scenario. This affects metrics and count functionalities. Please update you Flink/Telegraf setup accordingly (see nussknacker-quickstart for details). If you still want to use process tag (e.g. you have a lot of dashboards), please set countsSettings.metricsConfig.scenarioTag setting to process Also, dashboard links format changed, see documentation for the details.
  • #2645 Default models: genericModel.jar, liteModel.jar. were merged to defaultModel.jar, managementSample.jar was renamed to devModel.jar. If you use defaultModel.jar it's important to include flinkExecutor.jar explicitly on model classpath.

Scenario authoring changes​

  • #2564 Flink union now takes only 'Output expression' parameters for branches (previously 'value' parameter), output variable must be of the same type, if you want to distinguish source branch in output variable please use map variable, example in Basic Nodes docs.

Other changes​

  • #2554 Maven artifact nussknacker-kafka-flink-util become nussknacker-flink-kafka-util and nussknacker-avro-flink-util become nussknacker-flink-avro-util. General naming convention is nussknacker-$runtimeType-$moduleName. Components inside distribution changed layout to components(/$runtimeType)/componentName.jar e.g. components/flink/kafka.jar or components/openapi.jar KafkaSource become FlinkKafkaSource, ConsumerRecordBasedKafkaSource become FlinkConsumerRecordBasedKafkaSource, KafkaSink become FlinkKafkaSink, KafkaAvroSink become FlinkKafkaAvroSink
  • #2535, #2625, #2645 Rename standalone to request-response:
    • Renamed modules and artifacts
    • StandaloneMetaData is now RequestResponseMetaData
    • Move request-response modules to base dir.
    • standalone in package names changed to requestresponse
    • Standalone in class/variable names changed to RequestResponse
    • DeploymentManager/Service uses dedicated format of status DTO, instead of the ones from deployment-manager-api
    • Removed old, deprecated jarPath settings, in favour of classPath used in other places
    • Extracted nussknacker-lite-request-response-components module
  • #2582 KafkaUtils.toProducerProperties setup only basic properties now (bootstrap.servers and serializers) - before the change it was setting options which are not always good choice (for transactional producers wasn't)
  • #2600 ScenarioInterpreter, ScenarioInterpreterWithLifecycle now takes additional generic parameter: Input. ScenarioInterpreter.invoke takes ScenarioInputBatch which now contains list of SourceId -> Input instead of SourceId -> Context. Logic of Context preparation should be done in LiteSource instead of before ScenarioInterpreter.invoke. invocation It means that LiteSource also takes this parameter and have a new method createTransformation.
  • #2635 ContextInitializer.initContext now takes ContextIdGenerator instead of nodeId and returns just a function with strategy of context initialization instead of serializable function with Lifecycle. To use it with Flink engine, use FlinkContextInitializingFunction wrapper.
  • #2649 DeploymentManagerProvider takes new ProcessingTypeDeploymentService class as an implicit parameter
  • #2564 'UnionParametersMigration' available to migrate parameter name from 'value' to 'Output expression' - please turn it on you are using 'union' like component
  • #2645 Simplify structure of available models (implementations of ProcessConfigCreator). defaultModel.jar and components should be used instead of custom implementations of ProcessConfigCreator, the only exception is when one wants to customize ExpressionConfig. Also, nussknacker-flink-engine module became nussknacker-flink-executor.
  • #2651 ValidationContext.clearVariables now clears also parent reference. Important when invoked inside fragments.
  • #2673 KafkaZookeeperUtils renamed to KafkaTestUtils, it doesn't depend on ZooKeeper anymore.
  • #2686 ServiceWithStaticParameters renamed to EagerServiceWithStaticParameters.
  • #2695 nodeId replaced with NodeComponentInfo in NuExceptionInfo. Simple wrapper class which holds the same nodeId and also componentInfo. Migration is straightforward, just put nodeId into the new case class:
    • NuExceptionInfo(None, exception, context) => stays the same
    • NuExceptionInfo(Some(nodeId), exception, context) => NuExceptionInfo(Some(NodeComponentInfo(nodeId, None)), exception, context)
      • if an exception is thrown inside the component, additional information can be provided:
        • for base component (like filter or split): NodeComponentInfo.forBaseNode("nodeId", ComponentType.Filter)
        • for other: NodeComponentInfo("nodeId", ComponentInfo("kafka-avro-source", ComponentType.Source))
    • The same migration has to be applied to ExceptionHandler.handling() method.
  • #2824 'ProcessSplitterMigration' available to migrate node name from 'split' to 'for-each' (see #2781)- please turn it on if you are using 'split' component

In version 1.1.0​

info

Summary:

  • A lot of internal refactoring was made to separate code/API specific for Flink. If your deployment has custom components pay special attention to:
    • Lifecycle management
    • Kafka components
    • Differences in artifacts and packages
  • Some of the core dependencies: cats, cats-effect and circe were upgraded. It affects mainly code, but it may also have impact on state compatibility and performance.
  • Default Flink version was bumped do 1.14 - see https://github.com/TouK/nussknacker-flink-compatibility on how to run Nu on older Flink versions.
  • Execution of SpEL expressions is now checked more strictly, due to security considerations. These checks can be overridden with custom ExpressionConfig.
info
  • Apart from that:
    • minor configuration naming changes
    • removal of a few of minor, not documented features (e.g. SQL Variable)
  • #2208 Upgrade, cats, cats-effects, circe. An important nuisance: we didn't upgrade sttp, so we cannot depend on "com.softwaremill.sttp.client" %% "circe". Instead, the code is copied. Make sure you don't include sttp-circe integration as transitive dependency, but use class from http-utils instead.
  • #2176 EnrichDeploymentWithJarDataFactory was replaced with ProcessConfigEnricher.
  • #2278 SQL Variable is removed
  • #2280 Added optional defaultValue field to Parameter. In GenericNodeTransformation can be set to None - values will be determined automatically.
  • #2289 Savepoint path in /api/adminProcessManagement/deploy endpoint is passed as a savepointPath parameter instead of path segment.
  • #2293 Enhancement: change nodeCategoryMapping configuration to componentsGroupMapping
  • #2301 #2620 GenericNodeTransformation.initialParameters was removed - now GenericNodeTransformation.contextTransformation is used instead. To make Admin tab -> Invoke service form working, use WithLegacyStaticParameters trait
  • #2409 JsonValidator is now not determined by default based on JsonParameterEditor but must be explicitly defined by @JsonValidator annotation
  • #2304 Upgrade to Flink 1.14. Pay attention to Flink dependencies - in some (e.g. runtime) there is no longer scala version.
  • #2295 FlinkLazyParameterFunctionHelper allows (and sometimes requires) correct exception handling
  • #2307 Changed nussknacker-kafka module name to nussknacker-kafka-util
  • #2310 Changed nussknacker-process module name to nussknacker-flink-engine
  • #2300 #2343 Enhancement: refactor and improvements at components group:
    • Provided ComponentGroupName as VO
    • SingleNodeConfig was renamed to SingleComponentConfig and moved from pl.touk.nussknacker.engine.api.process package to pl.touk.nussknacker.engine.api.component
    • Configuration category in node configuration was replaced by componentGroup
    • Configuration nodes in model configuration was replaced by componentsUiConfig
    • Additional refactor: ProcessToolbarService moved from pl.touk.nussknacker.ui.service package to pl.touk.nussknacker.ui.process
    • Additional refactor: ProcessToolbarService moved from pl.touk.nussknacker.ui.service package to pl.touk.nussknacker.ui.process
    • DefinitionPreparer was renamed to ComponentDefinitionPreparer
    • NodesConfigCombiner was removed
    • REST API /api/processDefinitionData/* response JSON was changed:
      • nodesToAdd was renamed to componentGroups
      • posibleNode was renamed to components
      • nodesConfig was renamed to componentsConfig
      • config icon property from componentsConfig right now should be relative to http.publicPath e.g. /assets/components/Filter.svg (before was just Filter.svg) or url (with http / https)
  • #2346 Remove endResult from Sink in graph.
    • Sink no longer defines testOutput method - they should be handled by respective implementations
    • Change in definition of StandaloneSink previously StandaloneSinkWithParameters, as output always has to be computed with sink parameters now
    • Changes in definition of FlinkSink, to better handle capturing test data
    • Removal of .sink method in GraphBuilder - use .emptySink if suitable
  • #2331
    • KafkaAvroBaseTransformer companion object renamed to KafkaAvroBaseComponentTransformer
    • KryoGenericRecordSchemaIdSerializationSupport renamed to GenericRecordSchemaIdSerializationSupport
  • #2305 Enhancement: change processingTypeToDashboard configuration to scenarioTypeToDashboard
  • #2296 Scenarios & Fragments have separate TypeSpecificData implementations. Also, we remove isSubprocess field from process JSON, and respectively from MetaData constructor. See corresponding db migration V1_031__FragmentSpecificData.scala
  • #2368 WithCategories now takes categories as an Option[List[String]] instead of List[String]. You should wrap given list of categories with Some(...). None mean that component will be available in all categories.
  • #2360 union, union-memo and dead-end components were extracted from model/genericModel.jar to components/baseComponents.jar If you have your own application.conf which changes scenarioTypes, you should add "components/baseComponents.jar" entry into classPath array
  • #2337 Extract base engine from standalone
    • Common functionality of base engine (i.e. microservice based, without Flink) is extracted to base-api and base-runtime
    • new API for custom components (pl.touk.nussknacker.engine.baseengine.api.customComponentTypes)
    • StandaloneProcessInterpreter becomes StandaloneScenarioEngine
    • Replace Either[NonEmptyList[Error], _] with ValidatedNel[Error, _] as return type
    • StandaloneContext becomes EngineRuntimeContext
  • #2349 queryable-state module was removed, FlinkQueryableClient was moved to nussknacker-flink-manager. PrettyValidationErrors, CustomActionRequest and CustomActionResponse moved from nussknacker-ui to nussknacker-restmodel.
  • #2361 Removed security dependency from listener-api. LoggedUser replaced with dedicated class in listener-api.
  • #2385 Deprecated CustomStreamTransformer.clearsContext was removed. Use
@MethodToInvoke
def execute(...) = 
  ContextTransformation
    .definedBy(ctx => Valid(ctx.clearVariables ...))
    .implementedBy(...)
}

instead.

  • #2348 #2459 #2486 #2490 #2496 #2536 Introduce KafkaDeserializationSchema and KafkaSerializationSchema traits to decouple from flink dependency. move KeyedValue to nussknacker-util, move SchemaRegistryProvider to utils/avro-util To move between nussknacker's/flink's Kafka(De)serializationSchema use wrapToFlink(De)serializatioinSchema from FlinkSerializationSchemaConversions. SchemaRegistryProvider and ConfluentSchemaRegistryProvider is now in nussknacker-avro-util module. FlinkSourceFactory is gone - use SourceFactory instead. KafkaSourceFactory, KafkaAvroSourceFactory, KafkaSinkFactory, KafkaAvroSinkFactory, and ContextIdGenerator not depends on flink. Extracted KafkaSourceImplFactory, KafkaSinkImplFactory and KafkaAvroSinkImplFactory which deliver implementation of component (after all validations and parameters evaluation). Use respectively: FlinkKafkaSourceImplFactory, FlinkKafkaSinkImplFactory and FlinkKafkaAvroSinkImplFactory to deliver flink implementations. Moved non-flink specific serializers, deserializers, BestEffortAvroEncoder, ContextIdGenerators and RecordFormatters to kafka-util/avro-util KafkaDelayedSourceFactory is now DelayedKafkaSourceFactory. FixedRecordFormatterFactoryWrapper moved to RecordFormatterFactory
  • #2477 FlinkContextInitializer and FlinkGenericContextInitializer merged to ContextInitializer, BasicFlinkContextInitializer and BasicFlinkGenericContextInitializer merged to BasicContextInitializer. All of them moved to pl.touk.nussknacker.engine.api.process package. ContextInitializer.validationContext returns ValidatedNel - before this change errors during context initialization weren't accumulated. ContextInitializingFunction now is a scala's function instead of Flink's MapFunction. You should wrap it with RichLifecycleMapFunction to make sure that it will be opened correctly by Flink. InputMeta was moved to kafka-util module.
  • #2389 #2391 deployment-manager-api module was extracted and DeploymentManagerProvider, ProcessingTypeData and QueryableClient was moved from interpreter into it. DeploymentManager, CustomAction and ProcessState was moved from api to deployment-manager-api. ProcessingType was moved to rest-model package.
  • #2393 Added ActorSystem, ExecutionContext and SttpBackend into DeploymentManagerProvider.createDeploymentManager. During clean ups also was removed nussknacker-http-utils dependency to async-http-client-backend-future and added SttpBackend to CountsReporterCreator.createReporter arguments.
  • #2397 Common EngineRuntimeContext lifecycle and MetricsProvider. This may cause runtime consequences - make sure your custom services/listeners invoke open/close correctly - especially in complex inheritance scenarios.
    • Lifecycle has now EngineRuntimeContext as parameter, JobData is embedded in it.
    • TimeMeasuringService replaces GenericTimeMeasuringService, Flink/Standalone flavours of TimeMeasuringService are removed
    • EngineRuntimeContext and MetricsProvider moved to base API, RuntimeContextLifecycle moved to base API as Lifecycle
    • GenericInstantRateMeter is now InstantRateMeter
    • Flink RuntimeContextLifecycle should be replaced in most cases by Lifecycle
    • In Flink engine MetricsProvider (obtained with EngineRuntimeContext) should be used in most places instead of MetricUtils
  • #2486 Context.withInitialId is deprecated now - use EngineRuntimeContext.contextIdGenerator instead. EngineRuntimeContext can be accessible via FlinkCustomNodeContext.convertToEngineRuntimeContext
  • #2377 #2534 Removed clazz from SourceFactory. Remove generic parameter from Source and SourceFactory. Return type of source should be returned either by:
    • returnType field of @MethodToInvoke
    • ContextTransformation API
    • GenericNodeTransformer API
    • SourceFactory.noParam
  • #2453 Custom actions for PeriodicDeploymentManager now can be defined and implemented outside this class, in PeriodicCustomActionsProvider created by PeriodicCustomActionsProviderFactory. If you do not need them, just pass PeriodicCustomActionsProviderFactory.noOp to object's PeriodicDeploymentManager factory method.
  • #2501 nussknacker-baseengine-components module renamed to nussknacker-lite-base-components
  • #2221 ReflectUtils fixedClassSimpleNameWithoutParentModule renamed to simpleNameWithoutSuffix
  • #2495 TypeSpecificDataInitializer trait change to TypeSpecificDataInitializ
  • #2245 FailedEvent has been specified in FailedOnDeployEvent and FailedOnRunEvent

In version 1.0.0​

  • #1439 #2090 Upgrade do Flink 1.13.
    • setTimeCharacteristic is deprecated, and should be handled automatically by Flink.
    • UserClassLoader was removed, use appropriate Flink objects or context ClassLoader.
    • RocksDB configuration is turned on by rocksdb.enable instead of rocksdb.checkpointDataUri which is not used now.
  • #2133 SQL Variable is hidden in generic model, please look at comment in defaultModelConfig.conf
  • #2152 schedulePropertyExtractor parameter of PeriodicDeploymentManagerProvider was changed to a factory, replace with a lambda creating the original property extractor.
  • #2159 useTypingResultTypeInformation option is now enabled by default
  • #2108 Changes in ClassExtractionSettings:
    • Refactor of classes defining extraction rules,
    • TypedClass has private apply method, please use Typed.typedClass
    • Fewer classes/methods are accessible in SpEL, in particular Scala collections, internal time API, methods returning or having parameters from excluded classes
  • Changes in OAuth2 security components:
    • refactoring of OpenIdConnectService, now it's named GenericOidcService (it's best to use OidcService, which can handle most of the configuration automatically)
  • New security settings, in particular new flags in ExpressionConfig:
    • strictMethodsChecking
    • staticMethodInvocationsChecking
    • methodExecutionForUnknownAllowed
    • dynamicPropertyAccessAllowed
    • spelExpressionExcludeList
  • #2101 Global permissions can be arbitrary string, for admin user it's not necessary to return global permissions
  • #2182 To avoid classloader leaks during SQL DriverManager registration, HSQLDB (used e.g. for SQL Variable) is no longer included in model jars, it should be added in Flink lib dir

In version 0.4.0​

  • #1479 ProcessId and VersionId moved to API included in ProcessVersion, remove spurious ProcessId and ProcessVersionId in restmodel.

  • #1422 Removed ServiceReturningType and WithExplicitMethod, use EagerServiceWithStaticParameters, EnricherContextTransformation or SingleInputGenericNodeTransformation

  • #1845 AuthenticatorData has been renamed to AuthenticationResources and changed into a trait, apply construction has been preserved. AuthenticatorFactory and its createAuthenticator method has been renamed to AuthenticationProvider and createAuthenticationResources. It is recommended to rename the main class of any custom authentication module to <Something>AuthenticationProvider accordingly.

  • #1542 KafkaConfig now has new parameter topicsExistenceValidationConfig. When topicsExistenceValidationConfig.enabled = true Kafka sources/sinks do not validate if provided topic does not exist and cluster is configured with auto.create.topics.enable=false

  • #1416 OAuth2Service has changed. You can still use your old implementation by importing OAuth2OldService with an alias. OAuth2ServiceFactory.create method now accepts implicit parameters for an ExecutionContext and sttp.HttpBackend. You can ignore them to maintain previous behaviour, but it is always better to use them instead of locally defined ones.

  • #1346 AggregatorFunction now takes type of stored state that can be immutable.SortedMap (previous behaviour) or java.util.Map (using Flink's serialization) and validatedStoredType parameter for providing better TypeInformation for aggregated values

  • #1343 FirstAggregator changed serialized state, it is not compatible, Aggregator trait has new method computeStoredType

  • #1352 and #1568 AvroStringSettings class has been introduced, which allows control whether Avro type string is represented by java.lang.String (also in runtime) or java.lang.CharSequence (implemented in runtime by org.apache.avro.util.Utf8). This setting is available through environment variable AVRO_USE_STRING_FOR_STRING_TYPE - default is true. Please mind that this setting is global - it applies to all processes running on Flink and also requires restarting TaskManager when changing the value.

  • #1361 Lazy variables are removed, you should use standard enrichers for those cases. Their handling has been source of many problems and they made it harder to reason about the exeuction of process.

  • #1373 Creating ClassLoaderModelData directly is not allowed, use ModelData.apply with plain config, wrapping with ModelConfigToLoad by yourself is not needed.

  • #1406 ServiceReturningType is deprecated in favour of EagerService

  • #1445 RecordFormatter now handles TestDataSplit for Kafka sources. It is required in KafkaSource creation, instead of TestDataSplit

  • #1433 Pass DeploymentData to process via JobData, additional parameters to deployment methods are needed. Separate ExternalDeploymentId from DeploymentId (generated by NK)

  • #1466 ProcessManager.deploy can return ExternalDeploymentId

  • #1464

    • Slight change of API of StringKeyedValueMapper
    • Change of semantics of some parameters of AggregatorFunction, AggregatorFunctionMixin (storedAggregateType becomes aggregateElementType)

  • #1405 'KafkaAvroSink' requires more generic 'AvroSinkValue' as value parameter

  • #1510

    • Change of FlinkSource API: sourceStream produces stream of initialized Context (DataStream[Context]) This initialization step was previously performed within FlinkProcessRegistrar.registerSourcePart. Now it happens explicitly within the flink source.
    • FlinkIntermediateRawSource is used as an extension to flink sources, it prepares source with typical stream transformations (add source function, set uid, assign timestamp, initialize Context)
    • FlinkContextInitializer is used to initialize Context. It provides map function that transforms raw event (produced by flink source function) into Context variable. Default implementation of FlinkContextInitializer, see BasicFlinkContextInitializer, sets raw event value to singe "input" variable.
    • For sources based on GenericNodeTransformation it allows to initialize Context with more than one variable. Default implementation of initializer, see BasicFlinkGenericContextInitializer, provides default definition of variables as a ValidationContext with single "input" variable. The implementation requires to provide separately the definition of "input" variable type (TypingResult). See GenericSourceWithCustomVariablesSample.
    • To enable "test source" functionality, a source needs to be extended with SourceTestSupport.
    • For flink sources that use TestDataParserProvider switch to FlinkSourceTestSupport (which is used to provide "test source" functionality for flink sources).
    • Old TestDataParserProvider is renamed to SourceTestSupport
    • To enable test data generator for "test source" , a source needs to be extended with both SourceTestSupport and TestDataGenerator. What was related to "test source" functionality and was obsolete in FlinkSource now is excluded to FlinkSourceTestSupport.
    • FlinkCustomNodeContext has access to TypeInformationDetection, it allows to get TypeInformation for the node stream mapping from ValidationContext.
    • For kafka sources RecordFormatter parses raw test data to ConsumerRecord which fits into deserializer (instead of ProducerRecord that required another transformation).
    • Definitions of names of common Context variables are moved to VariableConstants (instead of Interpreter).

  • #1497 Changes in PeriodicProcessManager, change PeriodicProperty to ScheduleProperty

  • #1499

    • trait KafkaAvroDeserializationSchemaFactory uses both key and value ClassTags and schemas (instead of value-only), check the order of parameters.
    • ClassTag is provided in params in avro key-value deserialization schema factory: KafkaAvroKeyValueDeserializationSchemaFactory
    • BaseKafkaAvroSourceFactory is able to read both key and value schema determiner to build proper DeserializationSchema (support for keys is not fully introduced in this change)

  • #1514 ExecutionConfigPreparer has different method parameter - JobData, which has more info than previous parameters

  • #1532 TypedObjectTypingResult#fields uses now scala.collection.immutable.ListMap to keep fields order

  • #1546 StandaloneCustomTransformer now takes a list of Context objects, to process them in one go

  • #1557 Some classes from standalone engine were moved to standalone api to remove engine to (model) utils dependency: StandaloneContext, StandaloneContextLifecycle, MetricsProvider

  • #1558 FlinkProcessRegistrar takes configuration directly from FlinkProcessCompiler (this can affect some tests setup)

  • #1631 Introduction of nussknacker.config.locations property, drop use of standard config.file property. Model configuration no longer has direct access to root UI config.

  • #1512

    • Replaced KafkaSourceFactory with source based on GenericNodeTransformation, which gives access to setup of ValidationContext and Context initialization. To migrate KafkaSourceFactory:
      • provide deserializer factory (source factory requires deserialization to ConsumerRecord):
        • use ConsumerRecordDeserializationSchemaFactory with current DeserializationSchema as a value deserializer, add key deserializer (e.g. org.apache.kafka.common.serialization.StringDeserializer)
        • or use FixedValueDeserializationSchemaFactory with simple key-as-string deserializer
      • provide RecordFormatterFactory
        • use ConsumerRecordToJsonFormatterFactory for whole key-value-and-metadata serialization
        • or, for value-only-and-without-metadata scenario, you can use current RecordFormater wrapped in FixedRecordFormatterFactoryWrapper
      • provide timestampAssigner that is able to extract time from ConsumerRecord[K, V]
    • Removed BaseKafkaSourceFactory with multiple topics support: use KafkaSourceFactory instead, see "source with two input topics" test case
    • Removed SingleTopicKafkaSourceFactory: use KafkaSourceFactory with custom prepareInitialParameters, contextTransformation and extractTopics to alter parameter list and provide constant topic value.
    • TypingResultAwareTypeInformationCustomisation is moved to package pl.touk.nussknacker.engine.flink.api.typeinformation

    Example of source with value-only deserialization and custom timestampAssigner:

    // provide new deserializer factory with old schema definition for event's value
    val oldSchema = new EspDeserializationSchema[SampleValue](bytes => io.circe.parser.decode[SampleValue](new String(bytes)).toOption.get)
    val schemaFactory: KafkaDeserializationSchemaFactory[ConsumerRecord[String, SampleValue]] = new FixedValueDeserializationSchemaFactory(oldSchema)

    // ... provide timestampAssigner that extracts timestamp from SampleValue.customTimestampField
    // ... or use event's metadata: record.timestamp()
    def timestampExtractor(record: ConsumerRecord[String, SampleValue]): Long = record.value().customTimestampField
    val watermarkHandler = StandardTimestampWatermarkHandler.boundedOutOfOrderness[ConsumerRecord[String, SampleValue]](timestampExtractor, java.time.Duration.ofMinutes(10L))
    val timestampAssigner: Option[TimestampWatermarkHandler[ConsumerRecord[String, SampleValue]]] = Some(watermarkHandler)

    // ... provide RecordFormatterFactory that allows to generate and parse test data with key, headers and other metadata
    val formatterFactory: RecordFormatterFactory = new ConsumerRecordToJsonFormatterFactory[String, SampleValue]

    // ... and finally
    val sourceFactory = new KafkaSourceFactory[String, SampleValue](schemaFactory, timestampAssigner, formatterFactory, dummyProcessObjectDependencies)

  • #1651 KafkaAvroSourceFactory provides additional #inputMeta variable with event's metadata.

    • That source now has key and value type parameters. That parameters are relevant for sources handling SpecificRecords. For GenericRecords use explicitly KafkaAvroSourceFactory[Any, Any].
    • SpecificRecordKafkaAvroSourceFactory extends whole KafkaAvroSourceFactory with context validation and initialization
    • New flag in KafkaConfig: useStringForKey determines if event's key should be intepreted as ordinary String (which is default scenario). It is used in deserialization and for generating/parsing test data.
    • SchemaRegistryProvider now provides factories to produce SchemaRegistryClient and RecordFormatter.
    • For ConfluentSchemaRegistryProvider KafkaConfig and ProcessObjectDependencies (that contains KafkaConfig data) are no longer required. That configuration is required by factories in the moment the creation of requested objects that happens in KafkaAvroSourceFactory (and that makes that all objects within KafkaAvroSourceFactory see the same kafka configuration).
    • Removed:
      • BaseKafkaAvroSourceFactory, the class is incorporated into KafkaAvroSourceFactory to provide elastic approach to create KafkaSource
      • with ReturningType for generic types (this is defined by ValidationContext, see also KafkaContextInitializer that allows to return more than one variable)
      • KafkaAvroValueDeserializationSchemaFactory (source requires deserialization to ConsumerRecord[K, V], there are only deserializers based on KafkaAvroKeyValueDeserializationSchemaFactory)
      • ConfluentKafkaAvroDeserializationSchemaFactory, use ConfluentKeyValueKafkaAvroDeserializationFactory
      • TupleAvroKeyValueKafkaAvroDeserializerSchemaFactory, this approach is deprecated due to #inputMeta variable that contains key data

    To migrate KafkaAvroSourceFactory:

    • Provide KafkaConfig with correct useStringForKey flag value. By default we want to EvictableStatehandle keys as ordinary String and all topics related to such config require only value schema definitions (key schemas are ignored). For specific scenario, when complex key with its own schema is provided, this flag is false and all topics related to this config require both key and value schema definitions. Example of default KafkaConfig override: override protected def prepareKafkaConfig: KafkaConfig = super.prepareKafkaConfig.copy(useStringForKey = false)
    • provide your own SchemaRegistryProvider (or use ConfluentSchemaRegistryProvider)
    • custom RecordFormatter can be wrapped in FixedRecordFormatterFactoryWrapper (or keep ConfluentAvroToJsonFormatterFactory)
    • provide timestampAssigner that is able to extract time from ConsumerRecord[K, V] (see example above)

  • #1741 Minor changes in KafkaUtils, NonTransientException uses Instant instead of LocalDateTime

  • #1806 Remove old, deprecated API:

    • EvictableState, RichEvictableState - use EvictableStateFunction
    • checkpointInterval - use checkpointConfig.checkpointInterval
    • old versions of sampleTransformers - use newer ones
    • MiniClusterExecutionEnvironment.runningJobs() - use flinkMiniClusterHolder.runningJobs()

  • #1807 Removed jdbcServer, please use Postgres for production-ready setups

  • #1799

    • RecordFormatterFactory instead of one, uses two type parameters: K, V
    • ConfluentAvroToJsonFormatter is produced by ConfluentAvroToJsonFormatterFactory
    • ConfluentAvroToJsonFormatter produces test data in valid JSON format, does not use Separator
    • ConfluentAvroMessageFormatter has asJson method instead of writeTo
    • ConfluentAvroMessageReader has readJson method instead of readMessage Example test data object:
    {"keySchemaId":null,"valueSchemaId":1,"consumerRecord":{"key":null,"value":{"first":"Jan","last":"Kowalski"},"topic":"testAvroRecordTopic1","partition":0,"offset":0,"timestamp":1624279687756,"timestampType":"CreateTime","headers":{},"leaderEpoch":0}}

  • #1663 Default FlinkExceptionHandler implementations are deprecated, use ConfigurableExceptionHandler instead.

  • #1731 RockDB config's flag incrementalCheckpoints is turned on by default.

  • #1825 Default dashboard renamed from flink-esp to nussknacker-scenario

  • #1836 Change default kafka.consumerGroupNamingStrategy to processId-nodeId.

  • #1357 Run mode added to nodes. ServiceInvoker interface was extended with new, implicit runMode parameter.

  • #1836 Change default kafka.consumerGroupNamingStrategy to processId-nodeId.

  • #1886 aggregate-sliding with emitWhenEventLeft = true, aggregate-tumbling and aggregate-session components now doesn't emit full context of variables that were before node (because of performance reasons and because that wasn't obvious which one context is emitted). If you want to emit some information other than aggregated value and key (availabled via new #key variable), you should use #AGG.map expression in aggregateBy.

  • #1910 processTypes renamed to scenarioTypes. You can still use old processTypes configuration. Old configuration will be removed in version 0.5.0.

  • Various naming changes:

    • #1917 configuration of engineConfig to deploymentConfig
    • #1921 ProcessManager to DeploymentManager
    • #1927 Rename outer-join to single-side-join

In version 0.3.0​

  • #1313 Kafka Avro API passes KafkaConfig during TypeInformation determining

  • #1305 Kafka Avro API passes RuntimeSchemaData instead of Schema in various places

  • #1304 SerializerWithSpecifiedClass was moved to flink-api module.

  • #1044 Upgrade to Flink 1.11. Current watermark/timestamp mechanisms are deprectated in Flink 1.11, new API TimestampWatermarkHandler is introduced, with LegacyTimestampWatermarkHandler as wrapper for previous mechanisms.

  • #1244 Parameter has new parameter 'variablesToHide' with Set of variable names that will be hidden before parameter's evaluation

  • #1159 #1170 Changes in GenericNodeTransformation API:

    • Now implementation takes additional parameter with final state value determined during contextTransformation
    • DefinedLazyParameter and DefinedEagerParameter holds expression: TypedExpression instead of returnType: TypingResult
    • DefinedLazyBranchParameter and DefinedEagerBranchParameter holds expressionByBranchId: Map[String, TypedExpression] instead of returnTypeByBranchId: Map[String, TypingResult]

  • #1083

    • Now SimpleSlidingAggregateTransformerV2 and SlidingAggregateTransformer is deprecated in favour of SlidingAggregateTransformerV2
    • Now SimpleTumblingAggregateTransformer is deprecated in favour of TumblingAggregateTransformer
    • Now SumAggregator, MaxAggregator and MinAggregator doesn't change type of aggregated value (previously was changed to Double)
    • Now SumAggregator, MaxAggregator and MinAggregator return null instead of 0D/Double.MaxValue/Double.MinValue for case when there was no element added before getResult

  • #1149 FlinkProcessRegistrar refactor (can affect test code)

  • #1166 model.conf should be renamed to defaultModelConfig.conf

  • #1218 FlinkProcessManager is no longer bundled in ui uber-jar. In docker/tgz distribution

  • #1255 Moved displaying Metrics tab to customTabs

  • #1257 Improvements: Flink test util package

    • Added methods: cancelJob, submitJob, listJobs, runningJobs to FlinkMiniClusterHolder
    • Deprecated: runningJobs, from MiniClusterExecutionEnvironment
    • Removed: getClusterClient from FlinkMiniClusterHolder interface, because of flink compatibility at Flink 1.9
    • Renamed: FlinkStreamingProcessRegistrar to FlinkProcessManager

  • #1303 TypedObjectTypingResult has a new field: additionalInfo

In version 0.2.0​

  • #1104 Creation of FlinkMiniCluster is now extracted from StoppableExecutionEnvironment. You should create it using e.g.:

    val flinkMiniClusterHolder = FlinkMiniClusterHolder(FlinkTestConfiguration.configuration(parallelism))
    flinkMiniClusterHolder.start()

    and then create environment using:

    flinkMiniClusterHolder.createExecutionEnvironment()

    . At the end you should cleanup flinkMiniClusterHolder by:

    flinkMiniClusterHolder.stop()

    . FlinkMiniClusterHolder should be created once for test class - it is thread safe and resource expensive. MiniClusterExecutionEnvironment in the other hand should be created for each process. It is not thread safe because underlying StreamExecutionEnvironment is not. You can use FlinkSpec to achieve that.

  • #1077

    • pl.touk.nussknacker.engine.queryablestate.QueryableClient was moved from queryableState module to pl.touk.nussknacker.engine.api.queryablestate package in api module
    • pl.touk.nussknacker.engine.queryablestate.QueryableState was moved to pl.touk.nussknacker.engine.api.queryablestate
    • CustomTransformers from pl.touk.nussknacker.engine.flink.util.transformer in flinkUtil module were moved to new flinkModelUtil module.
    • pl.touk.nussknacker.engine.testing.EmptyProcessConfigCreator was moved from interpreter module to pl.touk.nussknacker.engine.util.process package in util module

  • #1039 Generic parameter of LazyParameter[T] has upper bound AnyRef now to avoid problems with bad type extraction. It caused changes Any to AnyRef in a few places - mainly FlinkLazyParameterFunctionHelper and FlinkCustomStreamTransformation

  • #1039 FlinkStreamingProcessRegistrar.apply has a new parameter of type ExecutionConfigPreparer. In production code you should pass ExecutionConfigPreparer.defaultChain() there and in test code you should pass ExecutionConfigPreparer.unOptimizedChain(). See scaladocs for more info. If you already have done some Flink's ExecutionConfig set up before you've registered process, you should consider create your own chain using ExecutionConfigPreparer.chain().

  • #1039 FlinkSourceFactory doesn't take TypeInformation type class as a generic parameter now. Instead of this, it takes ClassTag. TypeInformation is determined during source creation. typeInformation[T] method was moved from BasicFlinkSource to FlinkSource because still must be some place to determine it for tests purpose.

  • #965 'aggregate' node in generic model was renamed to 'aggregate-sliding'

  • #922 HealthCheck API has new structure, naming and JSON responses:

    • old /healthCheck is moved to /healthCheck/process/deployment
    • old /sanityCheck is moved to /healthCheck/process/validation
    • top level /healthCheck indicates general "app-is-running" state

  • #879 Metrics use variables by default, see docs to enable old mode, suitable for graphite protocol. To use old way of sending:

    • put globalParameters.useLegacyMetrics = true in each model configuration (to configure metrics sending in Flink)
    • put:
    countsSettings {
      user: ...
      password: ...
      influxUrl: ...
      metricsConfig {
        nodeCountMetric: "nodeCount.count"
        sourceCountMetric: "source.count"
        nodeIdTag: "action"
        countField: "value"

      }
    }

  • Introduction to KafkaAvro API: #871, #881, #903, #981, #989, #998, #1007, #1014, #1034, #1041

API for KafkaAvroSourceFactory was changed:

KafkaAvroSourceFactory old way:

val clientFactory = new SchemaRegistryClientFactory
val source = new KafkaAvroSourceFactory(
  new AvroDeserializationSchemaFactory[GenericData.Record](clientFactory, useSpecificAvroReader = false),
  clientFactory,
  None,
  processObjectDependencies = processObjectDependencies
)

KafkaAvroSourceFactory new way :

val schemaRegistryProvider = ConfluentSchemaRegistryProvider(processObjectDependencies)
val source = new KafkaAvroSourceFactory(schemaRegistryProvider, processObjectDependencies, None)

Provided new API for Kafka Avro Sink:

val kafkaAvroSinkFactory = new KafkaAvroSinkFactory(schemaRegistryProvider, processObjectDependencies)

Additional changes:

  • Bump up confluent package to 5.5.0
  • (Refactor Kafka API) Moved KafkaSourceFactory to pl.touk.nussknacker.engine.kafka.sink package
  • (Refactor Kafka API) Changed BaseKafkaSourceFactory, now it requires deserializationSchemaFactory: KafkaDeserializationSchemaFactory[T]
  • (Refactor Kafka API) Moved KafkaSinkFactory to pl.touk.nussknacker.engine.kafka.source package
  • (Refactor Kafka API) Renamed SerializationSchemaFactory to KafkaSerializationSchemaFactory
  • (Refactor Kafka API) Renamed DeserializationSchemaFactory to KafkaDeserializationSchemaFactory
  • (Refactor Kafka API) Renamed FixedDeserializationSchemaFactory to FixedKafkaDeserializationSchemaFactory
  • (Refactor Kafka API) Renamed FixedSerializationSchemaFactory to FixedKafkaSerializationSchemaFactory
  • (Refactor Kafka API) Removed KafkaSerializationSchemaFactoryBase
  • (Refactor Kafka API) Replaced KafkaKeyValueSerializationSchemaFactoryBase by KafkaAvroKeyValueSerializationSchemaFactory (it handles only avro case now)
  • (Refactor Kafka API) Removed KafkaDeserializationSchemaFactoryBase
  • (Refactor Kafka API) Replaced KafkaKeyValueDeserializationSchemaFactoryBase by KafkaAvroKeyValueDeserializationSchemaFactory (it handles only avro case now)
  • (Refactor KafkaAvro API) Renamed AvroDeserializationSchemaFactory to ConfluentKafkaAvroDeserializationSchemaFactory and moved to avro.schemaregistry.confluent package
  • (Refactor KafkaAvro API) Renamed AvroKeyValueDeserializationSchemaFactory to ConfluentKafkaAvroDeserializationSchemaFactory and moved to avro.schemaregistry.confluent package
  • (Refactor KafkaAvro API) Renamed AvroSerializationSchemaFactory to ConfluentAvroSerializationSchemaFactory and moved to avro.schemaregistry.confluent package
  • (Refactor KafkaAvro API) Renamed AvroKeyValueSerializationSchemaFactory to ConfluentAvroKeyValueSerializationSchemaFactory and moved to avro.schemaregistry.confluent package
  • (Refactor KafkaAvro API) Removed FixedKafkaAvroSourceFactory and FixedKafkaAvroSinkFactory (now we don't support fixed schema)
  • (Refactor Kafka API) Replaced topics: List[String] by List[PreparedKafkaTopic] and removed processObjectDependencies in KafkaSource

Be aware that we are using Avro 1.9.2 instead of default Flink's 1.8.2 (for Java time logical types conversions purpose).

  • #1013 Expression evaluation is synchronous now. It shouldn't cause any problems (all languages were synchronous anyway), but some internal code may have to change.

In version 0.1.2​

  • #957 Custom node aggregate from generic model has changed parameter from windowLengthInSeconds to windowLength with human friendly duration input. If you have used it in process, you need to insert correct value again.
  • #954 TypedMap is not a case class wrapping scala Map anymore. If you have done some pattern matching on it, you should use case typedMap: TypedMap => typedMap.asScala instead.

In version 0.1.1​

  • #930 DeeplyCheckingExceptionExtractor was moved from nussknacker-flink-util module to nussknacker-util module.
  • #919 KafkaSource constructor now doesn't take consumerGroup. Instead of this it computes consumerGroup on their own based on kafka.consumerGroupNamingStrategy in modelConfig (default set to processId). You can also override it by overriddenConsumerGroup optional parameter. Regards to this changes, KafkaConfig has new, optional parameter consumerGroupNamingStrategy.
  • #920 KafkaSource constructor now takes KafkaConfig instead of using one that was parsed by BaseKafkaSourceFactory.kafkaConfig. Also if you parse Typesafe Config to KafkaSource on your own, now you should use dedicated method KafkaConfig.parseConfig to avoid further problems when parsing strategy will be changed.
  • #914 pl.touk.nussknacker.engine.api.definition.Parameter has deprecated main factory method with runtimeClass parameter. Now should be passed isLazyParameter instead. Also were removed runtimeClass from variances of factory methods prepared for easy testing (optional method and so on).

In version 0.1.0​

  • #755 Default async execution context does not depend on parallelism. asyncExecutionConfig.parallelismMultiplier has been deprecated and should be replaced with asyncExecutionConfig.workers. 8 should be sane default value.
  • #722 Old way of configuring Flink and model (via flinkConfig and processConfig) is removed. processTypes configuration should be used from now on. Example: 
    flinkConfig {...}
    processConfig {...}
    becomes: 
    processTypes {
      "type e.g. streaming" {
        deploymentConfig { 
          type: "flinkStreaming"
          PUT HERE PROPERTIES OF flinkConfig FROM OLD CONFIG 
        }
        modelConfig {
          classPath: PUT HERE VALUE OF flinkConfig.classPath FROM OLD CONFIG
          PUT HERE PROPERTIES OF processConfig FROM OLD CONFIG
        }
      }
    }
  • #763 Some API traits (ProcessManager, DictRegistry DictQueryService, CountsReporter) now extend java.lang.AutoCloseable.
  • Old way of additional properties configuration should be replaced by the new one, which is now mapped to Map[String, AdditionalPropertyConfig]. Example in your config: 
    additionalFieldsConfig: {
      mySelectProperty {
        label: "Description"
        type: "select"
        isRequired: false
        values: ["true", "false"]
      }
    }
    becomes: 
    additionalPropertiesConfig {
      mySelectProperty {
        label: "Description"
        defaultValue: "false"
        editor: {
          type: "FixedValuesParameterEditor",
          possibleValues: [
            {"label": "Yes", "expression": "true"},
            {"label": "No", "expression": "false"}
          ]
        }
      }
    }
  • #588 #882 FlinkSource API changed, current implementation is now BasicFlinkSource
  • #839 #882 FlinkSink API changed, current implementation is now BasicFlinkSink
  • #841 ProcessConfigCreator API changed; note that currently all process objects are invoked with ProcessObjectDependencies as a parameter. The APIs of KafkaSinkFactory, KafkaSourceFactory, and all their implementations were changed. Config is available as property of ProcessObjectDependencies instance.
  • #863 restUrl in deploymentConfig need to be preceded with protocol. Host with port only is not allowed anymore.
  • Rename grafanaSettings to metricsSettings in configuration.

In version 0.0.12​

  • Upgrade to Flink 1.7
  • Refactor of custom transformations, dictionaries, unions, please look at samples in example or generic to see API changes
  • Considerable changes to authorization configuration, please look at sample config to see changes
  • Circe is now used by default instead of Argonaut, but still can use Argonaut in Displayable

In version 0.0.11​

  • Changes in CustomStreamTransformer implementation, LazyInterpreter became LazyParameter, please look at samples to see changes in API

In version 0.0.8​

  • Upgrade to Flink 1.4
  • Change of format of Flink cluster configuration
  • Parameters of sources and sinks are expressions now - automatic update of DB is available
  • Change of configuration of Grafana dashboards
  • Custom processes are defined in main configuration file now