Flink session cluster
This variant installs Designer and its bundled dependencies on Kubernetes and deploys streaming scenarios to an existing Flink session cluster. The chart does not install its JobManager or TaskManagers. Request-Response scenarios stay embedded in Designer.
Prepare Flink
Use a Flink runtime compatible with the Nussknacker image selected by your chart: its Flink and Scala versions, libraries and metrics plugins must agree. Obtain the compatible runtime from your Nussknacker provider rather than substituting an arbitrary upstream Flink image. Upgrading Designer does not upgrade the remote cluster.
- Designer must reach the JobManager REST API at
flinkRestUrl. - JobManagers and TaskManagers must reach Kafka, Schema Registry, any integrations used by scenarios, and Postgres for the SQL key-value store and database-backed live data preview.
- Configure durable checkpoint/savepoint storage on Flink itself. The Designer PVC is not mounted in remote pods.
- Provision enough TaskManager slots and resources for the scenarios you deploy.
The chart automatically qualifies bundled-service addresses with the installation namespace. A Flink cluster in
another Kubernetes namespace can use them if DNS, routing and NetworkPolicies permit access. Kafka clients
must also reach the broker addresses returned in Kafka metadata. For a cluster outside this Kubernetes
network, configure reachable external data services instead of relying on .svc.cluster.local addresses.
Values
Save this example as values.yaml. Replace all placeholders, including the full JobManager REST URL.
# Set the three hostnames to DNS names pointing at your Kubernetes ingress controller.
global:
ingress:
hosts:
designer: REPLACE_WITH_DESIGNER_HOST
requestResponse: REPLACE_WITH_REQUEST_RESPONSE_HOST
gateway: REPLACE_WITH_GATEWAY_HOST
tls:
enabled: false
imagePullSecrets:
- name: nussknacker-registry
license:
existingSecret: nussknacker-license
ingress:
enabled: true
className: REPLACE_WITH_INGRESS_CLASS
environment: trial
# Flink itself is managed separately. This is the JobManager REST endpoint reached by Designer.
nussknacker:
streaming:
deploymentTargets:
- name: Flink trial
useMiniClusterForDeployment: false
flinkRestUrl: https://REPLACE_WITH_FLINK_REST_HOST
metricsClusterId: trial-flink
jobManagerTimeout: "2 minutes"
scenarioStateVerification:
enabled: false
requestResponse:
deploymentTargets:
- name: Request-Response embedded
jobManagerTimeout controls deployment operations, including savepoints. The chart also exposes
scenarioStateRequestTimeout, shouldCheckAvailableSlots, and shared modelConfig settings such as the
checkpoint interval. See scenario deployment configuration
and Flink model configuration.
scenarioStateVerification is off because Designer cannot read the savepoints of a remote cluster: with it on,
every redeploy of a scenario that has state fails. Turn it on only if Designer mounts the savepoint directory at the
same path as Flink.
To offer more than one cluster, add entries to deploymentTargets, each with a unique name and its own
flinkRestUrl and metricsClusterId. Settings in streaming.deploymentConfig are shared defaults; all targets
share streaming.modelConfig and its component configuration.
Metrics
The compatible InfluxDB reporter plugin must be installed in Flink. Configure it to send to Telegraf in the
Nussknacker namespace. The reporter's cluster ID must match metricsClusterId: trial-flink in the values.
For the default namespace, use these entries in the Flink cluster configuration (merge them with its existing
settings; use your chosen namespace if different):
taskmanager.numberOfTaskSlots: 8
# Nu requires a little bit more metaspace than Flink default allocate based on process size
taskmanager.memory.process.size: 1500m
taskmanager.memory.jvm-metaspace.size: 500m
jobmanager.memory.process.size: 1800m
jobmanager.memory.jvm-metaspace.size: 800m
state.backend.type: hashmap
execution.checkpointing.dir: file:///opt/flink/data/checkpoints
execution.checkpointing.savepoint-dir: file:///opt/flink/data/savepoints
#Below are base settings for rocksdb metrics, that can be used for grafana dashboards
state.backend.rocksdb.metrics.estimate-num-keys: true
state.backend.rocksdb.metrics.estimate-live-data-size: true
state.backend.rocksdb.metrics.cur-size-all-mem-tables: true
state.backend.rocksdb.metrics.size-all-mem-tables: true
# We can have many jobs per cluster, in such setting managed memory is not easy to tune
state.backend.rocksdb.memory.managed: false
# For frequent writes increase the value as needed. Currently RocksDB settings can only be changed per Flink cluster
state.backend.rocksdb.writebuffer.size: 256m
# Must match deploymentConfig.metricsClusterId; use a distinct value for each cluster.
metrics.reporter.influxdb_reporter.scope.variables.additional: cluster_id:trial-flink
metrics.reporters: influxdb_reporter
metrics.reporter.influxdb_reporter.factory.class: org.apache.flink.metrics.influxdb.InfluxdbReporterFactory
metrics.reporter.influxdb_reporter.host: nussknacker-telegraf.nussknacker.svc.cluster.local
metrics.reporter.influxdb_reporter.port: 8087
metrics.reporter.influxdb_reporter.db: nussknacker_metrics
# Required by the reporter's Basic header builder; this local Telegraf input does not authenticate.
metrics.reporter.influxdb_reporter.username: nussknacker
metrics.reporter.influxdb_reporter.password: nussknacker
metrics.reporter.influxdb_reporter.scope.variables.excludes: task_attempt_id;operator_id;task_attempt_num;task_name
metrics.scope.jm: local.<host>.jobmanagerGlobal
metrics.scope.jm-job: local.<host>.jobmanagerJob.<job_name>
metrics.scope.tm: local.<host>.taskmanagerGlobal.<tm_id>
metrics.scope.tm-job: local.<host>.taskmanagerJob.<tm_id>.<job_name>
metrics.scope.task: local.<host>.taskmanagerTask.<tm_id>.<job_name>.<task_name>.<subtask_index>
metrics.scope.operator: local.<host>.taskmanagerTask.<tm_id>.<job_name>.<operator_name>.<subtask_index>
The example includes the scope patterns expected by the bundled Grafana dashboards. If NetworkPolicies are
enabled, add peers for the remote Flink pods to networkPolicy.additionalPeers in Nussknacker values, and allow
outbound connections in the Flink installation's own policies.
Install and verify
Create the registry/license Secrets and install the chart.
In Designer, deploy a streaming scenario to Flink trial. Check that its job is RUNNING in Flink and that
it processes a record. Verify scenario metrics when the reporter is configured. helm test alone does not
exercise deployment to the session cluster. Scenario tests may still use a MiniCluster inside Designer.
Request-Response scenarios deploy to Request-Response embedded and use the chart's Request-Response host.