Skip to main content
Version: Current

Flink session cluster

This variant installs Designer and its bundled dependencies on Kubernetes and deploys streaming scenarios to an existing Flink session cluster. The chart does not install its JobManager or TaskManagers. Request-Response scenarios stay embedded in Designer.

Use a Flink runtime compatible with the Nussknacker image selected by your chart: its Flink and Scala versions, libraries and metrics plugins must agree. Obtain the compatible runtime from your Nussknacker provider rather than substituting an arbitrary upstream Flink image. Upgrading Designer does not upgrade the remote cluster.

  • Designer must reach the JobManager REST API at flinkRestUrl.
  • JobManagers and TaskManagers must reach Kafka, Schema Registry, any integrations used by scenarios, and Postgres for the SQL key-value store and database-backed live data preview.
  • Configure durable checkpoint/savepoint storage on Flink itself. The Designer PVC is not mounted in remote pods.
  • Provision enough TaskManager slots and resources for the scenarios you deploy.

The chart automatically qualifies bundled-service addresses with the installation namespace. A Flink cluster in another Kubernetes namespace can use them if DNS, routing and NetworkPolicies permit access. Kafka clients must also reach the broker addresses returned in Kafka metadata. For a cluster outside this Kubernetes network, configure reachable external data services instead of relying on .svc.cluster.local addresses.

Values​

Save this example as values.yaml. Replace all placeholders, including the full JobManager REST URL.

# Set the three hostnames to DNS names pointing at your Kubernetes ingress controller.
global:
ingress:
hosts:
designer: REPLACE_WITH_DESIGNER_HOST
requestResponse: REPLACE_WITH_REQUEST_RESPONSE_HOST
gateway: REPLACE_WITH_GATEWAY_HOST
tls:
enabled: false
imagePullSecrets:
- name: nussknacker-registry
license:
existingSecret: nussknacker-license
ingress:
enabled: true
className: REPLACE_WITH_INGRESS_CLASS
environment: trial

# Flink itself is managed separately. This is the JobManager REST endpoint reached by Designer.
nussknacker:
streaming:
deploymentTargets:
- name: Flink trial
useMiniClusterForDeployment: false
flinkRestUrl: https://REPLACE_WITH_FLINK_REST_HOST
metricsClusterId: trial-flink
jobManagerTimeout: "2 minutes"
scenarioStateVerification:
enabled: false
requestResponse:
deploymentTargets:
- name: Request-Response embedded

jobManagerTimeout controls deployment operations, including savepoints. The chart also exposes scenarioStateRequestTimeout, shouldCheckAvailableSlots, and shared modelConfig settings such as the checkpoint interval. See scenario deployment configuration and Flink model configuration.

scenarioStateVerification is off because Designer cannot read the savepoints of a remote cluster: with it on, every redeploy of a scenario that has state fails. Turn it on only if Designer mounts the savepoint directory at the same path as Flink.

To offer more than one cluster, add entries to deploymentTargets, each with a unique name and its own flinkRestUrl and metricsClusterId. Settings in streaming.deploymentConfig are shared defaults; all targets share streaming.modelConfig and its component configuration.

Metrics​

The compatible InfluxDB reporter plugin must be installed in Flink. Configure it to send to Telegraf in the Nussknacker namespace. The reporter's cluster ID must match metricsClusterId: trial-flink in the values. For the default namespace, use these entries in the Flink cluster configuration (merge them with its existing settings; use your chosen namespace if different):

taskmanager.numberOfTaskSlots: 8
# Nu requires a little bit more metaspace than Flink default allocate based on process size
taskmanager.memory.process.size: 1500m
taskmanager.memory.jvm-metaspace.size: 500m

jobmanager.memory.process.size: 1800m
jobmanager.memory.jvm-metaspace.size: 800m

state.backend.type: hashmap
execution.checkpointing.dir: file:///opt/flink/data/checkpoints
execution.checkpointing.savepoint-dir: file:///opt/flink/data/savepoints

#Below are base settings for rocksdb metrics, that can be used for grafana dashboards
state.backend.rocksdb.metrics.estimate-num-keys: true
state.backend.rocksdb.metrics.estimate-live-data-size: true
state.backend.rocksdb.metrics.cur-size-all-mem-tables: true
state.backend.rocksdb.metrics.size-all-mem-tables: true
# We can have many jobs per cluster, in such setting managed memory is not easy to tune
state.backend.rocksdb.memory.managed: false
# For frequent writes increase the value as needed. Currently RocksDB settings can only be changed per Flink cluster
state.backend.rocksdb.writebuffer.size: 256m

# Must match deploymentConfig.metricsClusterId; use a distinct value for each cluster.
metrics.reporter.influxdb_reporter.scope.variables.additional: cluster_id:trial-flink
metrics.reporters: influxdb_reporter
metrics.reporter.influxdb_reporter.factory.class: org.apache.flink.metrics.influxdb.InfluxdbReporterFactory
metrics.reporter.influxdb_reporter.host: nussknacker-telegraf.nussknacker.svc.cluster.local
metrics.reporter.influxdb_reporter.port: 8087
metrics.reporter.influxdb_reporter.db: nussknacker_metrics
# Required by the reporter's Basic header builder; this local Telegraf input does not authenticate.
metrics.reporter.influxdb_reporter.username: nussknacker
metrics.reporter.influxdb_reporter.password: nussknacker
metrics.reporter.influxdb_reporter.scope.variables.excludes: task_attempt_id;operator_id;task_attempt_num;task_name
metrics.scope.jm: local.<host>.jobmanagerGlobal
metrics.scope.jm-job: local.<host>.jobmanagerJob.<job_name>
metrics.scope.tm: local.<host>.taskmanagerGlobal.<tm_id>
metrics.scope.tm-job: local.<host>.taskmanagerJob.<tm_id>.<job_name>
metrics.scope.task: local.<host>.taskmanagerTask.<tm_id>.<job_name>.<task_name>.<subtask_index>
metrics.scope.operator: local.<host>.taskmanagerTask.<tm_id>.<job_name>.<operator_name>.<subtask_index>

The example includes the scope patterns expected by the bundled Grafana dashboards. If NetworkPolicies are enabled, add peers for the remote Flink pods to networkPolicy.additionalPeers in Nussknacker values, and allow outbound connections in the Flink installation's own policies.

Install and verify​

Create the registry/license Secrets and install the chart. In Designer, deploy a streaming scenario to Flink trial. Check that its job is RUNNING in Flink and that it processes a record. Verify scenario metrics when the reporter is configured. helm test alone does not exercise deployment to the session cluster. Scenario tests may still use a MiniCluster inside Designer.

Request-Response scenarios deploy to Request-Response embedded and use the chart's Request-Response host.