Ververica Platform
This variant connects Designer to an existing Ververica Platform (VVP). VVP runs each streaming scenario as a per-job deployment on a stock Ververica Runtime engine. Its Flink pods load the model from a volume that Designer writes to. Request-Response scenarios stay embedded in Designer.
This page replaces step 3 of Getting started: do steps 1 and 2 there first. To deploy to an existing session cluster instead, see Session cluster.
Before you start
You need:
- The Designer and Blabber images of the Flink 1.20 / Scala 2.12 build, from your Nussknacker provider. The standard Designer build does not work with VVP.
- From your VVP administrator:
- the VVP API URL and the VVP console URL,
- the VVP namespace and workspace,
- a per-job deployment target, and the Kubernetes namespace that it deploys Flink pods into,
- an API token with the
editorrole in the VVP namespace.
- ReadWriteMany (RWX) storage, for example NFS. Designer and the Flink pods use it at the same time, from different nodes.
- Network access from Designer to the VVP API, and from the Flink pods to Kafka, the Schema Registry, Postgres and the other systems that your scenarios use.
- Checkpoint and savepoint storage, configured in VVP.
1. Prepare the shared model volume
The values create the claim nussknacker-model in Designer's namespace with extraDeploy, and mount
it in Designer with designer.extraVolumes and designer.extraVolumeMounts. Set its storageClassName to a
StorageClass with ReadWriteMany (RWX) access. The annotation helm.sh/resource-policy: keep keeps the claim when
the release is uninstalled.
- If the deployment target deploys into another Kubernetes namespace, that namespace needs a claim with the same name on the same directory. Two dynamically provisioned claims get two different directories, and a PersistentVolume binds only one claim. Ask your storage administrator for a PersistentVolume and claim pair in each namespace on the same storage, for example on the same NFS export.
- Designer runs as UID 1001 with fsGroup 1001. It must be able to create and rename directories on the volume. The Flink pods only read it.
- Designer deletes the model directories that no deployment in its VVP namespace uses. Do not share the directory with a Designer that deploys into another VVP namespace.
2. Mount the volume in VVP's Flink pods
Your VVP administrator does this step. The mount goes into the platform-wide deployment defaults of VVP, so
that VVP adds it to the Flink pods of every deployment. Save this as vvp-model-values.yaml, and add
--values vvp-model-values.yaml to the usual helm upgrade command of the VVP Helm release:
vvp-appmanager:
globalDeploymentDefaults:
spec:
template:
spec:
kubernetes:
pods:
volumeMounts:
- name: flink-logs
volume: { name: flink-logs, emptyDir: {} }
volumeMount: { name: flink-logs, mountPath: /opt/flink/log }
- name: flink-tmp-dir
volume: { name: flink-tmp-dir, emptyDir: {} }
volumeMount: { name: flink-tmp-dir, mountPath: /opt/flink/flink-tmp-dir }
- name: flink-sts-dir
volume: { name: flink-sts-dir, emptyDir: {} }
volumeMount: { name: flink-sts-dir, mountPath: /flink/sts-secrets/sts-credential }
- name: nussknacker-model
volume:
name: nussknacker-model
persistentVolumeClaim: { claimName: nussknacker-model, readOnly: true }
volumeMount: { name: nussknacker-model, mountPath: /flink/nussknacker-model, readOnly: true }
- The first three entries are the platform mounts of VVP: a
volumeMountslist replaces them. Compare them with Ververica's platform-wide deployment defaults for your VVP version. - This form needs VVP chart 3.1.3 or later. In older charts the value is a YAML string: copy the default of the
chart and add
kubernetesto it. - After the upgrade, check that the AppManager restarted, and restart it if not. Only VVP deployments created after the restart get the mount.
- A claim
nussknacker-modelmust exist in every Kubernetes namespace that VVP deploys into, also where no scenario runs. An empty claim is enough there. Without it, the Flink pods stayPending.
For namespace-level defaults and deployments created before the mount, see Configuring Ververica Platform.
3. Create the API token Secret
Create the Secret that designer.extraEnv refers to, replacing the placeholder with your VVP API token:
kubectl -n "$NU_NAMESPACE" create secret generic nussknacker-vvp \
--from-literal=VVP_API_TOKEN='REPLACE_WITH_VVP_API_TOKEN'
4. Values
Save this as values.yaml and replace all placeholders.
# Set the three hostnames to DNS names pointing at your Kubernetes ingress controller.
global:
ingress:
hosts:
designer: REPLACE_WITH_DESIGNER_HOST
requestResponse: REPLACE_WITH_REQUEST_RESPONSE_HOST
gateway: REPLACE_WITH_GATEWAY_HOST
tls:
enabled: false
imagePullSecrets:
- name: nussknacker-registry
license:
existingSecret: nussknacker-license
ingress:
enabled: true
className: REPLACE_WITH_INGRESS_CLASS
environment: trial
# The model shared with the Flink pods of VVP (step 1).
extraDeploy:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nussknacker-model
annotations:
helm.sh/resource-policy: keep
spec:
accessModes: [ReadWriteMany]
storageClassName: REPLACE_WITH_RWX_STORAGE_CLASS
resources:
requests:
storage: 10Gi
designer:
image:
repository: registry.nussknacker.io/nussknacker/nussknacker
# The Flink 1.20 / Scala 2.12 build: the tag ends with _scala-2.12.
tag: REPLACE_WITH_DESIGNER_IMAGE_TAG
extraVolumes:
- name: nussknacker-model
persistentVolumeClaim:
claimName: nussknacker-model
extraVolumeMounts:
- name: nussknacker-model
mountPath: /opt/nussknacker/nussknacker-model
extraEnv:
- name: VVP_API_TOKEN
valueFrom:
secretKeyRef:
name: nussknacker-vvp
key: VVP_API_TOKEN
extraConfig: |
# Replace the Flink target of the chart with the VVP target.
scenarioTypes.streaming.deploymentConfig = null
scenarioTypes.streaming.deploymentConfig {
targets: [
{
name: "Ververica Platform trial"
type: "ververica"
baseUrl: "https://REPLACE_WITH_VVP_API_HOST"
consoleUrl: "https://REPLACE_WITH_VVP_CONSOLE_HOST"
namespace: "REPLACE_WITH_VVP_NAMESPACE"
workspace: "REPLACE_WITH_VVP_WORKSPACE"
# Any stock engine version on Flink 1.20. Nussknacker is tested with this one.
engineVersion: "vera-4.5-jdk17-flink-1.20"
auth {
type: "apiToken"
token: ${VVP_API_TOKEN}
}
deploymentMode {
type: "perJob"
deploymentTargetName: "REPLACE_WITH_VVP_DEPLOYMENT_TARGET"
jobResources {
jobmanager { cpu: 1, memory: "1Gi" }
taskmanager { cpu: 1, memory: "2Gi" }
}
}
sharedModelDirectory {
designerPath: "/opt/nussknacker/nussknacker-model"
jobPath: "/flink/nussknacker-model"
}
flinkConfiguration {
"metrics.reporters": "influxdb_reporter"
"metrics.reporter.influxdb_reporter.factory.class": "org.apache.flink.metrics.influxdb.InfluxdbReporterFactory"
"metrics.reporter.influxdb_reporter.host": ${telegrafHost}
"metrics.reporter.influxdb_reporter.port": "8087"
"metrics.reporter.influxdb_reporter.db": "nussknacker_metrics"
"metrics.reporter.influxdb_reporter.username": "nussknacker"
"metrics.reporter.influxdb_reporter.password": "nussknacker"
"metrics.reporter.influxdb_reporter.scope.variables.additional": "cluster_id:vvp"
"metrics.reporter.influxdb_reporter.scope.variables.excludes": "task_attempt_id;operator_id;task_attempt_num;task_name"
}
}
]
}
blabber:
image:
repository: registry.nussknacker.io/nussknacker/nussknacker-blabber
# The same version as Designer, but Blabber is built for Scala 2.13 only: the tag ends with _scala-2.13.
tag: REPLACE_WITH_BLABBER_IMAGE_TAG
nussknacker:
streaming:
deploymentTargets: []
requestResponse:
deploymentTargets:
- name: Request-Response embedded
sharedModelDirectory.designerPath is the mountPath in designer.extraVolumeMounts, and jobPath is the
mountPath of step 2. For the other target parameters, see
Ververica Platform.
flinkConfiguration makes the jobs send their metrics to Telegraf of this installation, for the counts and the
Grafana dashboards. The Ververica Runtime image has this reporter. If your VVP already sets metrics.reporters
for its own monitoring, keep those reporters in the list too, for example "prom,influxdb_reporter".
The chart gives the Flink pods namespace-qualified addresses of the bundled Kafka, Schema Registry, Blabber, Telegraf and Postgres. If the Flink pods run outside this Kubernetes cluster, use external services that they can reach.
If you turn on networkPolicy.enabled, also let the Flink pods in, with a peer for the Kubernetes namespace of
the deployment target. The policies admit only the chart's own pods, so this applies also when the Flink pods
run in Designer's namespace:
networkPolicy:
enabled: true
additionalPeers:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: REPLACE_WITH_FLINK_PODS_NAMESPACE
Egress policies on the Flink pods must allow these connections too.
5. Install and verify
Install the chart and run its checks. helm test does not deploy to VVP,
so then deploy a streaming scenario to Ververica Platform trial, and check that it is RUNNING in VVP and
processes records.
If it does not run:
-
The Flink pods stay
Pending: the claimnussknacker-modelis missing in their namespace. -
The job fails at start: look at the volume in a JobManager pod. It must hold a directory for each deployed model.
kubectl -n <flink-pods-namespace> exec <jobmanager-pod> -c flink-main-container -- ls -l /flink/nussknacker-modelIf the volume is not there, the VVP deployment was created before step 2: delete it in VVP and deploy the scenario again. If the volume is empty, the two claims do not point at the same directory.
-
Designer logs a permission error: fix the ownership or access rights of the storage.
Request-Response scenarios deploy to Request-Response embedded and use the chart's Request-Response host.
Session cluster
To deploy to an existing VVP session cluster instead, skip steps 1 and 2, and change the values:
-
Replace the
deploymentModeblock with this one, and removesharedModelDirectory:deploymentMode {
type: "session"
clusterName: "REPLACE_WITH_VVP_SESSION_CLUSTER_NAME"
} -
Remove the
nussknacker-modelclaim fromextraDeploy, and its entries fromdesigner.extraVolumesanddesigner.extraVolumeMounts.
The session cluster must be RUNNING when you deploy. Its resources are set in VVP.
Without a shared volume, Designer uploads the full model JAR to VVP, about 400 MB for the bundled model. Your VVP administrator must raise the upload limit in the VVP Helm release above that size, see Ververica's artifact upload settings. A reverse proxy in front of the VVP API must accept that size too.
global:
upload:
maxFileSize: 500MB