AWS Glue (Iceberg REST catalog)

Overview

The AWS Glue integration connects Nussknacker to Apache Iceberg tables registered in the AWS Glue Data Catalog. It uses the Glue Iceberg REST endpoint (https://glue.<region>.amazonaws.com/iceberg) as a Flink Iceberg catalog, so the table schemas come from the catalog and no data type mappings or CREATE TABLE statements are needed. Table data is read from and written to S3 with the same credentials.

Once configured, the following components are added to the Component Palette, both labeled <integration name> Iceberg AWS Glue:

• Table (Source) - reads records from a selected Iceberg table
• Table (Sink) - writes records to a selected Iceberg table

In the component, choose the table from the tables discovered in the catalog.

To add the integration, open Integrations from the Designer's side menu and click Add new on the AWS Glue tile (category Iceberg).

Integration parameters

Parameter	Required	Description
Integration name	true	Name used to refer to this configuration in the Iceberg AWS Glue components; it prefixes the generated component names. At least 3 characters.
Region	true	AWS Region of your Glue Data Catalog, e.g. eu-central-1.
AWS Access Key ID	true	Access key ID of the AWS IAM user used by the integration.
AWS Secret Access Key	true	Secret access key of that IAM user.
Warehouse	false	Chooses a specific Glue catalog; called the prefix parameter in the Iceberg REST specification. If left empty, the default catalog of the AWS account is used. See AWS documentation.

Use the check connection button to verify the configuration before saving it.

Required AWS permissions

The IAM user needs at least the permissions below. You can narrow the Resource scope to the S3 buckets, Glue databases and tables you intend to access.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "glue:GetCatalog",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetTable",
        "glue:GetTables",
        "glue:UpdateTable"
      ],
      "Resource": [
        "arn:aws:glue:<region>:<account_id>:catalog",
        "arn:aws:glue:<region>:<account_id>:database/*",
        "arn:aws:glue:<region>:<account_id>:table/*/*"
      ]
    },
    {
      "Sid": "S3Access",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": ["*"]
    },
    {
      "Sid": "LakeFormationAccess",
      "Effect": "Allow",
      "Action": ["lakeformation:GetDataAccess"],
      "Resource": "*"
    }
  ]
}