Apache Iceberg

marmotdata/iceberg

Discover namespaces, tables and views from Iceberg catalogs (REST and AWS Glue)

Storage Assets Lineage Experimental

The Iceberg plugin discovers namespaces, tables and views from Iceberg catalogs. It supports both REST catalogs and AWS Glue Data Catalog as backends.

AWS Glue Catalog Permissions

When using catalog_type: "glue", the following IAM permissions are required:

The s3:GetObject permission is needed because Glue's LoadTable reads Iceberg metadata files from S3.

AWS Configuration

When using catalog_type: "glue", see AWS Configuration for the supported AWS configuration options.

Keyless authentication

On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of access keys. Register your Marmot instance as an IAM OIDC identity provider (client id sts.amazonaws.com), create a role whose trust policy allows sts:AssumeRoleWithWebIdentity for that provider with <issuer host>:sub equal to the pipeline's subject, pipeline:<name> as reported by the pipeline API, and set credentials.role_arn and credentials.region. No key exists anywhere; Marmot mints a short-lived token for each run.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Apache Iceberg from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-iceberg-pipeline
runs:
  - iceberg:
      # No required fields — see the Configuration tab.

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

17 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

credentials object

AWS credentials configuration

shown when: catalog_type = glue

use_default bool

Use AWS credentials from environment or default profile (recommended)

default: true

id string

AWS access key ID

secret password

AWS secret access key

token password

AWS session token

profile string

AWS profile to use from shared credentials file

role_arn string

IAM role to assume with the Marmot identity token (AssumeRoleWithWebIdentity); its trust policy names the Marmot issuer as an OIDC provider. Setting it federates: no keys are needed, and region is required. role, if also set, is assumed on top of it

role string

AWS IAM role ARN to assume with AssumeRole from the base credentials (static keys, a profile, the default chain, or role_arn)

role_external_id string

External ID for cross-account role assumption

region string

AWS region for services

endpoint string

Custom endpoint URL for AWS services

tags_to_metadata bool

Convert AWS tags to Marmot metadata

shown when: catalog_type = glue

include_tags multiselect

List of AWS tags to include as metadata. By default, all tags are included.

shown when: catalog_type = glue

audience string

Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another

catalog_type select

Catalog backend type

default: rest

rest glue

uri string

REST catalog URI (required for catalog_type=rest)

shown when: catalog_type = rest

warehouse string

Warehouse identifier

shown when: catalog_type = rest

credential password

Credential for OAuth2 client credentials authentication

shown when: catalog_type = rest

token password

Bearer token for authentication

shown when: catalog_type = rest

properties string

Additional catalog properties

shown when: catalog_type = rest

prefix string

Optional prefix for the REST catalog

shown when: catalog_type = rest

glue_catalog_id string

AWS Glue Data Catalog ID (defaults to caller's account)

shown when: catalog_type = glue

include_namespaces bool

Whether to discover namespaces as assets

default: true

include_views bool

Whether to discover views

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

Namespace

IcebergNamespaceFields

2 fields

Iceberg namespace metadata fields

namespace string

Namespace path

location string

Default location for tables

Table

IcebergTableFields

12 fields

Iceberg table metadata fields

table_uuid string

Table UUID

location string

Table data location

format_version int

Iceberg format version (1, 2, or 3)

current_snapshot_id string

Current snapshot ID

snapshot_count int

Number of snapshots

schema_field_count int

Number of schema fields

partition_spec string

Partition specification

sort_order string

Sort order specification

last_updated_ms int

Last update timestamp in milliseconds

total_records string

Total record count

total_data_files string

Total data file count

total_file_size string

Total file size in bytes

View

IcebergViewFields

6 fields

Iceberg view metadata fields

view_uuid string

View UUID

location string

View metadata location

format_version int

View format version

schema_field_count int

Number of schema fields

sql_dialect string

SQL dialect of the view definition

sql string

SQL definition of the view