Apache Iceberg
Official · maintained by Marmotmarmotdata/iceberg Discover namespaces, tables and views from Iceberg catalogs (REST and AWS Glue)
The Iceberg plugin discovers namespaces, tables and views from Iceberg catalogs. It supports both REST catalogs and AWS Glue Data Catalog as backends.
AWS Glue Catalog Permissions
When using catalog_type: "glue", the following IAM permissions are required:
The s3:GetObject permission is needed because Glue's LoadTable reads Iceberg metadata files from S3.
AWS Configuration
When using catalog_type: "glue", see AWS Configuration for the supported AWS configuration options.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Apache Iceberg from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-iceberg-pipeline
runs:
- iceberg:
# No required fields — see the Configuration tab.$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
16 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
credentials object AWS credentials configuration
- shown when
- catalog_type = glue
use_default bool Use AWS credentials from environment or default profile (recommended)
- default
- true
id string AWS access key ID
secret password AWS secret access key
token password AWS session token
profile string AWS profile to use from shared credentials file
role string AWS IAM role ARN to assume
role_external_id string External ID for cross-account role assumption
region string AWS region for services
endpoint string Custom endpoint URL for AWS services
tags_to_metadata bool Convert AWS tags to Marmot metadata
- shown when
- catalog_type = glue
include_tags multiselect List of AWS tags to include as metadata. By default, all tags are included.
- shown when
- catalog_type = glue
catalog_type select Catalog backend type
- default
- rest
uri string REST catalog URI (required for catalog_type=rest)
- shown when
- catalog_type = rest
warehouse string Warehouse identifier
- shown when
- catalog_type = rest
credential password Credential for OAuth2 client credentials authentication
- shown when
- catalog_type = rest
token password Bearer token for authentication
- shown when
- catalog_type = rest
properties string Additional catalog properties
- shown when
- catalog_type = rest
prefix string Optional prefix for the REST catalog
- shown when
- catalog_type = rest
glue_catalog_id string AWS Glue Data Catalog ID (defaults to caller's account)
- shown when
- catalog_type = glue
include_namespaces bool Whether to discover namespaces as assets
- default
- true
include_views bool Whether to discover views
- default
- true
Assets emitted
Metadata this plugin attaches to each discovered asset.
Iceberg Namespace
IcebergNamespaceFieldsIcebergNamespaceFields represents Iceberg namespace metadata fields
namespace stringNamespace path
location stringDefault location for tables
Iceberg Table
IcebergTableFieldsIcebergTableFields represents Iceberg table metadata fields
table_uuid stringTable UUID
location stringTable data location
format_version intIceberg format version (1, 2, or 3)
current_snapshot_id stringCurrent snapshot ID
snapshot_count intNumber of snapshots
schema_field_count intNumber of schema fields
partition_spec stringPartition specification
sort_order stringSort order specification
last_updated_ms intLast update timestamp in milliseconds
total_records stringTotal record count
total_data_files stringTotal data file count
total_file_size stringTotal file size in bytes
Iceberg View
IcebergViewFieldsIcebergViewFields represents Iceberg view metadata fields
view_uuid stringView UUID
location stringView metadata location
format_version intView format version
schema_field_count intNumber of schema fields
sql_dialect stringSQL dialect of the view definition
sql stringSQL definition of the view