Delta Lake
Official · maintained by Marmotmarmotdata/deltalake Discover tables from Delta Lake transaction logs on local filesystems
The Delta Lake plugin catalogues tables from Delta Lake transaction logs. It reads the _delta_log directory at each configured path to capture the table schema, partition columns and current version.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Delta Lake from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-deltalake-pipeline
runs:
- deltalake:
table_paths: []$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
6 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
s3_source object S3 file source configuration
- shown when
- source_type = s3
credentials object AWS credentials for S3 access
use_default bool Use AWS credentials from environment or default profile (recommended)
- default
- true
id string AWS access key ID
secret password AWS secret access key
token password AWS session token
profile string AWS profile to use from shared credentials file
role string AWS IAM role ARN to assume
role_external_id string External ID for cross-account role assumption
region string AWS region for services
endpoint string Custom endpoint URL for AWS services
git_source object Git repository file source configuration
- shown when
- source_type = git
url string Git repository URL
ref string Branch, tag or commit to check out
- default
- main
path string Subdirectory within the repository
token password Personal access token for HTTPS auth
ssh_key_path string Path to SSH private key for SSH auth
table_paths multiselect Paths to Delta Lake table directories (local paths, s3://bucket/prefix or git::url)
Assets emitted
Metadata this plugin attaches to each discovered asset.
Delta Lake Table
DeltaLakeTableFieldsDeltaLakeTableFields represents Delta Lake table metadata fields
table_id stringDelta table unique identifier
location stringTable directory path
format stringData format (e.g. parquet)
min_reader_version intMinimum reader protocol version
min_writer_version intMinimum writer protocol version
partition_columns stringComma-separated partition column names
schema_field_count intNumber of schema fields
created_time intTable creation timestamp in milliseconds
num_files intNumber of active data files
total_size intTotal size of active data files in bytes
current_version intCurrent Delta log version