AWS Glue
Official · maintained by Marmotmarmotdata/glue Discover jobs, databases, tables and crawlers from AWS Glue
The Glue plugin discovers and catalogs AWS Glue resources including jobs, databases, tables and crawlers. It captures metadata such as job configurations, table schemas, crawler schedules and database properties. Iceberg-managed tables are automatically skipped (use the dedicated Iceberg plugin instead).
Required Permissions
AWS Configuration
See AWS Configuration for the supported AWS configuration options.
Keyless authentication
On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of access keys. Register your Marmot instance as an IAM OIDC identity provider (client id sts.amazonaws.com), create a role whose trust policy allows sts:AssumeRoleWithWebIdentity for that provider with <issuer host>:sub equal to the pipeline's subject, pipeline:<name> as reported by the pipeline API, and set credentials.role_arn and credentials.region. No key exists anywhere; Marmot mints a short-lived token for each run.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick AWS Glue from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-glue-pipeline
runs:
- glue:
# No required fields — see the Configuration tab.$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
11 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
credentials object AWS credentials configuration
use_default bool Use AWS credentials from environment or default profile (recommended)
- default
- true
id string AWS access key ID
secret password AWS secret access key
token password AWS session token
profile string AWS profile to use from shared credentials file
role_arn string IAM role to assume with the Marmot identity token (AssumeRoleWithWebIdentity); its trust policy names the Marmot issuer as an OIDC provider. Setting it federates: no keys are needed, and region is required. role, if also set, is assumed on top of it
role string AWS IAM role ARN to assume with AssumeRole from the base credentials (static keys, a profile, the default chain, or role_arn)
role_external_id string External ID for cross-account role assumption
region string AWS region for services
endpoint string Custom endpoint URL for AWS services
tags_to_metadata bool Convert AWS tags to Marmot metadata
include_tags multiselect List of AWS tags to include as metadata. By default, all tags are included.
audience string Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another
discover_jobs bool Whether to discover Glue jobs
- default
- true
discover_databases bool Whether to discover Glue databases
- default
- true
discover_tables bool Whether to discover Glue tables
- default
- true
discover_crawlers bool Whether to discover Glue crawlers
- default
- true
Assets emitted
Metadata this plugin attaches to each discovered asset.
Job
GlueJobFieldsGlue Job metadata fields
role stringIAM role ARN assigned to the job
type stringJob command type (glueetl, pythonshell, gluestreaming)
glue_version stringGlue version used by the job
worker_type stringWorker type (Standard, G.1X, G.2X, etc.)
number_of_workers intNumber of workers allocated to the job
max_capacity floatMaximum number of DPU that can be allocated
timeout intJob timeout in minutes
max_retries intMaximum number of retries
script_location stringS3 location of the job script
connections stringConnections used by the job
created_on stringDate and time the job was created
last_modified_on stringDate and time the job was last modified
security_configuration stringSecurity configuration applied to the job
Database
GlueDatabaseFieldsGlue Database metadata fields
catalog_id stringID of the Data Catalog
location_uri stringLocation of the database
description stringDescription of the database
create_time stringDate and time the database was created
parameters stringDatabase parameters
Table
GlueTableFieldsGlue Table metadata fields
database_name stringName of the database containing the table
table_type stringType of table (EXTERNAL_TABLE, VIRTUAL_VIEW, etc.)
classification stringClassification of the table data (csv, parquet, json, etc.)
owner stringOwner of the table
location stringS3 location of the table data
input_format stringHadoop input format class
output_format stringHadoop output format class
serde stringSerialization/deserialization library
partition_keys stringPartition key columns
create_time stringDate and time the table was created
update_time stringDate and time the table was last updated
retention intRetention period in days
Crawler
GlueCrawlerFieldsGlue Crawler metadata fields
role stringIAM role ARN assigned to the crawler
database_name stringTarget database for the crawler
state stringCurrent state of the crawler (READY, RUNNING, STOPPING)
schedule stringCron schedule expression
targets stringSummary of crawler targets
schema_update_behavior stringBehavior when schema changes are detected
schema_delete_behavior stringBehavior when schema objects are deleted
recrawl_behavior stringRecrawl behavior policy
creation_time stringDate and time the crawler was created
last_updated stringDate and time the crawler was last updated
last_crawl_status stringStatus of the last crawl
last_crawl_time stringStart time of the last crawl
last_crawl_error stringError message from the last crawl
classifiers stringCustom classifiers used by the crawler