AWS Glue

marmotdata/glue

Discover jobs, databases, tables and crawlers from AWS Glue

Pipelines Assets Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The Glue plugin discovers and catalogs AWS Glue resources including jobs, databases, tables and crawlers. It captures metadata such as job configurations, table schemas, crawler schedules and database properties. Iceberg-managed tables are automatically skipped (use the dedicated Iceberg plugin instead).

Required Permissions

AWS Configuration

See AWS Configuration for the supported AWS configuration options.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick AWS Glue from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-glue-pipeline
runs:
  - glue:
      # No required fields — see the Configuration tab.

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

10 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

credentials object

AWS credentials configuration

use_default bool

Use AWS credentials from environment or default profile (recommended)

default: true

id string

AWS access key ID

secret password

AWS secret access key

token password

AWS session token

profile string

AWS profile to use from shared credentials file

role string

AWS IAM role ARN to assume

role_external_id string

External ID for cross-account role assumption

region string

AWS region for services

endpoint string

Custom endpoint URL for AWS services

tags_to_metadata bool

Convert AWS tags to Marmot metadata

include_tags multiselect

List of AWS tags to include as metadata. By default, all tags are included.

discover_jobs bool

Whether to discover Glue jobs

default: true

discover_databases bool

Whether to discover Glue databases

default: true

discover_tables bool

Whether to discover Glue tables

default: true

discover_crawlers bool

Whether to discover Glue crawlers

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

Glue Job

GlueJobFields

13 fields

GlueJobFields represents Glue Job metadata fields

role string

IAM role ARN assigned to the job

type string

Job command type (glueetl, pythonshell, gluestreaming)

glue_version string

Glue version used by the job

worker_type string

Worker type (Standard, G.1X, G.2X, etc.)

number_of_workers int

Number of workers allocated to the job

max_capacity float

Maximum number of DPU that can be allocated

timeout int

Job timeout in minutes

max_retries int

Maximum number of retries

script_location string

S3 location of the job script

connections string

Connections used by the job

created_on string

Date and time the job was created

last_modified_on string

Date and time the job was last modified

security_configuration string

Security configuration applied to the job

Glue Database

GlueDatabaseFields

5 fields

GlueDatabaseFields represents Glue Database metadata fields

catalog_id string

ID of the Data Catalog

location_uri string

Location of the database

description string

Description of the database

create_time string

Date and time the database was created

parameters string

Database parameters

Glue Table

GlueTableFields

12 fields

GlueTableFields represents Glue Table metadata fields

database_name string

Name of the database containing the table

table_type string

Type of table (EXTERNAL_TABLE, VIRTUAL_VIEW, etc.)

classification string

Classification of the table data (csv, parquet, json, etc.)

owner string

Owner of the table

location string

S3 location of the table data

input_format string

Hadoop input format class

output_format string

Hadoop output format class

serde string

Serialization/deserialization library

partition_keys string

Partition key columns

create_time string

Date and time the table was created

update_time string

Date and time the table was last updated

retention int

Retention period in days

Glue Crawler

GlueCrawlerFields

14 fields

GlueCrawlerFields represents Glue Crawler metadata fields

role string

IAM role ARN assigned to the crawler

database_name string

Target database for the crawler

state string

Current state of the crawler (READY, RUNNING, STOPPING)

schedule string

Cron schedule expression

targets string

Summary of crawler targets

schema_update_behavior string

Behavior when schema changes are detected

schema_delete_behavior string

Behavior when schema objects are deleted

recrawl_behavior string

Recrawl behavior policy

creation_time string

Date and time the crawler was created

last_updated string

Date and time the crawler was last updated

last_crawl_status string

Status of the last crawl

last_crawl_time string

Start time of the last crawl

last_crawl_error string

Error message from the last crawl

classifiers string

Custom classifiers used by the crawler