AWS Glue Pipelines

marmotdata/gluepipeline

Discover workflows, tasks and run history from AWS Glue

Pipelines Assets Lineage Run History Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The Glue Pipelines plugin discovers AWS Glue workflows as Pipeline assets, their steps as Task assets, and the run history of the workflows, jobs and crawlers in the account.

It shares the Glue provider with the Glue plugin, so a workflow step links to the job or crawler asset that plugin already created rather than a copy of it. This plugin creates no Job, Crawler, Database or Bucket assets, only the lineage edges and the run history that point at them.

Required Permissions

AWS Configuration

See AWS Configuration for the supported AWS configuration options.

Tests

Unit tests need nothing. The end-to-end tests run against a Glue endpoint:

docker run -d --name marmot-test-gluepipeline -p 15560:5000 motoserver/moto:latest
MARMOT_TEST_GLUEPIPELINE_ENDPOINT=http://localhost:15560 go test ./...

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick AWS Glue Pipelines from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-gluepipeline-pipeline
runs:
  - gluepipeline:
      # No required fields — see the Configuration tab.

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

11 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

credentials object

AWS credentials configuration

use_default bool

Use AWS credentials from environment or default profile (recommended)

default: true

id string

AWS access key ID

secret password

AWS secret access key

token password

AWS session token

profile string

AWS profile to use from shared credentials file

role string

AWS IAM role ARN to assume

role_external_id string

External ID for cross-account role assumption

region string

AWS region for services

endpoint string

Custom endpoint URL for AWS services

tags_to_metadata bool

Convert AWS tags to Marmot metadata

include_tags multiselect

List of AWS tags to include as metadata. By default, all tags are included.

include_workflows bool

Whether to discover Glue workflows

default: true

include_triggers bool

Whether to read trigger definitions

default: true

include_run_history bool

Whether to collect workflow, job and crawler runs

default: true

run_history_limit int

How many recent runs to read per workflow, job and crawler

default: 20

include_crawlers bool

Whether to read crawlers for lineage and runs

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

Pipeline

PipelineFields

16 fields

The metadata a workflow Pipeline asset carries.

description string

Workflow description

default_run_properties string

Run properties every run starts with, as key=value pairs

created_on string

When the workflow was created

last_modified_on string

When the workflow was last changed

max_concurrent_runs int

How many runs may overlap

node_count int

Number of steps in the workflow

job_count int

Number of job steps

crawler_count int

Number of crawler steps

trigger_count int

Number of trigger steps

last_run_id string

Identifier of the most recent run

last_run_status string

Status of the most recent run

last_run_started string

When the most recent run started

last_run_completed string

When the most recent run finished

last_run_statistics string

Action counters of the most recent run, as key=value pairs

region string

AWS region the workflow lives in

url string

AWS console link to the workflow

Task

TaskFields

13 fields

The metadata a workflow step Task asset carries.

workflow string

Workflow the step belongs to

node_name string

Step name inside the workflow

node_type string

Step kind (job, crawler, trigger)

unique_id string

Identifier AWS gives the step in the run graph

job string

Glue job the step runs

job_type string

Job command (glueetl, pythonshell, gluestreaming)

job_script_location string

Location of the job script

crawler string

Glue crawler the step runs

trigger_type string

Trigger kind (SCHEDULED, CONDITIONAL, ON_DEMAND, EVENT)

trigger_schedule string

Cron expression of a scheduled trigger

trigger_state string

Trigger state reported by AWS

trigger_predicate string

What a conditional trigger waits for

trigger_actions string

Jobs and crawlers the trigger starts

Run Facet

RunFacetFields

12 fields

The facets attached to run history events.

state string

Job or crawler run state reported by AWS

status string

Workflow run status reported by AWS

statistics string

Action counters of a workflow run

execution_time_seconds int

How long a job run took

attempt int

Retry attempt number of a job run

trigger string

Trigger that started the job run

worker_type string

Worker size the job run used

number_of_workers int

Workers the job run used

dpu_hour float

DPU hours a crawl consumed

summary string

What a crawl changed

log_group string

CloudWatch log group of a crawl

error_message string

Error reported by a failed run