Airflow

marmotdata/airflow

Ingest metadata from Apache Airflow including DAGs, tasks, and dataset lineage

Category orchestration

The Airflow plugin ingests metadata from Apache Airflow, including DAGs (Directed Acyclic Graphs), tasks, and dataset lineage. It connects to Airflow's REST API to discover your orchestration layer and track data dependencies through Airflow's native Dataset feature.

Prerequisites

Airflow 2.0+ for basic DAG and task discovery
Airflow 2.4+ for Dataset-based lineage tracking
REST API enabled with authentication configured

Authentication

The plugin supports two authentication methods:

Basic Auth: Username and password
API Token: For token-based authentication

Configure authentication in your Airflow instance via airflow.cfg:

[api]
auth_backends = airflow.api.auth.backend.basic_auth

In the UI

Point-and-click, no config file needed.

1 Open Runs → Create Pipeline
2 Pick Airflow from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the full UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml

name: my-airflow-pipeline
runs:
  - airflow:
      host: "<host>"

$ marmot ingest -c ingest.yaml

Read the full CLI guide

Not using plugins? Other ways to populate Marmot

Terraform

Declare assets as infrastructure-as-code.

Pulumi

Native TypeScript, Python, or Go resources.

Kubernetes Operator

Sync assets from Kubernetes custom resources.

Configuration

13 top-level fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

Airflow webserver URL (e.g., http://localhost:8080)

username string

Username for basic authentication

password password

Password for basic authentication

api_token password

API token for authentication (alternative to basic auth)

discover_dags bool

Discover Airflow DAGs as Pipeline assets

default true

discover_tasks bool

Discover tasks within DAGs

default true

discover_datasets bool

Discover Airflow Datasets for lineage (requires Airflow 2.4+)

default true

include_run_history bool

Include DAG run history in metadata

default true

run_history_days int

Number of days of run history to fetch

default 7

only_active bool

Only discover active (unpaused) DAGs

default true

Assets Emitted

Metadata fields this plugin attaches to each discovered asset.

Airflow DAG

AirflowDAGFields

14 fields

AirflowDAGFields describes the metadata fields Airflow emits for a DAG (Pipeline) asset. It is kept as a documentation-only struct so downstream tooling can introspect the shape of the metadata map.

dag_id string

Unique DAG identifier

description string

DAG description

file_path string

Path to DAG definition file

schedule_interval string

DAG schedule (cron expression or preset)

is_paused bool

Whether DAG is paused

is_active bool

Whether DAG is active

owners string

DAG owners (comma-separated)

last_run_state string

State of the last DAG run (success, failed, running)

last_run_id string

ID of the last DAG run

last_run_date string

Execution date of the last DAG run

next_run_date string

Next scheduled run date

last_parsed_time string

Last time the DAG file was parsed

success_rate float

Success rate percentage over the lookback period

run_count int

Number of runs in the lookback period

Airflow Task

AirflowTaskFields

7 fields

AirflowTaskFields describes the metadata fields for a Task asset.

task_id string

Task identifier within the DAG

dag_id string

Parent DAG ID

operator_name string

Airflow operator class name (e.g., BashOperator, PythonOperator)

trigger_rule string

Task trigger rule (e.g., all_success, one_success)

retries int

Number of retries configured for the task

pool string

Execution pool for the task

downstream_tasks string[]

List of downstream task IDs

Airflow Dataset

AirflowDatasetFields

5 fields

AirflowDatasetFields describes the metadata fields for a Dataset asset.

uri string

Dataset URI identifier

created_at string

Dataset creation timestamp

updated_at string

Dataset last update timestamp

producer_count int

Number of tasks that produce this dataset

consumer_count int

Number of DAGs that consume this dataset

Airflow DAG Run

AirflowDAGRunFields

6 fields

AirflowDAGRunFields describes the run facet fields emitted for DAG run history events.

dag_run_id string

Unique identifier for the DAG run

state string

Run state (queued, running, success, failed)

execution_date string

Logical execution date

start_date string

Actual start time of the run

end_date string

End time of the run

run_type string

Type of run (scheduled, manual, backfill)