Dagster
Official · maintained by Marmotmarmotdata/dagster Discover jobs, ops and software-defined assets from Dagster
The Dagster plugin discovers jobs, the ops inside them, and software-defined assets from a Dagster webserver.
It reads everything through the GraphQL endpoint at {host}/graphql, so it works against both open source Dagster and Dagster+. For Dagster+, set token and the plugin sends it as the Dagster-Cloud-Api-Token header.
Assets
Pipelineper job, named after the job. Schedules, sensors and a summary of recent runs are attached as metadata.Taskper op, named<job>/<op handle>.Datasetper software-defined asset, named after its asset key joined with/. ATableSchemametadata entry becomes the asset's columns.
Dagster's implicit asset jobs (the ones named __ASSET_JOB) are skipped.
Lineage
Pipelines contain their Tasks, Tasks depend on the Tasks feeding their inputs, Datasets feed the Datasets built from them, and a Pipeline produces every Dataset it materialises. When an asset names a warehouse table and its compute kind is Snowflake, BigQuery, Postgres or DuckDB, the Dataset also produces that native table.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Dagster from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-dagster-pipeline
runs:
- dagster:
host: "<host>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
11 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
host string Dagster webserver URL, for example http://localhost:3000
token password Dagster+ API token
verify_ssl bool Check the server's TLS certificate
- default
- true
include_ops bool Discover the ops inside each job as Task assets
- default
- true
include_assets bool Discover software-defined assets as Dataset assets
- default
- true
include_run_history bool Collect recent runs of each job
- default
- true
run_history_limit int How many recent runs to read per job
- default
- 10
code_locations multiselect Only discover these code locations. Empty means all of them
Assets emitted
Metadata this plugin attaches to each discovered asset.
Pipeline
DagsterPipelineFieldsThe metadata fields the Dagster plugin emits for Pipeline assets, one per job.
job_id stringDagster's internal identifier for the job
repository stringRepository the job is defined in
code_location stringCode location serving the job
description stringJob description
is_asset_job boolWhether the job exists to materialise assets
tags stringJob tags, keyed by tag name
schedules stringSchedules that launch the job, with cron expression and status
sensors stringSensors that launch the job, with status
op_count intNumber of ops in the job
last_run_status stringStatus of the most recent run
last_run_at stringStart time of the most recent run
run_count intNumber of runs read for this job
success_rate floatPercentage of finished runs that succeeded
url stringLink to the job in the Dagster UI
Task
DagsterTaskFieldsThe metadata fields emitted for Task assets, one per op in a job.
op stringName of the op definition
handle_id stringPath to the op within the job graph
job stringJob the op belongs to
description stringOp description
input_count intNumber of inputs the op declares
output_count intNumber of outputs the op declares
url stringLink to the op in the Dagster UI
Dataset
DagsterDatasetFieldsThe metadata fields emitted for Dataset assets, one per software-defined asset. Any metadata a user attaches to the asset in Dagster is flattened alongside these, keyed by its label.
asset_key stringAsset key path as declared in Dagster
description stringAsset description
compute_kind stringTechnology that computes the asset, for example duckdb
group stringAsset group the asset belongs to
code_location stringCode location serving the asset
repository stringRepository the asset is defined in
op_names stringOps that compute the asset
job_names stringJobs that can materialise the asset
is_partitioned boolWhether the asset is partitioned
last_materialized_at stringTime of the most recent materialization
last_materialization_run stringRun that produced the most recent materialization
url stringLink to the asset in the Dagster UI
Column
DagsterColumnFieldsThe per-column fields taken from an asset's TableSchema metadata entry and embedded in the asset's schema.
column_name stringColumn name
data_type stringColumn data type as declared in the table schema
is_nullable boolWhether null values are allowed
description stringColumn description