Apache Flink
Official · maintained by Marmotmarmotdata/flink Discover jobs and their vertices from an Apache Flink JobManager
The Flink plugin discovers jobs from an Apache Flink JobManager. Each job becomes a Pipeline, each vertex of its job graph a Task, with CONTAINS edges from the Pipeline to its Tasks and DEPENDS_ON edges between Tasks taken from the job plan. The timestamps Flink keeps for a job's state changes are recorded as run history.
It reads the JobManager REST API (/config, /jobs/overview, /jobs/{jid}, /jobs/{jid}/config, /jobs/{jid}/exceptions), which needs no credentials. username/password and token are for a proxy placed in front of the JobManager.
Naming
A Pipeline is named after the job. Flink lets several jobs share a name, in which case the most recently started job keeps the bare name and the others are named <name> (<jid>). A Task is named <pipeline name>/<vertex name>.
Jobs the JobManager still lists
The JobManager keeps listing finished, failed and cancelled jobs until it restarts. They are discovered by default; set include_completed: false to keep only jobs that are still running.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Apache Flink from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-flink-pipeline
runs:
- flink:
host: "<host>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
11 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
host string JobManager REST URL, for example http://localhost:8081
username string Username for basic auth, when a proxy in front of the JobManager requires it
password password Password for basic auth
token password Bearer token, when a proxy in front of the JobManager requires it
verify_ssl bool Verify the JobManager's TLS certificate
- default
- true
include_tasks bool Discover each job vertex as a Task asset
- default
- true
include_run_history bool Record each job's state changes as run history
- default
- true
include_completed bool Include the finished, failed and cancelled jobs the JobManager still lists
- default
- true
Assets emitted
Metadata this plugin attaches to each discovered asset.
Pipeline
FlinkPipelineFieldsThe metadata fields the Flink plugin emits for Pipeline assets (one per job).
jid stringFlink job id
state stringJob state (RUNNING, FINISHED, FAILED, CANCELED, ...)
start_time stringWhen the job was submitted (RFC3339)
end_time stringWhen the job reached its final state (RFC3339), absent while it runs
duration_ms intJob duration in milliseconds
is_stoppable boolWhether the job can be stopped with a savepoint
max_parallelism intConfigured maximum parallelism, absent when unset
parallelism intJob parallelism from the execution config
execution_mode stringExecution mode (Flink 1.x only)
restart_strategy stringRestart strategy description
task_counts objectNumber of tasks per state (running, finished, failed, ...)
vertex_count intNumber of vertices in the job graph
flink_version stringVersion of the Flink cluster
url stringJob page in the Flink web UI
error stringRoot cause of a failed job, trimmed to 500 characters
Task
FlinkTaskFieldsThe metadata fields the Flink plugin emits for Task assets (one per job vertex).
jid stringFlink job id
pipeline stringName of the Pipeline the vertex belongs to
vertex_id stringVertex id within the job graph
status stringVertex status (RUNNING, FINISHED, FAILED, CANCELED, ...)
parallelism intVertex parallelism
max_parallelism intVertex maximum parallelism
start_time stringWhen the vertex started (RFC3339)
end_time stringWhen the vertex finished (RFC3339), absent while it runs
duration_ms intVertex duration in milliseconds
operator stringOperator name from the job plan, when Flink reports one
description stringOperator chain from the job plan
read_records intRecords read by the vertex
write_records intRecords written by the vertex
read_bytes intBytes read by the vertex
write_bytes intBytes written by the vertex