Apache NiFi

marmotdata/nifi

Discover process groups, processors and data flow lineage from Apache NiFi

Pipelines Assets Lineage Experimental

The NiFi plugin discovers process groups and processors from Apache NiFi through its REST API. Each process group becomes a Pipeline, each processor a Task, and the connections between them become lineage. Well-known processors are also linked to the S3 and GCS buckets, Azure Blob containers, Kafka topics, database tables and Elasticsearch indexes they read or write.

Authentication

Single-user and LDAP installs take username and password; the plugin exchanges them for a bearer token. An existing bearer token goes in token. Installs secured with client certificates take client_cert and client_key. NiFi ships with a self-signed certificate, so verify_ssl: false is needed unless the certificate is trusted or ca_cert points at its CA.

The user needs view the user interface and read access to the process groups to discover, plus read access to controller services for table lineage through connection pools.

Run history

NiFi records provenance events, not runs, so the plugin emits no run history.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Apache NiFi from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-nifi-pipeline
runs:
  - nifi:
      host: "<host>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

15 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

NiFi base URL (e.g. https://nifi.example.com:8443)

username string

Username for single-user or LDAP login. Leave all credentials empty for an unsecured HTTP install

password password

Password for the username

token password

Bearer token to use instead of a username and password

client_cert string

Path to a PEM client certificate for mutual TLS (used instead of a login)

client_key string

Path to the PEM private key of the client certificate

ca_cert string

Path to a PEM CA certificate to trust

verify_ssl bool

Verify the NiFi TLS certificate. Set to false for the self-signed certificate NiFi ships with

default: true

include_processors bool

Discover processors as Task assets

default: true

include_ports bool

Discover input and output ports as Task assets

default: false

discover_lineage bool

Link well-known processors to the buckets, topics, tables and indexes they read or write

default: true

root_process_group string

Id of the process group to start from (defaults to the root group)

Assets emitted

Metadata this plugin attaches to each discovered asset.

Pipeline

NiFiPipelineFields

15 fields

The metadata fields the NiFi plugin emits for process group (Pipeline) assets.

id string

Process group id

parent_id string

Id of the parent process group

path string

Group names from the root, joined with /

comments string

Comments set on the process group

parameter_context string

Name of the bound parameter context

running_count int

Components running in the group and its children

stopped_count int

Components stopped in the group and its children

invalid_count int

Components that cannot start because their configuration is invalid

disabled_count int

Components disabled in the group and its children

processor_count int

Processors directly in the group

connection_count int

Connections directly in the group

input_port_count int

Input ports of the group

output_port_count int

Output ports of the group

nifi_version string

NiFi release the instance runs

url string

Link to the group in the NiFi UI

Task

NiFiTaskFields

12 fields

The metadata fields emitted for processor (Task) assets.

id string

Processor id

group_id string

Id of the process group holding the processor

pipeline string

Path of the process group holding the processor

type string

Processor class name (e.g. PutS3Object)

type_full string

Fully qualified processor class name

state string

RUNNING, STOPPED, DISABLED or INVALID

scheduling_period string

How often the processor is scheduled

scheduling_strategy string

TIMER_DRIVEN or CRON_DRIVEN

comments string

Comments set on the processor

properties object

Configured properties, with sensitive values masked

relationships string[]

Names of the processor's relationships

url string

Link to the processor in the NiFi UI

Port

NiFiPortFields

1 field

The extra metadata emitted for input and output ports when include_ports is on. Ports are Task assets and share id, group_id, pipeline, state, comments and url with processors.

port_type string

INPUT_PORT or OUTPUT_PORT

Topic

NiFiTopicFields

3 fields

The metadata emitted for the Kafka Topic assets created from PublishKafka and ConsumeKafka processors.

topic_name string

Kafka topic name

producers string[]

Tasks that publish to the topic

consumers string[]

Tasks that consume from the topic