Apache Pinot

marmotdata/pinot

Discover tables, schemas and stream lineage from Apache Pinot clusters

Warehouses Assets Lineage Experimental

The Pinot plugin discovers tables from Apache Pinot clusters through the controller REST API. It captures the schema (dimensions, metrics and date-time fields), table settings, segment counts, row counts and sizes, and links realtime tables to the Kafka topic or Kinesis stream they consume.

Connection

controller_url is required. Row counts and sample data run as SQL; they go through the controller's /sql proxy unless broker_url points at a broker directly. Basic auth (username, password) and a bearer token are both supported when the cluster has authentication enabled. Set database to discover one logical database on Pinot 1.1 or later.

Required Permissions

A read-only user with access to the table, schema, segment and query endpoints of the controller is enough:

GET  /health
GET  /version
GET  /tables
GET  /tables/{name}
GET  /tables/{name}/schema
GET  /tables/{name}/size
GET  /segments/{name}
GET  /schemas/{name}
POST /sql

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Apache Pinot from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-pinot-pipeline
runs:
  - pinot:
      controller_url: "<controller_url>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

15 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

controller_url string

Pinot controller URL, for example http://localhost:9000

broker_url string

Pinot broker URL for queries. When empty, queries go through the controller

username string

Username for basic authentication

password password

Password for basic authentication

token password

Bearer token, used instead of a username and password

database string

Logical database to discover (Pinot 1.1 and later). Empty means the default database

verify_ssl bool

Whether to verify the TLS certificate of the controller and broker

default: true

include_columns bool

Whether to include column information in table metadata

default: true

include_row_counts bool

Whether to count rows with a COUNT(*) query per table

default: true

include_sizes bool

Whether to include the segment size reported by the controller

default: true

discover_lineage bool

Whether to link realtime tables to the Kafka topic or Kinesis stream that feeds them

default: true

exclude_system_tables bool

Whether to skip tables whose name starts with an underscore

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

Table

PinotTableFields

23 fields

The metadata fields the Pinot plugin emits for table assets.

table_name string

Logical table name, without the _OFFLINE or _REALTIME suffix

table_types string[]

Table types present: OFFLINE, REALTIME or both for a hybrid table

ingestion_type string

How data arrives: batch, stream or hybrid

schema_name string

Name of the Pinot schema the table config points at, when it differs from the table name

time_column string

Column segments are partitioned and retained by

time_type string

Unit of the time column, for example DAYS or MILLISECONDS

replication string

Number of replicas per segment

retention string

How long segments are kept, as value and unit (for example 30 DAYS)

broker_tenant string

Broker tenant serving the table

server_tenant string

Server tenant hosting the table

load_mode string

How segments are loaded on servers (MMAP or HEAP)

is_dim_table bool

Whether the table is a dimension table replicated to every server

primary_key_columns string[]

Primary key columns declared in the schema

sorted_column string

Column segments are sorted by

inverted_index_columns string[]

Columns with an inverted index

segment_count int

Total number of segments across table types

offline_segment_count int

Number of offline segments

realtime_segment_count int

Number of realtime segments, including the consuming one

stream_type string

Stream type a realtime table consumes from, for example kafka or kinesis

stream_topic string

Topic or stream name a realtime table consumes from

stream_brokers string

Broker list a realtime table consumes from

pinot_version string

Pinot release reported by the controller

url string

Link to the table in the controller UI

Column

PinotColumnFields

9 fields

The per-column fields embedded in an asset's schema.

column_name string

Column name

data_type string

Pinot data type; multi-value columns carry a [] suffix

is_nullable bool

Whether null values are allowed (true unless the schema marks the field notNull)

is_primary_key bool

Whether the column is part of the primary key

field_type string

Role of the field in the schema: dimension, metric or datetime

single_value bool

Whether each row holds one value rather than an array

format string

Date-time format, for example 1:DAYS:EPOCH or TIMESTAMP

granularity string

Date-time granularity, for example 1:SECONDS

default_null_value object

Value stored in place of null

Stream

PinotStreamFields

2 fields

The metadata fields emitted on the Kafka topic or Kinesis stream asset that feeds a realtime table.

stream_type string

Stream type: kafka or kinesis

stream_brokers string

Broker list the table consumes from