Apache Doris

marmotdata/doris

Discover databases, tables, views and materialized views from Apache Doris clusters

Warehouses Assets Lineage Experimental

The Doris plugin discovers databases, tables, views and async materialized views from Apache Doris clusters. It reads the table model, partitioning, distribution and column aggregation types from the DDL, and emits row counts, data sizes, view lineage and foreign key relationships.

It connects to a frontend over the MySQL protocol (port 9030 by default) with github.com/go-sql-driver/mysql.

Required Permissions

The user needs to read the catalog and the definitions of views:

CREATE USER 'marmot_reader'@'%' IDENTIFIED BY 'your-password';
GRANT SELECT_PRIV, SHOW_VIEW_PRIV ON internal.*.* TO 'marmot_reader'@'%';

SHOW_VIEW_PRIV is what lets SHOW CREATE VIEW run; without it views are catalogued without their query or lineage.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Apache Doris from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-doris-pipeline
runs:
  - doris:
      host: "<host>"
      user: "<user>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

15 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

Doris frontend hostname or IP address

port int

Doris frontend MySQL protocol port

default: 9030

user string

Username for authentication

password password

Password for authentication

databases multiselect

Databases to discover (every database except exclude_databases when empty)

exclude_databases multiselect

Databases to skip when databases is empty

default: ["information_schema","mysql","__internal_schema"]

catalog string

Doris catalog to discover

default: internal

tls select

TLS configuration (false, true, skip-verify, preferred)

default: false

false true skip-verify preferred

include_columns bool

Whether to include column information in table metadata

default: true

include_views bool

Whether to discover views

default: true

include_materialized_views bool

Whether to discover async materialized views

default: true

include_statistics bool

Whether to include row counts, data sizes and column counts

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

Apache Doris

DorisFields

31 fields

The metadata fields the Doris plugin emits for database, table and view assets.

host string

Doris frontend hostname

port int

Doris frontend MySQL protocol port

catalog string

Doris catalog the object lives in

database string

Database name

doris_version string

Doris build the frontend reports

table_count int

Tables in the database

view_count int

Views and materialized views in the database

table_name string

Object name without the database

object_type string

Object type (table, view, materialized_view, external_table)

engine string

Storage engine (Doris, View, MATERIALIZED_VIEW, or the external system)

key_model string

Table model (DUPLICATE, UNIQUE, AGGREGATE, none)

key_columns string[]

Columns that make up the key

partition_type string

Partitioning (RANGE, LIST, none)

partition_columns string[]

Columns or expressions the table is partitioned by

partition_count int

Number of partitions

distribution_type string

Bucketing (HASH, RANDOM)

distribution_columns string[]

Columns the data is hashed on

buckets int

Number of buckets per partition

auto_bucket bool

Whether Doris picks the bucket count

replication string

Replication allocation or replica count

storage_medium string

Storage medium (hdd, ssd)

comment string

Table or view comment

created string

Creation timestamp

updated string

Last update timestamp

ddl string

SHOW CREATE output

materialized bool

Whether the view is an async materialized view

job_name string

Refresh job of a materialized view

state string

State of a materialized view (NORMAL, SCHEMA_CHANGE)

refresh_state string

Outcome of the last materialized view refresh

refresh_info string

Build and refresh settings of a materialized view

mv_partition_info string

How a materialized view is partitioned

Column

DorisColumnFields

9 fields

The per-column fields embedded in an asset's schema.

column_name string

Column name

data_type string

Declared type as Doris prints it (decimalv3(9, 2), array<int>)

is_nullable bool

Whether null values are allowed

is_primary_key bool

Key column of a UNIQUE or AGGREGATE table

is_sorting_key bool

Key column of a DUPLICATE table

is_key bool

Whether the column is part of the table key

aggregation_type string

Aggregation of a value column on an AGGREGATE table (SUM, MAX, REPLACE, HLL_UNION, BITMAP_UNION)

default_expression string

Default value

description string

Column comment