Unity Catalog

marmotdata/unitycatalog

Discover catalogs, tables, views, volumes, functions and models from Unity Catalog servers

Catalogs Assets Lineage Experimental

The Unity Catalog plugin discovers catalogs, tables, views, volumes, functions and registered models from a Unity Catalog server.

It talks to the open-source Unity Catalog REST API at {host}/api/2.1/unity-catalog, so it works against a self-hosted server and against anything else that serves the same API. Schemas are not assets of their own: every object is named by its three-part catalog.schema.object name and hangs off its catalog.

Authentication

token is sent as a bearer token when it is set. A server running without authentication needs no token.

Lineage

Each catalog CONTAINS every table, view, volume, function and model under it.
A view gets a VIEW_OF edge from each base table its definition reads, resolved against the view's own catalog and schema.
A table with foreign key constraints gets a FOREIGN_KEY edge to each parent table discovered in the same run. The open-source server does not store constraints, so these come from Databricks-compatible servers only.
A table or volume stored on cloud object storage gets a FEEDS edge from its bucket or container: s3:// from the S3 plugin, gs:// from GCS, abfss:// from Azure Blob. The bucket asset itself belongs to those plugins; this plugin only emits the edge.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Unity Catalog from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-unitycatalog-pipeline
runs:
  - unitycatalog:
      host: "<host>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

13 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

Unity Catalog server URL, for example http://localhost:8080

token password

Bearer token, when the server requires one

catalogs multiselect

Catalogs to discover (all when empty)

exclude_catalogs multiselect

Catalogs to skip

default: ["system","__databricks_internal"]

include_columns bool

Whether to include column information in table metadata

default: true

include_volumes bool

Whether to discover volumes

default: true

include_functions bool

Whether to discover functions

default: true

include_models bool

Whether to discover registered models

default: true

verify_ssl bool

Whether to verify the server's TLS certificate

default: true

page_size int

Objects requested per API page

default: 100

Assets emitted

Metadata this plugin attaches to each discovered asset.

Catalog

UnityCatalogCatalogFields

9 fields

The metadata fields emitted for catalog assets.

catalog_name string

Catalog name

catalog_id string

Catalog id

comment string

Catalog comment

owner string

Catalog owner

properties object

User-defined properties

schema_count int

Number of schemas in the catalog

table_count int

Number of tables and views in the catalog

created_at string

Creation time (RFC 3339)

updated_at string

Last update time (RFC 3339)

Table

UnityCatalogTableFields

15 fields

The metadata fields emitted for table and view assets.

catalog string

Catalog the table belongs to

schema string

Schema the table belongs to

table_name string

Table or view name

table_type string

Table type (MANAGED, EXTERNAL, VIEW, MATERIALIZED_VIEW, STREAMING_TABLE, FOREIGN)

data_source_format string

Data source format (DELTA, PARQUET, CSV, ...)

storage_location string

Storage location of the table data

owner string

Table owner

comment string

Table comment

properties object

User-defined properties

table_id string

Table id

partition_columns string[]

Partition columns in partition order

primary_key string[]

Primary key columns

foreign_keys object[]

Foreign key constraints (columns, parent_table, parent_columns)

created_at string

Creation time (RFC 3339)

updated_at string

Last update time (RFC 3339)

Column

UnityCatalogColumnFields

7 fields

The per-column fields embedded in an asset's schema.

column_name string

Column name

data_type string

Column type as written (type_text)

type_name string

Column type name (INT, STRING, DECIMAL, ...)

is_nullable bool

Whether null values are allowed

is_primary_key bool

Whether the column is part of the primary key

partition_index int

Position in the partition key, for partition columns

description string

Column comment

Volume

UnityCatalogVolumeFields

10 fields

The metadata fields emitted for volume assets.

catalog string

Catalog the volume belongs to

schema string

Schema the volume belongs to

volume_name string

Volume name

volume_type string

Volume type (MANAGED, EXTERNAL)

storage_location string

Storage location of the volume

comment string

Volume comment

owner string

Volume owner

volume_id string

Volume id

created_at string

Creation time (RFC 3339)

updated_at string

Last update time (RFC 3339)

Function

UnityCatalogFunctionFields

14 fields

The metadata fields emitted for function assets.

catalog string

Catalog the function belongs to

schema string

Schema the function belongs to

function_name string

Function name

parameters string[]

Input parameters as "name type", in declaration order

return_type string

Return type

routine_body string

Routine body kind (SQL, EXTERNAL)

language string

Language the body is written in (SQL, python, ...)

is_deterministic bool

Whether the function is deterministic

sql_data_access string

SQL data access (CONTAINS_SQL, READS_SQL_DATA, NO_SQL)

comment string

Function comment

owner string

Function owner

function_id string

Function id

created_at string

Creation time (RFC 3339)

updated_at string

Last update time (RFC 3339)

Model

UnityCatalogModelFields

12 fields

The metadata fields emitted for registered model assets.

catalog string

Catalog the model belongs to

schema string

Schema the model belongs to

model_name string

Model name

comment string

Model comment

owner string

Model owner

storage_location string

Storage location of the model artifacts

model_id string

Model id

version_count int

Number of registered versions

latest_version int

Highest registered version number

versions object

Version number to status (READY, PENDING_REGISTRATION, ...)

created_at string

Creation time (RFC 3339)

updated_at string

Last update time (RFC 3339)