marmotdata/openmetadata
Import tables, topics, dashboards, pipelines, models, glossary and lineage from an OpenMetadata instance
OpenMetadata The OpenMetadata plugin imports an entire OpenMetadata instance in one run: tables, views, stored procedures, topics, buckets, dashboards, charts, pipelines, models, search indices, API endpoints, drive files and spreadsheets, the business glossary, and the lineage between them.
OpenMetadata is a catalog, so everything in it describes something that lives somewhere else. This plugin catalogues each entity as the technology it belongs to rather than as an OpenMetadata thing: a table under a Postgres service becomes a PostgreSQL asset in Marmot, addressed exactly as Marmot's own PostgreSQL plugin would address it.
Supported OpenMetadata Versions
OpenMetadata 1.4 or newer. The plugin negotiates the field list with the server on the first call to each endpoint: fields the server does not know are dropped from later requests, so one plugin binary works across every version in that range. Entity kinds a server predates (drives, API collections, and so on) are skipped without failing the run.
Getting a Token
The plugin authenticates as a bot or as a user, with a JWT.
For a bot, open Settings → Bots in OpenMetadata, pick a bot such as ingestion-bot, and copy its token. For a user, open Settings → Members, pick the user, and create a personal access token. The token needs read access to the entities you want to import.
Cutting Over from OpenMetadata
Moving off OpenMetadata is not a single switch, so this plugin is built to run on a schedule for as long as the move takes. Each run brings across whatever changed in OpenMetadata; re-running is safe, and assets that have not changed are left alone.
Anything written in Marmot survives every re-sync. A description edited in Marmot is stored separately from the one the run imported, so the next sync refreshes the imported side and never overwrites the edit. The same holds for tags, owners and glossary terms added in Marmot.
When you are ready to catalogue a system directly, add its own pipeline (for example the PostgreSQL plugin against the database OpenMetadata was describing). Imported assets and native ones share an MRN, so the native run takes over the assets that already exist instead of creating a second copy. Nothing needs to be deleted or re-pointed.
When you are done, stop scheduling the run. The imported assets stay exactly as they are. Do not use marmot ingest --destroy: it deletes every asset the pipeline ever created, including ones another pipeline has since taken over.
Running Alongside Native Plugins
An imported asset lands on the same MRN the technology's native Marmot plugin would use, so the two runs contribute to one asset instead of creating two. A Postgres table becomes mrn://asset/table/postgresql/shop.public.orders whether Marmot read it from OpenMetadata or from the database itself. The path is the OpenMetadata name below the service, trimmed to the levels the native plugin uses: MySQL names a table <database>.<table>, BigQuery <project>.<dataset>.<table>, and so on.
Two systems of the same technology are told apart by zone. Map each OpenMetadata service to the Marmot zone of its native plugin with service_map, for example service_map: {prod-postgres: prod-eu}. A service that is not mapped carries the address from its OpenMetadata connection as a host hint, and Marmot picks the zone that owns that address; with neither, no zone is used and two services holding the same table merge into one asset. The run logs what it merged. This plugin never puts zone into an MRN, because it does not own the assets it describes.
Technologies Marmot has no plugin for yet, such as Snowflake or Looker, are imported under their own provider name. Nothing is invented: an entity is only imported when Marmot already has an asset type that means the same thing.
Container Prefixes and Drives
Object storage comes across as the bucket alone. OpenMetadata models the prefixes inside a bucket as containers of their own, but Marmot's S3, GCS and Azure Blob plugins catalogue the bucket and nothing below it, so an imported prefix would sit in the catalog forever without a native run ever updating it. Set include_container_prefixes: true to import the hierarchy anyway, worth doing when nothing else is going to catalogue that bucket.
Drives are different: a drive really is a tree of folders and Marmot's GoogleDrive plugin catalogues it as one, so folders come across in full.
In the UI
Point and click, no config file needed.
- Open Runs and choose Create pipeline.
- Pick OpenMetadata from the plugin list.
- Fill in the settings, set a schedule and save.
With the CLI
Save a YAML config, then run marmot ingest.
Not using plugins? Other ways to populate Marmot
Configuration
Fields marked Required must be set.
tagsList of textTags to apply to discovered assets
external_linksList of groupsExternal links to show on all assets
nameTextDisplay name for the link
iconTextIcon identifier for the link
urlTextURL to the external resource
filterGroup of settingsFilter discovered assets by name (regex)
includeList of textInclude patterns for resource names (regex)
excludeList of textExclude patterns for resource names (regex)
zoneTextTells this system apart from others of the same technology and becomes part of every asset's MRN
hostText RequiredOpenMetadata server URL, for example https://openmetadata.company.com
jwt_tokenSecret RequiredBot token or personal access token from OpenMetadata
timeout_secondsNumberPer-request timeout
insecure_skip_verifyYes or noSkip TLS certificate verification
servicesList of textOnly import these OpenMetadata services (all if empty)
exclude_servicesList of textOpenMetadata services to skip
service_typesList of textOnly import these OpenMetadata service types, for example Postgres or Kafka (all if empty)
exclude_service_typesList of textOpenMetadata service types to skip
include_deletedYes or noImport entities OpenMetadata has soft deleted
include_tablesYes or noImport databases, tables and views
include_stored_proceduresYes or noImport stored procedures as functions
include_topicsYes or noImport messaging topics
include_containersYes or noImport object storage buckets and containers
include_container_prefixesYes or noAlso import the prefixes and folders inside a storage container. Marmot's own object storage plugins catalogue only the container itself
include_drivesYes or noImport drive directories, files, spreadsheets and worksheets
include_dashboardsYes or noImport dashboards, charts and dashboard data models
include_pipelinesYes or noImport orchestration pipelines
include_tasksYes or noImport the individual tasks of each pipeline
include_run_historyYes or noImport recent pipeline executions as run history
run_history_daysNumberHow many days of pipeline executions to import
run_history_limitNumberMaximum executions to import per pipeline
include_mlmodelsYes or noImport machine learning models
include_search_indexesYes or noImport search indices
include_apisYes or noImport API collections and endpoints
include_columnsYes or noImport column, field and feature definitions
include_lineageYes or noImport lineage between imported assets
include_glossaryYes or noImport the business glossary as Marmot glossary terms, and assign them to the assets they are curated onto
service_mapTextZone of each OpenMetadata service, as service name to zone
tags_from_openmetadataYes or noCopy OpenMetadata classification tags onto assets
glossary_terms_as_tagsYes or noAlso copy assigned glossary terms onto assets as tags. They are imported as glossary terms either way
link_to_openmetadataYes or noAdd a link back to the entity in OpenMetadata on every asset
source_priorityNumberPriority of OpenMetadata against other sources of the same asset. Lower wins
page_sizeNumberEntities per API request
concurrencyNumberParallel lineage requests
Assets emitted
Metadata this plugin attaches to each discovered asset.
OpenMetadata
OpenMetadataFields records where in OpenMetadata an asset came from. It is set on every asset this plugin creates.
Curation
The curated context OpenMetadata holds that a Marmot ingestion run cannot yet create as first class objects, so it travels with the asset instead.
Table
Metadata on tables, views and stored procedures. The field names match Marmot's own database plugins so a merged asset reads the same whichever run contributed it.
Topic
Metadata on messaging topics.
Container
Metadata on object storage containers.
Dashboard
Metadata on dashboards, charts and dashboard data models.
Pipeline
Metadata on pipelines and their tasks.
MLModel
Metadata on machine learning models.
Search Index
Metadata on search indices.
API
Metadata on API collections and endpoints.
Drive
Metadata on drive directories, files, spreadsheets and worksheets.