marmotdata/amundsen
Import tables, columns, owners, usage, dashboards and lineage from an Amundsen metadata graph
The Amundsen plugin imports the contents of an Amundsen metadata graph over Bolt, reading Neo4j directly.
Amundsen is a catalog, so every entry in it describes something that lives somewhere else. Each table is projected onto the Marmot provider and MRN its own technology's plugin uses, so an Amundsen import and a later native run land on one asset instead of two. amundsen/projection.go holds that mapping; it agrees with the OpenMetadata plugin's projection for the technologies they share.
What it reads
Tables and views, their columns, owners, tags, badges and programmatic descriptions; dashboards and their charts; read counts as statistics; table to table lineage from HAS_UPSTREAM, and dashboard to table lineage from DASHBOARD_WITH_TABLE.
No database, cluster or schema assets are created. Those belong to the technology's own plugin.
Naming
The Amundsen hierarchy is Database -> Cluster -> Schema -> Table, where Database holds the technology name. The cluster enters an asset's name only where the technology's Marmot name has room for it, such as the leading part of a Snowflake database.schema.table. Everywhere else it is an environment label and is recorded in metadata. A table keeps its own name and its MRN carries the path, for example mrn://asset/table/postgresql/prod.public.orders, mrn://asset/table/mysql/shop.orders and mrn://asset/table/bigquery/acme.sales.orders.
Amundsen holds no address for the systems it describes, so its MRNs have no zone. If you run two systems of one technology and their own plugins set zone, Amundsen's copies of those assets stay separate from them.
Tests
Unit tests run without Neo4j. The end to end tests need a graph:
docker run -d --name marmot-test-amundsen -p 17687:7687 \
-e NEO4J_AUTH=neo4j/marmotpass neo4j:5-community
docker cp testdata/seed.cypher marmot-test-amundsen:/tmp/seed.cypher
docker exec marmot-test-amundsen cypher-shell -u neo4j -p marmotpass -f /tmp/seed.cypher
MARMOT_TEST_AMUNDSEN_URI=bolt://localhost:17687 \
MARMOT_TEST_AMUNDSEN_USER=neo4j \
MARMOT_TEST_AMUNDSEN_PASSWORD=marmotpass \
go test ./...
In the UI
Point and click, no config file needed.
- Open Runs and choose Create pipeline.
- Pick Amundsen from the plugin list.
- Fill in the settings, set a schedule and save.
With the CLI
Save a YAML config, then run marmot ingest.
Not using plugins? Other ways to populate Marmot
Configuration
Fields marked Required must be set.
tagsList of textTags to apply to discovered assets
external_linksList of groupsExternal links to show on all assets
nameTextDisplay name for the link
iconTextIcon identifier for the link
urlTextURL to the external resource
filterGroup of settingsFilter discovered assets by name (regex)
includeList of textInclude patterns for resource names (regex)
excludeList of textExclude patterns for resource names (regex)
zoneTextTells this system apart from others of the same technology and becomes part of every asset's MRN
uriText RequiredBolt address of Amundsen's Neo4j, for example bolt://neo4j.company.com:7687
usernameText RequiredNeo4j username
passwordSecret RequiredNeo4j password
databaseTextNeo4j database holding the Amundsen graph
encryptedYes or noConnect over TLS
trust_all_certificatesYes or noAccept any TLS certificate, including self signed ones
amundsen_urlTextAddress of the Amundsen web app, used to link each asset back to its page
include_usersYes or noRecord table owners from Amundsen
include_dashboardsYes or noImport dashboards and their charts
include_tagsYes or noCopy Amundsen tags onto assets
include_descriptionsYes or noCopy Amundsen descriptions onto assets and columns
include_usageYes or noImport Amundsen read counts as statistics
query_timeout_secondsNumberPer-query timeout
page_sizeNumberRecords per query. Every query is paged, so a large graph does not have to fit in memory
Assets emitted
Metadata this plugin attaches to each discovered asset.
Table
The metadata fields the plugin emits on a table or view asset.
Column
The per-column fields embedded in a table asset's schema.
Dashboard
The metadata fields the plugin emits on a dashboard asset.
Chart
The metadata fields the plugin emits on a chart asset.