Google Drive
Official · maintained by Marmotmarmotdata/googledrive Discover folders, files and spreadsheets from Google Drive
The Google Drive plugin catalogues the documents and sheets a team keeps in Drive, so the spreadsheet a report is actually built from is findable next to the warehouse tables feeding it.
A drive is imported as a tree: the drive itself at the root, folders nested under it, files linked to the folder holding them. Google Sheets can be imported as spreadsheets, with each sheet under them optionally imported as a table whose header row becomes the columns.
These are the same asset types and names the OpenMetadata plugin produces for a Google Drive service, so an organisation moving off OpenMetadata can import their drive from there and later point this plugin at Drive directly; the second run takes over the assets already in the catalog rather than creating a second copy.
Access
The plugin reads Drive with a Google service account.
A service account on its own only sees files that have been shared with its email address, which is a reasonable way to catalogue a handful of shared folders. To read a whole organisation's Drive, give the service account domain-wide delegation and set impersonate_user to a Workspace user to act as.
Enable the Google Drive API, and the Google Sheets API if you want the columns of each sheet. The scopes needed are:
https://www.googleapis.com/auth/drive.metadata.readonly
https://www.googleapis.com/auth/spreadsheets.readonlyCredentials follow Marmot's shared Google Cloud configuration: a key file, key JSON, or Application Default Credentials when nothing is set.
Keyless authentication
On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of a service account key. Set credentials.workload_identity_provider to a Workload Identity Federation provider that trusts your Marmot instance as an OIDC issuer, and grant the pipeline's subject, pipeline:<name> as reported by the pipeline API, the role above directly (principal://iam.googleapis.com/<pool>/subject/pipeline:<name>), or grant it roles/iam.workloadIdentityUser on a service account named in credentials.service_account. No key exists anywhere; Marmot mints a short-lived token for each run.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Google Drive from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-googledrive-pipeline
runs:
- googledrive:
# No required fields — see the Configuration tab.$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
17 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
credentials object GCP credentials configuration
credentials_json password Service account key JSON content
credentials_file string Path to a service account key JSON file
workload_identity_provider string Workload Identity Federation provider to exchange the Marmot identity token at, projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Setting it federates: the pipeline presents a Marmot identity token and no key is needed; bind the pipeline's subject on the Google Cloud side
service_account string Service account to impersonate after the exchange; empty acts as the federated principal directly
audience string Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another
impersonate_user string Workspace user to act as. Needed to read a whole organisation's Drive, and requires domain-wide delegation on the service account. Without it, only files shared with the service account are visible
drive_id string Shared drive to read. Empty means the user's My Drive
folder_id string Only discover this folder and everything under it
include_files bool Discover files, not just folders
- default
- true
include_spreadsheets bool Discover Google Sheets, including each of their sheets
- default
- true
include_worksheets bool Discover each sheet of a spreadsheet as a table, with its columns. Needs the Sheets read scope
- default
- true
include_trashed bool Discover files in the trash
- default
- false
exclude_mime_types multiselect MIME types to skip, for example application/vnd.google-apps.shortcut
max_files int Stop after this many files. 0 means no limit
- default
- 0
page_size int Files per API request
- default
- 1000
header_row int Row of a sheet that holds the column names
- default
- 1
concurrency int Parallel requests for the sheets of a spreadsheet
- default
- 8