Google Drive
Core pluginmarmotdata/googledrive
Discover folders, files and spreadsheets from Google Drive
The Google Drive plugin catalogues the documents and sheets a team keeps in Drive, so the spreadsheet a report is actually built from is findable next to the warehouse tables feeding it.
A drive is imported as a tree: the drive itself at the root, folders nested under it, files linked to the folder holding them. Google Sheets can be imported as spreadsheets, with each sheet under them optionally imported as a table whose header row becomes the columns.
These are the same asset types and names the OpenMetadata plugin produces for a Google Drive service, so an organisation moving off OpenMetadata can import their drive from there and later point this plugin at Drive directly; the second run takes over the assets already in the catalog rather than creating a second copy.
Access
The plugin reads Drive with a Google service account.
A service account on its own only sees files that have been shared with its email address, which is a reasonable way to catalogue a handful of shared folders. To read a whole organisation's Drive, give the service account domain-wide delegation and set impersonate_user to a Workspace user to act as.
Enable the Google Drive API, and the Google Sheets API if you want the columns of each sheet. The scopes needed are:
https://www.googleapis.com/auth/drive.metadata.readonly
https://www.googleapis.com/auth/spreadsheets.readonlyCredentials follow Marmot's shared Google Cloud configuration: a key file, key JSON, or Application Default Credentials when nothing is set.
In the UI
Point-and-click, no config file needed.
- 1 Open
Runs→Create Pipeline - 2 Pick Google Drive from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-googledrive-pipeline
runs:
- googledrive:
# No required fields — see the Configuration tab.marmot ingest -c ingest.yaml Not using plugins? Other ways to populate Marmot
Configuration
16 top-level fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
credentials object GCP credentials configuration
credentials_json password Service account key JSON content
credentials_file string Path to a service account key JSON file
impersonate_user string Workspace user to act as. Needed to read a whole organisation's Drive, and requires domain-wide delegation on the service account. Without it, only files shared with the service account are visible
drive_id string Shared drive to read. Empty means the user's My Drive
folder_id string Only discover this folder and everything under it
include_files bool Discover files, not just folders
true include_spreadsheets bool Discover Google Sheets, including each of their sheets
true include_worksheets bool Discover each sheet of a spreadsheet as a table, with its columns. Needs the Sheets read scope
true include_trashed bool Discover files in the trash
false exclude_mime_types multiselect MIME types to skip, for example application/vnd.google-apps.shortcut
max_files int Stop after this many files. 0 means no limit
0 page_size int Files per API request
1000 header_row int Row of a sheet that holds the column names
1 concurrency int Parallel requests for the sheets of a spreadsheet
8