Google BigQuery

marmotdata/bigquery

Discover datasets and tables from Google BigQuery projects

Warehouses Assets Lineage Experimental

The BigQuery plugin discovers datasets, tables, views, and external tables from Google BigQuery projects. It captures schemas, statistics, and lineage relationships.

Required Permissions

Assign roles/bigquery.metadataViewer to your service account, or these individual permissions:

bigquery.datasets.get
bigquery.tables.get
bigquery.tables.list

Keyless authentication

On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of a service account key. Set workload_identity_provider to a Workload Identity Federation provider that trusts your Marmot instance as an OIDC issuer, and grant the pipeline's subject, pipeline:<name> as reported by the pipeline API, the role above directly (principal://iam.googleapis.com/<pool>/subject/pipeline:<name>), or grant it roles/iam.workloadIdentityUser on a service account named in service_account. No key exists anywhere; Marmot mints a short-lived token for each run.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Google BigQuery from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-bigquery-pipeline
runs:
  - bigquery:
      project_id: "<project_id>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

16 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

audience string

Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another

project_id string

Google Cloud Project ID

credentials_path string

Path to service account credentials JSON file

credentials_json password

Service account credentials JSON content

use_default_credentials bool

Use default Google Cloud credentials

default: false

workload_identity_provider string

Workload Identity Federation provider to exchange the Marmot identity token at, projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Setting it federates: no key is needed; grant the pipeline's subject roles/bigquery.metadataViewer on the project

service_account string

Service account to impersonate after the exchange; empty acts as the federated principal directly

include_datasets bool

Whether to discover datasets

default: true

include_table_stats bool

Whether to include table statistics (row count, size)

default: true

include_views bool

Whether to discover views

default: true

include_external_tables bool

Whether to discover external tables

default: true

exclude_system_datasets bool

Whether to exclude system datasets (_script, _analytics, etc.)

default: true

max_concurrent_requests int

Maximum number of concurrent API requests

default: 10

Assets emitted

Metadata this plugin attaches to each discovered asset.

Dataset

BigQueryDatasetFields

10 fields

project_id string

Google Cloud Project ID

dataset_id string

Dataset ID

location string

Geographic location of the dataset

creation_time string

Dataset creation timestamp

last_modified string

Last modification timestamp

description string

Dataset description

default_table_expiration string

Default table expiration duration

default_partition_expiration string

Default partition expiration duration

labels object

Dataset labels

access_entries_count int

Number of access control entries

Table

BigQueryTableFields

18 fields

project_id string

Google Cloud Project ID

dataset_id string

Dataset ID

table_id string

Table ID

table_type string

Table type (TABLE, VIEW, EXTERNAL)

creation_time string

Table creation timestamp

last_modified string

Last modification timestamp

description string

Table description

expiration_time string

Table expiration timestamp

labels object

Table labels

num_rows int

Number of rows in the table

num_bytes int

Size of the table in bytes

time_partitioning_type string

Time partitioning type

time_partitioning_field string

Time partitioning field

partition_expiration string

Partition expiration duration

range_partitioning_field string

Range partitioning field

clustering_fields string[]

Clustering fields

view_query string

SQL query for views

external_data_config object

External data configuration for external tables

Column

BigQueryColumnFields

4 fields

name string

Column name

type string

Column data type

description string

Column description

nested_fields object[]

Nested fields for RECORD type columns