Google BigQuery
Official · maintained by Marmotmarmotdata/bigquery Discover datasets and tables from Google BigQuery projects
The BigQuery plugin discovers datasets, tables, and views from Google BigQuery projects. It captures schemas, statistics, and lineage relationships.
Required Permissions
Assign roles/bigquery.metadataViewer to your service account, or these individual permissions:
bigquery.datasets.getbigquery.tables.getbigquery.tables.list
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Google BigQuery from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-bigquery-pipeline
runs:
- bigquery:
project_id: "<project_id>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
13 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
project_id string Google Cloud Project ID
credentials_path string Path to service account credentials JSON file
credentials_json password Service account credentials JSON content
use_default_credentials bool Use default Google Cloud credentials
- default
- false
include_datasets bool Whether to discover datasets
- default
- true
include_table_stats bool Whether to include table statistics (row count, size)
- default
- true
include_views bool Whether to discover views
- default
- true
include_external_tables bool Whether to discover external tables
- default
- true
exclude_system_datasets bool Whether to exclude system datasets (_script, _analytics, etc.)
- default
- true
max_concurrent_requests int Maximum number of concurrent API requests
- default
- 10
Assets emitted
Metadata this plugin attaches to each discovered asset.
Big Query Dataset
BigQueryDatasetFieldsproject_id stringGoogle Cloud Project ID
dataset_id stringDataset ID
location stringGeographic location of the dataset
creation_time stringDataset creation timestamp
last_modified stringLast modification timestamp
description stringDataset description
default_table_expiration stringDefault table expiration duration
default_partition_expiration stringDefault partition expiration duration
labels objectDataset labels
access_entries_count intNumber of access control entries
Big Query Table
BigQueryTableFieldsproject_id stringGoogle Cloud Project ID
dataset_id stringDataset ID
table_id stringTable ID
table_type stringTable type (TABLE, VIEW, EXTERNAL)
creation_time stringTable creation timestamp
last_modified stringLast modification timestamp
description stringTable description
expiration_time stringTable expiration timestamp
labels objectTable labels
num_rows intNumber of rows in the table
num_bytes intSize of the table in bytes
time_partitioning_type stringTime partitioning type
time_partitioning_field stringTime partitioning field
partition_expiration stringPartition expiration duration
range_partitioning_field stringRange partitioning field
clustering_fields string[]Clustering fields
view_query stringSQL query for views
external_data_config objectExternal data configuration for external tables
Big Query Column
BigQueryColumnFieldsname stringColumn name
type stringColumn data type
description stringColumn description
nested_fields object[]Nested fields for RECORD type columns