Google Bigtable
Official · maintained by Marmotmarmotdata/bigtable Discover instances, tables and column families from Google Cloud Bigtable
The Bigtable plugin discovers instances, tables and column families from Google Cloud Bigtable.
Tables are named <instance>.<table>, because a table name is only unique within an instance. Each instance holds its tables through CONTAINS lineage.
Columns
Bigtable declares column families but not the qualifiers under them, and every value is raw bytes. The plugin reads sample_rows rows per table and records the family:qualifier pairs it saw, how many sampled rows held each one, and whether the values looked like text, an 8 byte number, or binary. Set include_columns to false to skip the sample.
Emulator
emulator_host connects to a local emulator without TLS or credentials. The emulator cannot list its own instances, so instances has to be listed alongside it. The plugin never reads BIGTABLE_EMULATOR_HOST, so one pipeline cannot change how another one connects.
Tests
Unit tests need nothing. The end to end tests need an emulator:
docker run -d --name bigtable-emulator -p 8086:8086 \
gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators \
gcloud beta emulators bigtable start --host-port=0.0.0.0:8086
MARMOT_TEST_BIGTABLE_EMULATOR=127.0.0.1:8086 go test ./...In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Google Bigtable from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-bigtable-pipeline
runs:
- bigtable:
project_id: "<project_id>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
12 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
credentials object GCP credentials configuration
credentials_json password Service account key JSON content
credentials_file string Path to a service account key JSON file
project_id string Google Cloud project ID
instances multiselect Instance IDs to discover. Leave empty to discover every instance in the project
emulator_host string host:port of a Bigtable emulator. Connects without credentials and requires instances to be listed
include_columns bool Read a sample of rows to find the columns each table holds
- default
- true
sample_rows int Rows to read per table when finding columns
- default
- 100
include_statistics bool Count the rows in each table. This reads the whole table
- default
- false
max_count_rows int Give up counting a table after this many rows
- default
- 100000
include_backups bool Count the backups held by each instance
- default
- false
Assets emitted
Metadata this plugin attaches to each discovered asset.
Instance
BigtableInstanceFieldsThe metadata fields the Bigtable plugin emits for instance assets.
project_id stringGoogle Cloud project holding the instance
instance_id stringInstance ID
display_name stringInstance display name
state stringInstance state (READY, CREATING)
instance_type stringInstance type (PRODUCTION, DEVELOPMENT)
labels objectLabels set on the instance
clusters BigtableCluster[]Clusters serving the instance
table_count intNumber of tables in the instance
backup_count intNumber of backups held by the instance's clusters
emulator boolWhether the instance was read from an emulator
Table
BigtableTableFieldsThe metadata fields the Bigtable plugin emits for table assets.
project_id stringGoogle Cloud project holding the table
instance_id stringInstance the table belongs to
table_name stringTable name within the instance
column_families string[]Column families defined on the table
gc_policies objectGarbage collection rule per column family
deletion_protection boolWhether the table is protected against deletion
change_stream_retention stringHow long change data is retained
sampled_rows intNumber of rows read to find the table's columns
url stringGoogle Cloud Console link to the table
Column
BigtableColumnFieldsThe per-column fields embedded in a table asset's schema. Bigtable declares column families but not the qualifiers under them, so everything below comes from a sample of rows.
column_name stringColumn name in Bigtable's family:qualifier notation
data_type stringStorage type, always bytes
is_nullable boolWhether the column may be missing from a row
is_primary_key boolWhether the column is the row key
column_family stringColumn family the column belongs to
qualifier stringQualifier within the column family
occurrence intNumber of sampled rows that held the column
inferred_type stringWhat the sampled values looked like (text, int64, binary)