Google Cloud Storage

marmotdata/gcs

Discover buckets from Google Cloud Storage

Storage

You're viewing v0.1.2. The latest release is v0.1.3. View latest

The Google Cloud Storage plugin discovers buckets from GCP projects. It captures bucket metadata including location, storage class, encryption settings, and lifecycle rules.

Connection Examples

Required Permissions

The service account needs the following IAM roles:

Storage Bucket Viewer (roles/storage.bucketViewer) - For discovering and listing buckets

Or use a custom role with these permissions:

storage.buckets.list
storage.buckets.get
storage.objects.list (if using object count)

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Google Cloud Storage from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-gcs-pipeline
runs:
  - gcs:
      project_id: "<project_id>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

10 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

project_id string

Google Cloud project ID

credentials_file string

Path to service account JSON file

credentials_json password

Service account JSON content

endpoint string

Custom endpoint URL (for fake-gcs-server or other emulators)

disable_auth bool

Disable authentication (for local emulators)

include_metadata bool

Include bucket metadata like labels

default: true

include_object_count bool

Count objects in each bucket (can be slow for large buckets)

default: false

Assets emitted

Metadata this plugin attaches to each discovered asset.

GCS Bucket

GCSBucketFields

13 fields

GCSBucketFields defines metadata fields for GCS buckets

bucket_name string

Name of the bucket

location string

Geographic location of the bucket

location_type string

Location type (region, dual-region, multi-region)

storage_class string

Default storage class (STANDARD, NEARLINE, COLDLINE, ARCHIVE)

created string

Bucket creation timestamp

versioning string

Whether object versioning is enabled

requester_pays bool

Whether requester pays for access

encryption string

Encryption type (google-managed or customer-managed)

kms_key string

Customer-managed encryption key name

logging_enabled bool

Whether access logging is enabled

lifecycle_rules_count int

Number of lifecycle rules configured

retention_period_seconds int

Retention period in seconds

object_count int

Number of objects in the bucket