Google Pub/Sub

marmotdata/pubsub

Discover topics and subscriptions from Google Cloud Pub/Sub

Streaming Assets Lineage Experimental

The Google Pub/Sub plugin discovers topics and subscriptions from a Google Cloud project, along with the links between them and the systems a subscription exports to.

Topics carry their labels, message storage policy, retention and schema. Subscriptions carry their delivery type (pull, push, BigQuery or Cloud Storage), acknowledgement settings, filter and dead letter policy.

Authentication

With neither credentials_file nor credentials_json set, Application Default Credentials are used. The service account needs roles/pubsub.viewer; without it, schemas are skipped and discovery continues.

Emulator

Set emulator_host to a host:port address to run against the Pub/Sub emulator. The connection then uses plaintext gRPC with no credentials, and assets get no Google Cloud console links.

Sample Messages

With include_sample_messages: true, asset previews read up to 20 messages from a pull subscription for 5 seconds. Every message is nacked, so Pub/Sub redelivers it to the real consumer straight after.

Keyless authentication

On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of a service account key. Set workload_identity_provider to a Workload Identity Federation provider that trusts your Marmot instance as an OIDC issuer, and grant the pipeline's subject, pipeline:<name> as reported by the pipeline API, the role above directly (principal://iam.googleapis.com/<pool>/subject/pipeline:<name>), or grant it roles/iam.workloadIdentityUser on a service account named in service_account. No key exists anywhere; Marmot mints a short-lived token for each run.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Google Pub/Sub from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-pubsub-pipeline
runs:
  - pubsub:
      project_id: "<project_id>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

14 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

audience string

Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another

project_id string

Google Cloud project ID

emulator_host string

Address of a Pub/Sub emulator, for example localhost:8085

credentials_file string

Path to a service account JSON file

credentials_json password

Service account JSON content

workload_identity_provider string

Workload Identity Federation provider to exchange the Marmot identity token at, projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Setting it federates: no key is needed; bind the pipeline's subject on the Google Cloud side

service_account string

Service account to impersonate after the exchange; empty acts as the federated principal directly

include_subscriptions bool

Whether to discover subscriptions

default: true

include_schemas bool

Whether to attach topic schemas and their fields

default: true

include_dead_letter_topics bool

Whether to record dead letter topics on subscriptions

default: true

include_sample_messages bool

Whether to allow reading sample messages for asset previews

default: false

Assets emitted

Metadata this plugin attaches to each discovered asset.

Topic

TopicFields

15 fields

The metadata fields the plugin emits for topic assets.

project_id string

Google Cloud project the topic belongs to

topic_name string

Full resource name, projects/{project}/topics/{topic}

subscription_count int

Number of subscriptions attached to the topic

subscriptions string[]

Ids of the subscriptions attached to the topic

labels object

Labels set on the topic

kms_key_name string

Cloud KMS key protecting published messages

retention string

How long published messages stay available to subscribers

allowed_persistence_regions string[]

Regions the message storage policy allows

state string

Topic state (ACTIVE, INGESTION_RESOURCE_ERROR)

schema string

Id of the schema published messages are validated against

schema_type string

Schema type (AVRO, PROTOCOL_BUFFER)

schema_encoding string

Message encoding the schema is applied to (JSON, BINARY)

schema_revision string

Revision id of the schema

ingestion_source string

External system Pub/Sub imports messages from (aws_kinesis, cloud_storage, azure_event_hubs, amazon_msk, confluent_cloud)

url string

Link to the topic in the Google Cloud console

Subscription

SubscriptionFields

25 fields

The metadata fields the plugin emits for subscription assets.

project_id string

Google Cloud project the subscription belongs to

subscription_name string

Full resource name, projects/{project}/subscriptions/{subscription}

topic string

Id of the topic the subscription reads

delivery_type string

How messages are delivered (pull, push, bigquery, cloud_storage)

push_endpoint string

URL messages are pushed to

bigquery_table string

BigQuery table messages are written to

bigquery_use_topic_schema bool

Whether the topic's schema is used to write the BigQuery rows

bigquery_state string

Whether the BigQuery export is working (ACTIVE when it is)

cloud_storage_bucket string

Cloud Storage bucket messages are written to

cloud_storage_filename_prefix string

Prefix of the objects written to the bucket

cloud_storage_filename_suffix string

Suffix of the objects written to the bucket

cloud_storage_state string

Whether the Cloud Storage export is working (ACTIVE when it is)

ack_deadline_seconds int

Seconds a subscriber has to acknowledge a message

message_retention string

How long unacknowledged messages are kept

retain_acked_messages bool

Whether acknowledged messages are kept for replay

expiration_ttl string

How long the subscription can be inactive before it is deleted

filter string

Expression selecting which messages are delivered

enable_message_ordering bool

Whether messages with the same ordering key are delivered in order

exactly_once_delivery bool

Whether exactly once delivery is enabled

dead_letter_topic string

Id of the topic undeliverable messages are forwarded to

max_delivery_attempts int

Deliveries attempted before a message goes to the dead letter topic

labels object

Labels set on the subscription

state string

Subscription state (ACTIVE, RESOURCE_ERROR)

detached bool

Whether the subscription is detached from its topic and no longer receives messages

url string

Link to the subscription in the Google Cloud console

Topic Column

TopicColumnFields

4 fields

The per-field entries the plugin derives from a topic's Avro schema. Only the top level of the record is expanded.

column_name string

Avro field name

data_type string

Avro field type, unions joined with a pipe

is_nullable bool

Whether the field's union includes null

description string

The Avro field's doc string