Google Pub/Sub
Official · maintained by Marmotmarmotdata/pubsub Discover topics and subscriptions from Google Cloud Pub/Sub
The Google Pub/Sub plugin discovers topics and subscriptions from a Google Cloud project, along with the links between them and the systems a subscription exports to.
Topics carry their labels, message storage policy, retention and schema. Subscriptions carry their delivery type (pull, push, BigQuery or Cloud Storage), acknowledgement settings, filter and dead letter policy.
Authentication
With neither credentials_file nor credentials_json set, Application Default Credentials are used. The service account needs roles/pubsub.viewer; without it, schemas are skipped and discovery continues.
Emulator
Set emulator_host to a host:port address to run against the Pub/Sub emulator. The connection then uses plaintext gRPC with no credentials, and assets get no Google Cloud console links.
Sample Messages
With include_sample_messages: true, asset previews read up to 20 messages from a pull subscription for 5 seconds. Every message is nacked, so Pub/Sub redelivers it to the real consumer straight after.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Google Pub/Sub from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-pubsub-pipeline
runs:
- pubsub:
project_id: "<project_id>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
11 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
project_id string Google Cloud project ID
emulator_host string Address of a Pub/Sub emulator, for example localhost:8085
credentials_file string Path to a service account JSON file
credentials_json password Service account JSON content
include_subscriptions bool Whether to discover subscriptions
- default
- true
include_schemas bool Whether to attach topic schemas and their fields
- default
- true
include_dead_letter_topics bool Whether to record dead letter topics on subscriptions
- default
- true
include_sample_messages bool Whether to allow reading sample messages for asset previews
- default
- false
Assets emitted
Metadata this plugin attaches to each discovered asset.
Topic
TopicFieldsThe metadata fields the plugin emits for topic assets.
project_id stringGoogle Cloud project the topic belongs to
topic_name stringFull resource name, projects/{project}/topics/{topic}
subscription_count intNumber of subscriptions attached to the topic
subscriptions string[]Ids of the subscriptions attached to the topic
labels objectLabels set on the topic
kms_key_name stringCloud KMS key protecting published messages
retention stringHow long published messages stay available to subscribers
allowed_persistence_regions string[]Regions the message storage policy allows
state stringTopic state (ACTIVE, INGESTION_RESOURCE_ERROR)
schema stringId of the schema published messages are validated against
schema_type stringSchema type (AVRO, PROTOCOL_BUFFER)
schema_encoding stringMessage encoding the schema is applied to (JSON, BINARY)
schema_revision stringRevision id of the schema
ingestion_source stringExternal system Pub/Sub imports messages from (aws_kinesis, cloud_storage, azure_event_hubs, amazon_msk, confluent_cloud)
url stringLink to the topic in the Google Cloud console
Subscription
SubscriptionFieldsThe metadata fields the plugin emits for subscription assets.
project_id stringGoogle Cloud project the subscription belongs to
subscription_name stringFull resource name, projects/{project}/subscriptions/{subscription}
topic stringId of the topic the subscription reads
delivery_type stringHow messages are delivered (pull, push, bigquery, cloud_storage)
push_endpoint stringURL messages are pushed to
bigquery_table stringBigQuery table messages are written to
bigquery_use_topic_schema boolWhether the topic's schema is used to write the BigQuery rows
bigquery_state stringWhether the BigQuery export is working (ACTIVE when it is)
cloud_storage_bucket stringCloud Storage bucket messages are written to
cloud_storage_filename_prefix stringPrefix of the objects written to the bucket
cloud_storage_filename_suffix stringSuffix of the objects written to the bucket
cloud_storage_state stringWhether the Cloud Storage export is working (ACTIVE when it is)
ack_deadline_seconds intSeconds a subscriber has to acknowledge a message
message_retention stringHow long unacknowledged messages are kept
retain_acked_messages boolWhether acknowledged messages are kept for replay
expiration_ttl stringHow long the subscription can be inactive before it is deleted
filter stringExpression selecting which messages are delivered
enable_message_ordering boolWhether messages with the same ordering key are delivered in order
exactly_once_delivery boolWhether exactly once delivery is enabled
dead_letter_topic stringId of the topic undeliverable messages are forwarded to
max_delivery_attempts intDeliveries attempted before a message goes to the dead letter topic
labels objectLabels set on the subscription
state stringSubscription state (ACTIVE, RESOURCE_ERROR)
detached boolWhether the subscription is detached from its topic and no longer receives messages
url stringLink to the subscription in the Google Cloud console
Topic Column
TopicColumnFieldsThe per-field entries the plugin derives from a topic's Avro schema. Only the top level of the record is expanded.
column_name stringAvro field name
data_type stringAvro field type, unions joined with a pipe
is_nullable boolWhether the field's union includes null
description stringThe Avro field's doc string