Confluent Cloud
Core pluginmarmotdata/confluent
Discover Kafka topics from Confluent Cloud clusters
The Confluent Cloud plugin discovers Kafka topics from Confluent Cloud clusters. It uses the same discovery engine as the Kafka plugin with defaults tuned for Confluent Cloud.
Connection
Confluent Cloud requires SASL/SSL authentication with an API key pair. You can create API keys in the Confluent Cloud Console.
bootstrap_servers: "pkc-xxxxx.us-west-2.aws.confluent.cloud:9092"
client_id: "marmot-discovery"
authentication:
type: "sasl_ssl"
username: "your-api-key"
password: "your-api-secret"
mechanism: "PLAIN"
tls:
enabled: trueSchema Registry
If your Confluent Cloud environment has Schema Registry enabled, add the following to pull schema metadata:
schema_registry:
url: "https://psrc-xxxxx.us-west-2.aws.confluent.cloud"
enabled: true
config:
basic.auth.user.info: "sr-key:sr-secret"In the UI
Point-and-click, no config file needed.
- 1 Open
Runs→Create Pipeline - 2 Pick Confluent Cloud from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-confluent-pipeline
runs:
- confluent:
bootstrap_servers: "pkc-xxxxx.us-west-2.aws.confluent.cloud:9092"marmot ingest -c ingest.yaml Not using plugins? Other ways to populate Marmot
Configuration
10 top-level fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
bootstrap_servers string Comma-separated list of bootstrap servers
pkc-xxxxx.us-west-2.aws.confluent.cloud:9092 client_id string Client ID for the consumer
authentication object Authentication configuration
username string SASL username
password password SASL password
client_timeout_seconds int Request timeout in seconds
schema_registry object Schema Registry configuration
url string Schema Registry URL
config string Additional Schema Registry configuration
enabled bool Whether to use Schema Registry
skip_verify bool Skip TLS certificate verification
include_partition_info bool Whether to include partition information in metadata
true include_topic_config bool Whether to include topic configuration in metadata
true