AWS Kinesis
Official · maintained by Marmotmarmotdata/kinesis Discover Kinesis Data Streams from AWS accounts
The Kinesis plugin discovers Amazon Kinesis Data Streams across your AWS accounts. Each stream becomes a Stream asset carrying its capacity mode, retention, shard and consumer counts, encryption settings and tags, with a data preview that reads the oldest retained records from the first open shard.
Required Permissions
GetShardIterator and GetRecords are only needed for the data preview. ListShards, ListStreamConsumers and ListTagsForStream can be left out when include_shards, include_consumers or tags_to_metadata are switched off; a denied call logs a warning and the stream is still catalogued.
AWS Configuration
See AWS Configuration for the supported AWS configuration options.
Keyless authentication
On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of access keys. Register your Marmot instance as an IAM OIDC identity provider (client id sts.amazonaws.com), create a role whose trust policy allows sts:AssumeRoleWithWebIdentity for that provider with <issuer host>:sub equal to the pipeline's subject, pipeline:<name> as reported by the pipeline API, and set credentials.role_arn and credentials.region. No key exists anywhere; Marmot mints a short-lived token for each run.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick AWS Kinesis from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-kinesis-pipeline
runs:
- kinesis:
# No required fields — see the Configuration tab.$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
9 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
credentials object AWS credentials configuration
use_default bool Use AWS credentials from environment or default profile (recommended)
- default
- true
id string AWS access key ID
secret password AWS secret access key
token password AWS session token
profile string AWS profile to use from shared credentials file
role_arn string IAM role to assume with the Marmot identity token (AssumeRoleWithWebIdentity); its trust policy names the Marmot issuer as an OIDC provider. Setting it federates: no keys are needed, and region is required. role, if also set, is assumed on top of it
role string AWS IAM role ARN to assume with AssumeRole from the base credentials (static keys, a profile, the default chain, or role_arn)
role_external_id string External ID for cross-account role assumption
region string AWS region for services
endpoint string Custom endpoint URL for AWS services
tags_to_metadata bool Convert AWS tags to Marmot metadata
include_tags multiselect List of AWS tags to include as metadata. By default, all tags are included.
audience string Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another
include_consumers bool Whether to list the enhanced fan-out consumers registered on each stream
- default
- true
include_shards bool Whether to list shards to count total and open shards per stream
- default
- true
Assets emitted
Metadata this plugin attaches to each discovered asset.
Stream
KinesisStreamFieldsThe metadata fields the Kinesis plugin emits for Stream assets.
arn stringStream ARN
status stringStream status (CREATING, DELETING, ACTIVE, UPDATING)
stream_mode stringCapacity mode (PROVISIONED or ON_DEMAND)
retention_hours intRecord retention period in hours
shard_count intTotal number of shards, closed ones included
open_shard_count intNumber of shards still accepting writes
encryption_type stringServer-side encryption type (NONE or KMS)
kms_key_id stringKMS key used for server-side encryption
enhanced_monitoring string[]Shard-level CloudWatch metrics that are enabled
consumer_count intNumber of registered enhanced fan-out consumers
consumers string[]Names of the registered enhanced fan-out consumers
created_at stringStream creation time (RFC 3339)
region stringAWS region the stream lives in
url stringLink to the stream in the AWS console
Sample
KinesisSampleFieldsThe columns of a stream's data preview.
sequence_number stringRecord sequence number within its shard
partition_key stringPartition key the producer wrote the record with
arrival_time stringApproximate time the record reached the stream (RFC 3339)
data stringRecord payload as text, or base64 when it is not valid UTF-8