AWS Firehose

marmotdata/firehose

Discover Amazon Data Firehose delivery streams from AWS accounts

Streaming Assets Lineage Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The Firehose plugin discovers Amazon Data Firehose delivery streams across your AWS accounts. It captures each stream's configuration and destination settings, and links the stream to the Kinesis stream or Kafka topic it reads from and to the tables and buckets it writes to.

Firehose creates no assets for those systems. Every lineage edge points at an asset another Marmot plugin catalogs, so run the Kinesis, Kafka, S3, Glue, Redshift, OpenSearch, Snowflake or Iceberg plugin alongside this one to see the edges.

Required Permissions

AWS Configuration

See AWS Configuration for the supported AWS configuration options.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick AWS Firehose from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-firehose-pipeline
runs:
  - firehose:
      # No required fields — see the Configuration tab.

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

8 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

credentials object

AWS credentials configuration

use_default bool

Use AWS credentials from environment or default profile (recommended)

default: true

id string

AWS access key ID

secret password

AWS secret access key

token password

AWS session token

profile string

AWS profile to use from shared credentials file

role string

AWS IAM role ARN to assume

role_external_id string

External ID for cross-account role assumption

region string

AWS region for services

endpoint string

Custom endpoint URL for AWS services

tags_to_metadata bool

Convert AWS tags to Marmot metadata

include_tags multiselect

List of AWS tags to include as metadata. By default, all tags are included.

discover_lineage bool

Link each stream to the systems it reads from and writes to

default: true

include_destination_config bool

Record the destination settings in metadata

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

AWS Firehose

FirehoseFields

17 fields

The metadata fields the Firehose plugin emits for a delivery stream asset.

arn string

ARN of the delivery stream

status string

Delivery stream status (ACTIVE, CREATING, DELETING)

stream_type string

Delivery stream type (DirectPut, KinesisStreamAsSource, MSKAsSource, DatabaseAsSource)

version_id string

Version of the delivery stream configuration

region string

AWS region the delivery stream lives in

created_at string

When the delivery stream was created

last_updated_at string

When the delivery stream was last updated

encryption_status string

Server-side encryption status

encryption_key_type string

Server-side encryption key type (AWS_OWNED_CMK, CUSTOMER_MANAGED_CMK)

source_type string

Where records come from (direct_put, kinesis, msk, database)

source_kinesis_stream string

Name of the Kinesis stream feeding the delivery stream

source_msk_cluster string

Name of the MSK cluster feeding the delivery stream

source_msk_topic string

Name of the MSK topic feeding the delivery stream

destination_type string

Where records are written (s3, extended_s3, redshift, elasticsearch, opensearch, opensearch_serverless, splunk, http_endpoint, snowflake, iceberg)

destination_count int

Number of destinations configured on the delivery stream

destination object

Destination settings, with any value that could carry a credential redacted

tags object

AWS resource tags

Destination

FirehoseDestinationFields

36 fields

The fields of the `destination` sub-map. Which of them are present depends on the destination type.

bucket string

S3 bucket name

prefix string

S3 key prefix records are written under

error_output_prefix string

S3 key prefix failed records are written under

compression_format string

S3 compression format

file_extension string

File extension of the delivered S3 objects

s3_backup_mode string

Whether records are also backed up to S3

buffering_size_mb int

Buffer size in MB before delivery

buffering_interval_seconds int

Buffer interval in seconds before delivery

format_conversion_enabled bool

Whether records are converted to a columnar format

input_format string

Deserializer for incoming records (HiveJsonSerDe, OpenXJsonSerDe)

output_format string

Serializer for delivered records (ParquetSerDe, OrcSerDe)

glue_catalog_id string

Glue catalog the conversion schema is read from

glue_database string

Glue database the conversion schema is read from

glue_table string

Glue table the conversion schema is read from

glue_region string

Region of the Glue catalog

cluster_endpoint string

Redshift cluster host, or Elasticsearch and OpenSearch cluster endpoint

database string

Redshift or Snowflake database

schema string

Snowflake schema

table string

Redshift or Snowflake table

username string

Redshift user the copy command runs as

copy_options string

Redshift copy command options

copy_columns string

Redshift columns the copy command targets

domain_arn string

Elasticsearch or OpenSearch domain ARN

index_name string

Elasticsearch or OpenSearch index name

type_name string

Elasticsearch or OpenSearch type name

index_rotation_period string

How often the index name is rotated

collection_endpoint string

OpenSearch Serverless collection endpoint

hec_endpoint string

Splunk HTTP event collector endpoint

hec_endpoint_type string

Splunk HTTP event collector endpoint type

url string

HTTP endpoint URL

name string

HTTP endpoint name

account_url string

Snowflake account URL

user string

Snowflake user

catalog_arn string

Iceberg catalog ARN

warehouse_location string

Iceberg warehouse location

tables string[]

Iceberg destination tables