AWS Athena

marmotdata/athena

Discover databases, tables, workgroups and saved queries from Amazon Athena

Warehouses Assets Lineage Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The Athena plugin discovers data catalogs, databases, tables, views, workgroups and saved queries from Amazon Athena.

Athena keeps no catalog of its own: the tables it queries live in the Glue Data Catalog. Databases, tables and views are therefore filed under the Glue provider with the same names the Glue plugin uses, so an Athena run and a Glue run land on one asset. Workgroups, saved queries and data catalogs are Athena's own objects and are filed under Athena.

Databases and tables are read through the Athena metadata API, which also serves federated catalogs. When that API does not answer for a Glue-backed catalog, the plugin reads the Glue Data Catalog directly instead. Set metadata_api to athena or glue to pin the choice.

Query history is not read, so this plugin produces no usage statistics or query-derived table-to-table lineage.

Required Permissions

sts:GetCallerIdentity is only needed with tags_to_metadata, because Athena does not return workgroup or catalog ARNs and the plugin builds them from the account id.

AWS Configuration

See AWS Configuration for the supported AWS configuration options.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick AWS Athena from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-athena-pipeline
runs:
  - athena:
      # No required fields — see the Configuration tab.

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

17 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

credentials object

AWS credentials configuration

use_default bool

Use AWS credentials from environment or default profile (recommended)

default: true

id string

AWS access key ID

secret password

AWS secret access key

token password

AWS session token

profile string

AWS profile to use from shared credentials file

role string

AWS IAM role ARN to assume

role_external_id string

External ID for cross-account role assumption

region string

AWS region for services

endpoint string

Custom endpoint URL for AWS services

tags_to_metadata bool

Convert AWS tags to Marmot metadata

include_tags multiselect

List of AWS tags to include as metadata. By default, all tags are included.

catalogs multiselect

Data catalogs to discover. All catalogs when empty

exclude_catalogs multiselect

Data catalogs to skip

databases multiselect

Databases to discover. All databases when empty

exclude_databases multiselect

Databases to skip

workgroups multiselect

Workgroups to discover. All workgroups when empty

include_workgroups bool

Whether to catalog workgroups

default: true

include_saved_queries bool

Whether to catalog saved queries

default: true

include_columns bool

Whether to include table columns

default: true

include_partitions bool

Whether to include partition keys

default: true

discover_lineage bool

Whether to discover lineage between catalogs, databases, tables, buckets and saved queries

default: true

metadata_api select

Which API reads databases and tables: auto, athena or glue

default: auto

auto athena glue

Assets emitted

Metadata this plugin attaches to each discovered asset.

Table

AthenaTableFields

15 fields

The metadata the plugin records for a table or view. The fields live under the `athena` key of the asset metadata, because the asset itself is owned by the Glue provider and shared with the Glue plugin.

catalog string

Data catalog holding the table

catalog_type string

Data catalog type (GLUE, HIVE, LAMBDA, FEDERATED)

database string

Database holding the table

table_type string

Table type (EXTERNAL_TABLE, VIRTUAL_VIEW, ...)

location string

Storage location of the table data

input_format string

Hadoop input format class

output_format string

Hadoop output format class

serde string

Serialization library

compression string

Compression codec of the table data

classification string

Data format of the table (parquet, csv, json, ...)

partition_keys string

Partition key columns

partition_projection bool

Whether partition projection is enabled

created string

Date and time the table was created

last_access string

Date and time the table was last accessed

comment string

Table comment

Database

AthenaDatabaseFields

5 fields

The metadata recorded for a database, under the `athena` key of the asset metadata.

catalog string

Data catalog holding the database

catalog_type string

Data catalog type (GLUE, HIVE, LAMBDA, FEDERATED)

description string

Description of the database

location_uri string

Storage location of the database

parameters object

Database parameters

Column

AthenaColumnFields

5 fields

The per-column fields embedded in a table asset's schema.

column_name string

Column name

data_type string

Hive type of the column, recorded verbatim

is_nullable bool

Whether null values are allowed, always true in Athena

is_partition_key bool

Whether the column is a partition key

description string

Column comment

Work Group

AthenaWorkGroupFields

14 fields

The metadata recorded for a workgroup.

state string

Workgroup state (ENABLED, DISABLED)

description string

Description of the workgroup

engine_version string

Effective Athena engine version

selected_engine_version string

Engine version the workgroup requested

output_location string

S3 location query results are written to

encryption string

Encryption option for query results (SSE_S3, SSE_KMS, CSE_KMS)

encryption_kms_key string

KMS key used to encrypt query results

bytes_scanned_cutoff int

Per-query limit on bytes scanned

enforce_configuration bool

Whether the workgroup settings override client settings

publish_metrics bool

Whether query metrics are published to CloudWatch

requester_pays bool

Whether queries may read requester pays buckets

created string

Date and time the workgroup was created

region string

AWS region of the workgroup

url string

Link to the workgroup in the Athena console

Saved Query

AthenaSavedQueryFields

4 fields

The metadata recorded for a saved query.

named_query_id string

Athena identifier of the saved query

database string

Database the query runs against by default

workgroup string

Workgroup holding the saved query

description string

Description of the saved query

Catalog

AthenaCatalogFields

3 fields

The metadata recorded for a data catalog.

type string

Catalog type (GLUE, HIVE, LAMBDA, FEDERATED)

description string

Description of the catalog

parameters object

Catalog parameters, such as the Lambda function backing a federated catalog