TimescaleDB

marmotdata/timescale

Discover databases, tables, hypertables and continuous aggregates from TimescaleDB instances

Databases Assets Lineage Experimental

The TimescaleDB plugin discovers databases, tables, views, hypertables and continuous aggregates from TimescaleDB instances. It captures column information, foreign key relationships, view lineage, and the TimescaleDB detail a plain PostgreSQL read cannot see: partitioning dimensions, chunk counts, compression settings and background policies.

Relationship to the PostgreSQL plugin

TimescaleDB is a PostgreSQL extension, so a TimescaleDB server is a PostgreSQL server. Assets are filed under the PostgreSQL provider by their bare object name, which is byte for byte the identity the PostgreSQL plugin produces. Running both plugins against the same instance updates one set of assets rather than creating two. This plugin is the PostgreSQL plugin plus hypertable knowledge.

Objects in the _timescaledb_internal, _timescaledb_catalog, _timescaledb_config, _timescaledb_cache, timescaledb_information and timescaledb_experimental schemas are skipped. They hold chunk tables, materialization hypertables and catalog views, and one busy hypertable puts thousands of chunks there.

Required Permissions

The user needs read access to the catalog and to the tables being described:

GRANT CONNECT ON DATABASE your_db TO marmot_reader;
GRANT USAGE ON SCHEMA public TO marmot_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO marmot_reader;

Tests

Unit tests run anywhere. The end to end tests need a real server:

docker run -d --name timescale -p 15436:5432 \
  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=shop \
  timescale/timescaledb:latest-pg16
docker exec -i timescale psql -U postgres -d shop < timescale/testdata/seed.sql

MARMOT_TEST_TIMESCALE_HOST=localhost MARMOT_TEST_TIMESCALE_PORT=15436 go test ./...

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick TimescaleDB from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-timescale-pipeline
runs:
  - timescale:
      host: "<host>"
      user: "<user>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

18 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

TimescaleDB server hostname or IP address

port int

TimescaleDB server port

default: 5432

user string

Username for authentication

password password

Password for authentication

database string

Database to discover. When empty, every database on the server is discovered

ssl_mode select

SSL mode (disable, require, verify-ca, verify-full)

default: disable

disable require verify-ca verify-full

include_columns bool

Whether to include column information in table metadata

default: true

include_statistics bool

Whether to collect row, column and size statistics

default: true

discover_foreign_keys bool

Whether to discover foreign key relationships

default: true

exclude_system_schemas bool

Whether to exclude PostgreSQL and TimescaleDB internal schemas

default: true

exclude_databases multiselect

Databases to skip

include_hypertables bool

Whether to record hypertable dimensions, chunk counts and policies

default: true

include_continuous_aggregates bool

Whether to record continuous aggregates and their source hypertable lineage

default: true

include_compression bool

Whether to record compression segment and order columns

default: true

include_chunks bool

Whether to read the oldest and newest chunk range. Off by default because a busy hypertable has thousands of chunks

default: false

Assets emitted

Metadata this plugin attaches to each discovered asset.

TimescaleDB

TimescaleFields

13 fields

The metadata every discovered object carries.

host string

TimescaleDB server hostname

port int

TimescaleDB server port

database string

Database name

schema string

Schema name

table_name string

Table or view name

object_type string

Object type (table, view, materialized_view)

owner string

Object owner

comment string

Object comment

encoding string

Database encoding

collate string

Database collation

ctype string

Database character classification

connection_limit int

Maximum allowed connections to the database

size int

Database size in bytes

Hypertable

TimescaleHypertableFields

18 fields

The metadata a table carries when TimescaleDB partitions it into chunks.

hypertable bool

Whether the table is a hypertable

timescaledb_version string

Version of the timescaledb extension

time_column string

Column the hypertable is partitioned by time on

time_column_type string

Data type of the time column

time_interval string

Time span covered by one chunk

time_interval_seconds int

Time span covered by one chunk, in seconds

integer_interval int

Chunk interval when the hypertable is partitioned on an integer column

integer_now_func string

Function that returns the current value of an integer time column

num_dimensions int

Number of partitioning dimensions

num_chunks int

Number of chunks the hypertable is split into

space_partitions string[]

Partitioning columns beyond time, with their partition counts

chunk_range string[]

Oldest and newest chunk boundary, when include_chunks is on

compression_enabled bool

Whether compression is enabled

compression_segment_by string[]

Columns compressed batches are grouped by

compression_order_by string[]

Columns rows are ordered by inside a compressed batch

policies string[]

Background policies attached to the object (compression, retention, refresh)

is_distributed bool

Whether the hypertable is distributed, on TimescaleDB versions that support it

tablespaces string[]

Tablespaces the hypertable's chunks are placed in

Aggregate

TimescaleAggregateFields

6 fields

The metadata a view carries when it is a continuous aggregate.

continuous_aggregate bool

Whether the view is a continuous aggregate

materialized_only bool

Whether the view reads only materialized data, without the most recent rows

finalized bool

Whether the aggregate uses the finalized form, on TimescaleDB versions that report it

source_hypertable string

Hypertable the aggregate reads from

materialization_hypertable string

Internal hypertable the results are stored in

refresh_policy string[]

Policy that refreshes the aggregate

Column

TimescaleColumnFields

8 fields

The per-column metadata attached to an asset's schema.

column_name string

Column name

data_type string

Data type

is_nullable bool

Whether null values are allowed

is_primary_key bool

Whether the column is part of the primary key

default_expression string

Default value expression

description string

Column comment

identity string

Identity kind: a for always, d for by default

is_generated bool

Whether the column is generated from other columns