PostgreSQL

marmotdata/postgresql

Discover databases, schemas, and tables from PostgreSQL instances

Databases Assets Lineage Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The PostgreSQL plugin discovers databases, schemas, and tables from PostgreSQL instances. It captures column information, table metrics, and foreign key relationships for lineage.

Required Permissions

The user needs read access to the information schema:

GRANT CONNECT ON DATABASE your_db TO marmot_reader;
GRANT USAGE ON SCHEMA public TO marmot_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO marmot_reader;

Example Configuration


host: "prod-postgres.company.com"
port: 5432
user: "marmot_reader"
password: "secure_password_123"
ssl_mode: "require"
tags:
  - "postgres"
  - "production"

Configuration

The following configuration options are available:

Property	Type	Required	Description
discover_foreign_keys	bool	false	Whether to discover foreign key relationships
enable_metrics	bool	false	Whether to include table metrics
exclude_system_schemas	bool	false	Whether to exclude system schemas (pg_*)
external_links	[]ExternalLink	false	External links to show on all assets
filter	Filter	false	Filter discovered assets by name (regex)
host	string	false	PostgreSQL server hostname or IP address
include_columns	bool	false	Whether to include column information in table metadata
include_databases	bool	false	Whether to discover databases
password	string	false	Password for authentication
port	int	false	PostgreSQL server port
ssl_mode	string	false	SSL mode (disable, require, verify-ca, verify-full)
tags	TagsConfig	false	Tags to apply to discovered assets
user	string	false	Username for authentication

Available Metadata

The following metadata fields are available:

Field	Type	Description
allow_connections	bool	Whether connections to this database are allowed
collate	string	Database collation
column_default	string	Default value expression
column_name	string	Column name
comment	string	Column comment/description
comment	string	Object comment/description
connection_limit	int	Maximum allowed connections
constraint_name	string	Foreign key constraint name
created	string	Creation timestamp
ctype	string	Database character classification
data_type	string	Data type
database	string	Database name
encoding	string	Database encoding
host	string	PostgreSQL server hostname
is_nullable	bool	Whether null values are allowed
is_primary_key	bool	Whether column is part of primary key
is_template	bool	Whether database is a template
object_type	string	Object type (table, view, materialized_view)
owner	string	Object owner
port	int	PostgreSQL server port
row_count	int64	Approximate row count
schema	string	Schema name
size	int64	Object size in bytes
source_column	string	Column in the referencing table
source_schema	string	Schema of the referencing table
source_table	string	Name of the referencing table
table_name	string	Object name
target_column	string	Column in the referenced table
target_schema	string	Schema of the referenced table
target_table	string	Name of the referenced table

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick PostgreSQL from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-postgresql-pipeline
runs:
  - postgresql:
      host: "<host>"
      user: "<user>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

13 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

PostgreSQL server hostname or IP address

port int

PostgreSQL server port

default: 5432

user string

Username for authentication

password password

Password for authentication

ssl_mode select

SSL mode (disable, require, verify-ca, verify-full)

default: disable

disable require verify-ca verify-full

include_databases bool

Whether to discover databases

default: true

include_columns bool

Whether to include column information in table metadata

default: true

enable_metrics bool

Whether to include table metrics

default: true

discover_foreign_keys bool

Whether to discover foreign key relationships

default: true

exclude_system_schemas bool

Whether to exclude system schemas (pg_*)

default: true

Assets emitted

Metadata this plugin attaches to each discovered asset.

Postgres

PostgresFields

17 fields

PostgresFields represents PostgreSQL-specific metadata fields

host string

PostgreSQL server hostname

port int

PostgreSQL server port

database string

Database name

schema string

Schema name

table_name string

Object name

object_type string

Object type (table, view, materialized_view)

owner string

Object owner

size int

Object size in bytes

row_count int

Approximate row count

created string

Creation timestamp

comment string

Object comment/description

encoding string

Database encoding

collate string

Database collation

ctype string

Database character classification

is_template bool

Whether database is a template

allow_connections bool

Whether connections to this database are allowed

connection_limit int

Maximum allowed connections

Postgres Column

PostgresColumnFields

6 fields

PostgresColumnFields represents PostgreSQL column-specific metadata fields

column_name string

Column name

data_type string

Data type

is_nullable bool

Whether null values are allowed

column_default string

Default value expression

is_primary_key bool

Whether column is part of primary key

comment string

Column comment/description

Postgres Foreign Key

PostgresForeignKeyFields

7 fields

PostgresForeignKeyFields represents PostgreSQL foreign key relationship fields

constraint_name string

Foreign key constraint name

source_schema string

Schema of the referencing table

source_table string

Name of the referencing table

source_column string

Column in the referencing table

target_schema string

Schema of the referenced table

target_table string

Name of the referenced table

target_column string

Column in the referenced table