SQLite
Official · maintained by Marmotmarmotdata/sqlite v0.1.0Discover tables, views and foreign key relationships from SQLite database files
The SQLite plugin discovers tables, views and foreign key relationships from SQLite database files.
It uses the pure-Go modernc.org/sqlite driver, so it needs no cgo and opens the database read-only. Turso and libSQL files are stored in the SQLite on-disk format, so a local copy of one works the same as any other SQLite file.
File Sources
The path field accepts local paths, S3 URIs (s3://bucket/key) or Git URIs (git::https://...). For S3 and Git sources, the file is downloaded to a temporary directory before discovery and cleaned up afterwards.
See File Sources for the full list of supported backends, authentication options and configuration examples.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick SQLite from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-sqlite-pipeline
runs:
- sqlite:
path: "<path>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
10 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
s3_source object S3 file source configuration
- shown when
- source_type = s3
credentials object AWS credentials for S3 access
use_default bool Use AWS credentials from environment or default profile (recommended)
- default
- true
id string AWS access key ID
secret password AWS secret access key
token password AWS session token
profile string AWS profile to use from shared credentials file
role string AWS IAM role ARN to assume
role_external_id string External ID for cross-account role assumption
region string AWS region for services
endpoint string Custom endpoint URL for AWS services
git_source object Git repository file source configuration
- shown when
- source_type = git
url string Git repository URL
ref string Branch, tag or commit to check out
- default
- main
path string Subdirectory within the repository
token password Personal access token for HTTPS auth
ssh_key_path string Path to SSH private key for SSH auth
path string Path to the SQLite database file (local path, s3://bucket/key or git::url)
include_columns bool Whether to include column information in table metadata
- default
- true
enable_metrics bool Whether to include table metrics (row and column counts)
- default
- true
discover_foreign_keys bool Whether to discover foreign key relationships
- default
- true
exclude_system_tables bool Whether to exclude SQLite internal tables (sqlite_*)
- default
- true
Assets emitted
Metadata this plugin attaches to each discovered asset.
SQ Lite
SQLiteFieldsSQLiteFields describes the metadata fields the SQLite plugin emits for table and view assets. It is kept as a documentation-only struct so downstream tooling can introspect the shape of the metadata map.
path stringPath to the SQLite database file
table_name stringTable or view name
object_type stringObject type (table, view)
SQ Lite Column
SQLiteColumnFieldsSQLiteColumnFields describes the per-column fields embedded in an asset's schema.
column_name stringColumn name
data_type stringDeclared column data type
is_nullable boolWhether null values are allowed
is_primary_key boolWhether the column is part of the primary key
column_default stringDefault value expression
SQ Lite Foreign Key
SQLiteForeignKeyFieldsSQLiteForeignKeyFields describes the fields of a discovered foreign key relationship.
source_table stringName of the referencing table
source_column stringColumn in the referencing table
target_table stringName of the referenced table
target_column stringColumn in the referenced table