SFTP

marmotdata/sftp

Discover directories and files from an SFTP server

Storage Assets Lineage Experimental

The SFTP plugin discovers directories and files from an SFTP server, and infers columns for delimited and JSON files.

It connects with golang.org/x/crypto/ssh and github.com/pkg/sftp, using a password or a private key (RSA, Ed25519 or ECDSA).

Names

A Folder or File is named by its path below the configured root, with no leading slash: incoming/2026-09 and incoming/2026-09/orders.csv. A root other than / contributes nothing to the name, so the same tree served from /data or from / produces the same assets.

The root itself is not an asset. The directories directly below each configured root are the top of the tree.

Columns

.csv and .tsv files get columns typed INT, FLOAT, BOOLEAN, DATETIME or STRING from up to sample_rows rows. .json (an array of objects) and .jsonl files get the union of their top-level keys plus one level of nesting, written parent.child. .parquet and .avro files are catalogued but get no columns.

No file is read beyond max_read_bytes. A row count taken from a truncated read is marked row_count_exact: false.

Limits

max_depth and max_files bound a run. Symlinks are skipped unless follow_symlinks is set, and a directory already walked is never walked again, so a link pointing back up the tree cannot loop.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick SFTP from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-sftp-pipeline
runs:
  - sftp:
      host: "<host>"
      username: "<username>"

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

19 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

host string

SFTP server hostname or IP address

port int

SFTP server port

default: 22

username string

User to log in as

password password

Password for the user

private_key password

PEM private key for the user, RSA, Ed25519 or ECDSA

private_key_passphrase password

Passphrase protecting the private key

host_key string

Server public key to trust, as written in known_hosts. Empty means any key is accepted

root_directories multiselect

Directories to walk

default: ["/"]

max_depth int

Levels below each root to walk

default: 10

max_files int

Stop after this many files

default: 20000

follow_symlinks bool

Walk into symlinks instead of skipping them

default: false

structured_only bool

Only catalogue csv, tsv, json, jsonl, parquet and avro files

default: false

include_columns bool

Infer columns for delimited and JSON files

default: true

include_statistics bool

Emit size, row count and column count statistics

default: true

sample_rows int

Rows read from a file to infer its columns

default: 200

max_read_bytes int

Most bytes to read from a single file

default: 33554432

Assets emitted

Metadata this plugin attaches to each discovered asset.

Folder

FolderFields

7 fields

The metadata fields the SFTP plugin emits for a Folder asset. The counts and the size cover a folder's immediate contents, not its whole subtree.

path string

Absolute path of the directory on the server

parent string

Name of the folder above this one, absent at the top of a root

root string

Configured root directory this folder was reached from

modified string

Last modification time, RFC3339

file_count int

Files directly in this directory

directory_count int

Subdirectories directly in this directory

size_bytes int

Total size of the files directly in this directory

File

FileFields

13 fields

The metadata fields the SFTP plugin emits for a File asset.

path string

Absolute path of the file on the server

directory string

Name of the folder holding the file, absent at the top of a root

size int

File size in bytes

modified string

Last modification time, RFC3339

mode string

Permission bits, as ls prints them

owner_uid int

Numeric user id of the owner

owner_gid int

Numeric group id of the owner

extension string

Lowercase file extension without the dot

mime_type string

MIME type guessed from the extension

file_type string

File kind: csv, tsv, json, jsonl, parquet, avro or other

row_count_exact bool

Whether the row count covers the whole file or stopped at max_read_bytes

host string

Server the file was read from

port int

Port the server was reached on

Delimited Column

DelimitedColumnFields

3 fields

The per-column fields inferred for a csv or tsv file.

column_name string

Column name, taken from the header row

data_type string

Inferred type: INT, FLOAT, BOOLEAN, DATETIME or STRING

is_nullable bool

Whether any sampled row left the column empty

JSONColumn

JSONColumnFields

4 fields

The per-column fields inferred for a json or jsonl file. A nested key is written parent.child.

column_name string

Key name, with one level of nesting written parent.child

data_type string

JSON types seen for the key, joined by | when mixed

is_nullable bool

Whether the key was null or missing in any sampled record

occurrence int

Sampled records that carried the key