AWS SageMaker

marmotdata/sagemaker

Discover models, endpoints, feature groups and training jobs from Amazon SageMaker

Machine learning Assets Lineage Run History Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The SageMaker plugin discovers models, inference endpoints, model registry groups, feature groups and training jobs from Amazon SageMaker.

Models and model package groups are both catalogued as Models, told apart by the kind metadata field. A registry group named the same as a deployed model resolves to the same asset.

Training jobs are off by default because an account keeps a long job history. Turn them on with include_training_jobs: true.

Lineage

Edge	Meaning
S3 bucket FEEDS model	The bucket holding the model artifact
Model FEEDS endpoint	A production variant serves the model
S3 bucket FEEDS training job	A training channel reads from the bucket
Training job PRODUCES model	The model's artifact is the one the job wrote
Feature group PRODUCES Glue table	The offline store is queryable through the table
Feature group PRODUCES S3 bucket	The offline store writes to the bucket

Edges into S3 and Glue name assets those plugins own. Marmot drops an edge whose other end is not catalogued.

Required Permissions

AWS Configuration

See AWS Configuration for the supported AWS configuration options.

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick AWS SageMaker from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-sagemaker-pipeline
runs:
  - sagemaker:
      # No required fields — see the Configuration tab.

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

10 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

credentials object

AWS credentials configuration

use_default bool

Use AWS credentials from environment or default profile (recommended)

default: true

id string

AWS access key ID

secret password

AWS secret access key

token password

AWS session token

profile string

AWS profile to use from shared credentials file

role string

AWS IAM role ARN to assume

role_external_id string

External ID for cross-account role assumption

region string

AWS region for services

endpoint string

Custom endpoint URL for AWS services

tags_to_metadata bool

Convert AWS tags to Marmot metadata

include_tags multiselect

List of AWS tags to include as metadata. By default, all tags are included.

include_endpoints bool

Whether to discover inference endpoints

default: true

include_model_packages bool

Whether to discover model package groups from the model registry

default: true

include_feature_groups bool

Whether to discover feature groups from the feature store

default: true

include_training_jobs bool

Whether to discover training jobs. Accounts keep a long job history, so this is off by default

default: false

Assets emitted

Metadata this plugin attaches to each discovered asset.

Model

SageMakerModelFields

26 fields

The metadata a Model asset carries. Both a deployed model and a model registry group are filed as Models, told apart by the kind field.

kind string

Which kind of model this is (model, model_package_group)

arn string

The ARN of the model or model package group

image string

Container image the model is served from

model_data_url string

S3 location of the model artifact

mode string

Container mode (SingleModel or MultiModel)

environment string

Container environment variables, with credential-looking values masked

containers string

Containers of an inference pipeline, each with its image, model_data_url and mode

execution_role_arn string

IAM role the model runs under

network_isolation bool

Whether the model container runs without network access

vpc_subnet_count int

Number of VPC subnets the model is attached to

description string

Description of the model package group

status string

Status of the model package group

version_count int

Number of versions in the model package group

latest_version int

Newest version number in the model package group

latest_approval_status string

Approval status of the newest version

latest_image string

Container image of the newest approved version

latest_model_data_url string

S3 artifact of the newest approved version

domain string

Machine learning domain the model package belongs to

task string

Machine learning task the model package performs

sample_payload_url string

S3 location of a sample inference payload

supported_content_types string

Content types the model package accepts

supported_response_mime_types string

Response MIME types the model package returns

model_quality_statistics_s3_uri string

S3 location of the model quality statistics report

created_at string

When the model or group was created

region string

AWS region the resource lives in

url string

Link to the model in the AWS console. Not set for a model package group

Endpoint

SageMakerEndpointFields

10 fields

The metadata an Endpoint asset carries.

arn string

The ARN of the endpoint

status string

Endpoint status (InService, Creating, Failed)

endpoint_config string

Name of the endpoint configuration in use

variants string

Production variants, each with its name, model, instance_type, instance_count, weight and serverless flag

data_capture_s3_uri string

S3 location captured requests and responses are written to

kms_key_id string

KMS key the endpoint storage volume is encrypted with

created_at string

When the endpoint was created

last_modified_at string

When the endpoint was last modified

region string

AWS region the endpoint lives in

url string

Link to the endpoint in the AWS console

Feature Group

SageMakerFeatureGroupFields

10 fields

The metadata a feature group Dataset asset carries.

arn string

The ARN of the feature group

record_identifier string

Feature that identifies a record

event_time_feature string

Feature holding the event timestamp

online_store bool

Whether the online store is enabled

offline_store_s3_uri string

S3 location of the offline store

glue_table string

Glue table the offline store is queryable through, as database.table

status string

Feature group status

description string

Description of the feature group

created_at string

When the feature group was created

region string

AWS region the feature group lives in

Feature

SageMakerFeatureFields

4 fields

The per-feature fields embedded in a feature group's schema.

column_name string

Feature name

data_type string

Feature type (String, Integral, Fractional)

is_nullable bool

False for the record identifier and event time features, which every record must carry

is_primary_key bool

Whether the feature is the record identifier

Training Job

SageMakerTrainingJobFields

17 fields

The metadata a training Job asset carries.

arn string

The ARN of the training job

status string

Training job status (InProgress, Completed, Failed, Stopped)

failure_reason string

Why the training job failed

algorithm_image string

Container image the job trained with

algorithm_name string

Marketplace algorithm the job trained with

hyperparameters string

Hyperparameters the job ran with

instance_type string

Instance type the job ran on

instance_count int

Number of instances the job ran on

input_channels string

Training channels, each mapping a channel name to its S3 location

output_s3_path string

S3 prefix the job wrote its output to

model_artifacts_s3_uri string

S3 location of the model artifact the job produced

final_metrics string

Last value the job reported for each metric

started_at string

When training started

ended_at string

When training ended

created_at string

When the training job was created

region string

AWS region the training job ran in

url string

Link to the training job in the AWS console