Google Vertex AI

marmotdata/vertexai

Discover models, endpoints, datasets, feature groups and pipeline jobs from Google Vertex AI

Machine learning Assets Lineage Run History Experimental

You're viewing v0.1.1. The latest release is v0.1.2. View latest

The Vertex AI plugin discovers models, prediction endpoints, managed datasets, feature groups and pipeline jobs from a Google Cloud project.

Managed datasets and feature groups are both catalogued as Datasets, with a feature group's features as the schema. Pipeline jobs are catalogued as Jobs with run history.

Pipeline jobs are off by default because a project keeps a long job history. Turn them on with include_pipeline_jobs: true.

Vertex AI has no wildcard location and serves regional resources from the location's own host, so locations is required and each one is scanned through its own endpoint.

Naming

Vertex AI display names are not unique. An asset is named after its display name, and when more than one resource of the same kind shares one, every one of them is named display name (resource id). A feature group is named after its id.

Lineage

Edge	Meaning
GCS bucket FEEDS model	The bucket holding the model artifact
Model FEEDS endpoint	The endpoint serves the model
Pipeline job PRODUCES model	The job the model was produced by
BigQuery table FEEDS dataset	The table a tabular or time series dataset reads
GCS bucket FEEDS dataset	The bucket a dataset reads its files or blobs from
BigQuery table FEEDS feature group	The table the feature group reads its features from

Edges into BigQuery and Cloud Storage name assets those plugins own. Marmot drops an edge whose other end is not catalogued.

Required Permissions

roles/aiplatform.viewer covers every call this plugin makes. With no credentials set, Application Default Credentials are used.

Testing

Google publishes no Vertex AI emulator. The end to end tests serve the aiplatform API themselves, built from the generated API structs, and drive the plugin binary over the gRPC wire:

MARMOT_TEST_VERTEXAI_ENDPOINT=http://127.0.0.1:18821 go test ./...

In the UI

Point-and-click, no config file needed.

1 Open Runs Create pipeline
2 Pick Google Vertex AI from the plugin list.
3 Fill in the wizard, set a schedule, save.

Read the UI guide

With the CLI

Save a YAML config, then run marmot ingest.

ingest.yaml yaml

name: my-vertexai-pipeline
runs:
  - vertexai:
      project_id: "<project_id>"
      locations: []

$ marmot ingest -c ingest.yaml

Read the CLI guide

Not using plugins? Other ways to populate Marmot

Terraform Declare assets as infrastructure-as-code. Pulumi Native TypeScript, Python, or Go resources. Kubernetes Operator Sync assets from Kubernetes custom resources.

Configuration

14 top-level fields. * marks required fields.

tags multiselect

Tags to apply to discovered assets

external_links object[]

External links to show on all assets

name string

Display name for the link

icon string

Icon identifier for the link

url string

URL to the external resource

filter object

Filter discovered assets by name (regex)

include multiselect

Include patterns for resource names (regex)

exclude multiselect

Exclude patterns for resource names (regex)

project_id string

Google Cloud project ID

locations multiselect

Regions to scan

credentials_file string

Path to service account JSON file

credentials_json password

Service account JSON content

endpoint string

Custom endpoint URL, for testing against a local server

disable_auth bool

Disable authentication, for local testing

include_endpoints bool

Whether to discover prediction endpoints

default: true

include_datasets bool

Whether to discover managed datasets

default: true

include_feature_groups bool

Whether to discover feature groups from the feature store

default: true

include_pipeline_jobs bool

Whether to discover pipeline jobs. Projects keep a long job history, so this is off by default

default: false

max_pipeline_jobs int

How many recent pipeline jobs to read

default: 50

Assets emitted

Metadata this plugin attaches to each discovered asset.

Common

VertexAICommonFields

5 fields

The metadata every Vertex AI asset carries, whatever its kind.

resource_id string

Vertex AI id of the resource, unique within its project and location

location string

Region the resource lives in

project_id string

Google Cloud project the resource belongs to

create_time string

When the resource was created

update_time string

When the resource was last updated

Model

VertexAIModelFields

18 fields

The metadata a Model asset carries.

display_name string

Display name of the model, which Vertex AI does not require to be unique

version_id string

Version of the model this entry describes

version_aliases string

Aliases the version can be referenced by

version_description string

Description of this version

version_create_time string

When this version was created

artifact_uri string

Cloud Storage directory holding the model artifact

metadata_schema_uri string

Schema describing the model's additional metadata

container_image string

Container image the model is served from

predict_schemata_instance string

Schema of a single prediction instance

predict_schemata_parameters string

Schema of the prediction parameters

predict_schemata_prediction string

Schema of a single prediction

supported_deployment_resources_types string

Resource types the model can be deployed with

supported_input_storage_formats string

Input formats the model accepts for batch prediction

supported_output_storage_formats string

Output formats the model writes for batch prediction

training_pipeline string

Id of the training pipeline that uploaded the model

pipeline_job string

Id of the pipeline job that produced the model

base_model_source string

Model Garden name or Genie URI of the model this one is derived from

deployed_model_count int

Number of endpoints the model is deployed to

Endpoint

VertexAIEndpointFields

8 fields

The metadata an Endpoint asset carries.

display_name string

Display name of the endpoint, which Vertex AI does not require to be unique

network string

VPC network the endpoint is peered with

dedicated_endpoint_enabled bool

Set when the endpoint has a dedicated DNS name, absent otherwise

dedicated_endpoint_dns string

DNS name of the dedicated endpoint

traffic_split string

How traffic is shared between deployed models, as id=percent pairs

deployed_model_count int

Number of models deployed to the endpoint

deployed_models string

Display names of the models deployed to the endpoint

model_deployment_monitoring_job string

Id of the monitoring job watching the endpoint

Dataset

VertexAIDatasetFields

7 fields

The metadata a managed dataset carries. Feature groups are Datasets too, and carry the feature group fields below instead.

display_name string

Display name of the dataset, which Vertex AI does not require to be unique

metadata_schema_uri string

Schema the dataset's metadata follows

data_item_count int

Number of data items in the dataset

dataset_kind string

Kind of dataset, read from the metadata schema URI, for example image_1.0.0

saved_query_count int

Number of saved queries defined on the dataset

model_reference string

Model the dataset was created for

source_uris string

BigQuery and Cloud Storage URIs the dataset reads from

Feature Group

VertexAIFeatureGroupFields

8 fields

The metadata a feature group Dataset asset carries.

display_name string

Id of the feature group, which is what Vertex AI shows as its name

feature_group_id string

Id of the feature group

entity_id_columns string

Columns of the source table that identify an entity

big_query_source_uri string

BigQuery table the features are read from

dense bool

Set when every feature is written on every row, absent otherwise

static_data_source bool

Set when the source table does not change, absent otherwise

service_account_email string

Service account the feature group reads its source with

feature_count int

Number of features in the group

Feature

VertexAIFeatureFields

4 fields

The per-feature fields embedded in a feature group's schema.

column_name string

Feature id

data_type string

Feature value type, lowercased, for example string or int64

is_nullable bool

Always true: Vertex AI does not record which features are mandatory

description string

Description of the feature

Pipeline Job

VertexAIPipelineJobFields

8 fields

The metadata a pipeline Job asset carries.

display_name string

Display name of the pipeline job, which Vertex AI does not require to be unique

state string

Pipeline state, for example PIPELINE_STATE_SUCCEEDED

start_time string

When the pipeline started running

end_time string

When the pipeline finished

schedule_name string

Schedule that created the run

template_uri string

Location of the pipeline template the run was compiled from

service_account string

Service account the pipeline ran as

error_message string

Why the pipeline failed