Google Vertex AI
Official · maintained by Marmotmarmotdata/vertexai Discover models, endpoints, datasets, feature groups and pipeline jobs from Google Vertex AI
The Vertex AI plugin discovers models, prediction endpoints, managed datasets, feature groups and pipeline jobs from a Google Cloud project.
Managed datasets and feature groups are both catalogued as Datasets, with a feature group's features as the schema. Pipeline jobs are catalogued as Jobs with run history.
Pipeline jobs are off by default because a project keeps a long job history. Turn them on with include_pipeline_jobs: true.
Vertex AI has no wildcard location and serves regional resources from the location's own host, so locations is required and each one is scanned through its own endpoint.
Naming
Vertex AI display names are not unique. An asset is named after its display name, and when more than one resource of the same kind shares one, every one of them is named display name (resource id). A feature group is named after its id.
Lineage
| Edge | Meaning |
|---|---|
| GCS bucket FEEDS model | The bucket holding the model artifact |
| Model FEEDS endpoint | The endpoint serves the model |
| Pipeline job PRODUCES model | The job the model was produced by |
| BigQuery table FEEDS dataset | The table a tabular or time series dataset reads |
| GCS bucket FEEDS dataset | The bucket a dataset reads its files or blobs from |
| BigQuery table FEEDS feature group | The table the feature group reads its features from |
Edges into BigQuery and Cloud Storage name assets those plugins own. Marmot drops an edge whose other end is not catalogued.
Required Permissions
roles/aiplatform.viewer covers every call this plugin makes. With no credentials set, Application Default Credentials are used.
Testing
Google publishes no Vertex AI emulator. The end to end tests serve the aiplatform API themselves, built from the generated API structs, and drive the plugin binary over the gRPC wire:
MARMOT_TEST_VERTEXAI_ENDPOINT=http://127.0.0.1:18821 go test ./...Keyless authentication
On Marmot Cloud or Marmot Enterprise the pipeline can present its own identity instead of a service account key. Set workload_identity_provider to a Workload Identity Federation provider that trusts your Marmot instance as an OIDC issuer, and grant the pipeline's subject, pipeline:<name> as reported by the pipeline API, the role above directly (principal://iam.googleapis.com/<pool>/subject/pipeline:<name>), or grant it roles/iam.workloadIdentityUser on a service account named in service_account. No key exists anywhere; Marmot mints a short-lived token for each run.
In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Google Vertex AI from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-vertexai-pipeline
runs:
- vertexai:
project_id: "<project_id>"
locations: []$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
17 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
audience string Audience of the Marmot identity token. Derived from the credentials' federation settings; set it only when the cloud side expects another
project_id string Google Cloud project ID
locations multiselect Regions to scan
credentials_file string Path to service account JSON file
credentials_json password Service account JSON content
workload_identity_provider string Workload Identity Federation provider to exchange the Marmot identity token at, projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Setting it federates: no key is needed; bind the pipeline's subject on the Google Cloud side
service_account string Service account to impersonate after the exchange; empty acts as the federated principal directly
endpoint string Custom endpoint URL, for testing against a local server
disable_auth bool Disable authentication, for local testing
include_endpoints bool Whether to discover prediction endpoints
- default
- true
include_datasets bool Whether to discover managed datasets
- default
- true
include_feature_groups bool Whether to discover feature groups from the feature store
- default
- true
include_pipeline_jobs bool Whether to discover pipeline jobs. Projects keep a long job history, so this is off by default
- default
- false
max_pipeline_jobs int How many recent pipeline jobs to read
- default
- 50
Assets emitted
Metadata this plugin attaches to each discovered asset.
Common
VertexAICommonFieldsThe metadata every Vertex AI asset carries, whatever its kind.
resource_id stringVertex AI id of the resource, unique within its project and location
location stringRegion the resource lives in
project_id stringGoogle Cloud project the resource belongs to
create_time stringWhen the resource was created
update_time stringWhen the resource was last updated
Model
VertexAIModelFieldsThe metadata a Model asset carries.
display_name stringDisplay name of the model, which Vertex AI does not require to be unique
version_id stringVersion of the model this entry describes
version_aliases stringAliases the version can be referenced by
version_description stringDescription of this version
version_create_time stringWhen this version was created
artifact_uri stringCloud Storage directory holding the model artifact
metadata_schema_uri stringSchema describing the model's additional metadata
container_image stringContainer image the model is served from
predict_schemata_instance stringSchema of a single prediction instance
predict_schemata_parameters stringSchema of the prediction parameters
predict_schemata_prediction stringSchema of a single prediction
supported_deployment_resources_types stringResource types the model can be deployed with
supported_input_storage_formats stringInput formats the model accepts for batch prediction
supported_output_storage_formats stringOutput formats the model writes for batch prediction
training_pipeline stringId of the training pipeline that uploaded the model
pipeline_job stringId of the pipeline job that produced the model
base_model_source stringModel Garden name or Genie URI of the model this one is derived from
deployed_model_count intNumber of endpoints the model is deployed to
Endpoint
VertexAIEndpointFieldsThe metadata an Endpoint asset carries.
display_name stringDisplay name of the endpoint, which Vertex AI does not require to be unique
network stringVPC network the endpoint is peered with
dedicated_endpoint_enabled boolSet when the endpoint has a dedicated DNS name, absent otherwise
dedicated_endpoint_dns stringDNS name of the dedicated endpoint
traffic_split stringHow traffic is shared between deployed models, as id=percent pairs
deployed_model_count intNumber of models deployed to the endpoint
deployed_models stringDisplay names of the models deployed to the endpoint
model_deployment_monitoring_job stringId of the monitoring job watching the endpoint
Dataset
VertexAIDatasetFieldsThe metadata a managed dataset carries. Feature groups are Datasets too, and carry the feature group fields below instead.
display_name stringDisplay name of the dataset, which Vertex AI does not require to be unique
metadata_schema_uri stringSchema the dataset's metadata follows
data_item_count intNumber of data items in the dataset
dataset_kind stringKind of dataset, read from the metadata schema URI, for example image_1.0.0
saved_query_count intNumber of saved queries defined on the dataset
model_reference stringModel the dataset was created for
source_uris stringBigQuery and Cloud Storage URIs the dataset reads from
Feature Group
VertexAIFeatureGroupFieldsThe metadata a feature group Dataset asset carries.
display_name stringId of the feature group, which is what Vertex AI shows as its name
feature_group_id stringId of the feature group
entity_id_columns stringColumns of the source table that identify an entity
big_query_source_uri stringBigQuery table the features are read from
dense boolSet when every feature is written on every row, absent otherwise
static_data_source boolSet when the source table does not change, absent otherwise
service_account_email stringService account the feature group reads its source with
feature_count intNumber of features in the group
Feature
VertexAIFeatureFieldsThe per-feature fields embedded in a feature group's schema.
column_name stringFeature id
data_type stringFeature value type, lowercased, for example string or int64
is_nullable boolAlways true: Vertex AI does not record which features are mandatory
description stringDescription of the feature
Pipeline Job
VertexAIPipelineJobFieldsThe metadata a pipeline Job asset carries.
display_name stringDisplay name of the pipeline job, which Vertex AI does not require to be unique
state stringPipeline state, for example PIPELINE_STATE_SUCCEEDED
start_time stringWhen the pipeline started running
end_time stringWhen the pipeline finished
schedule_name stringSchedule that created the run
template_uri stringLocation of the pipeline template the run was compiled from
service_account stringService account the pipeline ran as
error_message stringWhy the pipeline failed