Google Firebase
Official · maintained by Marmotmarmotdata/firebase Discover Firestore collections and Realtime Database instances from Firebase projects
The Firebase plugin discovers Firestore databases with their collections and subcollections, and Realtime Database instances, from a Firebase project.
Firestore stores no schema, so each collection's fields are inferred from a sample of its documents. A field missing from any sampled document, or null in one, is nullable; a field seen with two different types is mixed.
A subcollection exists once per parent document, so the copies are catalogued as one asset with the document id left out of the name: firestore/(default)/orders/lines.
Cloud Storage for Firebase buckets are not discovered here. They are GCS buckets and the gcs plugin already owns that identity.
Databases
Leaving databases empty lists every Firestore database through the admin API. The Firestore emulator does not implement that call, so name the databases explicitly when running against one.
Testing against the emulator
docker run -d --name firestore-emulator -p 8080:8080 \
gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators \
gcloud emulators firestore start --host-port=0.0.0.0:8080
MARMOT_TEST_FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 go test ./...In the UI
Point-and-click, no config file needed.
- 1 Open Runs Create pipeline
- 2 Pick Google Firebase from the plugin list.
- 3 Fill in the wizard, set a schedule, save.
With the CLI
Save a YAML config, then run marmot ingest.
name: my-firebase-pipeline
runs:
- firebase:
project_id: "<project_id>"$ marmot ingest -c ingest.yamlNot using plugins? Other ways to populate Marmot
Configuration
14 top-level fields. * marks required fields.
tags multiselect Tags to apply to discovered assets
external_links object[] External links to show on all assets
name string Display name for the link
icon string Icon identifier for the link
url string URL to the external resource
filter object Filter discovered assets by name (regex)
include multiselect Include patterns for resource names (regex)
exclude multiselect Exclude patterns for resource names (regex)
project_id string Google Cloud project ID
databases multiselect Firestore database IDs to scan. Every database is listed from the API when this is empty. The Firestore emulator has no such API, so name the databases here when pointing at one
credentials_file string Path to service account JSON file
credentials_json password Service account JSON content
endpoint string Custom endpoint URL, for testing against a local server
disable_auth bool Disable authentication, for local testing
include_realtime_database bool Whether to discover Realtime Database instances
- default
- true
include_project_details bool Whether to read the Firebase project name and number
- default
- true
include_subcollections bool Whether to descend into subcollections
- default
- true
max_collection_depth int How many levels of subcollection to descend
- default
- 2
sample_documents int How many documents to read per collection to infer its fields
- default
- 20
Assets emitted
Metadata this plugin attaches to each discovered asset.
Project
FirebaseProjectFieldsThe project-level metadata fields the plugin adds to every asset. They are kept as documentation-only structs so downstream tooling can introspect the shape of the metadata map.
firebase_project_id stringFirebase project ID
firebase_project_number intGoogle-assigned project number
firebase_project_display_name stringProject display name
Firestore Database
FirestoreDatabaseFieldsThe metadata fields the plugin emits for a Firestore database asset.
database_id stringFirestore database ID, (default) for the default database
database_kind stringWhich Firebase database this is: firestore or realtime
location_id stringRegion the database runs in
database_type stringFIRESTORE_NATIVE or DATASTORE_MODE
concurrency_mode stringDefault transaction concurrency control mode
point_in_time_recovery stringWhether point in time recovery is enabled
delete_protection stringWhether the database is protected from deletion
version_retention_period stringHow long past versions of the data are readable
earliest_version_time stringOldest timestamp a read can ask for
create_time stringWhen the database was created
update_time stringWhen the database resource was last changed
uid stringSystem-generated database UUID
free_tier boolWhether the database is eligible for the free tier
Realtime Database
RealtimeDatabaseFieldsThe metadata fields the plugin emits for a Realtime Database instance asset.
instance_id stringRealtime Database instance ID
database_kind stringWhich Firebase database this is: firestore or realtime
database_url stringHostname the instance is served on
instance_type stringDEFAULT_DATABASE or USER_DATABASE
state stringLifecycle state, for example ACTIVE or DISABLED
Firestore Collection
FirestoreCollectionFieldsThe metadata fields the plugin emits for a Firestore collection asset.
database_id stringFirestore database the collection lives in
collection_id stringCollection ID, the last segment of the path
collection_path stringCollection path with parent document IDs left out
depth intNesting level, 1 for a root collection
sampled_documents intHow many documents were read to infer the fields
collection_group_id stringID a collection group query uses to reach every copy of a subcollection
parent_path stringPath of the parent collection, for subcollections
Firestore Column
FirestoreColumnFieldsThe per-field columns inferred from the sampled documents and stored in an asset's schema.
column_name stringDocument field name
data_type stringInferred Firestore type, or mixed when the sample disagreed
is_nullable boolWhether the field was absent or null in any sampled document