Databricks Integration Discovered Resources

The Euno Databricks integration discovers and updates the following resource families:

• Databricks workspace

• Unity Catalog databases (catalogs)

• Unity Catalog schemas

• Databricks tables (including views, materialized views, and external tables)

• Databricks table columns

• Databricks notebooks

• Usage and lineage properties across the resources above

Discovery inputs

The integration combines multiple Databricks data sources:

• system.information_schema.* for metadata

• system.information_schema.column_masks and the tag views (catalog_tags, schema_tags, table_tags, column_tags) for column masking

• system.query.history for usage and notebook execution evidence

• system.access.table_lineage for table lineage evidence

• system.access.column_lineage for regular column-level lineage and notebook reporting evidence

• Workspace API for notebook discovery

• Unity Catalog policies API for ABAC column-mask policies

Resource hierarchy

The Databricks integration follows this container hierarchy:

Databricks Workspace
└── Catalog (Database)
    └── Schema
        ├── Table / View / Materialized View / External Table
        │   └── Column
        └── ...

Databricks Workspace
└── Notebook

Ingested Resources & Properties

Databricks workspace (databricks_workspace)

Property	Description
name	Workspace display name (from configuration; defaults to hostname if Workspace Name is not set).
type	Always databricks_workspace.
subtype	Always databricks_workspace.
description	Short generated description that includes the workspace hostname.
database_technology	Always databricks.

URI pattern

platform.databricks.databricks_workspace.<hostname>

Example: platform.databricks.databricks_workspace.dbc-50e7cad0-c2f9.cloud.databricks.com

Unity Catalog database (database)

Property	Description
name	Catalog name.
type	Always database.
subtype	Always database.
parent_container	Parent workspace URI.
database_technology	Always databricks.
description	Catalog comment from Unity Catalog, when present.
native_owners	Catalog owner from Unity Catalog metadata.
created_at, updated_at	Catalog timestamps when available.
created_by, updated_by	Creator and last modifier when available.
tags	Tag names with empty or null values.
meta	Tag key–value pairs when values are non-empty.

URI pattern

databricks.<hostname>.<catalog_name>

Unity Catalog schema (database_schema)

Property	Description
name	Schema name.
type	Always database_schema.
subtype	Always database_schema.
parent_container	Parent catalog URI.
database_technology	Always databricks.
database	Catalog name.
description	Schema comment when present.
native_owners	Schema owner when present.
created_at, updated_at	Schema timestamps when available.
created_by, updated_by	Creator and last modifier when available.
tags	Tag names with empty or null values.
meta	Tag key–value pairs when values are non-empty.

URI pattern

databricks.<hostname>.<catalog_name>.<schema_name>

Databricks table (table)

Databricks table resources use these subtypes: base_table, external_table, view, materialized_view.

Property	Description
name	Table or view name.
type	Always table.
subtype	One of the Databricks subtypes above.
parent_container	Parent schema URI.
database	Catalog name.
database_schema	Schema name.
database_technology	Always databricks.
description	Table or view comment when present.
native_owners	Owner when present.
created_at, updated_at	Object timestamps when available.
created_by, updated_by	Creator and last modifier when available.
tags	Tag names with empty or null values.
meta	Tag key–value pairs when values are non-empty.
sql_dialect	SQL dialect (databricks).
platform_uri_prefix	URI prefix for this workspace (databricks.<hostname>).
table_properties.materialized	Whether the object is materialized, derived from Databricks table type and view metadata.
table_schema	Structured column list for observed columns.
table_dependencies	Upstream tables and views for lineage and dependencies. Values combine warehouse lineage from system.access.table_lineage, dependencies parsed from view SQL when applicable, and other merged inputs; Euno exposes the result as this resource’s upstream table relationships.
databricks_contains_masked_columns	Indicates that at least one column in the table or view is protected by a column mask.
databricks_masked_column_count	Number of columns in the table or view protected by a column mask.
databricks_masking_refreshed_at	Timestamp when Euno last refreshed Databricks column-mask metadata for the resource.

URI pattern

databricks.<hostname>.<catalog_name>.<schema_name>.<table_name>

System exclusions

The crawler skips catalog discovery for information_schema and hive_metastore. Metadata queries for tables, schemas, and related objects exclude the system and samples catalogs and the information_schema schema.

Table usage metrics

Usage windows are emitted for 14d, 30d, and 60d intervals.

Read usage properties

Applicable to table resources (base_table, external_table, view, materialized_view).

Property family	Description
total_read_queries_14d, total_read_queries_30d, total_read_queries_60d	Number of read queries.
total_read_runtime_14d, total_read_runtime_30d, total_read_runtime_60d	Total active compute runtime (ms) for read queries.
total_read_bytes_processed_14d, total_read_bytes_processed_30d, total_read_bytes_processed_60d	Total read bytes.
distinct_users_14d, distinct_users_30d, distinct_users_60d	Distinct users running read queries.
total_read_dbu_14d, total_read_dbu_30d, total_read_dbu_60d	Total Databricks read DBU-seconds (compute-time proxy).
average_read_dbu_14d, average_read_dbu_30d, average_read_dbu_60d	Average read DBU-seconds per read query.

Read DBU values are derived from query-history active compute time.

Write usage properties

Write applicability follows Databricks table_type classification:

Observed subtype	Databricks table_type values	Write usage applicable?	Behavior
view	VIEW	No	Write metrics are emitted as zeros each crawl to clear stale values.
materialized_view	MATERIALIZED_VIEW	Yes	Write metrics from write statements; zero-filled when there is no in-window activity.
base_table	MANAGED, STREAMING_TABLE, MANAGED_SHALLOW_CLONE	Yes	Same as materialized view.
external_table	EXTERNAL, EXTERNAL_SHALLOW_CLONE, EXTERNAL_TABLE	Yes	Same as materialized view.
external_table	FOREIGN	No	Write metrics are not applicable.

Property family	Meaning
total_write_queries_14d, total_write_queries_30d, total_write_queries_60d	Number of write queries.
total_write_runtime_14d, total_write_runtime_30d, total_write_runtime_60d	Total active compute runtime (ms) for write queries.
total_write_bytes_processed_14d, total_write_bytes_processed_30d, total_write_bytes_processed_60d	Total bytes processed by write queries.

Databricks column (column)

Property	Description
name	Column name.
type	Always column.
subtype	Always column.
description	Column comment when present.
parent_container	Parent table URI.
database	Catalog name.
database_schema	Schema name.
database_technology	Always databricks.
native_data_type	Native Databricks data type.
normalized_data_type	Euno-normalized data type.
upstream_fields	Upstream column URIs for lineage. Values combine warehouse column lineage from system.access.column_lineage with other merged inputs; Euno exposes the result as this column’s upstream field relationships.
databricks_masking_policy	Column masks protecting the column: the mask function for directly assigned masks, or the ABAC masking policy for tag-driven masks.
databricks_masking_source	How the mask is attached: direct (assigned with SET MASK) or abac_policy (applied by an ABAC column-mask policy through a governed tag).
databricks_is_masked	Indicates that the column is protected by a column mask.
databricks_masking_refreshed_at	Timestamp when Euno last refreshed Databricks column-mask metadata for the resource.

Detailed masking information — the effective mask, the attachment source, the governed tag that triggered an ABAC mask, and any additional attachments — appears in the Data Model's shared Masking Policy section, which presents Databricks and Snowflake masking side by side.

Column masks are discovered from system.information_schema.column_masks (direct masks) and from ABAC column-mask policies listed through the Unity Catalog policies API on each observed schema and table. Schema listing includes inherited catalog policies, while table listing captures policies attached directly to that table. ABAC policies are resolved to columns by evaluating their MATCH COLUMNS conditions against the governed tags set directly on each column (governed tags do not propagate to individual columns in Databricks); WHEN conditions are evaluated against table tags, including tags inherited from the parent catalog and schema.

URI pattern

databricks.<hostname>.<catalog_name>.<schema_name>.<table_name>.column.<column_name>

Column usage metrics

Column usage is emitted for 14d, 30d, and 60d windows.

Property family	Meaning
total_read_queries_14d, total_read_queries_30d, total_read_queries_60d	Distinct statements that read the column.
distinct_users_14d, distinct_users_30d, distinct_users_60d	Distinct users reading the column.

Databricks notebook (databricks_notebook)

Property	Description
name	Notebook display name (path basename).
type	Always databricks_notebook.
subtype	Always databricks_notebook.
native_id	Databricks notebook object ID.
parent_container	Parent workspace URI.
created_at, updated_at	Notebook object timestamps from the Workspace API.
description	Notebook description when the API provides it.
native_last_data_update	Latest observed execution time from query history evidence.
defines	Table URIs the notebook is observed to define or update from table-lineage evidence (defines on the notebook).
table_dependencies	Table URIs the notebook reads from (system.access.table_lineage with notebook context); stored on the notebook.
upstream_fields	Column URIs referenced in notebook column-lineage evidence for reporting-style relationships; stored on the notebook (not on the column rows).

URI pattern

platform.databricks.databricks_workspace.<hostname>.notebook.<notebook_object_id>

Statement-level column dependencies for views

For Databricks views, Euno also captures statement-level column dependencies and reflects them in table-level upstream_fields.

That means a view can show upstream field relationships to columns used only in SQL logic (for example filters), not only columns in the SELECT list.

Included SQL logic clauses:

• JOIN conditions

• WHERE

• GROUP BY

• HAVING

• QUALIFY

• ORDER BY

Example: for SELECT order_id FROM orders_raw WHERE order_status = 'COMPLETE', the view can relate upstream to orders_raw.column.order_status even when order_status is not projected.

Relationships

Source type(s)	Relationship	Target type(s)	Notes
database	has parent	databricks_workspace	parent_container on the catalog.
database_schema	has parent	database	parent_container on the schema.
table	has parent	database_schema	parent_container on the table.
column	has parent	table	parent_container on the column.
databricks_notebook	has parent	databricks_workspace	parent_container on the notebook.
table	has upstream	table	Upstream warehouse tables and views (merged table_dependencies / lineage).
databricks_notebook	has upstream	table	Tables the notebook reads (table_dependencies on the notebook).
column	has upstream_fields	column	Column-to-column lineage (merged upstream_fields).
databricks_notebook	has upstream_fields	column	Notebook reporting lineage: the notebook lists source column URIs in upstream_fields.
databricks_notebook	defines	table	Tables the notebook is observed to define or update (defines on the notebook).
table	has definer	databricks_notebook	Inverse of defines when the table is tied to a defining notebook.

Notes and caveats

• Notebook execution evidence and notebook-derived relationships use system.query.history, system.access.table_lineage, and system.access.column_lineage with a 30-day lookback window.

• Regular Databricks column lineage reads from system.access.column_lineage with a 30-day lookback; graph relationships use the merged column upstream field model described above.

• Table and column usage metrics use 14d, 30d, and 60d windows.

• Usage and lineage relationships are emitted only for resources observed in the current integration scope.