Databricks

Euno's Databricks integration connects to Unity Catalog metadata, Databricks system lineage and query history, and the Workspace API so you can discover catalogs, schemas, tables, columns, usage metrics, lineage, and notebooks in one source.

What Euno Discovers

Euno's Databricks integration supports auto-discovery of:

• Databricks workspaces

• Unity Catalog databases (catalogs)

• Unity Catalog schemas

• Unity Catalog tables, views, materialized views, and external tables

• Table and column metadata (including tags)

• Table usage metrics for 14/30/60 day windows

• Column usage metrics for 14/30/60 day windows

• Query-history-backed column lineage

• Databricks notebooks and notebook-driven relationships

For resource properties and relationships in detail, see Databricks Integration Discovered Resources.

Prerequisites / Requirements

Before you add the source, make sure you have:

• A Databricks workspace with Unity Catalog enabled

• At least one running SQL warehouse and its HTTP path

• The workspace hostname (for example dbc-xxxxxxxx-xxxx.cloud.databricks.com)

• A Databricks personal access token for a user that can run metadata queries and access the objects you want to observe

The token user should have:

• Access to the Databricks workspace

• Permission to use the configured SQL warehouse

• SELECT access to the required system.information_schema and system.access / system.query.history objects used by the integration

• Access to the catalogs, schemas, and tables you want to observe

• Workspace API visibility to notebook paths you want to observe (under the default discovery roots)

• To ingest column masking: the MANAGE privilege on, or ownership of, the observed schemas and tables whose ABAC policies must be read. The Unity Catalog policies API is queried on both levels so Euno can discover inherited catalog/schema policies and policies attached directly to a table. Column masking is not ingested for a scope whose policies cannot be read, and the run report identifies the affected scopes.

Stage 1: Configure Databricks

Step 1: SQL warehouse and hostname

1. Verify that Unity Catalog is enabled.

2. Ensure at least one SQL warehouse is available and running.

3. Note your workspace hostname (for example dbc-xxxxxxxx-xxxx.cloud.databricks.com).

4. Note the SQL warehouse HTTP path (for example /sql/1.0/warehouses/warehouse-id).

Step 2: Personal access token

1. In Databricks, go to Settings → Developer → Access tokens.

2. Click Generate new token.

3. Set a comment (for example Euno integration).

4. Set token lifetime as needed.

5. Click Generate.

6. Copy and store the token securely.

In Euno, the integration combines:

• Unity Catalog metadata (system.information_schema.*)

• Databricks system lineage and usage evidence (system.access.*, system.query.history)

• Workspace notebook discovery (Workspace API)

Stage 2: Configure New Databricks Source in Euno

Step 1: Access the Sources Page

1. Open the Sources page.

2. Click Add New Source and choose Databricks.

Step 2: General Configuration

Asterisk (*) means a mandatory field.

Configuration	Description
Server Hostname*	Databricks workspace hostname (for example dbc-xxxxxxxx-xxxx.cloud.databricks.com).
HTTP Path*	Databricks SQL warehouse HTTP path (for example /sql/1.0/warehouses/warehouse-id).
Access Token*	Databricks personal access token (stored as a secret).
Workspace Name	Optional display name in Euno. If left blank, the hostname is used.

Step 3: Schedule

• Enable scheduling.

• Choose one option:

  1. Weekly with a specific day and time.

  2. Hourly with an interval.

Step 4: Resource Cleanup

• Immediate Cleanup: remove resources not detected in the latest successful run.

• No Cleanup: keep resources even if they are no longer detected.

For cleanup semantics, see Resource Sponsorship in Euno.

Step 5: Advanced Settings

Open the Advanced section to configure optional filters.

Configuration	Description
Override Base URI	Override the hostname used when Euno generates resource URIs. Leave blank to use Server Hostname.
Database Pattern	Allow/deny regular expressions for catalog names. Defaults to include all catalogs; system catalogs such as system and samples are always excluded from metadata queries.

Pattern examples

• .* — include all names that match.

• production_.* — include only names starting with production_.

• Allow .* and deny test_.* — include all except names starting with test_.

Step 6: Save

Click Test & Save to validate connectivity and save the source.

Important behavior notes

• Table read usage metrics are emitted for 14/30/60 day windows.

• Databricks read DBU metrics are emitted as total_read_dbu_14d, total_read_dbu_30d, total_read_dbu_60d and average_read_dbu_14d, average_read_dbu_30d, average_read_dbu_60d.

• Column usage metrics are emitted for 14/30/60 day windows.

• Notebook execution evidence and notebook-derived relationships use system.query.history, system.access.table_lineage, and system.access.column_lineage with a 30-day lookback window.

• Regular Databricks column lineage is read from system.access.column_lineage with a 30-day lookback; Euno represents it as column-level upstream field relationships after processing (see the discovered-resources page).

• Notebook observation runs by default; scope follows the integration’s discovery rules described in the hint under Advanced Settings.