> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/data-warehouses/databricks-integration.md).

# Databricks

Euno's Databricks integration connects to Unity Catalog metadata, Databricks system lineage and query history, and the Workspace API so you can discover catalogs, schemas, tables, columns, usage metrics, lineage, and notebooks in one source.

## What Euno Discovers

Euno's Databricks integration supports auto-discovery of:

* Databricks workspaces
* Unity Catalog databases (catalogs)
* Unity Catalog schemas
* Unity Catalog tables, views, materialized views, and external tables
* Table and column metadata (including tags)
* Table usage metrics for 14/30/60 day windows
* Column usage metrics for 14/30/60 day windows
* Query-history-backed column lineage
* Databricks notebooks and notebook-driven relationships

For resource properties and relationships in detail, see [Databricks Integration Discovered Resources](/sources/data-warehouses/databricks-integration/databricks-integration-discovered-resources.md).

## Prerequisites / Requirements

Before you add the source, make sure you have:

* A Databricks workspace with Unity Catalog enabled
* At least one running SQL warehouse and its HTTP path
* The workspace hostname (for example `dbc-xxxxxxxx-xxxx.cloud.databricks.com`)
* A Databricks personal access token for a user that can run metadata queries and access the objects you want to observe

The token user should have:

* Access to the Databricks workspace
* Permission to use the configured SQL warehouse
* `SELECT` access to the required `system.information_schema` and `system.access` / `system.query.history` objects used by the integration
* Access to the catalogs, schemas, and tables you want to observe
* Workspace API visibility to notebook paths you want to observe (under the default discovery roots)
* The Databricks `workspace` scope on the personal access token when notebook crawl is enabled (the default). A token that can query the SQL warehouse but lacks this scope still completes the warehouse crawl; see the behavior note below
* To ingest column masking: the `MANAGE` privilege on, or ownership of, the observed schemas and tables whose ABAC policies must be read. The Unity Catalog policies API is queried on both levels so Euno can discover inherited catalog/schema policies and policies attached directly to a table. Column masking is not ingested for a scope whose policies cannot be read, and the run report identifies the affected scopes.

## Stage 1: Configure Databricks

### Step 1: SQL warehouse and hostname

1. Verify that Unity Catalog is enabled.
2. Ensure at least one SQL warehouse is available and running.
3. Note your workspace hostname (for example `dbc-xxxxxxxx-xxxx.cloud.databricks.com`).
4. Note the SQL warehouse HTTP path (for example `/sql/1.0/warehouses/warehouse-id`).

### Step 2: Personal access token

1. In Databricks, go to **Settings** → **Developer** → **Access tokens**.
2. Click **Generate new token**.
3. Set a comment (for example `Euno integration`).
4. Set token lifetime as needed.
5. Click **Generate**.
6. Copy and store the token securely.

In Euno, the integration combines:

* Unity Catalog metadata (`system.information_schema.*`)
* Databricks system lineage and usage evidence (`system.access.*`, `system.query.history`)
* Workspace notebook discovery (Workspace API)

## Stage 2: Configure New Databricks Source in Euno

### Step 1: Access the Sources Page

1. Open the **Sources** page.
2. Click **Add New Source** and choose **Databricks**.

### Step 2: General Configuration

Asterisk (\*) means a mandatory field.

| Configuration         | Description                                                                           |
| --------------------- | ------------------------------------------------------------------------------------- |
| **Server Hostname**\* | Databricks workspace hostname (for example `dbc-xxxxxxxx-xxxx.cloud.databricks.com`). |
| **HTTP Path**\*       | Databricks SQL warehouse HTTP path (for example `/sql/1.0/warehouses/warehouse-id`).  |
| **Access Token**\*    | Databricks personal access token (stored as a secret).                                |
| **Workspace Name**    | Optional display name in Euno. If left blank, the hostname is used.                   |

### Step 3: Schedule

* Enable scheduling.
* Choose one option:
  1. **Weekly** with a specific day and time.
  2. **Hourly** with an interval.

### Step 4: Resource Cleanup

* **Immediate Cleanup**: remove resources not detected in the latest successful run.
* **No Cleanup**: keep resources even if they are no longer detected.

For cleanup semantics, see [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno.md).

### Step 5: Advanced Settings

Open the **Advanced** section to configure optional filters.

| Configuration         | Description                                                                                                                                                                   |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Override Base URI** | Override the hostname used when Euno generates resource URIs. Leave blank to use **Server Hostname**.                                                                         |
| **Database Pattern**  | Allow/deny regular expressions for catalog names. Defaults to include all catalogs; system catalogs such as `system` and `samples` are always excluded from metadata queries. |

**Pattern examples**

* `.*` — include all names that match.
* `production_.*` — include only names starting with `production_`.
* Allow `.*` and deny `test_.*` — include all except names starting with `test_`.

### Step 6: Save

Click **Test & Save** to validate connectivity and save the source.

## Important behavior notes

* Table read usage metrics are emitted for 14/30/60 day windows.
* Databricks read DBU metrics are emitted as `total_read_dbu_14d`, `total_read_dbu_30d`, `total_read_dbu_60d` and `average_read_dbu_14d`, `average_read_dbu_30d`, `average_read_dbu_60d`.
* Column usage metrics are emitted for 14/30/60 day windows.
* Notebook execution evidence and notebook-derived relationships use `system.query.history`, `system.access.table_lineage`, and `system.access.column_lineage` with a 30-day lookback window.
* Regular Databricks column lineage is read from `system.access.column_lineage` with a 30-day lookback; Euno represents it as column-level upstream field relationships after processing (see the discovered-resources page).
* Notebook observation runs by default; scope follows the integration’s discovery rules described in the hint under **Advanced Settings**.
* If the personal access token lacks the Databricks `workspace` scope, notebook discovery does not run. Euno reports a non-fatal warning, does not update notebook metadata, execution evidence, or notebook relationships, and leaves previously observed notebooks in place. Warehouse metadata, table lineage, column lineage, and usage are still collected. Resource cleanup is skipped for that run, so resources this crawl could not list are not removed. Add the `workspace` scope to the token and run again to resume notebook crawl and normal cleanup.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.euno.ai/sources/data-warehouses/databricks-integration.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.