> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/azure-data-factory.md).

# Azure Data Factory

Euno's Azure Data Factory (ADF) integration auto-discovers your data factories and resolves the lineage of the tables your pipelines and data flows read and write. It discovers the following resources:

* Data factories
* Pipelines
* Datasets
* Mapping Data Flows
* Power Query (Wrangling) Data Flows
* The warehouse tables and columns your pipelines and data flows read and write

{% hint style="info" %}
Euno reads factory metadata through the Azure Resource Manager API only. It never modifies your factories, and it never runs pipelines.
{% endhint %}

{% hint style="info" %}
This integration supports **Azure Data Factory** only. Azure Synapse Pipelines / Synapse workspaces are not supported.
{% endhint %}

## Prerequisites

To connect a source, you need:

* A **Microsoft Entra (Azure AD) application** (service principal) with a **client secret**
* The built-in **Reader** role at a scope that covers the factories you want Euno to discover — the subscription, a resource group, or an individual factory. A single assignment is enough; Euno lists factories and reads their pipelines, datasets, linked services, and data flows through that one grant.
* One **Subscription ID** per Euno source (you cannot list multiple subscriptions on one source)

## Stage 1: Configure Microsoft Entra and Azure RBAC

### Step 1: Create an Entra app registration

1. Open [Microsoft Entra](https://entra.microsoft.com/) and sign in.
2. In the sidebar, pick **App registrations**.
3. Click **New registration**.
4. Fill in the name (for example, "Euno ADF").
5. Under **Supported account types**, choose **Single tenant only** (your directory name appears on the option, for example **Single tenant only - Contoso**). This is the single-tenant choice — older Entra UIs labeled it **Accounts in this organizational directory only**.
6. Leave **Redirect URI** empty.
7. Click **Register**.
8. On the Overview page, copy the **Application (client) ID** and the **Directory (tenant) ID**. You will need these when configuring the source in Euno.
9. In the sidebar, pick **Certificates & secrets**.
10. Under **Client secrets**, click **New client secret**.
11. Set an expiry and optional description, then click **Add**. Copy the secret **Value** immediately — it is shown only once.

### Step 2: Grant RBAC to the service principal

Assign the built-in **Reader** role to the app from Step 1 at a scope that covers the factories you want to crawl (subscription, resource group, or individual factory).

Azure scopes nest like this:

```
Subscription
  └── Resource group (e.g. rg-analytics)
        └── Data Factory (e.g. adf-prod)
              ├── Pipelines
              ├── Datasets
              └── Data flows
```

Grant **Reader** at the **subscription** (simplest), a **resource group**, or a single **factory** (most least-privilege). A broader assignment covers everything under it.

| Role       | Scope                                                   | Purpose                                                                            |
| ---------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| **Reader** | Subscription, resource group, **or** individual factory | List factories and read their pipelines, datasets, linked services, and data flows |

{% hint style="info" %}
Azure’s ADF-specific built-in role is **Data Factory Contributor** (includes write). Euno only needs read access — use **Reader**, not Contributor. There is no built-in “Data Factory Reader” role.
{% endhint %}

#### Option A — Azure portal

1. Open the [Azure portal](https://portal.azure.com) and sign in.
2. Open the scope where you want to grant access:
   * **Subscription** — search for **Subscriptions**, then open the subscription that contains your factories; or
   * **Resource group** — open the resource group that contains the factories; or
   * **Factory** — open the specific Azure Data Factory resource.
3. In the left menu, select **Access control (IAM)**.
4. Click **Add** → **Add role assignment**.
5. On the **Role** tab, search for and select **Reader**, then click **Next**.
6. On the **Members** tab, for **Assign access to**, select **User, group, or service principal**.
7. Click **Select members**, search for the app name from Step 1 (for example, "Euno ADF"), select it, then click **Select**.
8. Click **Review + assign**, confirm the assignment, then click **Review + assign** again.

{% hint style="info" %}
App registrations do not always appear in the member list until you search by name. If nothing shows up, type the exact app display name from Step 1.
{% endhint %}

Repeat at a narrower scope (for example, a single factory) only if you did not already grant **Reader** at a broader scope that includes that factory.

#### Option B — Azure CLI

```bash
# Reader at subscription scope (lists factories and can read factory children)
az role assignment create \
  --assignee <client-id> \
  --role "Reader" \
  --scope /subscriptions/<subscription-id>

# Or Reader at a single factory when you prefer least privilege
az role assignment create \
  --assignee <client-id> \
  --role "Reader" \
  --scope /subscriptions/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.DataFactory/factories/<factory-name>
```

{% hint style="warning" %}
Rotate the client secret per your security policy; expired secrets cause crawl failures.
{% endhint %}

## Stage 2: Configure a new Azure Data Factory source in Euno

### Step 1: Access the Sources page

1. Go to the **Sources** page.
2. Click **Add New Source** and select **Azure Data Factory** from the list of supported platforms.

### Step 2: General configuration

Asterisk (\*) means a mandatory field.

| Configuration     | Description                                                                                                   |
| ----------------- | ------------------------------------------------------------------------------------------------------------- |
| Name\*            | A name for your Azure Data Factory source (e.g., "Azure Data Factory")                                        |
| Tenant ID\*       | Your Microsoft Entra tenant identifier                                                                        |
| Client ID\*       | The Azure AD application (service principal) client identifier                                                |
| Client Secret\*   | The Azure AD application secret                                                                               |
| Subscription ID\* | The Azure subscription ID that contains the factories to crawl. Each source crawls **one** subscription only. |

{% hint style="info" %}
**Multiple Azure subscriptions?** Create a separate Azure Data Factory source for each subscription. You can reuse the same Entra app registration (tenant ID, client ID, and client secret) if that app has **Reader** on every subscription you configure.
{% endhint %}

### Step 3: Advanced configuration (optional)

In the UI, the two allow/deny blocks are each titled **Pattern** (distinguished by their subtitles), and runtime parameter overrides appear under **JSON Mapping**.

| Configuration                              | Description                                                                                                                                                                                                                                                                        |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Crawl Data Flows                           | Whether to discover Mapping Data Flows and Power Query (Wrangling) Data Flows and resolve their lineage. Enabled by default.                                                                                                                                                       |
| Pattern (resource groups)                  | Allow/deny regular expressions on resource group names. `.*` matches everything; add a deny pattern to exclude specific resource groups.                                                                                                                                           |
| Pattern (factory names)                    | Allow/deny regular expressions on factory names within the filtered resource groups.                                                                                                                                                                                               |
| Linked Service to Platform Instance        | Manual mapping from a Linked Service to the warehouse location prefix Euno uses to match tables, when Azure does not provide enough information automatically (see below).                                                                                                         |
| JSON Mapping (runtime parameter overrides) | Per-pipeline overrides for runtime-only parameters whose defaults aren't representative, as a nested object: `{ "pipelineName": { "paramName": "value" } }`. Keys may be factory-scoped as `factoryName/pipelineName` when the same pipeline name exists in more than one factory. |

{% hint style="info" %}
**When is a Linked Service mapping needed?** For most warehouses, Euno can tell which database or project a dataset points at from the Linked Service alone. You only need a mapping when Azure does not give Euno enough information — most often **Azure Blob Storage** (the storage account name is hidden) or **Azure Databricks Delta Lake** (the dataset does not include the catalog). Enter only the missing location prefix; Euno adds the database, schema, and table from the dataset. Use the Linked Service name to apply the mapping across all factories, or `factoryName/linkedServiceName` when two factories share the same Linked Service name.
{% endhint %}

#### Supported Linked Services (where tables are found)

| Linked Service type                 | Setup                | Notes                                                                                                                                                                                  |
| ----------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AzureSqlDatabase`, `AzureSqlMI`    | Automatic            | Azure SQL Database / Managed Instance                                                                                                                                                  |
| `AzureSqlDW`                        | Automatic            | Azure Synapse dedicated SQL pool                                                                                                                                                       |
| `SnowflakeV2`                       | Automatic            | Snowflake **V2** connector only                                                                                                                                                        |
| `GoogleBigQueryV2`                  | Automatic            | BigQuery **V2** connector only                                                                                                                                                         |
| `AzureBlobFS`                       | Automatic            | ADLS Gen2 (Blob FS) via the storage account URL                                                                                                                                        |
| `AzureBlobStorage`                  | **Mapping required** | Storage account name is hidden by Azure — see [Azure Blob Storage mapping](#azure-blob-storage-mapping)                                                                                |
| `AzureDatabricksDeltaLake`          | **Mapping required** | Dataset does not carry the catalog; supply a prefix such as `databricks.<workspace-host>.<catalog>` (use the workspace host the same way your Databricks source is identified in Euno) |
| `Snowflake` / `GoogleBigQuery` (V1) | Not supported        | Use the V2 connectors. A Linked Service mapping does not enable V1 — Euno only reads table locations from V2 dataset types                                                             |
| `AzureDataLakeStoreGen2`            | Not supported        | Distinct from `AzureBlobFS`                                                                                                                                                            |

#### Azure Blob Storage mapping

When a Linked Service uses **Azure Blob Storage**, Azure hides the storage account name from Euno (and often shows it as `**********` in the ADF Linked Service editor). Without that name, Euno cannot connect ADF datasets on that Linked Service to the right place in your data model. Add a one-time mapping under **Linked Service to Platform Instance**.

**Example**

Your factory has a Linked Service named `SalesBlob` that points at the Azure storage account `contosoanalytics`. In Euno, add:

| Field                | Value                            |
| -------------------- | -------------------------------- |
| Linked Service name  | `SalesBlob`                      |
| Warehouse URI prefix | `azure_storage.contosoanalytics` |

After you save and re-run the source, Euno can tie **table-style** datasets that use `SalesBlob` to that storage account (it adds database, schema, and table names from each dataset). **File** datasets on the same Linked Service (CSV, Parquet, JSON, and similar) still appear in Euno as ADF datasets, but Euno does not build table-level lineage for them.

If the same Linked Service name exists in more than one factory, scope the key, for example `my-factory/SalesBlob`.

**How to find the storage account name**

Do not use the masked **Storage account name** field on an **Azure Blob Storage** Linked Service — Azure hides it there. Prefer one of these:

**Option A — From an ADLS Gen2 Linked Service in Azure Data Factory Studio (often easiest)**

If the factory also has an [Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage) Linked Service for the **same** storage account, the account name appears in the URL (it is not masked the way Blob Storage is).

1. In the [Azure portal](https://portal.azure.com), open [Data factories](https://portal.azure.com/#browse/Microsoft.DataFactory%2FdataFactories) and select your factory.
2. On the factory Overview page, select **Launch studio** to open [Azure Data Factory Studio](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory#create-a-data-factory-in-the-azure-portal).
3. In Studio, open the **Manage** hub (toolbox icon in the left rail) → **Linked services**. See [Management hub](https://learn.microsoft.com/en-us/azure/data-factory/author-management-hub).
4. Open the **Azure Data Lake Storage Gen2** Linked Service (not the Blob Storage one).
5. Copy the account from the **URL** field. Example: `https://eunolake.dfs.core.windows.net` → use `eunolake`.

Only use this when Blob and ADLS Gen2 point at the same account. If they differ, use Option B for the Blob account.

**Option B — From the Azure portal Storage accounts list**

1. Open [Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) (or search for **Storage accounts**).
2. Prefer accounts in the same subscription (and resource group, when possible) as your data factory.
3. Open a candidate account → **Containers** (blob containers — not Tables) and confirm it holds the paths your ADF datasets use (for example a `raw` container).
4. Copy **Name** from the account Overview. Guide: [Get storage account configuration information](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-get-info).

**Option C — From your own docs**

Use the short account name your team already records for that Linked Service — not a connection string or key.

Then enter `azure_storage.` followed by that name in the Euno mapping (as in the example above). For the URL in Option A, that would be `azure_storage.eunolake`.

### Step 4: Schedule

Set how often Euno re-crawls the source to keep discovered resources and lineage current.

### Step 5: Resource cleanup

To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource is removed if it is no longer detected by a source run. For a detailed explanation, see [Resource cleanup in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno.md).

* **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run.
* **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected.

### Step 6: Save configuration

Click **Save** to complete the setup.

## What Euno discovers

Once configured, the integration discovers, for each factory it can access:

* **Factories, pipelines, datasets, and data flows** — the full inventory, with each pipeline and data flow linked to its parent factory.
* **Table-level lineage** — for every table a pipeline or data flow writes, the source tables it reads. Euno resolves this from Copy activities, Script activities, Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded native SQL queries.
* **Column-level lineage** — where the transformation names its columns (for example a Power Query rename, or a Copy activity with a column mapping), the source column each output column derives from.
* **Which ADF resource writes a table** — the pipeline or data flow that produces a table is linked as its **definer**, so you can navigate from a table to the ADF resource that writes it.

Warehouse tables and columns discovered from ADF are matched to the same tables from your warehouse integration (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse, Azure Storage), so ADF lineage appears alongside the rest of your graph.

{% hint style="info" %}
**Current limitations**

* **Mapping Data Flows** — table-level lineage only (no column-level lineage). Flowlets are inventory only.
* **Copy** — column lineage only when the activity has explicit `translator.mappings` **and** a single resolved source and single resolved sink (fan-in/out Copies keep table-level lineage only). Only mapped columns are emitted.
* **Script** — table lineage from analyzed SQL when the Linked Service is a supported SQL warehouse. Column lineage only when the SQL names destination columns (for example `INSERT` with an explicit column list, or CTAS). Bare `INSERT … SELECT`, MERGE, and multi-statement scripts are limited or table-level only.
* **Power Query (Wrangling)** — table and column lineage for supported M connectors (`Snowflake.Databases`, `GoogleBigQuery.Database`, `DatabricksMultiCloud.Catalogs`, `Sql.Database`), sources that combine those connectors with a Linked Service, and `Value.NativeQuery` SQL. Other M connectors may leave a sink without resolved sources.
* **Stored Procedure** — ADF does not expose the procedure body, so Euno cannot resolve the tables it writes. No table or column lineage is produced.
* **No run history** — design-time factory metadata only; pipeline run and activity execution metrics are not collected.
* **Other activities** (Lookup, Databricks Notebook, Azure Function, Web, and similar) — discovered in the pipeline inventory only; Euno does not resolve lineage for them. The run report lists them under `tier_2_activities_skipped`.
* Activities inside **ForEach**, **IfCondition**, **Until**, and **Switch** **are** walked for lineage and attributed to the outer pipeline.
  {% endhint %}

For the full list of discovered resource types, their properties, and their relationships, see [Azure Data Factory Integration Discovered Resources](/sources/transformation-etl/azure-data-factory/azure-data-factory-discovered-resources.md).

## Understanding the run report

After each crawl, the run report helps you interpret coverage and gaps:

| Field                                                                                                                         | Meaning                                                                                                                                                                                          |
| ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `datasets_resolved` / `datasets_opaque`                                                                                       | How many datasets resolved to a warehouse table versus remained unresolved                                                                                                                       |
| `unresolved_linked_service_types`                                                                                             | Linked Service types that blocked warehouse resolution                                                                                                                                           |
| `linked_service_types_seen`                                                                                                   | Distribution of every Linked Service type observed (supported and not)                                                                                                                           |
| `unsupported_dataset_types`                                                                                                   | Non-tabular datasets (DelimitedText, Parquet, and similar) — expected for file-style datasets                                                                                                    |
| `mapping_dataflows_total` / `flowlets_total` / `power_queries_total`                                                          | Counts of Mapping Data Flows, Flowlets, and Power Query (Wrangling) Data Flows                                                                                                                   |
| `dataflow_inline_endpoints_resolved` / `dataflow_inline_endpoints_unsupported` / `dataflow_inline_endpoints_unresolved`       | Inline Mapping Data Flow source/sink endpoints (no Dataset reference)                                                                                                                            |
| `tier_2_activities_skipped`                                                                                                   | Activity types seen in pipelines for which Euno does not resolve lineage (Lookup, Azure Function, and similar)                                                                                   |
| `tier1_activities_seen` / `tier1_activity_counts`                                                                             | Lineage-supported activities encountered (Copy, Script, Execute Pipeline, Execute Data Flow, Execute Wrangling Data Flow, Stored Procedure) versus those that produced resolved lineage, by type |
| `unresolved_parameter_references`                                                                                             | Design-time expressions that could not be evaluated (system variables, missing defaults, unsupported functions)                                                                                  |
| `pipelines_with_lineage` / `pipelines_with_no_lineage`                                                                        | Pipelines that produced at least one lineage outcome (Copy, Script, Execute\*, or Stored Procedure note) versus none                                                                             |
| `copy_activities_total` / `copy_activities_with_column_mappings` / `copy_columns_with_lineage`                                | Copy column-lineage coverage (activities seen, activities with mappings, columns emitted)                                                                                                        |
| `scripts_with_lineage` / `script_columns_with_lineage` / `scripts_without_destination` / `scripts_unsupported_linked_service` | Script activity SQL analysis outcomes                                                                                                                                                            |
| `eunolizer_errors_total`                                                                                                      | Script or Power Query native SQL that Euno could not analyze                                                                                                                                     |
| `power_query_source_edges` / `power_query_sinks_without_source` / `power_query_columns_with_lineage`                          | Power Query table and column lineage coverage                                                                                                                                                    |

{% hint style="info" %}
Mapping Data Flow lineage outcomes (`dataflows_with_lineage`, `dataflows_with_no_lineage`, `unresolved_dataflow_dataset_references`) are computed after the crawl completes, so they appear on a separate run report rather than the crawler report above.
{% endhint %}

High `datasets_opaque` with Blob Storage or Databricks Delta Lake types usually means adding a **Linked Service to Platform Instance** mapping. High `tier_2_activities_skipped` means those activity types are inventory-only today.

If a crawl **succeeds but discovers no factories**, the service principal has no **Reader** assignment on any factory in the configured subscription. Azure returns an empty list rather than an error in this case, so the run will not show an authentication failure. Confirm the role assignment scope and that the **Subscription ID** matches the subscription containing the factories. Note that new role assignments can take a few minutes to take effect.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.euno.ai/sources/transformation-etl/azure-data-factory.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.