Azure Data Factory
Euno's Azure Data Factory (ADF) integration auto-discovers your data factories and resolves the lineage of the tables your pipelines and data flows read and write. It discovers the following resources:
Data factories
Pipelines
Datasets
Mapping Data Flows
Power Query (Wrangling) Data Flows
The warehouse tables and columns your pipelines and data flows read and write
Prerequisites
To connect a source, you need:
A Microsoft Entra (Azure AD) application (service principal) with a client secret. You can enter the secret directly in Euno or select a Secret Handle served by a local agent.
The built-in Reader role at a scope that covers the factories you want Euno to discover — the subscription, a resource group, or an individual factory. A single assignment is enough; Euno lists factories and reads their pipelines, datasets, linked services, and data flows through that one grant.
One Subscription ID per Euno source (you cannot list multiple subscriptions on one source)
For the Secret Handle path, a connected local agent whose host has a managed identity with Key Vault Secrets User on the vault that contains the client secret
Stage 1: Configure Microsoft Entra and Azure RBAC
Step 1: Create an Entra app registration
Open Microsoft Entra and sign in.
In the sidebar, pick App registrations.
Click New registration.
Fill in the name (for example, "Euno ADF").
Under Supported account types, choose Single tenant only (your directory name appears on the option, for example Single tenant only - Contoso). This is the single-tenant choice — older Entra UIs labeled it Accounts in this organizational directory only.
Leave Redirect URI empty.
Click Register.
On the Overview page, copy the Application (client) ID and the Directory (tenant) ID. You will need these when configuring the source in Euno.
In the sidebar, pick Certificates & secrets.
Under Client secrets, click New client secret.
Set an expiry and optional description, then click Add. Copy the secret Value immediately. It is shown only once. Enter it directly in Euno, or store it in Azure Key Vault for the local-agent Secret Handle path.
Step 2: Grant RBAC to the service principal
Assign the built-in Reader role to the app from Step 1 at a scope that covers the factories you want to crawl (subscription, resource group, or individual factory).
Azure scopes nest like this:
Grant Reader at the subscription (simplest), a resource group, or a single factory (most least-privilege). A broader assignment covers everything under it.
Reader
Subscription, resource group, or individual factory
List factories and read their pipelines, datasets, linked services, and data flows
Option A — Azure portal
Open the Azure portal and sign in.
Open the scope where you want to grant access:
Subscription — search for Subscriptions, then open the subscription that contains your factories; or
Resource group — open the resource group that contains the factories; or
Factory — open the specific Azure Data Factory resource.
In the left menu, select Access control (IAM).
Click Add → Add role assignment.
On the Role tab, search for and select Reader, then click Next.
On the Members tab, for Assign access to, select User, group, or service principal.
Click Select members, search for the app name from Step 1 (for example, "Euno ADF"), select it, then click Select.
Click Review + assign, confirm the assignment, then click Review + assign again.
Repeat at a narrower scope (for example, a single factory) only if you did not already grant Reader at a broader scope that includes that factory.
Option B — Azure CLI
Rotate the client secret per your security policy; expired secrets cause crawl failures. If you enter the secret directly, update the source in Euno after rotation. If you use a Secret Handle, update the Key Vault secret instead. A handle without a pinned Key Vault version reads the latest value.
Azure identity and role summary
ADF service principal
Reader on the subscription, resource group, or factory
Authenticates to Microsoft Entra and reads ADF metadata through Azure Resource Manager.
Agent host's managed identity
Key Vault Secrets User on the vault or client-secret entry
Retrieves the service principal's client secret from Key Vault.
The managed identity only retrieves the client secret. It does not replace the ADF service principal, authenticate the ADF crawl, or read factory metadata.
Stage 2: Configure a new Azure Data Factory source in Euno
Step 1: Access the Sources page
Go to the Sources page.
Click Add New Source and select Azure Data Factory from the list of supported platforms.
Step 2: General configuration
Asterisk (*) means a mandatory field.
Name*
A name for your Azure Data Factory source (e.g., "Azure Data Factory")
Tenant ID*
Your Microsoft Entra tenant identifier
Client ID*
The Azure AD application (service principal) client identifier
Client Secret*
The application secret, entered directly or selected from a local-agent Secret Handle.
Subscription ID*
The Azure subscription ID that contains the factories to crawl. Each source crawls one subscription only.
Step 3: Advanced configuration (optional)
In the UI, the two allow/deny blocks are each titled Pattern (distinguished by their subtitles), and runtime parameter overrides appear under JSON Mapping.
Local Agent
Select a local agent to route Microsoft Entra token and Azure Resource Manager requests through its network.
Client Secret
Select the Secret Handle that resolves to the ADF service principal's client secret. Manual entry in the general configuration remains available.
Crawl Data Flows
Whether to discover Mapping Data Flows and Power Query (Wrangling) Data Flows and resolve their lineage. Enabled by default.
Pattern (resource groups)
Allow/deny regular expressions on resource group names. .* matches everything; add a deny pattern to exclude specific resource groups.
Pattern (factory names)
Allow/deny regular expressions on factory names within the filtered resource groups.
Linked Service to Platform Instance
Manual mapping from a Linked Service to the warehouse location prefix Euno uses to match tables, when Azure does not provide enough information automatically (see below).
JSON Mapping (runtime parameter overrides)
Per-pipeline overrides for runtime-only parameters whose defaults aren't representative, as a nested object: { "pipelineName": { "paramName": "value" } }. Keys may be factory-scoped as factoryName/pipelineName when the same pipeline name exists in more than one factory.
Use a client-secret handle from Azure Key Vault
Follow Fetching credentials from your vault (secret handles) to grant the agent host's managed identity access to Key Vault, configure the canonical /etc/euno/secrets.yaml file, mount it, and restart the agent. An ADF entry can be as small as:
In Euno, select the agent under Local Agent, then select adf-client-secret under the advanced Client Secret field. The agent retrieves the value with its managed identity. Euno uses that value with the ADF service principal's tenant ID and client ID, and routes both the Microsoft Entra token request and all Azure Resource Manager metadata requests through the selected agent.
To keep using an inline secret, leave Local Agent unselected and enter the value in the general Client Secret field. You can also select an agent while keeping an inline secret when you need agent routing but do not need Key Vault retrieval.
Supported Linked Services (where tables are found)
AzureSqlDatabase, AzureSqlMI
Automatic
Azure SQL Database / Managed Instance
AzureSqlDW
Automatic
Azure Synapse dedicated SQL pool
SnowflakeV2
Automatic
Snowflake V2 connector only
GoogleBigQueryV2
Automatic
BigQuery V2 connector only
AzureBlobFS
Automatic
ADLS Gen2 (Blob FS) via the storage account URL
AzureBlobStorage
Mapping required
Storage account name is hidden by Azure — see Azure Blob Storage mapping
AzureDatabricksDeltaLake
Mapping required
Dataset does not carry the catalog; supply a prefix such as databricks.<workspace-host>.<catalog> (use the workspace host the same way your Databricks source is identified in Euno)
Snowflake / GoogleBigQuery (V1)
Not supported
Use the V2 connectors. A Linked Service mapping does not enable V1 — Euno only reads table locations from V2 dataset types
AzureDataLakeStoreGen2
Not supported
Distinct from AzureBlobFS
Azure Blob Storage mapping
When a Linked Service uses Azure Blob Storage, Azure hides the storage account name from Euno (and often shows it as ********** in the ADF Linked Service editor). Without that name, Euno cannot connect ADF datasets on that Linked Service to the right place in your data model. Add a one-time mapping under Linked Service to Platform Instance.
Example
Your factory has a Linked Service named SalesBlob that points at the Azure storage account contosoanalytics. In Euno, add:
Linked Service name
SalesBlob
Warehouse URI prefix
azure_storage.contosoanalytics
After you save and re-run the source, Euno can tie table-style datasets that use SalesBlob to that storage account (it adds database, schema, and table names from each dataset). File datasets on the same Linked Service (CSV, Parquet, JSON, and similar) still appear in Euno as ADF datasets, but Euno does not build table-level lineage for them.
If the same Linked Service name exists in more than one factory, scope the key, for example my-factory/SalesBlob.
How to find the storage account name
Do not use the masked Storage account name field on an Azure Blob Storage Linked Service — Azure hides it there. Prefer one of these:
Option A — From an ADLS Gen2 Linked Service in Azure Data Factory Studio (often easiest)
If the factory also has an Azure Data Lake Storage Gen2 Linked Service for the same storage account, the account name appears in the URL (it is not masked the way Blob Storage is).
In the Azure portal, open Data factories and select your factory.
On the factory Overview page, select Launch studio to open Azure Data Factory Studio.
In Studio, open the Manage hub (toolbox icon in the left rail) → Linked services. See Management hub.
Open the Azure Data Lake Storage Gen2 Linked Service (not the Blob Storage one).
Copy the account from the URL field. Example:
https://eunolake.dfs.core.windows.net→ useeunolake.
Only use this when Blob and ADLS Gen2 point at the same account. If they differ, use Option B for the Blob account.
Option B — From the Azure portal Storage accounts list
Open Storage accounts (or search for Storage accounts).
Prefer accounts in the same subscription (and resource group, when possible) as your data factory.
Open a candidate account → Containers (blob containers — not Tables) and confirm it holds the paths your ADF datasets use (for example a
rawcontainer).Copy Name from the account Overview. Guide: Get storage account configuration information.
Option C — From your own docs
Use the short account name your team already records for that Linked Service — not a connection string or key.
Then enter azure_storage. followed by that name in the Euno mapping (as in the example above). For the URL in Option A, that would be azure_storage.eunolake.
Step 4: Schedule
Set how often Euno re-crawls the source to keep discovered resources and lineage current.
Step 5: Resource cleanup
To keep your data relevant and free of outdated resources, Euno provides automatic resource cleanup options. These settings determine when a resource is removed if it is no longer detected by a source run. For a detailed explanation, see Resource cleanup in Euno.
Immediate Cleanup: Remove resources not detected in the most recent successful source integration run.
No Cleanup: Keep all resources indefinitely, even if they are no longer detected.
Step 6: Test and save the configuration
Click Test & Save. Euno resolves the Secret Handle when one is selected, requests an Azure token with the ADF service principal, and lists the factories available through Azure Resource Manager before saving. If the selected agent is disconnected or the handle cannot be resolved, the test fails without falling back to a direct connection.
What Euno discovers
Once configured, the integration discovers, for each factory it can access:
Factories, pipelines, datasets, and data flows — the full inventory, with each pipeline and data flow linked to its parent factory.
Table-level lineage — for every table a pipeline or data flow writes, the source tables it reads. Euno resolves this from Copy activities, Script activities, Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded native SQL queries.
Column-level lineage — where the transformation names its columns (for example a Power Query rename, or a Copy activity with a column mapping), the source column each output column derives from.
Which ADF resource writes a table — the pipeline or data flow that produces a table is linked as its definer, so you can navigate from a table to the ADF resource that writes it.
Warehouse tables and columns discovered from ADF are matched to the same tables from your warehouse integration (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse, Azure Storage), so ADF lineage appears alongside the rest of your graph.
For the full list of discovered resource types, their properties, and their relationships, see Azure Data Factory Integration Discovered Resources.
Understanding the run report
After each crawl, the run report helps you interpret coverage and gaps:
datasets_resolved / datasets_opaque
How many datasets resolved to a warehouse table versus remained unresolved
unresolved_linked_service_types
Linked Service types that blocked warehouse resolution
linked_service_types_seen
Distribution of every Linked Service type observed (supported and not)
unsupported_dataset_types
Non-tabular datasets (DelimitedText, Parquet, and similar) — expected for file-style datasets
mapping_dataflows_total / flowlets_total / power_queries_total
Counts of Mapping Data Flows, Flowlets, and Power Query (Wrangling) Data Flows
dataflow_inline_endpoints_resolved / dataflow_inline_endpoints_unsupported / dataflow_inline_endpoints_unresolved
Inline Mapping Data Flow source/sink endpoints (no Dataset reference)
tier_2_activities_skipped
Activity types seen in pipelines for which Euno does not resolve lineage (Lookup, Azure Function, and similar)
tier1_activities_seen / tier1_activity_counts
Lineage-supported activities encountered (Copy, Script, Execute Pipeline, Execute Data Flow, Execute Wrangling Data Flow, Stored Procedure) versus those that produced resolved lineage, by type
unresolved_parameter_references
Design-time expressions that could not be evaluated (system variables, missing defaults, unsupported functions)
pipelines_with_lineage / pipelines_with_no_lineage
Pipelines that produced at least one lineage outcome (Copy, Script, Execute*, or Stored Procedure note) versus none
copy_activities_total / copy_activities_with_column_mappings / copy_columns_with_lineage
Copy column-lineage coverage (activities seen, activities with mappings, columns emitted)
scripts_with_lineage / script_columns_with_lineage / scripts_without_destination / scripts_unsupported_linked_service
Script activity SQL analysis outcomes
eunolizer_errors_total
Script or Power Query native SQL that Euno could not analyze
power_query_source_edges / power_query_sinks_without_source / power_query_columns_with_lineage
Power Query table and column lineage coverage
High datasets_opaque with Blob Storage or Databricks Delta Lake types usually means adding a Linked Service to Platform Instance mapping. High tier_2_activities_skipped means those activity types are inventory-only today.
Troubleshooting
Test & Save reports that the agent is disconnected
Start the selected agent and confirm that its status is Connected under Account Settings → Local Agents. ADF does not bypass a selected agent when it is unavailable.
Secret resolution returns provider_auth_failed or provider_denied
Check that the agent host's managed identity has Key Vault Secrets User and that the configured secret name exists. This role belongs to the managed identity, not the ADF service principal.
Microsoft Entra authentication fails
Check the tenant ID, client ID, and current service-principal client secret. For a Secret Handle, confirm that the Key Vault value matches the active client secret.
A crawl succeeds but discovers no factories
Confirm that the ADF service principal has Reader on a factory, its resource group, or the subscription, and that the Subscription ID contains that factory. Azure can return an empty list when no factories are visible. New role assignments can take a few minutes to take effect.
Last updated