> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/azure-data-factory/azure-data-factory-discovered-resources.md). # Azure Data Factory Discovered Resources ## Overview The Azure Data Factory integration discovers the factories in the configured Azure subscription, their pipelines, datasets, and data flows, and the warehouse tables and columns ADF reads and writes. It discovers the following resources: * Data Factory (`adf_factory`) * Pipeline (`adf_pipeline`) * Dataset (`adf_dataset`, subtype is the ADF dataset type) * Mapping Data Flow (`adf_dataflow`, subtypes `MappingDataFlow`, `Flowlet`) * Power Query / Wrangling Data Flow (`adf_power_query`, subtype `WranglingDataFlow`) * Table (`table`) and Column (`column`) — the warehouse resources ADF reads and writes ADF resources use the URI prefix `adf..`. Warehouse tables and columns use the platform URI prefix of the warehouse the Linked Service points to (for example `snowflake.`, `bigquery`, or `mssql.`), so they match the resources discovered by your warehouse integration. ## Resource hierarchy ``` Data Factory ├── Pipeline ├── Dataset ├── Mapping Data Flow └── Power Query Table └── Column ``` ## Data Factory (`adf_factory`) The top-level resource representing a single Azure Data Factory. All pipelines, datasets, and data flows discovered from one factory are children of it. | Property | Type | Description | | ----------- | ------ | ------------------------------------------------ | | `name` | string | The data factory name. | | `type` | string | Always `adf_factory`. | | `native_id` | string | The factory's Azure Resource Manager identifier. | **URI pattern** ``` adf.. ``` Example: `adf.3f2b1c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d.acme-adf-prod` ## Pipeline (`adf_pipeline`) An ADF pipeline (an orchestration of activities such as Copy, Data Flow, Execute Pipeline, and Stored Procedure). | Property | Type | Description | | ----------------------- | ------ | ------------------------------------------------- | | `name` | string | The pipeline name. | | `type` | string | Always `adf_pipeline`. | | `native_id` | string | The pipeline's Azure Resource Manager identifier. | | `parent_container_name` | string | The name of the parent data factory. | **URI pattern** ``` adf...pipeline. ``` Example: `adf.3f2b1c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d.acme-adf-prod.pipeline.load_orders` ## Dataset (`adf_dataset`) An ADF dataset — a named pointer to a warehouse table or file. | Property | Type | Description | | ----------------------- | ------ | --------------------------------------------------------------------------------------------------- | | `name` | string | The dataset name. | | `type` | string | Always `adf_dataset`. | | `subtype` | string | The ADF dataset type (e.g. `AzureSqlTable`, `SnowflakeV2Table`, `AzureDatabricksDeltaLakeDataset`). | | `native_id` | string | The dataset's Azure Resource Manager identifier. | | `parent_container_name` | string | The name of the parent data factory. | **URI pattern** ``` adf...dataset. ``` Example: `adf.3f2b1c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d.acme-adf-prod.dataset.orders_sql` ## Mapping Data Flow (`adf_dataflow`) An ADF Mapping Data Flow or Flowlet (`subtype`). A Mapping Data Flow resolves the tables its sinks write; Flowlets are inventory only. | Property | Type | Description | | ----------------------- | ------ | -------------------------------------------------- | | `name` | string | The data flow name. | | `type` | string | Always `adf_dataflow`. | | `subtype` | string | `MappingDataFlow` or `Flowlet`. | | `native_id` | string | The data flow's Azure Resource Manager identifier. | | `parent_container_name` | string | The name of the parent data factory. | **URI pattern** ``` adf...dataflow. ``` Example: `adf.3f2b1c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d.acme-adf-prod.dataflow.clean_orders` ## Power Query / Wrangling Data Flow (`adf_power_query`) An ADF Power Query (Wrangling Data Flow), whose transformations are authored in Power Query M. Euno parses the M to resolve the warehouse tables and columns its queries read and write, including embedded native SQL queries. | Property | Type | Description | | ----------------------- | ------ | -------------------------------------------------- | | `name` | string | The Power Query data flow name. | | `type` | string | Always `adf_power_query`. | | `subtype` | string | Always `WranglingDataFlow`. | | `native_id` | string | The data flow's Azure Resource Manager identifier. | | `parent_container_name` | string | The name of the parent data factory. | Power Query data flows share the data flow URI pattern. **URI pattern** ``` adf...dataflow. ``` Example: `adf.3f2b1c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d.acme-adf-prod.dataflow.wrangle_customers` ## Table (`table`) A warehouse table referenced by an ADF dataset, data flow, or activity. When the source resolves to a supported warehouse (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse, Azure Storage), Euno matches it to the same table discovered by your warehouse integration. | Property | Type | Description | | --------------------- | ------ | ----------------------------------------------------------------------------------------------------------------- | | `name` | string | The table name. | | `type` | string | Always `table`. | | `database_database` | string | The database the table belongs to. | | `database_schema` | string | The schema the table belongs to. | | `database_technology` | string | The warehouse technology (e.g. `snowflake`, `bigquery`, `databricks`, `mssql`, `azure_synapse`, `azure_storage`). | | `defined_by` | string | Set to `adf` when the table is currently written (defined) by an ADF pipeline or data flow. | **URI pattern** ``` ... ``` Coordinates the Linked Service does not provide are omitted, so the URI has the same shape as the one your warehouse integration emits. Example: `snowflake.xy12345.us-east-1.analytics.public.orders` ## Column (`column`) A warehouse column discovered to carry column-level lineage. | Property | Type | Description | | --------- | ------ | ---------------- | | `name` | string | The column name. | | `type` | string | Always `column`. | | `subtype` | string | Always `column`. | **URI pattern** ``` .column. ``` Example: `snowflake.xy12345.us-east-1.analytics.public.orders.column.order_id` ## Relationships Each row is one relationship edge, written in the canonical direction for its family. Use the predicate in each heading to search for these relationships in EQL. ### Containment (`has parent` / `has child`) | Source type | Relationship | Target type | Notes | | ----------------- | ------------ | ------------- | -------------------------------------- | | `adf_pipeline` | has parent | `adf_factory` | | | `adf_dataset` | has parent | `adf_factory` | | | `adf_dataflow` | has parent | `adf_factory` | | | `adf_power_query` | has parent | `adf_factory` | | | `column` | has parent | `table` | Columns are grouped under their table. | ### Lineage (`has upstream` / `has downstream`) | Source type | Relationship | Target type | Notes | | -------------- | ------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `adf_pipeline` | has upstream | `adf_pipeline` | Pipelines the pipeline runs with Execute Pipeline activities. | | `adf_pipeline` | has upstream | `adf_dataflow` | Mapping Data Flows the pipeline runs with Execute Data Flow activities. | | `adf_pipeline` | has upstream | `adf_power_query` | Power Query data flows the pipeline runs with Execute Wrangling Data Flow activities. | | `table` | has upstream | `table` | The source tables an ADF-written table reads, resolved from Copy, Script, Mapping Data Flow, and Power Query lineage (including native SQL). | ### Field-level lineage (`has upstream_fields` / `has downstream_fields`) | Source type | Relationship | Target type | Notes | | ----------- | -------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `column` | has upstream\_fields | `column` | Source columns a written column derives from when named by the transformation (Copy `translator.mappings` on single-source/single-sink Copies; Script SQL where destination columns are named; Power Query M-named columns and native-query columns). | ### Definition (`defines` / `has definer`) | Source type | Relationship | Target type | Notes | | ----------------- | ------------ | ----------- | ------------------------------------------------------------------------- | | `adf_pipeline` | defines | `table` | Tables the pipeline's Copy and Script activities write. | | `adf_dataflow` | defines | `table` | Sink tables a Mapping Data Flow writes (Flowlets do not resolve lineage). | | `adf_power_query` | defines | `table` | Sink tables a Power Query (Wrangling) Data Flow writes. | ## Lineage Euno resolves ADF lineage at two levels so ADF-written tables and columns appear alongside the rest of your lineage: * **Table level** — for each table an ADF pipeline or data flow writes, Euno records the source tables it reads (`has upstream`) and the pipeline or data flow that produces it (`has definer`). Sources are resolved from Copy activities, Script activities (SQL analysis), Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded `Value.NativeQuery` SQL. A transform that joins or appends several sources attributes the written table to all of them. * **Column level** — where the transformation names its output columns, Euno records the source column each output column derives from (`has upstream_fields`). Because warehouse tables and columns from ADF are matched to the same resources from your warehouse integration, a single table can show both its ADF-resolved lineage and any lineage from your warehouse or other tools. ## Usage in EQL ``` # Tables written by an ADF pipeline or data flow type = 'table' AND defined_by = 'adf' # Pipelines that run a given Mapping Data Flow type = 'adf_pipeline' AND has upstream(uri = '') ``` ## Known limitations * Stored Procedure activities are opaque writes: ADF does not expose the procedure body, so they produce no table-level lineage. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.euno.ai/sources/transformation-etl/azure-data-factory/azure-data-factory-discovered-resources.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.