> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/azure-data-factory/azure-data-factory-discovered-resources.md). # Azure Data Factory Discovered Resources ## Overview The Azure Data Factory integration discovers the following resources: * Data Factory (`adf_factory`) * Pipeline (`adf_pipeline`) * Dataset (`adf_dataset`) * Mapping Data Flow (`adf_dataflow`) * Power Query / Wrangling Data Flow (`adf_power_query`) * Table (`table`) and Column (`column`) — the warehouse resources ADF reads and writes ### Ingested Resources & Properties #### Data Factory (`adf_factory`) The top-level resource representing a single Azure Data Factory. All pipelines, datasets, and data flows discovered from one factory are children of it. | Property | Description | | ----------- | ----------------------------------------------- | | `name` | The data factory name | | `type` | Always `adf_factory` | | `native_id` | The factory's Azure Resource Manager identifier | #### Pipeline (`adf_pipeline`) An ADF pipeline (an orchestration of activities such as Copy, Data Flow, Execute Pipeline, and Stored Procedure). | Property | Description | | ------------------ | ------------------------------------------------ | | `name` | The pipeline name | | `type` | Always `adf_pipeline` | | `native_id` | The pipeline's Azure Resource Manager identifier | | `parent_container` | Parent **`adf_factory`** URI | #### Dataset (`adf_dataset`) An ADF dataset — a named pointer to a warehouse table or file. | Property | Description | | ------------------ | -------------------------------------------------------------------------------------------------- | | `name` | The dataset name | | `type` | Always `adf_dataset` | | `native_id` | The dataset's Azure Resource Manager identifier | | `subtype` | The ADF dataset type (e.g. `AzureSqlTable`, `SnowflakeV2Table`, `AzureDatabricksDeltaLakeDataset`) | | `parent_container` | Parent **`adf_factory`** URI | #### Mapping Data Flow (`adf_dataflow`) An ADF Mapping Data Flow or Flowlet (`subtype`). A Mapping Data Flow resolves the tables its sinks write; Flowlets are inventory only. | Property | Description | | ------------------ | ------------------------------------------------- | | `name` | The data flow name | | `type` | Always `adf_dataflow` | | `subtype` | `MappingDataFlow` or `Flowlet` | | `native_id` | The data flow's Azure Resource Manager identifier | | `parent_container` | Parent **`adf_factory`** URI | #### Power Query / Wrangling Data Flow (`adf_power_query`) An ADF Power Query (Wrangling Data Flow), whose transformations are authored in Power Query M. Euno parses the M to resolve the warehouse tables and columns its queries read and write, including embedded native SQL queries. | Property | Description | | ------------------ | ------------------------------------------------- | | `name` | The Power Query data flow name | | `type` | Always `adf_power_query` | | `subtype` | Always `WranglingDataFlow` | | `native_id` | The data flow's Azure Resource Manager identifier | | `parent_container` | Parent **`adf_factory`** URI | #### Table (`table`) A warehouse table referenced by an ADF dataset, data flow, or activity. When the source resolves to a supported warehouse (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse, Azure Storage), Euno matches it to the same table discovered by your warehouse integration. | Property | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | `name` | The table name | | `type` | Always `table` | | `database_database` | The database the table belongs to | | `database_schema` | The schema the table belongs to | | `database_technology` | The warehouse technology (e.g. `snowflake`, `bigquery`, `databricks`, `mssql`, `azure_synapse`, `azure_storage`) | | `defined_by` | Set to `adf` when the table is currently written (defined) by an ADF pipeline or data flow | #### Column (`column`) A warehouse column discovered to carry column-level lineage. `parent_container` links it to its table. | Property | Description | | ------------------ | ---------------------- | | `name` | The column name | | `type` | Always `column` | | `parent_container` | Parent **`table`** URI | *** ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ----------------- | -------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `adf_factory` | has child | `adf_pipeline`, `adf_dataset`, `adf_dataflow`, `adf_power_query` | Pipelines, datasets, and data flows are direct children of their factory | | `adf_pipeline` | defines | `table` | Tables the pipeline's Copy and Script activities write | | `adf_pipeline` | has upstream | `adf_pipeline`, `adf_dataflow`, `adf_power_query` | The pipelines and data flows the pipeline executes (Execute Pipeline, Execute Data Flow, Execute Wrangling Data Flow) | | `adf_dataflow` | defines | `table` | Sink tables a Mapping Data Flow writes (Flowlets do not resolve lineage) | | `adf_power_query` | defines | `table` | Sink tables a Power Query (Wrangling) Data Flow writes | | `column` | has parent | `table` | Columns are grouped under their table | | `table` | has definer | `adf_pipeline`, `adf_dataflow`, `adf_power_query` | The pipeline or data flow that writes the table (`defined_by = adf`) | | `table` | has upstream | `table` | The source tables an ADF-written table reads, resolved from Copy, Script, Mapping Data Flow, and Power Query lineage (including native SQL) | | `column` | has upstream\_fields | `column` | Source columns a written column derives from when named by the transformation (Copy `translator.mappings` on single-source/single-sink Copies; Script SQL where destination columns are named; Power Query M-named columns and native-query columns) | ### Lineage Euno resolves ADF lineage at two levels so ADF-written tables and columns appear alongside the rest of your lineage: * **Table level** — for each table an ADF pipeline or data flow writes, Euno records the source tables it reads (`has upstream`) and the pipeline or data flow that produces it (`has definer`). Sources are resolved from Copy activities, Script activities (SQL analysis), Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded `Value.NativeQuery` SQL. A transform that joins or appends several sources attributes the written table to all of them. * **Column level** — where the transformation names its output columns, Euno records the source column each output column derives from (`has upstream_fields`). Because warehouse tables and columns from ADF are matched to the same resources from your warehouse integration, a single table can show both its ADF-resolved lineage and any lineage from your warehouse or other tools. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.euno.ai/sources/transformation-etl/azure-data-factory/azure-data-factory-discovered-resources.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.