For the complete documentation index, see llms.txt. This page is also available as Markdown.

Azure Data Factory Discovered Resources

Overview

The Azure Data Factory integration discovers the following resources:

  • Data Factory (adf_factory)

  • Pipeline (adf_pipeline)

  • Dataset (adf_dataset)

  • Mapping Data Flow (adf_dataflow)

  • Power Query / Wrangling Data Flow (adf_power_query)

  • Table (table) and Column (column) โ€” the warehouse resources ADF reads and writes

Ingested Resources & Properties

Data Factory (adf_factory)

The top-level resource representing a single Azure Data Factory. All pipelines, datasets, and data flows discovered from one factory are children of it.

Property
Description

name

The data factory name

type

Always adf_factory

native_id

The factory's Azure Resource Manager identifier

Pipeline (adf_pipeline)

An ADF pipeline (an orchestration of activities such as Copy, Data Flow, Execute Pipeline, and Stored Procedure).

Property
Description

name

The pipeline name

type

Always adf_pipeline

native_id

The pipeline's Azure Resource Manager identifier

parent_container

Parent adf_factory URI

adf_stored_procedure_writes

The stored procedures the pipeline executes, listed as annotations (a stored procedure is an opaque write, so no table edge is drawn)

Dataset (adf_dataset)

An ADF dataset โ€” a named pointer to a warehouse table or file.

Property
Description

name

The dataset name

type

Always adf_dataset

native_id

The dataset's Azure Resource Manager identifier

parent_container

Parent adf_factory URI

linked_service_name

The Linked Service the dataset connects through

linked_service_type

The Linked Service type (e.g. AzureSqlDatabase, Snowflake, AzureDatabricksDeltaLake)

resolved_warehouse_uri

The canonical warehouse table URI the dataset resolves to, when its Linked Service maps to a supported warehouse

Mapping Data Flow (adf_dataflow)

An ADF Mapping Data Flow or Flowlet (subtype). A Mapping Data Flow resolves the tables its sinks write; Flowlets are inventory only.

Property
Description

name

The data flow name

type

Always adf_dataflow

subtype

MappingDataFlow or Flowlet

native_id

The data flow's Azure Resource Manager identifier

parent_container

Parent adf_factory URI

Power Query / Wrangling Data Flow (adf_power_query)

An ADF Power Query (Wrangling Data Flow), whose transformations are authored in Power Query M. Euno parses the M to resolve the warehouse tables and columns its queries read and write, including embedded native SQL queries.

Property
Description

name

The Power Query data flow name

type

Always adf_power_query

subtype

Always WranglingDataFlow

native_id

The data flow's Azure Resource Manager identifier

parent_container

Parent adf_factory URI

Table (table)

A warehouse table referenced by an ADF dataset, data flow, or activity. Emitted by reference when the source resolves to a supported warehouse (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse), so it merges with the same table discovered by your warehouse integration.

Property
Description

name

The table name

type

Always table

database_database

The database the table belongs to

database_schema

The schema the table belongs to

database_technology

The warehouse technology (e.g. snowflake, bigquery, databricks, mssql, azure_synapse)

defined_by

Set to adf when the table is currently written (defined) by an ADF pipeline or data flow

Column (column)

A warehouse column, emitted by reference to carry column-level lineage. parent_container links it to its table.

Property
Description

name

The column name

type

Always column

parent_container

Parent table URI


Relationships

Source type(s)
Relationship
Target type(s)
Notes

adf_factory

has child

adf_pipeline, adf_dataset, adf_dataflow, adf_power_query

Pipelines, datasets, and data flows are direct children of their factory

adf_pipeline

defines

table

Tables the pipeline's Copy activities write

adf_pipeline

has upstream

adf_pipeline, adf_dataflow, adf_power_query

The pipelines and data flows the pipeline executes (Execute Pipeline, Execute Data Flow, Execute Wrangling Data Flow)

adf_dataflow

defines

table

Sink tables a Mapping Data Flow writes (Flowlets do not resolve lineage)

adf_power_query

defines

table

Sink tables a Power Query (Wrangling) Data Flow writes

column

has parent

table

Columns are grouped under their table

table

has definer

adf_pipeline, adf_dataflow, adf_power_query

The pipeline or data flow that writes the table (defined_by = adf)

table

has upstream

table

The source tables an ADF-written table reads, resolved from Copy, Script, Mapping Data Flow, and Power Query lineage (including native SQL)

column

has upstream_fields

column

The source columns a written column derives from, where the transformation names its columns (Copy column mappings, Script and Power Query column-level lineage)

Lineage

Euno resolves ADF lineage at two levels and expresses both as standard graph relationships, so ADF-written tables and columns appear alongside the rest of your lineage:

  • Table level โ€” for each table an ADF pipeline or data flow writes, Euno records the source tables it reads (has upstream) and the pipeline or data flow that produces it (has definer). Sources are resolved from Copy activities, Script activities (SQL analysis), Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded Value.NativeQuery SQL. A transform that joins or appends several sources attributes the written table to all of them.

  • Column level โ€” where the transformation names its output columns, Euno records the source column each output column derives from (has upstream_fields).

Because the warehouse tables and columns are emitted by reference, they merge with the same resources discovered by your warehouse integration โ€” so a single table shows both its ADF-resolved lineage and any lineage from your warehouse or other tools.

Last updated