> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/airflow-integration/airflow-integration-discovered-resources.md). # Airflow Integration Discovered Resources ## Overview The Airflow integration discovers the DAGs, tasks, and datasets (assets in Airflow 3) of an Airflow deployment from the Airflow REST API, and the warehouse databases, schemas, and tables that datasets and supported operators read and write. It discovers the following resources: * Airflow instance (`airflow_instance`) * DAG (`airflow_dag`) * Task (`airflow_task`, subtype is the operator class name) * Dataset / Asset (`airflow_dataset`, subtypes `dataset`, `asset`, or the dataset group name) * Database (`database`) — non-authoritative * Schema (`database_schema`) — non-authoritative * Table (`table`) — non-authoritative Airflow resources use the URI prefix `airflow.`, where `` is the host (and port, if any) of the configured Airflow webserver URL. Warehouse resources use the platform URI prefix of the warehouse they belong to (for example `snowflake.` or `bigquery`), so they match the resources discovered by the warehouse integration. ## Resource hierarchy ``` Airflow Instance ├── DAG │ └── Task └── Dataset / Asset Database └── Schema └── Table ``` ## Airflow instance (`airflow_instance`) The top-level resource representing a single Airflow deployment. All DAGs and datasets discovered from one webserver are children of this instance. | Property | Type | Description | | --------------------- | ------- | ------------------------------------------------------------------- | | `name` | string | Hostname extracted from the Airflow webserver URL. | | `type` | string | Always `airflow_instance`. | | `subtype` | string | Always `airflow_instance`. | | `native_id` | string | Same as `name`; the hostname used as the stable identifier. | | `airflow_base_url` | string | Canonical Airflow webserver URL. | | `airflow_api_version` | string | Detected REST API version (`v1` for Airflow 2, `v2` for Airflow 3). | | `airflow_version` | string | Airflow version string reported by the Airflow version endpoint. | | `is_active` | boolean | `true` when all Airflow health components report a healthy status. | | `external_links` | list | Link to the Airflow UI. | **URI pattern** ``` airflow. ``` Example: `airflow.airflow.acme.com` ## DAG (`airflow_dag`) An Airflow DAG observed from the Airflow REST API. | Property | Type | Description | | -------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | DAG display name, or the DAG ID when no display name is set. | | `type` | string | Always `airflow_dag`. | | `subtype` | string | Always `airflow_dag`. | | `native_id` | string | DAG ID. | | `is_active` | boolean | `true` when the Airflow scheduler still sees the DAG file; independent of `is_paused`. For Airflow 3, this is derived from `is_stale`. | | `description` | string | DAG description (when available). | | `source_path` | string | Relative or absolute path to the DAG file. | | `tags` | list | List of DAG tags. | | `native_owners` | list | List of DAG owners, taken from the DAG's `owners` (owner names only; no email is set). | | `external_links` | list | Link to the DAG in the Airflow UI. | | `airflow_last_run_pass` | boolean | `true` if the most recent DAG run succeeded, `false` if it failed, `null` if unknown. | | `airflow_last_run_id` | string | Run ID of the most recent DAG run. | | `airflow_last_run_duration` | number | Duration of the most recent DAG run in seconds. | | `airflow_last_run_failing_tasks` | list | List of task IDs that failed in the most recent run. | | `airflow_last_successful_run` | timestamp | Timestamp of the most recent successful run. | | `airflow_failure_streak` | number | Number of consecutive failed runs (resets to 0 on success). | | `airflow_total_runs_14d` / `_30d` / `_60d` | number | Total DAG runs in the last 14, 30, or 60 days. | | `airflow_success_rate_14d` / `_30d` / `_60d` | number | Fraction of successful runs (0–1) over the last 14, 30, or 60 days. | | `airflow_avg_duration_14d` / `_30d` / `_60d` | number | Average DAG run duration in seconds over the last 14, 30, or 60 days. | **URI pattern** ``` airflow..dag. ``` Example: `airflow.airflow.acme.com.dag.daily_orders` {% hint style="info" %} Execution metrics (`airflow_last_run_*`, `airflow_last_successful_run`, `airflow_total_runs_*`, `airflow_success_rate_*`, `airflow_avg_duration_*`, `airflow_failure_streak`) are derived from stored DAG run events and require **Observe execution history** to be enabled. {% endhint %} ## Task (`airflow_task`) An individual task within an Airflow DAG. | Property | Type | Description | | ---------------------------------------- | ------ | ---------------------------------------------------------------------------------------- | | `name` | string | Task display name, or the task ID when no display name is set. | | `type` | string | Always `airflow_task`. | | `subtype` | string | Operator class name (e.g. `BashOperator`, `PythonOperator`, `SnowflakeOperator`). | | `native_id` | string | Task ID. | | `parent_container_name` | string | Display name of the parent DAG. | | `airflow_task_avg_duration_14d` / `_30d` | number | Average task instance duration in seconds over the last 14 or 30 days. | | `airflow_task_retry_rate_14d` / `_30d` | number | Fraction of task instances that required at least one retry over the last 14 or 30 days. | **URI pattern** ``` airflow..dag..task. ``` Example: `airflow.airflow.acme.com.dag.daily_orders.task.load_orders` {% hint style="info" %} Task execution metrics (`airflow_task_avg_duration_*`, `airflow_task_retry_rate_*`) are derived from stored task instance events and require **Observe execution history** to be enabled. {% endhint %} ## Dataset / Asset (`airflow_dataset`) An Airflow dataset (Airflow 2) or asset (Airflow 3) observed from the Airflow REST API. Dataset and asset payloads are kept as raw source evidence for lineage resolution. | Property | Type | Description | | ----------- | ------ | ------------------------------------------------------------------------------------------------------------------- | | `name` | string | Dataset name, or the URI string when no explicit name is set. | | `type` | string | Always `airflow_dataset`. | | `subtype` | string | `dataset` for Airflow 2 resources, `asset` for Airflow 3 resources, or the dataset group name when one is assigned. | | `native_id` | string | Dataset URI (Airflow 2) or asset URI (Airflow 3). | **URI pattern** ``` airflow..dataset. ``` Example: `airflow.airflow.acme.com.dataset.s3___acme-data_orders.parquet` Characters other than letters, digits, `-`, `_`, and `.` in the dataset URI are replaced with `_`. {% hint style="info" %} `airflow_dataset` resources are only discovered when **Observe datasets** is enabled. {% endhint %} ## Database (`database`) A warehouse database inferred from an Airflow dataset/asset URI or from a supported SQL/operator input or output target. These observations are non-authoritative and are enriched or overridden by the primary warehouse integration when one is configured. | Property | Type | Description | | --------------------- | ------ | ----------------------------------------------------------------------- | | `name` | string | Database name extracted from the dataset URI or operator target. | | `type` | string | Always `database`. | | `database_technology` | string | Warehouse technology (e.g. `snowflake`, `bigquery`, `databricks`). | | `platform_uri_prefix` | string | Warehouse platform URI prefix (e.g. `snowflake.`, `bigquery`). | **URI pattern** ``` . ``` Example: `snowflake.xy12345.us-east-1.analytics` ## Schema (`database_schema`) A warehouse schema inferred from an Airflow dataset/asset URI or from a supported SQL/operator input or output target. These observations are non-authoritative. | Property | Type | Description | | --------------------- | ------ | -------------------------------------------------------------- | | `name` | string | Schema name extracted from the dataset URI or operator target. | | `type` | string | Always `database_schema`. | | `database_database` | string | Database name used for warehouse hierarchy matching. | | `database_technology` | string | Warehouse technology (e.g. `snowflake`, `bigquery`). | | `platform_uri_prefix` | string | Warehouse platform URI prefix. | **URI pattern** ``` .. ``` Example: `snowflake.xy12345.us-east-1.analytics.public` ## Table (`table`) A warehouse table inferred from an Airflow dataset/asset URI or from a supported SQL/operator input or output target. Temporary tables created inside operator SQL are skipped. These observations are non-authoritative; final lineage and property resolution is handled by global processors. | Property | Type | Description | | --------------------- | ------ | ------------------------------------------------------------------------------- | | `name` | string | Table name extracted from the dataset URI or operator target. | | `type` | string | Always `table`. | | `database_database` | string | Database name. | | `database_schema` | string | Schema name. | | `database_technology` | string | Warehouse technology. | | `platform_uri_prefix` | string | Warehouse platform URI prefix. | | `defined_by` | string | Set to `airflow` when the table is currently defined by an Airflow DAG or task. | **URI pattern** ``` ... ``` Example: `snowflake.xy12345.us-east-1.analytics.public.orders` {% hint style="info" %} When the lineage global processor reprocesses a previous Airflow-defined output and sees that Airflow no longer defines it, stale Airflow-owned output lineage is cleared and the table's upstream lineage is recomputed. {% endhint %} ## Relationships Each row is one relationship edge, written in the canonical direction for its family. Use the predicate in each heading to search for these relationships in EQL. ### Containment (`has parent` / `has child`) | Source type | Relationship | Target type | Notes | | ----------------- | ------------ | ------------------ | --------------------------------------------------------------------------------------------------- | | `airflow_dag` | has parent | `airflow_instance` | | | `airflow_dataset` | has parent | `airflow_instance` | | | `airflow_task` | has parent | `airflow_dag` | | | `database_schema` | has parent | `database` | Non-authoritative warehouse hierarchy emitted from dataset URIs and supported SQL/operator targets. | | `table` | has parent | `database_schema` | Non-authoritative warehouse hierarchy emitted from dataset URIs and supported SQL/operator targets. | ### Lineage (`has upstream` / `has downstream`) | Source type | Relationship | Target type | Notes | | ------------------- | ------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `airflow_dag` | has upstream | `airflow_dataset` | Datasets and assets the DAG consumes or is scheduled on. | | `airflow_dag` | has upstream | `table` | Tables the DAG's tasks read, from dataset metadata (when the dataset URI maps to a warehouse table) and operator SQL. | | `airflow_task` | has upstream | `table` | Tables the task reads, from consumed datasets that map to a warehouse table and from operator SQL. | | `table` | has upstream | `table` | Airflow-defined output tables inherit the upstream warehouse tables of the task (or, when no task-level lineage exists, the DAG) that defines them. | | `salesforce_object` | has upstream | `table` | Warehouse tables read by an Airflow task that loads into the Salesforce object (for example with `SalesforceBulkOperator`). Only the loading task's own reads are used. | ### Task dependencies (`has upstream_task` / `has downstream_task`) | Source type | Relationship | Target type | Notes | | -------------- | ------------------ | -------------- | ---------------------------------------------------------------------------------------------- | | `airflow_task` | has upstream\_task | `airflow_task` | Direct upstream tasks within the same DAG, from Airflow's `downstream_task_ids` task metadata. | ### Definition (`defines` / `has definer`) | Source type | Relationship | Target type | Notes | | ----------------- | ------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------- | | `airflow_dag` | defines | `airflow_dataset` | Datasets and assets the DAG's tasks produce. | | `airflow_dag` | defines | `table` | Tables the DAG's tasks write, from dataset metadata and operator SQL. | | `airflow_dag` | defines | `salesforce_object` | Salesforce objects the DAG's tasks load into. | | `airflow_task` | defines | `table` | Tables the task writes, from produced datasets that map to a warehouse table and from operator SQL. | | `airflow_task` | defines | `salesforce_object` | Salesforce objects the task loads into (e.g. via `SalesforceBulkOperator`). | | `airflow_dataset` | defines | `table` | Only when the dataset or asset URI maps to a supported warehouse table URI (e.g. Snowflake or BigQuery table path). | {% hint style="info" %} Lineage and definition edges to tables are created only for tables that exist in Euno (discovered by a warehouse integration or observed by Airflow). Edges to Salesforce objects require the object to be discovered by a Salesforce integration. {% endhint %} ## Usage in EQL ``` # Tables written by Airflow DAGs type = 'table' AND has definer(type = 'airflow_dag') # Airflow DAGs that read a given warehouse table type = 'airflow_dag' AND has upstream(uri = '') # Tasks that run directly after a given task type = 'airflow_task' AND has upstream_task(uri = '') ``` ## Known limitations * A Salesforce object name that matches objects in more than one Salesforce org is skipped, so no Salesforce edge is created for it. * A Salesforce-loading task whose data comes from upstream tasks (for example through XCom) produces a `defines` edge to the Salesforce object but no upstream tables for it. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.euno.ai/sources/transformation-etl/airflow-integration/airflow-integration-discovered-resources.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.