> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/openlineage-integration.md). # OpenLineage Euno's OpenLineage integration enables seamless ingestion of data lineage events from any system that produces [OpenLineage](https://openlineage.io/) compliant events. This integration automatically processes lineage metadata, table information, and tags to build a comprehensive view of your data pipeline dependencies and transformations. ### How It Works The integration follows these steps: 1. **Provides a secure endpoint** Euno generates a unique trigger secret and endpoint URL for receiving OpenLineage events 2. **Accepts OpenLineage Events** The integration accepts both single events and arrays of events in standard OpenLineage format via HTTP POST 3. **Processes Lineage and Metadata** * Extracts table information from input and output datasets * Creates lineage relationships between input and output tables * Observes the transformation job itself as an OpenLineage Job resource on `COMPLETE` events, with a `defines` relationship to its output tables * Processes tags and metadata from dataset facets, and job-level metadata from the `euno_job` job facet * Validates naming conventions for supported data warehouses ### Supported Data Warehouses Currently, the Euno's OpenLineage integration supports: * **Snowflake** - with naming convention: * namespace: `snowflake://org-account` * name: `database.schema.table` * **BigQuery** - with naming convention: * namespace: `bigquery` * name: `project.dataset.table` For detailed naming conventions, see the [OpenLineage Naming Specification](https://openlineage.io/docs/spec/naming). ## Setting up Euno's OpenLineage Integration ### Step 1: Configure New OpenLineage Source in Euno #### Access the Sources Page 1. Navigate to the **Sources** page in the Euno application 2. Click on the **Add New Source** button 3. Select **OpenLineage** from the available integrations ### Step 2: General Configuration 1. **Name**: Enter a descriptive name for your OpenLineage source (e.g., "Data Pipeline Lineage") 2. **Configuration Details**: * OpenLineage integration requires minimal configuration as it's a push-based integration * No schedule configuration is needed since events are pushed in real-time ### Step 3: Resource Retention OpenLineage POSTs are incremental events, not complete snapshots of every known job and dataset. Euno therefore keeps previously observed OpenLineage resources when later POSTs omit them: the resource cleanup options shown for the source do not apply to OpenLineage runs. Use explicit resource invalidation when a job or dataset is intentionally retired. Job-to-output relationships have a separate 60-day window based on the `COMPLETE` event's `eventTime`. Euno removes expired relationships during subsequent processing cycles, even when those cycles contain no new OpenLineage outputs. An idle account is updated on its next processing cycle; the job and table resources themselves remain. ### Step 4: Save Configuration Click the **Save** button, and Euno will generate a trigger secret. **Copy and save this secret securely** as it will not be displayed again. ### Step 5: Get the Upload Endpoint 1. Click **"Reset Trigger Key"** to get the endpoint URL 2. Copy the provided endpoint URL where you'll send OpenLineage events 3. Use the trigger secret from Step 4 as the Bearer token in your Authorization header ## Sending OpenLineage Events ### Example: Single Event with Lineage and Tags Here's a complete example of an OpenLineage event with input/output lineage and tags: ```json { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": { "runId": "my-etl-run-12345" }, "job": { "namespace": "production-pipeline", "name": "customer-analytics-etl" }, "inputs": [ { "namespace": "snowflake://myorg-account123", "name": "raw_data.public.customer_events" }, { "namespace": "bigquery", "name": "external_data.staging.product_catalog" } ], "outputs": [ { "namespace": "snowflake://myorg-account123", "name": "analytics.public.customer_analytics", "facets": { "tags": [ { "key": "environment", "value": "production" }, { "key": "team", "value": "data-engineering" }, { "key": "contains_pii" }, { "key": "data_classification", "value": "sensitive" } ], "schema": { "fields": [ { "name": "customer_id", "type": "BIGINT", "description": "Unique customer identifier" }, { "name": "total_purchases", "type": "DECIMAL(10,2)", "description": "Total purchase amount" } ] } } } ], "producer": "https://my-etl-system.com/v1.2.0", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } ``` ### cURL Command Examples #### Single Event Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '{ "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "simple-etl-123"}, "job": {"namespace": "my-pipeline", "name": "daily-aggregation"}, "inputs": [{ "namespace": "snowflake://myorg-account123", "name": "raw.public.events" }], "outputs": [{ "namespace": "snowflake://myorg-account123", "name": "analytics.public.daily_stats", "facets": { "tags": [ {"key": "environment", "value": "prod"}, {"key": "automated"} ] } }], "producer": "my-pipeline-v1.0", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" }' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Multiple Events Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '[ { "eventType": "START", "eventTime": "2024-01-15T10:00:00.001Z", "run": {"runId": "batch-job-456"}, "job": {"namespace": "etl", "name": "batch-processor"}, "producer": "scheduler-v2.1", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" }, { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:15:00.001Z", "run": {"runId": "batch-job-456"}, "job": {"namespace": "etl", "name": "batch-processor"}, "inputs": [{"namespace": "bigquery", "name": "raw.events.user_actions"}], "outputs": [{"namespace": "bigquery", "name": "processed.analytics.user_metrics"}], "producer": "scheduler-v2.1", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } ]' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Upload from File ```bash # Save your event to a file cat > event.json << 'EOF' { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "file-upload-test"}, "job": {"namespace": "testing", "name": "file-upload"}, "outputs": [{ "namespace": "snowflake://myorg-account123", "name": "test.public.sample_table" }], "producer": "test-script", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } EOF # Upload the file curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d @event.json \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` A `200` response means the run was accepted for asynchronous processing. If Euno returns `503`, retry the same submission with exponential backoff. ## Job Resources and the `euno_job` Facet On every `COMPLETE` event, Euno observes the transformation process itself as an **OpenLineage Job** resource (`openlineage_job`), in addition to the input and output tables: * **Identity**: the job is keyed by the exact, case-sensitive pair (`job.namespace`, `job.name`), not by `run.runId`. Its stable URI is `openlineage.job.{identity_digest}`, using the full SHA-256 digest of the UTF-8 byte-length-prefixed namespace and name. Case, punctuation, and the boundary between namespace and name remain significant, so distinct jobs keep distinct identities. * **Defines relationship**: Euno combines observable output tables across the job's `COMPLETE` events from the last 60 days, including separate and concurrent POSTs. An event reporting A followed by one reporting B produces both relationships after asynchronous processing. Replaying the same event does not duplicate a relationship. The window uses `eventTime`, which must include a timezone; replaying an old event does not make its output recent again. * **No observable outputs**: an empty output list or unsupported dataset contributes no output evidence and does not immediately clear earlier relationships. Outputs remain until their last supporting event falls outside the 60-day window. If all outputs expire, subsequent processing clears the entire `defines` list. * **Shared table ownership**: OpenLineage does not set a table's single-valued `defined_by` property, so a table built by dbt or another integration keeps that definer. The job's `defines` relationship and the table's inverse `has definer` relationship carry the OpenLineage provenance. * Events of other types (`START`, `RUNNING`, `FAIL`, etc.) still observe tables and lineage, but do not observe the job and do not set `defines`. ### Job-Level Metadata with the `euno_job` Facet Producers can attach a custom job facet named `euno_job` to send job-level metadata for Euno to ingest. Like every OpenLineage custom facet, it must include these BaseFacet fields: * `_producer`: a URI that identifies the producer implementation. * `_schemaURL`: the corresponding versioned facet schema. Because `euno_job` uses the extensible BaseFacet contract, set this to `https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet`. The Euno-specific fields below are optional and map 1:1 onto Euno properties: | `euno_job` field | Type | Euno property | Fallback when omitted | | ---------------- | ---------------------------------------------------------------------------------- | --------------------- | --------------------------------- | | `name` | string | `name` (display name) | `job.name` | | `subtype` | string | `subtype` | — | | `description` | string | `description` | `documentation.description` facet | | `raw_code` | string | `raw_code` | `sql.query` facet | | `tags` | array of strings | `tags` | — | | `external_links` | array of `{"url": string, "label": string}` | `external_links` | — | | `native_owners` | array of owner objects (`name`, `display_name`, `email`, `id`, `native_user_type`) | `native_owners` | — | The job's identity always comes from `job.namespace` + `job.name` — `euno_job.name` only changes the display name. Each POST supplies the job's current metadata: when the facet omits `name`, the job is displayed as `job.name`, even if an earlier POST sent a custom display name. Malformed optional fields are skipped without failing the event: the job is still observed with its identity and `defines` relationship, and the skipped fields are reported in the run report. For `tags`, `external_links`, and `native_owners`, valid list entries are retained when sibling entries are malformed, and the run report identifies the dropped zero-based indexes. A native owner must provide at least one non-empty `name`, `display_name`, `email`, or `id`. ### Example: `COMPLETE` Event with the `euno_job` Facet ```json { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "0191f3a2-8c4d-7e11-b2a1-0242ac120002"}, "job": { "namespace": "production-pipeline", "name": "customer-analytics-etl", "facets": { "euno_job": { "_producer": "https://my-etl.example/v1.2.0", "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet", "name": "Customer Analytics ETL", "subtype": "spark_job", "description": "Builds the customer analytics mart from raw events and the product catalog.", "raw_code": "INSERT INTO analytics.public.customer_analytics SELECT ...", "tags": ["pii", "tier-1", "customer-360"], "external_links": [ {"label": "Airflow DAG", "url": "https://airflow.example.com/dags/customer_analytics/grid"}, {"label": "Source code", "url": "https://github.com/acme/data-pipelines/blob/main/jobs/customer_analytics.py"} ], "native_owners": [ {"name": "analytics", "display_name": "Analytics Team", "email": "analytics@acme.com", "id": "team-analytics-001", "native_user_type": "team"} ] } } }, "inputs": [ {"namespace": "snowflake://myorg-account123", "name": "raw_data.public.customer_events"} ], "outputs": [ {"namespace": "snowflake://myorg-account123", "name": "analytics.public.customer_analytics"} ], "producer": "https://my-etl.example/v1.2.0", "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent" } ``` This event observes the `raw_data.public.customer_events` and `analytics.public.customer_analytics` tables, lineage between them, and the OpenLineage Job resource `openlineage.job.fece0f925338e641b174a10f6681ed0579e4b3fb877b6711ce2e88131634b9b2` that defines the output table. The job is displayed as **Customer Analytics ETL**. ## What Gets Observed in Euno When OpenLineage events are processed, Euno observes: ### Table Resources * **Tables** from both inputs and outputs with properties: * `name`: Table name * `database`: Database name * `schema`: Schema name * `database_technology`: `snowflake` or `bigquery` * `type`: `table` * `meta` : see below * `tags` : see below ### Job Resources * **OpenLineage Jobs** from `COMPLETE` events with properties: * `name`: Display name (from `euno_job.name`, or `job.name`) * `type`: `openlineage_job` * `subtype`, `description`, `raw_code`, `tags`, `external_links`, `native_owners`: from the `euno_job` facet (see above) * `defines`: the deduplicated output tables from `COMPLETE` events in the last 60 days, combined across POSTs and updated asynchronously ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ----------------- | ------------ | -------------- | ---------------------------------------------------------------------------------------------------------- | | `openlineage_job` | defines | `table` | Created for observable outputs of `COMPLETE` events; tables expose the inverse `has definer` relationship. | | `table` | has upstream | `table` | Output tables depend on the input tables of the same event. | ### Tags and Metadata * **Tags with values** (e.g., `{"key": "environment", "value": "prod"}`) become **meta properties** * **Tags without values** (e.g., `{"key": "pii"}`) become **simple tags** **Generating a new trigger secret:** If you need to rotate the secret, go to the **Sources** page and click on the three-dot menu next to your OpenLineage source. Select **"Reset Trigger Key"** to generate a new trigger secret and endpoint URL. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.euno.ai/sources/transformation-etl/openlineage-integration.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.