> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/openlineage-integration.md). # OpenLineage Euno's OpenLineage integration enables seamless ingestion of data lineage events from any system that produces [OpenLineage](https://openlineage.io/) compliant events. This integration automatically processes lineage metadata, table information, and tags to build a comprehensive view of your data pipeline dependencies and transformations. ### How It Works The integration follows these steps: 1. **Provides a secure endpoint** Euno generates a unique trigger secret and endpoint URL for receiving OpenLineage events 2. **Accepts OpenLineage Events** The integration accepts both single events and arrays of events in standard OpenLineage format via HTTP POST 3. **Processes Lineage and Metadata** * Extracts table information from input and output datasets * Creates lineage relationships between input and output tables * Observes the transformation job itself as an OpenLineage Job resource on `COMPLETE` events, with a `defines` relationship to its output tables * Processes tags and metadata from dataset facets, and job-level metadata from the `euno_job` job facet * Validates naming conventions for supported data warehouses ### Supported Data Warehouses Currently, the Euno's OpenLineage integration supports: * **Snowflake** - with naming convention: * namespace: `snowflake://org-account` * name: `database.schema.table` * **BigQuery** - with naming convention: * namespace: `bigquery` * name: `project.dataset.table` For detailed naming conventions, see the [OpenLineage Naming Specification](https://openlineage.io/docs/spec/naming). ## Setting up Euno's OpenLineage Integration ### Step 1: Configure New OpenLineage Source in Euno #### Access the Sources Page 1. Navigate to the **Sources** page in the Euno application 2. Click on the **Add New Source** button 3. Select **OpenLineage** from the available integrations ### Step 2: General Configuration 1. **Name**: Enter a descriptive name for your OpenLineage source (e.g., "Data Pipeline Lineage") 2. **Configuration Details**: * OpenLineage integration requires minimal configuration as it's a push-based integration * No schedule configuration is needed since events are pushed in real-time ### Step 3: Resource Retention OpenLineage POSTs are incremental events, not complete snapshots of every known job and dataset. Euno therefore keeps previously observed OpenLineage resources when later POSTs omit them. Automatic source-run cleanup is not available for this integration; use explicit resource invalidation when a job or dataset is intentionally retired. ### Step 4: Save Configuration Click the **Save** button, and Euno will generate a trigger secret. **Copy and save this secret securely** as it will not be displayed again. ### Step 5: Get the Upload Endpoint 1. Click **"Reset Trigger Key"** to get the endpoint URL 2. Copy the provided endpoint URL where you'll send OpenLineage events 3. Use the trigger secret from Step 4 as the Bearer token in your Authorization header ## Sending OpenLineage Events ### Example: Single Event with Lineage and Tags Here's a complete example of an OpenLineage event with input/output lineage and tags: ```json { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": { "runId": "my-etl-run-12345" }, "job": { "namespace": "production-pipeline", "name": "customer-analytics-etl" }, "inputs": [ { "namespace": "snowflake://myorg-account123", "name": "raw_data.public.customer_events" }, { "namespace": "bigquery", "name": "external_data.staging.product_catalog" } ], "outputs": [ { "namespace": "snowflake://myorg-account123", "name": "analytics.public.customer_analytics", "facets": { "tags": [ { "key": "environment", "value": "production" }, { "key": "team", "value": "data-engineering" }, { "key": "contains_pii" }, { "key": "data_classification", "value": "sensitive" } ], "schema": { "fields": [ { "name": "customer_id", "type": "BIGINT", "description": "Unique customer identifier" }, { "name": "total_purchases", "type": "DECIMAL(10,2)", "description": "Total purchase amount" } ] } } } ], "producer": "https://my-etl-system.com/v1.2.0", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } ``` ### cURL Command Examples #### Single Event Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '{ "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "simple-etl-123"}, "job": {"namespace": "my-pipeline", "name": "daily-aggregation"}, "inputs": [{ "namespace": "snowflake://myorg-account123", "name": "raw.public.events" }], "outputs": [{ "namespace": "snowflake://myorg-account123", "name": "analytics.public.daily_stats", "facets": { "tags": [ {"key": "environment", "value": "prod"}, {"key": "automated"} ] } }], "producer": "my-pipeline-v1.0", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" }' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Multiple Events Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '[ { "eventType": "START", "eventTime": "2024-01-15T10:00:00.001Z", "run": {"runId": "batch-job-456"}, "job": {"namespace": "etl", "name": "batch-processor"}, "producer": "scheduler-v2.1", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" }, { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:15:00.001Z", "run": {"runId": "batch-job-456"}, "job": {"namespace": "etl", "name": "batch-processor"}, "inputs": [{"namespace": "bigquery", "name": "raw.events.user_actions"}], "outputs": [{"namespace": "bigquery", "name": "processed.analytics.user_metrics"}], "producer": "scheduler-v2.1", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } ]' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Upload from File ```bash # Save your event to a file cat > event.json << 'EOF' { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "file-upload-test"}, "job": {"namespace": "testing", "name": "file-upload"}, "outputs": [{ "namespace": "snowflake://myorg-account123", "name": "test.public.sample_table" }], "producer": "test-script", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } EOF # Upload the file curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d @event.json \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` A `200` response means the run was accepted for asynchronous processing. If Euno returns `503`, retry the same submission with exponential backoff. ## Job Resources and the `euno_job` Facet On every `COMPLETE` event, Euno observes the transformation process itself as an **OpenLineage Job** resource (`openlineage_job`), in addition to the input and output tables: * **Identity**: the job is keyed by the exact, case-sensitive pair (`job.namespace`, `job.name`), not by `run.runId`. Euno hashes an unambiguous encoding of that pair into `openlineage.job.{identity_digest}`, so dots, punctuation, and case remain identity-significant. * **Defines relationship**: the job gets a `defines` relationship to the output table URIs of its `COMPLETE` events. Output tables accumulate across both batched and separate POSTs, including concurrent pushes, so sending one event per request produces the same relationships as sending an array. A POST with no observable outputs leaves the job's previous `defines` untouched. OpenLineage events never remove an earlier output implicitly; retire stale relationships through explicit resource invalidation. * **Shared table ownership**: OpenLineage does not overwrite a table's single-valued `defined_by` property. The job's `defines` relationship and the table's inverse `has definer` relationship preserve exact OpenLineage provenance alongside dbt or other definers. * Events of other types (`START`, `RUNNING`, `FAIL`, etc.) still observe tables and lineage, but do not observe the job and do not set `defines`. ### Job-Level Metadata with the `euno_job` Facet Producers can attach a custom job facet named `euno_job` to send job-level metadata for Euno to ingest. Like every OpenLineage custom facet, it must include these BaseFacet fields: * `_producer`: a URI that identifies the producer implementation. * `_schemaURL`: the corresponding versioned facet schema. Because `euno_job` uses the extensible BaseFacet contract, set this to `https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet`. The Euno-specific fields below are optional and map 1:1 onto Euno properties: | `euno_job` field | Type | Euno property | Fallback when omitted | | ---------------- | ---------------------------------------------------------------------------------- | --------------------- | --------------------------------- | | `name` | string | `name` (display name) | `job.name` | | `subtype` | string | `subtype` | — | | `description` | string | `description` | `documentation.description` facet | | `raw_code` | string | `raw_code` | `sql.query` facet | | `tags` | array of strings | `tags` | — | | `external_links` | array of `{"url": string, "label": string}` | `external_links` | — | | `native_owners` | array of owner objects (`name`, `display_name`, `email`, `id`, `native_user_type`) | `native_owners` | — | The job's identity always comes from `job.namespace` + `job.name` — `euno_job.name` only changes the display name. For a newly observed job, Euno uses `job.name` when the custom facet does not provide a name. That fallback does not replace a custom display name from an earlier event or POST. A later explicit `euno_job.name` does replace the stored display name. Malformed optional fields are skipped without failing the event: the job is still observed with its identity and `defines` relationship, and the skipped fields are reported in the run report. For `tags`, `external_links`, and `native_owners`, valid list entries are retained when sibling entries are malformed, and the run report identifies the dropped zero-based indexes. A native owner must provide at least one non-empty `name`, `display_name`, `email`, or `id`. ### Example: `COMPLETE` Event with the `euno_job` Facet ```json { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "0191f3a2-8c4d-7e11-b2a1-0242ac120002"}, "job": { "namespace": "production-pipeline", "name": "customer-analytics-etl", "facets": { "euno_job": { "_producer": "https://my-etl.example/v1.2.0", "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet", "name": "Customer Analytics ETL", "subtype": "spark_job", "description": "Builds the customer analytics mart from raw events and the product catalog.", "raw_code": "INSERT INTO analytics.public.customer_analytics SELECT ...", "tags": ["pii", "tier-1", "customer-360"], "external_links": [ {"label": "Airflow DAG", "url": "https://airflow.example.com/dags/customer_analytics/grid"}, {"label": "Source code", "url": "https://github.com/acme/data-pipelines/blob/main/jobs/customer_analytics.py"} ], "native_owners": [ {"name": "analytics", "display_name": "Analytics Team", "email": "analytics@acme.com", "id": "team-analytics-001", "native_user_type": "team"} ] } } }, "inputs": [ {"namespace": "snowflake://myorg-account123", "name": "raw_data.public.customer_events"} ], "outputs": [ {"namespace": "snowflake://myorg-account123", "name": "analytics.public.customer_analytics"} ], "producer": "https://my-etl.example/v1.2.0", "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent" } ``` This event observes the `raw_data.public.customer_events` and `analytics.public.customer_analytics` tables, lineage between them, and the OpenLineage Job resource `openlineage.job.fece0f925338e641b174a10f6681ed0579e4b3fb877b6711ce2e88131634b9b2` that defines the output table. ## What Gets Observed in Euno When OpenLineage events are processed, Euno observes: ### Table Resources * **Tables** from both inputs and outputs with properties: * `name`: Table name * `database`: Database name * `schema`: Schema name * `database_technology`: `snowflake` or `bigquery` * `type`: `table` * `meta` : see below * `tags` : see below ### Job Resources * **OpenLineage Jobs** from `COMPLETE` events with properties: * `name`: Display name (from `euno_job.name`, or `job.name`) * `type`: `openlineage_job` * `subtype`, `description`, `raw_code`, `tags`, `external_links`, `native_owners`: from the `euno_job` facet (see above) * `defines`: the output tables of the job's `COMPLETE` events ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ----------------- | ------------ | -------------- | ---------------------------------------------------------------------------------------------------------- | | `openlineage_job` | defines | `table` | Created for observable outputs of `COMPLETE` events; tables expose the inverse `has definer` relationship. | | `table` | has upstream | `table` | Output tables depend on the input tables of the same event. | ### Tags and Metadata * **Tags with values** (e.g., `{"key": "environment", "value": "prod"}`) become **meta properties** * **Tags without values** (e.g., `{"key": "pii"}`) become **simple tags** **Generating a new trigger secret:** If you need to rotate the secret, go to the **Sources** page and click on the three-dot menu next to your OpenLineage source. Select **"Reset Trigger Key"** to generate a new trigger secret and endpoint URL. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.euno.ai/sources/transformation-etl/openlineage-integration.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.