OpenLineage

Euno's OpenLineage integration enables seamless ingestion of data lineage events from any system that produces OpenLineage compliant events. This integration automatically processes lineage metadata, table information, and tags to build a comprehensive view of your data pipeline dependencies and transformations.

How It Works

The integration follows these steps:

1. Provides a secure endpoint Euno generates a unique trigger secret and endpoint URL for receiving OpenLineage events

2. Accepts OpenLineage Events The integration accepts both single events and arrays of events in standard OpenLineage format via HTTP POST

3. Processes Lineage and Metadata

   • Extracts table information from input and output datasets

   • Creates lineage relationships between input and output tables

   • Observes the transformation job itself as an OpenLineage Job resource on COMPLETE events, with a defines relationship to its output tables

   • Processes tags and metadata from dataset facets, and job-level metadata from the euno_job job facet

   • Validates naming conventions for supported data warehouses

Supported Data Warehouses

Currently, the Euno's OpenLineage integration supports:

• Snowflake - with naming convention:

  • namespace: snowflake://org-account

  • name: database.schema.table

• BigQuery - with naming convention:

  • namespace: bigquery

  • name: project.dataset.table

For detailed naming conventions, see the OpenLineage Naming Specification.

Setting up Euno's OpenLineage Integration

Step 1: Configure New OpenLineage Source in Euno

Access the Sources Page

1. Navigate to the Sources page in the Euno application

2. Click on the Add New Source button

3. Select OpenLineage from the available integrations

Step 2: General Configuration

1. Name: Enter a descriptive name for your OpenLineage source (e.g., "Data Pipeline Lineage")

2. Configuration Details:

   • OpenLineage integration requires minimal configuration as it's a push-based integration

   • No schedule configuration is needed since events are pushed in real-time

Step 3: Resource Retention

OpenLineage POSTs are incremental events, not complete snapshots of every known job and dataset. Euno therefore keeps previously observed OpenLineage resources when later POSTs omit them. Automatic source-run cleanup is not available for this integration; use explicit resource invalidation when a job or dataset is intentionally retired.

Step 4: Save Configuration

Click the Save button, and Euno will generate a trigger secret. Copy and save this secret securely as it will not be displayed again.

Step 5: Get the Upload Endpoint

1. Click "Reset Trigger Key" to get the endpoint URL

2. Copy the provided endpoint URL where you'll send OpenLineage events

3. Use the trigger secret from Step 4 as the Bearer token in your Authorization header

Sending OpenLineage Events

Example: Single Event with Lineage and Tags

Here's a complete example of an OpenLineage event with input/output lineage and tags:

{
  "eventType": "COMPLETE",
  "eventTime": "2024-01-15T10:30:00.001Z",
  "run": {
    "runId": "my-etl-run-12345"
  },
  "job": {
    "namespace": "production-pipeline",
    "name": "customer-analytics-etl"
  },
  "inputs": [
    {
      "namespace": "snowflake://myorg-account123",
      "name": "raw_data.public.customer_events"
    },
    {
      "namespace": "bigquery", 
      "name": "external_data.staging.product_catalog"
    }
  ],
  "outputs": [
    {
      "namespace": "snowflake://myorg-account123",
      "name": "analytics.public.customer_analytics",
      "facets": {
        "tags": [
          {
            "key": "environment",
            "value": "production"
          },
          {
            "key": "team", 
            "value": "data-engineering"
          },
          {
            "key": "contains_pii"
          },
          {
            "key": "data_classification",
            "value": "sensitive"
          }
        ],
        "schema": {
          "fields": [
            {
              "name": "customer_id",
              "type": "BIGINT",
              "description": "Unique customer identifier"
            },
            {
              "name": "total_purchases",
              "type": "DECIMAL(10,2)",
              "description": "Total purchase amount"
            }
          ]
        }
      }
    }
  ],
  "producer": "https://my-etl-system.com/v1.2.0",
  "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
}

cURL Command Examples

Single Event Upload

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \
  -d '{
    "eventType": "COMPLETE",
    "eventTime": "2024-01-15T10:30:00.001Z",
    "run": {"runId": "simple-etl-123"},
    "job": {"namespace": "my-pipeline", "name": "daily-aggregation"},
    "inputs": [{
      "namespace": "snowflake://myorg-account123", 
      "name": "raw.public.events"
    }],
    "outputs": [{
      "namespace": "snowflake://myorg-account123",
      "name": "analytics.public.daily_stats",
      "facets": {
        "tags": [
          {"key": "environment", "value": "prod"},
          {"key": "automated"}
        ]
      }
    }],
    "producer": "my-pipeline-v1.0",
    "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
  }' \
  https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run

Multiple Events Upload

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \
  -d '[
    {
      "eventType": "START", 
      "eventTime": "2024-01-15T10:00:00.001Z",
      "run": {"runId": "batch-job-456"},
      "job": {"namespace": "etl", "name": "batch-processor"},
      "producer": "scheduler-v2.1",
      "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
    },
    {
      "eventType": "COMPLETE",
      "eventTime": "2024-01-15T10:15:00.001Z", 
      "run": {"runId": "batch-job-456"},
      "job": {"namespace": "etl", "name": "batch-processor"},
      "inputs": [{"namespace": "bigquery", "name": "raw.events.user_actions"}],
      "outputs": [{"namespace": "bigquery", "name": "processed.analytics.user_metrics"}],
      "producer": "scheduler-v2.1", 
      "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
    }
  ]' \
  https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run

Upload from File

# Save your event to a file
cat > event.json << 'EOF'
{
  "eventType": "COMPLETE",
  "eventTime": "2024-01-15T10:30:00.001Z",
  "run": {"runId": "file-upload-test"},
  "job": {"namespace": "testing", "name": "file-upload"},
  "outputs": [{
    "namespace": "snowflake://myorg-account123",
    "name": "test.public.sample_table"
  }],
  "producer": "test-script", 
  "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
}
EOF

# Upload the file
curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \
  -d @event.json \
  https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run

A 200 response means the run was accepted for asynchronous processing. If Euno returns 503, retry the same submission with exponential backoff.

Job Resources and the euno_job Facet

On every COMPLETE event, Euno observes the transformation process itself as an OpenLineage Job resource (openlineage_job), in addition to the input and output tables:

• Identity: the job is keyed by the exact, case-sensitive pair (job.namespace, job.name), not by run.runId. Euno hashes an unambiguous encoding of that pair into openlineage.job.{identity_digest}, so dots, punctuation, and case remain identity-significant.

• Defines relationship: the job gets a defines relationship to the output table URIs of its COMPLETE events. Output tables accumulate across both batched and separate POSTs, including concurrent pushes, so sending one event per request produces the same relationships as sending an array. A POST with no observable outputs leaves the job's previous defines untouched. OpenLineage events never remove an earlier output implicitly; retire stale relationships through explicit resource invalidation.

• Shared table ownership: OpenLineage does not overwrite a table's single-valued defined_by property. The job's defines relationship and the table's inverse has definer relationship preserve exact OpenLineage provenance alongside dbt or other definers.

• Events of other types (START, RUNNING, FAIL, etc.) still observe tables and lineage, but do not observe the job and do not set defines.

Job-Level Metadata with the euno_job Facet

Producers can attach a custom job facet named euno_job to send job-level metadata for Euno to ingest. Like every OpenLineage custom facet, it must include these BaseFacet fields:

• _producer: a URI that identifies the producer implementation.

• _schemaURL: the corresponding versioned facet schema. Because euno_job uses the extensible BaseFacet contract, set this to https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet.

The Euno-specific fields below are optional and map 1:1 onto Euno properties:

euno_job field	Type	Euno property	Fallback when omitted
name	string	name (display name)	job.name
subtype	string	subtype	—
description	string	description	documentation.description facet
raw_code	string	raw_code	sql.query facet
tags	array of strings	tags	—
external_links	array of {"url": string, "label": string}	external_links	—
native_owners	array of owner objects (name, display_name, email, id, native_user_type)	native_owners	—

The job's identity always comes from job.namespace + job.name — euno_job.name only changes the display name. For a newly observed job, Euno uses job.name when the custom facet does not provide a name. That fallback does not replace a custom display name from an earlier event or POST. A later explicit euno_job.name does replace the stored display name.

Malformed optional fields are skipped without failing the event: the job is still observed with its identity and defines relationship, and the skipped fields are reported in the run report. For tags, external_links, and native_owners, valid list entries are retained when sibling entries are malformed, and the run report identifies the dropped zero-based indexes. A native owner must provide at least one non-empty name, display_name, email, or id.

Example: COMPLETE Event with the euno_job Facet

{
  "eventType": "COMPLETE",
  "eventTime": "2024-01-15T10:30:00.001Z",
  "run": {"runId": "0191f3a2-8c4d-7e11-b2a1-0242ac120002"},
  "job": {
    "namespace": "production-pipeline",
    "name": "customer-analytics-etl",
    "facets": {
      "euno_job": {
        "_producer": "https://my-etl.example/v1.2.0",
        "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet",
        "name": "Customer Analytics ETL",
        "subtype": "spark_job",
        "description": "Builds the customer analytics mart from raw events and the product catalog.",
        "raw_code": "INSERT INTO analytics.public.customer_analytics SELECT ...",
        "tags": ["pii", "tier-1", "customer-360"],
        "external_links": [
          {"label": "Airflow DAG", "url": "https://airflow.example.com/dags/customer_analytics/grid"},
          {"label": "Source code", "url": "https://github.com/acme/data-pipelines/blob/main/jobs/customer_analytics.py"}
        ],
        "native_owners": [
          {"name": "analytics", "display_name": "Analytics Team", "email": "[e-mail]", "id": "team-analytics-001", "native_user_type": "team"}
        ]
      }
    }
  },
  "inputs": [
    {"namespace": "snowflake://myorg-account123", "name": "raw_data.public.customer_events"}
  ],
  "outputs": [
    {"namespace": "snowflake://myorg-account123", "name": "analytics.public.customer_analytics"}
  ],
  "producer": "https://my-etl.example/v1.2.0",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent"
}

This event observes the raw_data.public.customer_events and analytics.public.customer_analytics tables, lineage between them, and the OpenLineage Job resource openlineage.job.fece0f925338e641b174a10f6681ed0579e4b3fb877b6711ce2e88131634b9b2 that defines the output table.

What Gets Observed in Euno

When OpenLineage events are processed, Euno observes:

Table Resources

• Tables from both inputs and outputs with properties:

  • name: Table name

  • database: Database name

  • schema: Schema name

  • database_technology: snowflake or bigquery

  • type: table

  • meta : see below

  • tags : see below

Job Resources

• OpenLineage Jobs from COMPLETE events with properties:

  • name: Display name (from euno_job.name, or job.name)

  • type: openlineage_job

  • subtype, description, raw_code, tags, external_links, native_owners: from the euno_job facet (see above)

  • defines: the output tables of the job's COMPLETE events

Relationships

Source type(s)	Relationship	Target type(s)	Notes
openlineage_job	defines	table	Created for observable outputs of COMPLETE events; tables expose the inverse has definer relationship.
table	has upstream	table	Output tables depend on the input tables of the same event.

Tags and Metadata

• Tags with values (e.g., {"key": "environment", "value": "prod"}) become meta properties

• Tags without values (e.g., {"key": "pii"}) become simple tags

Generating a new trigger secret: If you need to rotate the secret, go to the Sources page and click on the three-dot menu next to your OpenLineage source. Select "Reset Trigger Key" to generate a new trigger secret and endpoint URL.