STRUCT Support

Overview

Euno provides column-level lineage tracking for BigQuery STRUCT columns, enabling you to trace data flow through nested and complex data structures. This enhancement allows you to understand exactly which nested fields within STRUCT columns are being used, transformed, and propagated throughout your data pipeline.

Capabilities

1. Nested Field Discovery

Euno automatically discovers and catalogs all nested fields within STRUCT columns, creating individual column resources for each field path.

Example Table Structure:

CREATE TABLE users (
  user_id INT64,
  profile STRUCT<
    name STRING,
    email STRING,
    street STRING,
    city STRING,
    zip STRING
  >
)

What You See in Euno:

In the Data Model screen, you'll see these columns:

• user_id (INT64)

• profile (STRUCT) - marked with subtype: struct

• profile.name (STRING)

• profile.email (STRING)

• profile.street (STRING)

• profile.city (STRING)

• profile.zip (STRING)

Each nested field appears as a separate, searchable column resource with its own URI, allowing you to track it independently.

2. Column-Level Lineage

When queries reference specific fields within STRUCT columns, Euno traces lineage to the exact nested field, not just the parent STRUCT.

Example Query:

SELECT 
  user_id,
  profile.name AS user_name,
  profile.city AS user_city
FROM users

Lineage Tracking:

• user_name column traces back to users.profile.name (not just users.profile)

3. Accurate Usage Metrics

Column-level usage statistics now include nested STRUCT fields, showing you:

• How many queries reference each nested field

• Usage frequency for each field path

Example Usage Report:

Column	Total Queries	Distinct Users
profile.name	1,247	23
profile.email	892	18
profile.city	456	12
profile.street	89	5
profile.zip	67	4

This data helps you:

• Optimize schemas: Identify rarely-used nested fields that could be removed

• Understand access patterns: See which parts of your STRUCT columns are most valuable

• Plan migrations: Know which fields are critical before restructuring

4. Lineage Through Complex Transformations

Euno tracks nested field lineage through complex SQL operations:

Example: Extracting and Transforming Nested Fields

CREATE TABLE customer_summary AS
SELECT 
  user_id,
  profile.name AS customer_name,
  UPPER(profile.city) AS city_upper,
  profile.zip AS postal_code
FROM users
WHERE profile.city IS NOT NULL

Lineage Captured:

• customer_summary.customer_name ← users.profile.name

• customer_summary.city_upper ← users.profile.city

• customer_summary.postal_code ← users.profile.zip

Example: Aggregating Nested Fields

CREATE TABLE city_stats AS
SELECT 
  profile.city,
  COUNT(*) AS user_count,
  COUNT(DISTINCT profile.email) AS unique_emails
FROM users
GROUP BY profile.city

Lineage Captured:

• city_stats.city ← users.profile.city

• city_stats.user_count ← (aggregation, no specific column)

• city_stats.unique_emails ← users.profile.email (with aggregation: COUNT DISTINCT)

Viewing STRUCT Columns in Euno

In the Data Model Screen

1. Navigate to a table with STRUCT columns

2. Click on the Columns tab in the resource sidepane

3. Identify STRUCT columns by the struct badge

4. Expand nested fields to see all field paths

5. Click on any nested field to view its details and lineage

Column Properties

Root STRUCT Column:

• Type: column

• Subtype: struct

• Data Type: STRUCT<...> (full schema definition)

• Normalized Type: object

Nested Field:

• Type: column

• Name: Full path (e.g., profile.city)

• Data Type: The field's actual type (e.g., STRING, INT64)

• Parent Container: The table URI

Lineage View

When viewing lineage for a nested field:

1. Upstream: Shows the source nested field (e.g., source_table.profile.city)

2. Downstream: Shows all columns and resources that depend on this nested field

3. Usage: Shows query count and user count for the field

Limitations

1. ARRAY of STRUCT: Nested paths through UNNEST are supported, but element-level instance/index lineage is not tracked

2. Dynamic Field Access: Fields accessed via dynamic SQL or UDFs may not be captured

3. Complex Unnesting: Some complex UNNEST operations with multiple levels may not fully resolve

4. Cross-Project References: Nested field lineage is most accurate within a single BigQuery project