# Welcome to Euno
Euno is a metadata management platform that powers governance automation and provides automated context for AI. It maps your existing metadataβ€”stitching together column-level lineage across warehouses, dashboards, and metrics, enriched with field-level usage to highlight what matters, and custom properties with automated rule based labels or tags. Then it activates this metadata: enabling data teams to declutter, cut costs, and automate governance at scale, while surfacing the real-time context AI agents need to understand the data and deliver trusted resultsβ€”every time.
# Getting Started Welcome to Euno! This section will help you get up and running quickly. ## What You'll Learn * **Core Concepts** - Understand how Euno works and what resources are * **Quickstart Guide** - Get your first integration connected in 30 minutes ## Start Here 1. **New to Euno?** β†’ Read [Core Concepts](/getting-started/core-concepts) first 2. **Ready to connect?** β†’ Follow the [Quickstart Guide](/getting-started/quickstart-guide) *** ## Quick Links * [What is Euno?](/) * [Connect Your First Source](/getting-started/quickstart-guide#step-1-connect-your-first-source-10-minutes) * [Explore the Data Model](/getting-started/quickstart-guide#step-2-explore-your-data-model-10-minutes) * [Try the AI Assistant](/getting-started/quickstart-guide#step-3-ask-the-ai-assistant-5-minutes) # Core Concepts Understanding these fundamental concepts will help you get the most out of Euno. *** ## What is a Resource? In Euno, everything in your data stack is represented as a **resource**. A resource can be: * πŸ“Š A database table or view * πŸ“ˆ A dashboard or chart * πŸ”„ A dbt model or transformation * πŸ“ A schema or database * πŸ”’ A column within a table * πŸ“ A Looker Explore or Tableau workbook **Key Point:** Euno discovers resources automatically from your connected data sources. You don't need to manually catalog anything. ### Resource Properties Each resource has properties that describe it: * **Identity Properties:** Name, URI, native ID * **Descriptive Properties:** Description, owner, tags * **Usage Properties:** Query counts, view counts, costs * **Metadata Properties:** Last updated, created date, source *** ## Universal Resource Identifiers (URIs) Every resource in Euno has a unique identifier called a **URI**. ### Why URIs Matter URIs allow Euno to: * Uniquely identify every resource across your entire data stack * Connect resources across different platforms * Track resources even when they're renamed * Build accurate lineage relationships ### URI Structure URIs follow a hierarchical pattern: ``` platform.technology.identifier.path.to.resource ``` **Examples:** **Snowflake Table:** ``` table.snowflake.us_west_2.abc123.analytics_db.public.users ``` **dbt Model:** ``` dbt_model.abc123.my_dbt_project.main.stg_users ``` **Tableau Dashboard:** ``` tableau_workbook.my_server.sales_metrics ``` **Looker Dashboard:** ``` looker_dashboard.my_instance.executive_overview ``` ### URI Components Explained Let's break down a Snowflake table URI: ``` table.snowflake.us_west_2.abc123.analytics_db.public.users β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ └─ Table name β”‚ β”‚ β”‚ β”‚ β”‚ └──────── Schema name β”‚ β”‚ β”‚ β”‚ └────────────────── Database name β”‚ β”‚ β”‚ └──────────────────────────── Account ID β”‚ β”‚ └──────────────────────────────────── Region β”‚ └─────────────────────────────────────────────── Technology └────────────────────────────────────────────────────── Resource type ``` > πŸ’‘ **Pro Tip:** URIs are always lowercase. Euno automatically normalizes names to ensure consistency. For detailed URI formats by resource type, see [URI Structure Deep Dive](/developer-reference/technical-concepts/universal-resource-identifier-uri). *** ## Relationships Between Resources Resources don't exist in isolationβ€”they're connected through **relationships**. ### Types of Relationships Euno tracks three primary relationship types: #### 1. **Dependency** (Lineage) Shows data flow from upstream β†’ downstream. **Example:** ``` raw.users β†’ stg_users β†’ dim_customers β†’ revenue_dashboard ``` * `raw.users` is **upstream of** `stg_users` * `revenue_dashboard` is **downstream of** `dim_customers` **Questions this answers:** * "What tables feed into this dashboard?" * "If I change this table, what breaks?" * "Where does this data come from?" #### 2. **Containment** (Hierarchy) Shows parent-child relationships. **Example:** ``` Database └─ Schema └─ Table └─ Column ``` **Questions this answers:** * "What columns are in this table?" * "What tables are in this schema?" * "What's the full path to this resource?" #### 3. **Defined-by** (Logical Definition) Shows which tool defines a resource's logic. **Example:** ``` dim_customers (table) is defined by dim_customers (dbt model) ``` This means: * The table exists in Snowflake * But its logic is managed by dbt * Changes should be made in dbt, not directly in Snowflake **Questions this answers:** * "Where is this table's transformation logic?" * "Is this table managed by dbt?" * "What's the source of truth for this resource?" ### Visualizing Relationships Euno provides two ways to explore relationships: 1. **Graph View:** Visual representation of lineage 2. **Resource Detail Page:** Lists of upstream/downstream resources For detailed relationship types and examples, see [Relationships Reference](/developer-reference/technical-concepts/relationships). *** ## Column-Level Lineage Euno tracks relationships not just at the table/model level, but at the **column level**. ### Why Column Lineage Matters Understanding column-level lineage helps you: * Trace sensitive data (PII) through your entire pipeline * Understand calculation dependencies * Perform precise impact analysis * Document data transformations ### Example ``` Source Table: raw.orders └─ Column: customer_id Transformation: stg_orders (dbt model) └─ SQL: SELECT customer_id, ... └─ Column: customer_id Final Table: dim_customers └─ Column: customer_key (derived from customer_id) Dashboard: Tableau - Sales Report └─ Field: Customer Key ``` You can see exactly how `customer_id` flows from raw data β†’ staging β†’ dimensional model β†’ BI dashboard. *** ## Resource Sponsorship & Lifecycle Euno needs to know how to handle resources when they're no longer detected in a source integration. ### What is a Sponsor? A **sponsor** is the source integration that discovered a resource. **Example:** * Your dbt integration discovers the `dim_customers` model * dbt is now the **sponsor** of that resource * If dbt stops detecting it, Euno can automatically clean it up ### Cleanup Strategies You can configure how Euno handles resources that disappear: #### **Time-Based Cleanup** (Default) Remove resources that haven't been detected in X days (default: 7 days) **Use when:** You want a grace period for temporary issues #### **Immediate Cleanup** Remove resources as soon as they're not detected **Use when:** You want your catalog to always reflect current state #### **No Cleanup** Keep all resources indefinitely **Use when:** You want to preserve historical resources for auditing For more details, see [Resource Sponsorship & Cleanup](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). *** ## Resource Types Euno supports many resource types across different platforms: * Databases, Schemas, Tables, Views, Columns, etc. from Data Warehouse resources * Models, Metrics, Pipelines, etc. from Semantic Layers & ETL Tools * Dashboards, Reports, Visualizations, Data Sources/Models, etc. from BI Tools *** ## Usage & Performance Metadata Euno automatically collects usage data from your sources: ### Warehouse Usage * **Query Count:** How many times was this queried? * **Query Cost / Runtime:** (For supported integrations) How much did queries cost? * **Storage:** How much space does this use? ### BI Usage * **Impressions:** How many times was this viewed? * **Users:** Who viewed this? * **Last Accessed:** When was this last used? ### Transformation Performance * **Build Time:** How long does this model take to build? * **Build Status:** Did the last run succeed or fail? * **Freshness:** Is the data up to date? ### Why This Matters Usage data helps you: * **Identify unused resources** β†’ Candidates for deprecation * **Find expensive queries** β†’ Optimization opportunities * **Prioritize work** β†’ Focus on high-impact resources * **Understand adoption** β†’ See what people actually use *** ## Active Metadata Management You can enrich resources with custom metadata: ### Fixed Tags Pre-defined tags with specific values: * Owner * Description * Status (Active, Deprecated, etc.) * Certification Level ### Active Tags Live tags that are calculated based on the latest state of your data: * Relies on PII * Certified Dashboard * Usage Level *** ## Euno Query Language (EQL) EQL (Euno Query Language) is a powerful query language for finding and filtering resources in Euno's data model. ### What is EQL? EQL allows you to: * **Filter resources** using specific property values * **Explore relationships** such as dependencies and hierarchies * **Query metrics** and usage patterns * **Build complex queries** by combining conditions with logical operators ### Key EQL Concepts **Resource Filtering:** * Filter by resource type: `type = 'dbt_model'` * Filter by properties: `database_schema = 'sales'` * Combine conditions: `type = 'table' AND database = 'analytics'` **Relationship Queries:** * Find upstream dependencies: `has upstream(type='dbt_source')` * Find downstream resources: `has downstream(type='tableau_dashboard')` * Explore parent-child relationships: `has parent(type='schema')` **Logical Operations:** * AND: `type = 'table' AND database = 'analytics'` * OR: `type = 'dbt_model' OR type = 'dbt_source'` * NOT: `NOT (type = 'looker_look')` ### Example EQL Queries **Find dbt models that depend on specific sources:** ``` type="dbt_model" AND has upstream((type='dbt_source' AND database_schema='raw'), 1) ``` **Find dashboards with no upstream dependencies:** ``` type = 'looker_dashboard' AND NOT has upstream(TRUE) ``` **Find tables with high query costs:** ``` type = 'table' AND total_query_cost_30d > 1000 ``` For complete EQL documentation, see [Understanding EQL](/using-euno/euno-query-language-eql). *** ## Metadata Activation Euno doesn't just catalog metadataβ€”it **activates** it: ### What is Metadata Activation? Using metadata to trigger automated actions: **Examples:** * **Workflow:** Notify #data-team when new ungoverned resources appear * **Data Model Sync:** Auto-update Looker when dbt changes This is what makes Euno more than just a catalogβ€”it's an active governance and automation platform. *** ## Key Takeaways 1. βœ“ **Everything is a Resource** - Tables, dashboards, models, columns, etc. 2. βœ“ **URIs Uniquely Identify Resources** - Across all platforms 3. βœ“ **Relationships Connect Resources** - Dependency, containment, definition 4. βœ“ **Column-Level Lineage** - Track data at the most granular level 5. βœ“ **Usage Data Drives Decisions** - Optimize based on actual behavior 6. βœ“ **Metadata Activation** - Turn passive metadata into automated actions *** ## Next Steps Now that you understand the core concepts: 1. **Try the Quickstart Guide** - See these concepts in action 2. **Explore the Data Model Screen** - Navigate your actual resources 3. **Use the AI Assistant** - Ask questions about your metadata 4. **Set Up Your First Workflow** - Activate your metadata *** ## Related Documentation * [Universal Resource Identifier (URI)](/developer-reference/technical-concepts/universal-resource-identifier-uri) - Detailed URI structure * [Relationships](/developer-reference/technical-concepts/relationships) - Complete relationship reference * [Resource Sponsorship & Cleanup](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno) - Lifecycle management * [Quickstart Guide](/getting-started/quickstart-guide) - Hands-on tutorial # Quickstart Guide Welcome! This guide will help you get started with Euno and see value in your first session. ## What You'll Accomplish By the end of this guide, you will: * βœ“ Connect your first data source to Euno * βœ“ Explore your data model and understand lineage * βœ“ Ask your first question using the AI Assistant **Time Required:** 30 minutes\ **Recommended First Integration:** dbt or DWH (you'll see results fastest) *** ## Before You Begin Make sure you have: * [ ] An Euno account with Admin or Maintainer permissions * [ ] Access credentials for at least one of your data tools (dbt, Snowflake, Tableau, etc.) * [ ] 30 minutes of uninterrupted time *** ## Step 1: Connect Your First Source (10 minutes) ### Choose Your Starting Point We recommend starting with one of these: **Option A: dbt** (Best if you use dbt) * Shows your transformations and logic * Provides immediate value with model relationships * Quick to set up with GitHub or dbt Cloud * [Setup Guides](/sources/transformation-etl) **Option B: DWH** (Best if you don't use dbt) * Maps your entire warehouse structure * Provides usage and cost data * Shows which tables are actually being queried * [Setup Guides](/sources/data-warehouses) **Option C: BI Tool** (Best for understanding BI usage) * Shows what dashboards exist * Reveals which data sources are most used * Identifies optimization opportunities * [Setup Guides](/sources/business-intelligence) ### Quick Setup 1. **Navigate to Sources** * Click "Sources" in the left sidebar * Click "Add New Source" 2. **Select Your Integration** * Choose the platform you want to connect * Click "Configure" 3. **Enter Credentials** * Follow the platform-specific instructions * Test the connection * Click "Save" 4. **Run Your First Sync** * Click "Run Now" to start discovering resources * This may take 5-15 minutes depending on your data stack size * You can continue to Step 2 while it runs > πŸ’‘ **Pro Tip:** While your first source is syncing, read through the [Core Concepts](/getting-started/core-concepts) page to understand how Euno maps your data. *** ## Step 2: Explore Your Data Model (10 minutes) Once your first source has finished syncing, it's time to explore. ### View Your Resources 1. **Open the Data Model Screen** * Click "Data Model" in the sidebar * You'll see a list of all discovered resources 2. **Understand What You're Seeing** * **Resources:** Every table, column, dashboard, model, etc. * **Properties:** Metadata like owner, description, last updated * **Usage Data:** How often each resource is queried or viewed ### Explore a Resource 1. **Click on any resource** to see its detail information 2. **Notice these key sections:** * **Description & Properties:** What is this resource? * **Code/Logic:** How is it defined? (for transformations) * **Lineage:** What does it depend on? * **Usage:** How much is it being used? * **Downstream Impact:** What depends on it? ### Try the Graph View 1. **Click "View Lineage"** from any resource detail page 2. **Explore the graph:** * Blue arrows show data flowing upstream β†’ downstream * Click nodes to expand and see more connections * Use the controls to filter by resource type ### Exercise: Find a Heavily Used Resource 1. Go to Data Model screen 2. Sort by usage (impressions or queries) 3. Click on the top resource 4. View its lineage to understand its data flow *** ## Step 3: Ask the AI Assistant (5 minutes) Let's use natural language to explore your metadata. ### Open the AI Assistant 1. Click "AI Assistant" in the sidebar 2. You'll see a chat interface ### Try These Starter Queries Copy and paste these questions (or create your own): **Understanding Your Data:** ``` Show me the 10 most queried tables in my warehouse ``` **Finding Specific Resources:** ``` What tables are in the [your schema name] schema? ``` **Exploring Relationships:** ``` What dashboards use the [table name] table? ``` **Identifying Issues:** ``` Show me dbt models that don't have any description ``` ### Understanding the Results * The AI Assistant converts your natural language to EQL queries * Results are shown in a table format * You can click any resource to see its detail page * You can ask follow-up questions to dig deeper *** ## Step 4: Identify Your First Opportunity (5 minutes) Now let's find something actionable. ### Find Ungoverned Data Ask the AI Assistant: ``` Show me tables that aren't defined by dbt models ``` If you see results: * These are tables that exist in your warehouse but aren't managed by your transformation tool * They might be: * Legacy tables that can be deprecated * Ad-hoc tables that should be formalized * External tables that need documentation ### Find Unused Resources Ask the AI Assistant: ``` Show me dashboards with zero views in the last 60 days ``` If you see results: * These are candidates for deprecation * Cleaning them up can reduce maintenance burden * Archive them or discuss with owners *** ## Next Steps Congratulations! You've completed the Euno quickstart. Here's what to do next: 1. **Connect More Sources** * Add your other data tools to get a complete picture * [View All Integrations](/sources) 2. **Set Up Your First Workflow** * Get notified when new ungoverned resources appear * [Learn About Workflows](/using-euno/metadata-activation/workflows) 3. **Add Metadata Tags** * Start classifying your resources * [Metadata Tagging Guide](/using-euno/metadata-activation/metadata-tags) 4. **Perform an Impact Analysis** * Understand what would be affected by a change * [Impact Analysis Guide](/using-euno/impact-analysis) 5. **Set Up Data Model Sync** (if you use dbt + Looker) * Keep your BI tool in sync with your models * [Data Model Sync Guide](/using-euno/metadata-activation/data-application-sync) 6. **Explore Advanced EQL Queries** * Build custom queries for your specific needs * [EQL Documentation](/using-euno/euno-query-language-eql) 7. **Enable MCP Integration** * Use Euno directly in Claude, Cursor, or VSCode * [MCP Setup Guide](/ai/mcp-setup) *** ## Common Questions **Q: How often does Euno sync my data sources?**\ A: By default, sources sync daily. You can configure this in the source settings or trigger manual syncs anytime. **Q: Does Euno affect my data warehouse performance?**\ A: No. Euno only reads metadata and query logs. It doesn't query your actual data. **Q: Can I control what resources are discovered?**\ A: Yes. Each source integration has filters to include/exclude specific schemas, databases, or projects. **Q: How do I invite my team?**\ A: Go to Account Management β†’ Team Members β†’ Invite. You can set roles and permissions for each user. **Q: What if I make a mistake?**\ A: Most actions are reversible. You can disconnect sources, delete resources, or reset configurations. However, be careful with bulk deletions. *** ## Need Help? * **In-App Support:** Click the chat icon in the bottom right * **Email:** * **Documentation:** [docs.euno.ai](https://docs.euno.ai) *** ## Give Feedback We'd love to hear about your experience! Let us know: * What worked well? * What was confusing? * What would you like to see in Euno? Email us at # Setup & Configuration Administrative tasks and account setup for Euno. ## Sections ### πŸ‘₯ [User Management](/setup-configuration/user-management) Invite team members, assign personas, and manage access. ### 🎭 [Personas](/setup-configuration/personas) Roles and permissions: built-in roles, initial roles, custom personas, and domain scoping. ### βš™οΈ [Account Settings](/setup-configuration/account-settings) General configuration, notifications, and API keys. ### πŸ“‚ [Domains](/setup-configuration/domains) Logical groupings of resources and metadata tag organization. *** ## Quick Links * [User Management](/setup-configuration/user-management) - Manage users and access * [Personas](/setup-configuration/personas) - Roles, permissions, and persona management * [Account Settings](/setup-configuration/account-settings) - Configure your account * [Domains](/setup-configuration/domains) - Organize resources and metadata tags by domain * [Creating and Managing Sources](/sources) - Source management *** ## Getting Started If you're setting up Euno for the first time: 1. Review [User Management](/setup-configuration/user-management) for user setup 2. Configure [Account Settings](/setup-configuration/account-settings) 3. Connect your first source - see [Getting Started](/getting-started) # User Management User Management lets you invite team members, assign [personas](/setup-configuration/personas), and control who has access to your Euno account. ## Account Admin and Initial Setup Every account must have at least one user with **Account Admin** privileges. Account Admins have full access to all Euno functionality at the account level. The Account Admin persona cannot be updated or deleted. When an account is created, **initial personas** are also set up: Maintainer, Contributor, and Viewer. These are scoped to the `default` [domain](/setup-configuration/domains) and can be assigned to users. You can create additional custom [personas](/setup-configuration/personas) as needed. ## Adding Users To add a new team member: 1. Navigate to **User Menu β†’ Account Settings β†’ User Management** 2. Click **Invite new user** 3. Enter the email address of the user you wish to invite 4. Select one or more **personas** to assign. Choose from built-in personas (Account Admin), initial personas (Maintainer, Contributor, Viewer), or custom personas you have created 5. Click **Add user** to send the invitation The user will receive an email invitation to join Euno. They can log in via OAuth (GitHub, Gmail, or Oktaβ€”contact for Okta setup) or via a magic link. Magic links expire 30 minutes after creation. ## User Permissions Permissions are determined by the **permission set** of the persona (admin, maintainer, contributor, or viewer). See [Personas](/setup-configuration/personas) for the built-in and initial personas. The following permissions are available: | Permission | Description | | ---------------------------- | ------------------------------------------------------------------------ | | manage\_members | Manage member access: invite, remove, and update personas of other users | | list\_stakeholders | View the list of stakeholders and members in the account | | manage\_account\_settings | Update and manage global account settings | | manage\_source\_integrations | Add, remove, and configure source integrations | | view\_sources | View metadata and configuration of connected sources | | trigger\_source\_run | Manually initiate ingestion runs for integrations | | manage\_integrations | Set up and manage integrations with external tools | | view\_resource | View data models, metrics, and metadata in the Data Model screen | | search\_resource | Search across data resources | | create\_custom\_property | Define new custom properties | | edit\_custom\_property | Modify existing custom properties | | delete\_custom\_property | Delete custom properties | | read\_custom\_property | View custom property values (read-only) | | set\_custom\_property | Assign or update custom property values on resources | | view\_dashboards | Access and view dashboards | | view\_targets | View git repositories and dbt projects configured as sync targets | | manage\_targets | Create and configure sync targets | | view\_runs | View history, status, and logs of data model sync runs | | manage\_runs | Initiate, cancel, and manage sync runs | | view\_jobs | View job definitions and execution details | | manage\_jobs | Create, configure, and schedule sync jobs | ## Permission sets | Archtype | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **Admin** (Account Admin, Super Admin) | Full administrative control. Can manage members and personas, configure integrations, and access all functionality. | | **Maintainer** | Comprehensive access to manage all aspect of the account or scoped domains, except managing personas and assigning personas to users. | | **Contributor** | Can manage most of Euno's resources, assign values to manual properties in the scoped domains | | **Viewer** | Read-only access. Can view resources, active tags and their values in the scoped domains | Permissions can be further scoped by [domain](/setup-configuration/domains) when using domain-scoped personas. See [Personas](/setup-configuration/personas) for details. ## Switching a User's Personas To change the personas assigned to a user: 1. Navigate to **User Menu β†’ Account Settings β†’ User Management** 2. Search for the user using the search bar 3. Click on the user's current persona(s) to display a dropdown of available personas 4. Select or deselect personas to update the user's assignments Users can have multiple personas and switch between them using the [User Menu](/setup-configuration/personas#active-persona-and-switching). ## Removing a User from an Account To remove a user: 1. Navigate to **User Menu β†’ Account Settings β†’ User Management** 2. Search for the user 3. Click the vertical ellipsis in the user's row 4. Select **Remove from account** 5. Confirm the action in the dialog

Remove a User from an Account

You cannot remove the last Account Admin from an account. ## Service Accounts Account Admins can create **service accounts** from **User Menu β†’ Account Settings β†’ User Management**. Click **Create service account** (next to **Add user**), provide a name, and confirm. Service accounts are used to generate **Personal Access Tokens (PATs)** for programmatic access to Euno. Tokens inherit the service account's persona and permissions. Grant only the minimum required permissions and store tokens securely. For assistance, contact your Euno account manager. # Domains Domains allow you to logically group resources in your data model based on EQL rules. Each domain defines a subset of resources (e.g., "Sales Data", "PII Assets", "Production Models") and metadata tags are scoped to domains, so you can organize and manage tags separately within each domain. ## What Are Domains? A **domain** is a named grouping of resources defined by one or more EQL conditions. When you create a domain, you specify: * **Name**: A descriptive name (e.g., "Sales Data", "PII Assets") * **Description**: Optional text explaining the domain's purpose * **Defining EQLs**: A list of EQL expressions. A resource belongs to the domain if it satisfies **any** of these EQLs. Each account starts with a **universal** domain that includes all resources (`true`). The universal domain is immutable (you cannot edit or delete it). You can create additional domains to segment your data model. {% hint style="info" %} **Recommendation**: Leave the universal domain as-is so it continues to include all resources. New resources remain covered by universal even when you add more specific domains. {% endhint %} ## The "domains" Property Euno maintains a built-in property called **domains** for every resource. This property is a list of domain names that the resource belongs to. For example, a dbt model in the "Sales" schema might have `domains = ["universal", "Sales Data"]`. You can use the `domains` property in EQL to filter resources by domain membership. See [EQL Examples Using Domains](#eql-examples-using-domains) below. A Resource can belong to any number of domains, or even to no domains. ## Managing Domains ### Creating a Domain 1. Navigate to **User Menu -> Account Settings -> Domains** 2. To create a new domain, click *Create New Domain*\* When creating a domain, you must provide: * **Name**: Unique within the account (e.g., "Sales Data") * **Defining EQLs**: At least one EQL. Each EQL is validated against built-in indexed properties and metadata tags that belong to the **universal** domain (fixed and active tags). Tags from other domains are not allowed in defining EQLs, even if those tags are Shared. Examples: * `database_schema = 'sales'` β€” resources in the sales schema * `type = 'dbt_model' AND 'production' in tags` β€” dbt models tagged as production * `has upstream(type = 'dbt_source')` β€” resources that depend on dbt sources * `my_universal_fixed_tag is true` β€” resources matching a fixed/active tag defined in universal * Optionally, you can provide a **description** ### Listing and Editing Domains You can list all domains for your account, view a domain's details, and update its name, description, or defining EQLs (except for the universal domain). Deleting a domain removes it and its associated metadata tags. ## Examples of Domains | Domain Name | Defining EQL(s) | Purpose | | ------------- | --------------------------------------------------------- | ---------------------------------- | | Sales Data | `database_schema = 'sales'` | Resources in the sales schema | | PII Assets | `contains_pii is true` | Resources marked as containing PII | | Production | `'production' in tags` | Production-tagged resources | | dbt Models | `type = 'dbt_model'` | All dbt models | | BI Dashboards | `type = 'looker_dashboard' OR type = 'tableau_dashboard'` | Looker and Tableau dashboards | ## EQL Examples Using Domains The `domains` property is a list of strings. You can use it in EQL to filter resources by domain membership. For full EQL syntax, see the [Euno Query Language (EQL)](/using-euno/euno-query-language-eql) documentation. **Filter resources in a specific domain:** ``` 'Sales Data' in domains ``` **Filter resources in multiple domains (OR):** ``` 'Sales Data' in domains OR 'PII Assets' in domains ``` **Filter resources that are in Sales Data but not PII:** ``` 'Sales Data' in domains AND NOT ('PII Assets' in domains) ``` **Filter resources in any domain except universal:** ``` domains is not null AND NOT ('universal' in domains) ``` **Combine with other properties:** ``` type = 'dbt_model' AND 'Production' in domains ``` For more EQL examples and operators, see [Example of Full EQL Statements](/using-euno/euno-query-language-eql/example-of-full-eql-statements). ## Metadata Tags and Domains Metadata tags are scoped to domains. When you create a metadata tag, you assign it to a domain. Tags in one domain are separate from tags in another, so you can organize tags by business area or use case. If your account has more than one domain, you will see a domain selector when creating or editing metadata tags. Tags in the **universal** domain may only depend on built-in properties and other tags in universal. Tags in other domains may depend on Shared tags from other domains. Domain-defining EQLs can use universal tags (including fixed tags and coalesce tags built from them). See [Metadata Tags](/using-euno/metadata-activation/metadata-tags) for details. ## Personas and Domains [Personas](/setup-configuration/personas) (roles) can be scoped to specific domains. When a user has a domain-scoped persona, they can only access resources and metadata tags within those domains when using that persona. This lets you align permissions with your domain structure. See [Personas](/setup-configuration/personas) for how to create and manage domain-scoped personas. # Personas Personas in Euno define what a user can see and do. Personas also personalize how Euno's AI features respond, whether through the [AI Assistant](/ai) or when using Euno as the context layer for another agent via [MCP](/ai/mcp-setup). ## What Are Personas? A **persona** is a named role with: * **Name**: A descriptive name (e.g., "Sales Analyst", "Data Engineer") * **Permission Set**: The permission level - maintainer, contributor, or viewer - deciding which Euno functionalities they are authorized to perform * **Domain scope**: Which [domains](/setup-configuration/domains) the persona can access * **AI Instructions**: Custom instructions that can be used to control the AI's behavior; for example, an Engineer persona could get instructions to be verbose about technical details whereas an Analyst persona could get instructions to avoid diving into too many technical details. Personas allow you to tailor access by team or use case. For example, a "Sales Analyst" persona might have contributor permissions scoped to the Sales domain, while a "Data Engineer" persona might have maintainer permissions across all domains. ## Built-in and Initial Personas Every account has the `Account Admin` persona. This is a system managed persona, and cannot be updated or deleted. The `Account Admin` persona has all permissions across all domains. When an account is set up, three additional personas are created and scoped to the universal domain: | Persona | Permission Set | Description | | --------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------- | | **Maintainer** | maintainer | Comprehensive access to contribute to the data model. Can create and manage pull requests, folders, projects, and branches. | | **Contributor** | contributor | Can create change drafts, edit folders and projects, and configure custom properties. Cannot propose or merge pull requests. | | **Viewer** | viewer | Read-only access. Can inspect projects, dashboards, resources, and filters but cannot make modifications. | The Viewer persona is typically set as the default for new users (e.g., via SSO). You can create additional custom personas beyond these initial personas. ## Personas and Domains * A persona has a set of domains which it can operate on * Account Admin and personas have access to all domains regardless of scope If your account has multiple domains, creating domain-scoped personas helps you align permissions with your data organization. See [Domains](/setup-configuration/domains) for how domains are defined and used. ## Managing Personas ### Viewing Personas 1. Navigate to **User Menu β†’ Account Settings β†’ Personas** 2. You will see all personas for the account, including built-in, initial, and custom personas ### Creating a Custom Persona 1. Navigate to **User Menu β†’ Account Settings β†’ Personas** 2. Click **Create New Persona** 3. Provide: * **Name**: Unique within the account (e.g., "Sales Analyst") * **Permission Set**: maintainer, contributor, or viewer * **Domains**: Optional. Select which domains this persona can access. Leave empty for account-wide access * **Description**: Optional text explaining the persona's purpose ### Editing and Deleting Personas You can edit the name, description, and domain scope of custom personas. Built-in personas (Account Admin) and initial personas (Maintainer, Contributor, Viewer) can be edited but not deleted. Custom personas you create can be deleted if no users are assigned to them. ## Active Persona and Switching When you use Euno, you always operate under one of the personas assigned to you. ### Switching Your Active Persona 1. Expand the **User Menu** in the top-right corner (click your avatar or name) 2. Your current active persona is shown when the menu is expanded 3. Click the persona name to open a dropdown of personas you are assigned to 4. Select a different persona to switch; the page will reload with the new permissions ### Default Persona When you have multiple personas, you can set one as your **default**. Your default persona is used when: * You first log in * No active persona is explicitly selected (e.g., from a new session) Set your default persona from **User Menu β†’ Account Settings β†’ Personas** (or **User Management**) by selecting "Set as default" next to the persona you want. ## Assigning Personas to Users Personas are assigned to users in **User Management**. When you invite a user or edit their access, you select one or more personas for them. Users can switch between any personas they are assigned to. See [User Management](/setup-configuration/user-management) for details on inviting users and managing their persona assignments. # Account Settings ## Account Settings Screen The **Account Settings** screen is divided into two primary tabs: **Linked Accounts**: * **Purpose**: Linked accounts enable integration with external platforms for resource ingestion and automations, including metadata import, usage statistics, and lineage information. They also facilitate the Looker data model sync. * **Supported Linked Accounts**: * **GitHub**: For version control and collaboration. * **GitLab**: For managing repositories and CI/CD workflows. * **Jira:** * **Secret key generation**: This screen is a central hub for managing integrations and monitoring account activities, providing essential tools for troubleshooting and automation. # Webhooks ## Overview Euno can send real-time event notifications to your own HTTP endpoints whenever something significant happens in your account β€” for example, when a workflow rule triggers. Webhook delivery is powered by [Svix](https://www.svix.com/), which handles queueing, retries, delivery logs, and signature signing on your behalf. Every Euno account gets its own isolated Svix application, so webhook endpoints and events are never shared across accounts. *** ## Prerequisites * You must have the **Manage Webhooks** permission on your account. * You need an HTTPS endpoint that can receive `POST` requests and return a `2xx` response. {% hint style="info" %} If you don't have a live endpoint yet, you can use [Svix Play](https://play.svix.com/) in the App Portal to generate a temporary test URL. {% endhint %} *** ## Configure a Webhook Endpoint ### Step 1: Open the Webhooks page Navigate to **Account Settings β†’ Webhooks** (`/account?tab=webhooks`). ### Step 2: Open the Svix App Portal Click **Open Webhook Portal**. Euno generates a short-lived magic link and opens the [Svix Consumer App Portal](https://docs.svix.com/app-portal) in a new tab. No separate Svix account is needed. ### Step 3: Add an endpoint Inside the App Portal, click **Add Endpoint** and enter your HTTPS endpoint URL. See [Adding Endpoints in the App Portal](https://docs.svix.com/receiving/using-app-portal/adding-endpoints) for details. You can optionally: * **Filter by event type** β€” select which event types the endpoint should receive. If you leave this blank, the endpoint receives all event types. * **Filter by channel** β€” subscribe to a specific channel to receive only events tagged with it (see [Event Filtering with Channels](#event-filtering-with-channels) below). ### Step 4: Verify delivery After Euno sends the next matching event, use the **Delivery Logs** tab in the App Portal to inspect the request, response, and full payload. You can also manually send a test event from there. *** ## Available Event Types | Event type | When it fires | | -------------------- | ------------------------------------------------------ | | `workflow.triggered` | A workflow rule evaluated and its conditions were met. | {% hint style="info" %} See [Euno Workflows](/using-euno/metadata-activation/workflows) for details on creating and managing workflow rules. {% endhint %} *** ## `workflow.triggered` Payload Every webhook request body is a JSON object. Below is a full example for `workflow.triggered`: ```json { "type": "workflow.triggered", "workflow_rule_id": "550e8400-e29b-41d4-a716-446655440000", "workflow_rule_name": "Undocumented dbt models", "account_id": 42, "trigger_type": "threshold", "results_count": 2, "triggered_at": "2026-05-12T08:30:00+00:00", "result_uris": [ "euno://dbt_model/my_project/my_schema/model_a", "euno://dbt_model/my_project/my_schema/model_b" ], "result_uris_truncated": false } ``` > `result_uris` is capped at 1,000 entries. When the relevant resource count exceeds this, `result_uris_truncated` is `true` and the full list can be found via the Euno UI. | Field | Type | Description | | ----------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | Always `"workflow.triggered"`. | | `workflow_rule_id` | string (UUID) | Unique identifier for the workflow rule. | | `workflow_rule_name` | string | Display name of the workflow rule. | | `account_id` | integer | Euno account ID. | | `trigger_type` | string | Trigger type: `"threshold"`, `"result_diff"`, or `"result_added"`. | | `results_count` | integer | Number of resources represented by the event: all matches for `threshold`, additions for `result_added`, or additions and removals for `result_diff`. | | `triggered_at` | string (ISO 8601) | UTC timestamp when the workflow triggered. | | `result_uris` | array of strings | Euno URIs represented by the event: all matches for `threshold`, additions for `result_added`, or additions and removals for `result_diff` (capped at 1,000). | | `result_uris_truncated` | boolean | `true` when the event represents more than 1,000 resources. | ### Additional fields for `result_diff` triggers For `"result_diff"` events, the payload also labels each change by direction, so consumers can act differently on additions and removals (for example, certify resources that entered the results and revoke certification from resources that left): ```json { "trigger_type": "result_diff", "results_count": 4, "result_uris": [ "euno://dbt_model/my_project/my_schema/model_a", "euno://dbt_model/my_project/my_schema/model_b", "euno://dbt_model/my_project/my_schema/model_c", "euno://dbt_model/my_project/my_schema/model_d" ], "added_count": 3, "removed_count": 1, "added_uris": [ "euno://dbt_model/my_project/my_schema/model_a", "euno://dbt_model/my_project/my_schema/model_b", "euno://dbt_model/my_project/my_schema/model_c" ], "removed_uris": [ "euno://dbt_model/my_project/my_schema/model_d" ] } ``` | Field | Type | Description | | --------------- | ---------------- | --------------------------------------------------------------------------------- | | `added_count` | integer | Number of resources that entered the query results since the last successful run. | | `removed_count` | integer | Number of resources that left the query results since the last successful run. | | `added_uris` | array of strings | URIs that entered the query results, in the current run's order. | | `removed_uris` | array of strings | URIs that left the query results, in the previous run's order. | > `added_uris` and `removed_uris` share the same 1,000-entry cap as `result_uris`: additions are listed first, and after truncation `result_uris` always equals `added_uris` followed by `removed_uris`. The counts are never truncated, and `results_count` equals `added_count + removed_count`. *** ## Event Filtering with Channels Every `workflow.triggered` event is tagged with two [Svix channels](https://docs.svix.com/channels): | Channel | Meaning | | ----------------------------- | ----------------------------------------------- | | `workflows` | All workflow-triggered events for this account. | | `workflow:` | Events from one specific workflow rule. | Channels are **optional** β€” by default an endpoint with no channel filter receives all events. You add a channel filter in the App Portal when adding or editing an endpoint. ### Practical example: subscribe to a single workflow 1. Open the workflow in Euno. The URL looks like: ``` /workflows/550e8400-e29b-41d4-a716-446655440000?accountId=42 ``` 2. Copy the UUID from the URL (`550e8400-e29b-41d4-a716-446655440000`). This is the `workflow_rule_id`. 3. In the App Portal, add or edit your endpoint and set the **Channel** filter to: ``` workflow:550e8400-e29b-41d4-a716-446655440000 ``` 4. The endpoint will now receive events only when that specific workflow triggers. The `workflow_rule_id` is also present in every `workflow.triggered` payload, so you can also filter on the receiving side if you prefer a single catch-all endpoint. {% hint style="info" %} Channel strings are case-sensitive. Always use lowercase. {% endhint %} *** ## Delivery, Retries, and Troubleshooting Svix delivers each webhook to your endpoint and retries automatically on failure using exponential back-off: | Attempt | Delay after previous attempt | | ------- | ---------------------------- | | 1 | Immediately | | 2 | 5 seconds | | 3 | 5 minutes | | 4 | 30 minutes | | 5 | 2 hours | | 6 | 5 hours | | 7 | 10 hours | | 8 | 10 hours | After all attempts fail the message is marked as **Failed**. You can recover failed messages or replay any past message from the **Delivery Logs** tab in the App Portal. See [Svix Retry Schedule](https://docs.svix.com/retries) for full details. {% hint style="warning" %} Your endpoint must respond with a `2xx` HTTP status code within the timeout window. If it returns any other code or times out, Svix treats the delivery as failed and will retry. {% endhint %} *** ## Security Svix signs every outgoing webhook request using an HMAC-SHA256 signature. The signature is sent in the `svix-signature` request header alongside `svix-id` and `svix-timestamp`. You can verify these headers in your receiving code to confirm the request genuinely came from Euno. The signing secret for your endpoint is shown in the App Portal under the endpoint's **Signing Secret** section. See the [Svix signature verification guide](https://docs.svix.com/receiving/verifying-payloads/how) for code examples in Python, Node.js, Go, and other languages. *** ## References * [Svix Consumer App Portal](https://docs.svix.com/app-portal) * [Adding Endpoints in the App Portal](https://docs.svix.com/receiving/using-app-portal/adding-endpoints) * [Svix Event Types](https://docs.svix.com/event-types) * [Svix Channels](https://docs.svix.com/channels) * [Svix Retry Schedule](https://docs.svix.com/retries) * [Verifying Webhook Signatures](https://docs.svix.com/receiving/verifying-payloads/how) # Local Agents A local agent lets Euno run supported source-integration network requests from your environment without opening inbound firewall ports. The agent connects **outbound** to Euno's relay, and Euno routes the selected source's traffic through the agent's encrypted tunnel. {% hint style="info" %} Local agents are currently supported for **MySQL**, **SQL Server**, and **Azure Data Factory** integrations. {% endhint %} ## Step 1: Create a local agent in Euno 1. Navigate to **Account Settings β†’ Local Agents**. 2. Click **+ Create agent**. 3. Enter a **name** for the agent (e.g. `my-private-network-agent`) and optionally a **description** (e.g. `Production network - US East`). 4. Save. On the confirmation screen, copy the **Agent ID** and **Agent Secret**. {% hint style="warning" %} The Agent Secret is shown only once. Store it securely before closing the dialog β€” it cannot be retrieved afterwards. {% endhint %} ## Step 2: Deploy the agent container ### Quick test ```bash docker run -d --restart unless-stopped \ -e AGENT_ID= \ -e AGENT_SECRET= \ eunoai/euno-local-agent:latest ``` ### Persistent setup (recommended) Create a `docker-compose.yml` file on a machine that can reach the required private services, secret stores, or external APIs: ```yaml services: euno-local-agent: image: eunoai/euno-local-agent:latest environment: AGENT_ID: AGENT_SECRET: restart: unless-stopped ``` Then start the agent: ```bash docker compose up -d ``` ### Environment variables | Variable | Required | Description | | -------------- | -------- | ---------------------------------------- | | `AGENT_ID` | Yes | Agent UUID from the Euno Local Agents UI | | `AGENT_SECRET` | Yes | Secret from agent creation (shown once) | ## Verifying the agent image (recommended) Euno publishes the agent image signed, so you can confirm you are running exactly the image Euno built. Pin to an immutable digest instead of the floating `:latest` tag: ```bash # resolve the current digest docker buildx imagetools inspect eunoai/euno-local-agent:latest --format '{{.Manifest.Digest}}' # then reference it by digest in your compose/run # image: eunoai/euno-local-agent@sha256: ``` Save Euno's public signing key as `euno-agent.pub`: ``` -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEtC8el1SduKyqU7Dgcc2crmhHkubI H1gAS7PcerXGYgfgV98B3n9S1QPhGWOxEovBSChyV/+YNfAzTBUaagnn1A== -----END PUBLIC KEY----- ``` Then verify the signature against it: ```bash cosign verify --key euno-agent.pub \ eunoai/euno-local-agent@sha256: ``` A successful verification confirms the image was signed by Euno's key and has not been altered. ## Step 3: Verify the connection 1. Go back to **Account Settings β†’ Local Agents** and click **Refresh**. 2. The agent's status should change from **Disconnected** to **Connected**. You can also verify by checking the container logs: ```bash docker logs # Expected: agent connected to relay ``` ## Step 4: Configure a source to use the agent When creating or editing a supported source, expand the **Advanced** section and select your agent from the **Local agent** dropdown. The **Need a local agent?** link under the credential fields also opens this section. * For **MySQL** and **SQL Server**, set the hostname to the database address that the agent can reach. Euno routes the database connection through the agent. * For **Azure Data Factory**, Euno routes both the Microsoft Entra token request and Azure Resource Manager metadata requests through the agent. If you select a client-secret handle, the agent also resolves that handle. You can still enter the client secret directly. Click **Test & Save**. Euno will validate the connection through the agent before saving. ## Fetching credentials from your vault (secret handles) Instead of typing supported source credentials into Euno, the agent can fetch them from your secret store at connection time. Euno stores only a **handle** (an opaque name you define); the value is fetched through the agent when a connection is tested or a sync runs, used in memory, and never persisted in Euno. {% hint style="info" %} Secret handles require an up-to-date agent image. Pull the latest image and restart the container before following this section; agents on older images keep tunneling normally but cannot serve handles. {% endhint %} Agent images use the full repository Git SHA as their immutable version tag; semantic version tags are not published. The provider-specific troubleshooting codes below require the agent image published from the same Euno release as this documentation or a later release. Older secret-handle images return the generic `cannot_fetch_dynamic_secret` code instead. ### 1. Give the agent host access to your vault The agent authenticates to Azure Key Vault with the host's **managed identity** β€” no credential is ever written into Euno or the agent's configuration. 1. Enable a system-assigned managed identity on the machine running the agent (for a VM: `az vm identity assign -n -g `, or the Identity blade in the portal). Container Apps, AKS, and Arc-enabled servers work through their platform identity equivalents. 2. Grant that identity the **Key Vault Secrets User** role on the vault (or on the individual secrets): ```bash az role assignment create \ --assignee-object-id --assignee-principal-type ServicePrincipal \ --role "Key Vault Secrets User" \ --scope $(az keyvault show -n --query id -o tsv) ``` For vaults using legacy access policies instead of RBAC, an access policy with secret **Get** permission is the equivalent. The vault must be reachable from the agent host; a private endpoint is recommended. ### 2. Create `secrets.yaml` on the agent host ```yaml version: 1 secrets: - handle: sql-prod-password type: azure_kv directions: vault_uri: https://.vault.azure.net/ secret_name: # version: optional; omit to always use the latest (recommended for rotation) ``` * `handle` β€” the name shown in the Euno UI. It is the only thing Euno stores. * `type` β€” use `azure_kv` (Azure Key Vault) in production. The `static` provider is available only for development and testing. * `directions` β€” where the secret lives. No credentials go in this file: the agent authenticates to the vault with the host's **managed identity**, which needs the **Key Vault Secrets User** role on the vault. * The file is an **allowlist**: Euno can only fetch handles listed here. Keep the file owned by root with `chmod 600`, and mount it read-only into the container (add to the `docker-compose.yml` above): ```yaml volumes: - /etc/euno/secrets.yaml:/etc/euno/secrets.yaml:ro ``` The agent reads the file at startup β€” restart the container after changing it. If an explicitly configured file is missing, or a present file is invalid (for example, malformed YAML, missing or duplicate handles, or an unknown provider type), the agent logs the reason and keeps its tunnel connected. Secret-handle listing and resolution return `secrets_file_missing_or_invalid` until the file is fixed and the agent is restarted; integrations that use inline credentials are unaffected. If the default file is absent, the agent treats secret handles as unconfigured and returns an empty handle list. ### 3. Pick handles in the source form In the source's **Advanced** section, once a local agent is selected, each credential field gains a **from agent** dropdown listing the agent's handles. Picking one disables the matching inline field β€” the value now comes from your vault. Choose **Enter value manually** to switch back. ### Security notes * The value travels from the agent to Euno through the agent's outbound encrypted tunnel β€” it is never exposed to your network and never written to Euno storage, logs, or error reports. * Every fetch is written to the agent's container logs as an audit line (handle and outcome only, never the value), so your team can export access records from your own log pipeline. * The agent refuses to start if long-lived cloud credentials (for example `AZURE_CLIENT_SECRET`) are present in its environment β€” vault access is by machine identity only. ## Managing agents | Action | How | | ------ | ----------------------------------------------------------------------------------------------------------- | | Rename | Click **β‹―** next to the agent β†’ **Edit**. Change the name in the form. This does not affect the connection. | | Stop | Stop the agent container. The UI will show **Disconnected** after a refresh. | | Delete | Click **β‹―** next to the agent β†’ **Delete**. Update any sources that used the agent before deleting. | {% hint style="warning" %} Deleting an agent clears the local agent selection and any credential references effectively using the deleted agent from sources that used it. Those sources cannot connect until you open them, select a new agent or enter credentials inline, and save. {% endhint %} ## Troubleshooting | Symptom | Fix | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Status stays **Disconnected** after starting the container | Check that `AGENT_ID` and `AGENT_SECRET` are correct. Inspect container logs for connection errors. | | Source run fails with "agent not connected" | Ensure the agent container is running. Restart it if needed, then refresh the Local Agents page to confirm the status is **Connected** before re-running. | | Source run fails with "agent not found" | The agent was deleted from Euno. Create a new agent and update the source's Advanced settings to select it. | | Source run fails with `provider_auth_failed` | Confirm that the agent host has a managed identity and can obtain a token, then grant that identity the **Key Vault Secrets User** role on the configured vault. Also confirm that the vault belongs to the identity's Azure tenant. | | Source run fails with `provider_denied` | Confirm that the configured secret name and optional version exist in the vault and are available to read. | | Source run fails with `fetch_timeout` | Confirm that the vault URL names an existing vault and is reachable from the agent host, including DNS, firewall, and private-endpoint rules. A managed identity metadata timeout can also produce this code. | | Source run fails with `cannot_fetch_dynamic_secret` | Pull the latest agent image and restart the container. If the failure continues, inspect the agent audit outcome and contact Euno support. | | Handle listing or a source run fails with `secrets_file_missing_or_invalid` | Fix `/etc/euno/secrets.yaml`, then restart the agent. The tunnel and integrations that use inline credentials remain available while the file is invalid. | | Source run fails with "Failed to connect to \[database]" | Verify that the database hostname is reachable from the machine running the agent container (not the Euno cloud). Check firewall rules and credentials. | | `no matching manifest for linux/arm64/v8` | Add `platform: linux/amd64` to your `docker-compose.yml` under the `euno-local-agent` service. This applies when running on Apple Silicon Macs for local testing. | # IP Whitelisting ### Overview IP Whitelisting lets account admins restrict which IP addresses can reach Euno β€” without contacting Euno support. Two scopes are enforced independently: | Scope | Covers | | ------------------- | -------------------------------------------------------------------------------------------- | | **Interactive** | Human sign-ins to the Euno web app (SSO / browser sessions). | | **Non-Interactive** | Machine-to-machine traffic: personal API keys, service accounts, ETL jobs, and CI pipelines. | Each scope has its own list of allowed IP ranges and its own **Enforce**toggle. For example, you can restrict which cloud environments your API keys may be called from while leaving browser sign-in open, or the reverse. {% hint style="info" %} IP Whitelisting is rolling out gradually. If you don't see it under Account Settings, ask your Euno contact to enable it for your account. {% endhint %} ### Prerequisites * Viewing the page requires access to Account Settings (any account member). * Adding, editing, or deleting ranges, and toggling enforcement, requires the **Manage Security** permission. This is assigned to the **Account Admin** persona by default. ### Where to find it Navigate to **Account Settings β†’ IP Whitelisting** ### Managing IP ranges Each scope β€” **Interactive (Human users)** and **Non-Interactive (M2M / API)** β€” shows: * A status row with enforcement status (**Active** / **Disabled**), plus, for Interactive, your current IP address, or for Non-Interactive, allowed/blocked request counts from the last 24 hours. * A table of configured ranges: CIDR, label, (Non-Interactive only) last used, created date, and who created it. #### Add a range 1. Click **Add IP range**. 2. Enter a **CIDR or IP address** β€” for example `203.0.113.0/24`, or `203.0.113.10/32` for a single IP β€” and a **Label** (1–80 characters) to describe it. 3. Click **Save**. A given CIDR can only be added once per scope; adding a duplicate is rejected. There is no limit on the number of ranges per scope. #### Edit or delete a range Use the row actions next to a range to **Edit label** (the CIDR itself is immutable once saved β€” delete and re-add the range if it needs to change) or **Delete** (asks for confirmation first). #### Search Use the search box above each table to filter ranges by label or CIDR. ### Turning enforcement on or off Each scope has its own **Enforce** toggle. * **Turning enforcement off** applies immediately. * **Turning enforcement on** with no ranges configured for that scope is blocked, with the error *"Cannot turn on enforcement while no ranges have been configured."* Add at least one range first. * **Turning on Interactive enforcement** while your current IP isn't covered by any configured range prompts a confirmation, since doing so will sign you out of your current session. You can confirm anyway or cancel. * **Turning on Non-Interactive enforcement** while your current IP isn't covered prompts a similar confirmation, since it may immediately block API keys and service accounts calling from IPs that aren't allowlisted. {% hint style="warning" %} When a scope is disabled, or has no ranges configured, traffic of that type is allowed from any source. {% endhint %} ### Recent blocked attempts Admins with the Manage Security permission see a **Recent blocked attempts** panel under the Non-Interactive section, listing the most recent requests rejected by that scope's enforcement: source IP, credential identifier (the API key or service account name β€” never the secret itself), endpoint, and timestamp. Use this to spot legitimate automation that needs a new range added. # Resource Audit Log ### Overview The Resource Audit Log records a history of changes to your data catalog, so you can answer questions like *"when did the owner of this table change?"* or *"when did this resource first show up in our catalog?"* It's searchable and filterable from a single page in Account Settings. ### Prerequisites * Your account must have the Audit Log feature enabled by Euno. * Viewing the Audit Log requires the **View Resource Audit Log** permission, assigned to the **Account Admin** persona by default. ### Where to find it Navigate to **Account Settings β†’ Audit Log**. ### What's tracked Every catalog change recorded by the Audit Log falls into one of three event types: | Event type | What it captures | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Fixed tag** | A user manually edits a fix tag value on a resource. | | **Built-in property** | An observer ingests a new value for one of Euno's built-in catalog properties (for example, owner or description) that differs from the previously stored value. | | **Lifecycle** | A resource is added to the catalog for the first time (**Created**) or removed from it (**Deleted**). Repeated observations of an unchanged resource don't produce additional rows. | {% hint style="warning" %} History starts from the day the Audit Log is enabled for your account. Changes made before that date are not retroactively logged β€” including resources that already existed in your catalog, which are not logged as newly "Created." {% endhint %} Each row shows the resource's URI and type, the property that changed (where applicable), the actor responsible (a user, or the observer that ingested the change), the timestamp, and the new value. Euno does not store the prior value directly, so only the new value is shown. ### Searching the log 1. Enter at least one filter β€” **URI**, **Property**, **Event type**, **Resource type** (multi-select), and/or **Actor** (comma-separated for multiple) β€” the search requires at least one of these. 2. Set a **Start time** and **End time**. The window defaults to the last 7 days and can be widened up to 365 days (or your account's configured retention, if different). 3. Click **Search**. Results appear in a table with these columns: **Time**, **Event** (the event type), **Actor**, **Property**, **Type** (resource type), **URI**, and **New Value**. Use **Previous** / **Next** to page through results (50 rows per page); the total number of matching rows is shown above the table. ### Retention The catalog history's retention period is customized per account. If you are unsure about the duration, contact your account manager for additional information . Searches cannot span a longer range than your account's configured retention. # SCIM Provisioning ## πŸ” SCIM Provisioning SCIM (System for Cross-domain Identity Management) lets your Identity Provider (IdP) create, update, and deactivate Euno users and groups automatically. Once configured, your IT admin manages access to Euno from the IdP β€” Okta, Microsoft Entra ID, or any other SCIM 2.0–compliant IdP β€” without having to invite users individually from Euno. > Compatible with **Okta** and **Microsoft Entra ID** (Azure AD). ### What SCIM Provisioning Does When SCIM is enabled and your IdP is connected: * **User lifecycle is mirrored**: when a user is added, renamed, deactivated, or removed in your IdP, the same change is applied in Euno on the next provisioning cycle. * **Group memberships drive personas**: you map IdP groups to Euno personas. Anyone in a mapped group automatically gets that persona; remove them from the group and the persona is revoked. * **Manual personas still work**: existing manually-assigned personas are preserved alongside SCIM-derived ones. SCIM-derived personas are visibly locked in the persona picker so admins know they're managed by the IdP.
### Prerequisites SCIM requires **Single Sign-On (SSO) to be configured first** for your account. SCIM is governed by the same SSO configuration β€” when SSO is removed, SCIM is treated as torn down: SCIM-derived personas disappear and provisioning halts on the next cycle. If SSO is not yet configured, see [SSO login](/setup-configuration/linked-accounts/sso-login) before continuing. ### Stage 1: Enable SCIM in Euno #### Step 1: Open SCIM Provisioning Settings 1. Navigate to **User Menu β†’ Account Settings β†’ SCIM Provisioning** 2. If SSO is not yet configured for your account, you'll see a warning and the **Enable SCIM** toggle will be disabled
#### Step 2: Enable SCIM Click the **Enable SCIM provisioning** toggle. The change is saved immediately. A green **SCIM enabled** indicator appears, and the rest of the configuration form expands. #### Step 3: Generate a Bearer Token The bearer token is the credential your IdP uses to authenticate with Euno. Generate it now so you can paste it into your IdP in Stage 2. 1. Click **Generate token** 2. The token is shown **once** in a modal β€” copy it immediately. You will not be able to see it again. 3. Store the token somewhere secure until you've finished IdP setup.
You can rotate the token at any time. Rotating immediately invalidates the existing token β€” any IdP currently using the old token will start receiving 401s until you paste the new one. Click **Rotate token** to replace it; **Disable token** to revoke it entirely. #### Step 4: Copy the SCIM Base URL Click **Copy** next to the **SCIM base URL**. You'll paste it into your IdP in Stage 2. #### Step 5: Configure Optional Settings Two additional fields control SCIM behavior: | Field | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Default persona** | The persona assigned to SCIM-provisioned users who are not in any mapped group. Leave unset to deny access to such users; set it to grant a baseline persona (typically Viewer). | | **Account Admin mapping** | A safety knob. While set to **Disabled**, no group can be mapped to the Account Admin persona. To allow it, change to **Allow explicit mapped group only**. | ### Stage 2: Configure Your Identity Provider > **⚠️ Important**: The SCIM `userName` attribute must match the user's primary email β€” the exact identifier they use to sign in via SSO. > > Both Okta and Microsoft Entra ID default to email-shaped values for `userName` (Okta's user login and Entra's `userPrincipalName`), which works for standard setups. **Verify the mapping on your IdP before going live.** #### Stage 2A: Microsoft Entra ID (Azure AD) 1. In the Microsoft Entra admin center, go to **Enterprise applications β†’ New application** 2. Choose a **non-gallery / custom SCIM application** (or your existing SCIM app) 3. Open the **Provisioning** blade β†’ **Get started** β†’ set Mode to **Automatic** 4. Under **Admin Credentials**, paste: * **Tenant URL**: the SCIM base URL from Stage 1, Step 4 * **Secret token**: the bearer token from Stage 1, Step 3 5. Click **Test Connection** β€” you should see a green success message 6. Save 7. Under **Settings**, set Scope to **Sync only assigned users and groups** 8. Assign users and groups to the application (Users and groups blade) 9. Click **Start provisioning** The initial provisioning cycle can take up to 40 minutes. Subsequent incremental cycles run every \~40 minutes. To push a specific change immediately, use **Provision on demand** from the Provisioning blade.
#### Stage 2B: Okta 1. In the Okta admin console, go to **Applications** 2. Open the app you want to enable SCIM provisioning on 3. Open the **Provisioning** tab β†’ **integration** 4. Set **SCIM connector base URL** to the SCIM base URL from Stage 1, Step 4 5. Set **Unique identifier field for users** to `userName` 6. Under **Supported provisioning actions**, check **Push New Users**, **Push Profile Updates**, and **Push Groups**. Leave the Import options unchecked. 7. Set **Authentication Mode** to **HTTP Header** 8. Under **HTTP Header β†’ Authorization**, paste the bearer token from Stage 1, Step 3 (the **Bearer** prefix is added automatically) 9. Click **Test Connector Configuration** β€” you should see a green success message 10. Save 11. On the **Provisioning β†’ To App** settings page, enable **Create Users**, **Update User Attributes**, and **Deactivate Users** 12. Assign users to the application (Assignments tab) and push groups (Push Groups tab)
### Stage 3: Map IdP Groups to Personas After your IdP has pushed groups to Euno, you can map each group to a Euno persona. #### Step 1: Open the Mappings Section In **SCIM Provisioning**, scroll to **Group to persona mappings**. Pushed groups appear in the dropdown. #### Step 2: Add a Mapping 1. Click **Add mapping** 2. Select an IdP group from the dropdown 3. Select a persona to grant to its members 4. Click **Save** The mapping is applied within \~10 seconds across all open sessions.
#### Multiple Group Memberships If a user is a member of multiple mapped groups, they receive **all** the mapped personas β€” switchable from the User Menu like any other multi-persona user. See [Personas](/setup-configuration/personas) for how persona switching works. #### Default Persona Fallback When a SCIM-provisioned user is in **no mapped group** and you've set a **Default persona** in Stage 1 Step 5, that persona is granted automatically. If no default is set, the user has no access until they're added to a mapped group or assigned a manual persona. ### Account Admin Mapping (Safety Knob) Granting the Account Admin persona via a group mapping is an exceptional operation β€” it gives anyone in that group full administrative control. To prevent accidental escalation, the **Account Admin mapping** setting is **Disabled** by default. To grant Account Admin via SCIM: 1. Change **Account Admin mapping** to **Allow explicit mapped group only** 2. The Account Admin persona now appears as an option in the Mapping dialog and in the Default persona dropdown 3. Add a mapping for the appropriate IdP group β†’ Account Admin If you later want to remove the Account Admin mapping option entirely, you must first remove any existing mapping that references it. Euno will block the flip with an error otherwise. ### Coexistence: Manual + SCIM-Derived Personas A user can have both **manually-assigned personas** (via the Stakeholders page) and **SCIM-derived personas** (via group mapping or default persona) at the same time. In the Stakeholders persona picker: * **SCIM-derived persona chips are locked** with a "Managed by SCIM" subtitle and a lock icon. They cannot be toggled β€” only the IdP can change them. * **Manually-assigned persona chips are editable** β€” admins can still add or remove them via the picker. * The **Remove from account** action is hidden for SCIM-managed users. To deprovision, remove the user from your IdP.
### What Triggers a Sync | Event in your IdP | What happens in Euno | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | User added or assigned to the app | New SCIM user created; SCIM-derived personas applied based on group memberships | | User profile updated (name change) | User's display name updated in Euno | | User deactivated or unassigned | User's SCIM-derived personas removed; user remains visible only if they have manually-assigned personas | | User permanently deleted from IdP | User's SCIM record removed; any manually-assigned personas are preserved | | User added to or removed from a mapped group | SCIM-derived personas adjusted; change applies within \~10 seconds | | New group pushed | Group becomes available in the Mappings dropdown | | Group renamed | Mapping persists, name updates in the Mappings table | | Group deleted | Any mapping referencing it is removed; affected users lose those personas | ### Audit Log Every SCIM operation β€” both IdP-pushed and admin-driven changes β€” is recorded in the **Events** section of the SCIM Provisioning page. Events include the operation type, the affected user or group, the actor (the IdP for pushes; the admin email for config changes), and a timestamp.
The events page is paginated and ordered most-recent-first. Use it for forensic review or to verify that recent IdP changes have propagated correctly. ### Troubleshooting #### A user can sign in via SSO but has no personas The most common cause is a `userName` mismatch between SCIM and SSO. Verify in your IdP that the SCIM `userName` attribute maps to the same email the user signs in with. Both Okta and Entra default to this; the mismatch usually appears when an admin has explicitly customized the mapping (e.g. mapping `userName` to `samAccountName` or `employeeId` instead of email). #### Provisioning is failing with 401 errors The bearer token may have been rotated or disabled. Re-generate the token in Stage 1 Step 3 and paste the new value into your IdP. If SSO has been removed from your Euno account, SCIM is also considered torn down β€” all incoming provisioning requests will fail with 401 until SSO is restored. #### A group mapping was added but users don't see the persona Group-to-persona mappings apply within \~10 seconds across all open sessions. If the persona still doesn't appear after that, verify: 1. The user is actually a member of the mapped group on the IdP side 2. The user has been provisioned (visible in the Stakeholders list with a **Managed by SCIM** badge) 3. The user's SCIM record is `active` (not deactivated) #### I want to test what my IdP is sending The Events page captures every SCIM operation with the full (redacted) request payload. Filter by recent operations and inspect the payload to confirm what your IdP is actually pushing. ### Removing SCIM To turn off SCIM provisioning for your account: 1. In your IdP, **stop provisioning** to disconnect the integration 2. In Euno SCIM Provisioning, click **Disable token** to revoke the bearer token 3. Disable the **Enable SCIM** toggle if you want to fully turn off SCIM features Existing SCIM-provisioned users retain their manually-assigned personas (if any). To fully remove a user, remove them from the IdP β€” Euno will deprovision them on the next sync β€” or use the User Management Remove flow once SCIM is fully turned off. # Linked Accounts **Linked accounts** in Euno serve dual purposes: 1. **Ingestion of resources**: Similar to sources, linked accounts facilitate the ingestion of resources into Euno. 2. **Enabling data model sync**: Linked accounts allow Euno to keep your dbt resources synced with Looker. For more details see [Looker Data Model Sync](/using-euno/metadata-activation/data-application-sync). Currently, Euno supports three types of linked accounts: * [**GitHub**](/setup-configuration/linked-accounts/github-integration) * [**GitLab**](/setup-configuration/linked-accounts/gitlab-integration) * [**Slack**](/ai/slack-assistant), including the Euno Slack Assistant

Linked Accounts

# GitHub Euno integrates with GitHub to efficiently handle new dbt builds and automate the generation and deployment of dbt code. For further details on "GitHub Apps" and their permissions, refer to the linked [article](https://docs.github.com/en/apps/using-github-apps/about-using-github-apps). Install the Euno GitHub app at the "organization" level on GitHub. This integration is crucial to facilitate automated code generation, encompassing the generation of code modifications in the dbt repository and the synchronization of certain data applications, like Looker, and writing changes to a LookML repository. **Note:** The user installing the app does not require push and pull code permissions. Euno authenticates actions as itself, not as the user who installed it. User’s who wish to [create a new project](/setup-configuration/linked-accounts/github-integration), connect an existing project to a new branch, or [configure the data application sync to Looker](https://github.com/delphiio/delphi_backend/blob/dev/using-euno/metadata-activation/data-application-sync/README.md#rupq6wm679lu-1), will be required to sign into their Github account. ## Integration Steps To integrate with GitHub please follow the below steps:
1Click on the "Account settings" found in the left side menu.
2Locate the Github tile under the Linked Accounts tab and click β€œConfigure”.
3In the dialog, click β€œLink new organization” to be redirected to Github authentication flow.
4Follow the instructions on Github to link your account to Euno.
5Your organization is now linked to Euno. You can repeat step 3 if you want to link another organization.
# GitLab Euno integrates with your GitLab groups and projects, necessitating read and write access. These permissions allow Euno to interact with your dbt projects and repositories on GitLab, facilitating the retrieval of essential information and enabling automated code generation and deployment tasks. The GitLab integration can be configured at both the group and project levels, allowing for tailored access controls to meet your specific requirements. ## Integration Steps To integrate with GitLab please follow the below steps:
1Click on the "Account settings" found in the left side menu.
2Locate the GitLab tile under the Linked Accounts tab and click β€œConfigure”.
3

In the dialog box, enter the following details:

  • Domain - Only edit if your GitLab instance is self-hosted
  • Group ID
  • Group token - This is optional and can instead be managed at the project creation level using a project token
4Your organization is now linked to Euno. You can repeat step 3 if you want to link another instance.
# SSO login How to set up SSO login for Euno Euno supports SSO integration through Okta and Entra. To setup SSO for your account, go to Account Settings -> Linked accounts
Under SSO, click "Configure" This will generate a link to our SSO setup wizard. Follow the wizard to setup SSO in your account. # Source Integrations The Sources page in Euno lets you connect and manage a variety of platforms, integrating their resources into your Euno data model. By setting up these integrations, Euno can observe resources metadata, usage statistics, lineage information, and any other relevant data, providing a comprehensive and interactive view of your data ecosystem. ## Overview Source integrations are organized by type to help you find the right integration for your stack: * [**πŸ’Ύ Data Warehouses**](/sources/data-warehouses) - Connect your data warehouses * **πŸ”’** [**Data Security**](/sources/data-security) - Connect data security and classification platforms * [**πŸ”„ Transformation & ETL**](/sources/transformation-etl) - Connect transformation and ETL tools * [**πŸ“Š Business Intelligence**](/sources/business-intelligence) - Connect BI and analytics platforms * [**πŸ““ Analytics & Notebooks**](/sources/analytics-notebooks) - Connect analytics and notebook platforms * [**πŸ”§ Custom Integrations**](/sources/custom-integrations) - Build custom integrations ## Quick Links ### Data Warehouses * [Snowflake](/sources/data-warehouses/snowflake-integration) * [BigQuery](/sources/data-warehouses/bigquery-integration) * [Databricks](/sources/data-warehouses/databricks-integration) * [MySQL](/sources/data-warehouses/mysql-integration) * [SQL Server](/sources/data-warehouses/sql-server-integration) ### Transformation & ETL * [dbt Core](/sources/transformation-etl/dbt-core) * [dbt Cloud](/sources/transformation-etl/dbt-cloud) * [Matillion DPC](/sources/transformation-etl/matillion) * [Fivetran](/sources/transformation-etl/fivetran) * [OpenLineage](/sources/transformation-etl/openlineage-integration) ### Business Intelligence * [Tableau](/sources/business-intelligence/tableau-integration) * [Looker](/sources/business-intelligence/looker-integration) * [Power BI](/sources/business-intelligence/powerbi-integration) * [Thoughtspot](/sources/business-intelligence/thoughtspot-integration) * [Sigma](/sources/business-intelligence/sigma-integration) * [Omni](/sources/business-intelligence/omni-integration) * [Cube Core](/sources/business-intelligence/cube-core-integration) ### Analytics & Notebooks * [Hex Technologies](/sources/analytics-notebooks/hex-technologies) * [Hex Semantic Project](/sources/analytics-notebooks/hex-semantic-project) ### Data Security * [Cyera](/sources/data-security/cyera-integration) * [BigID](/sources/data-security/bigid-integration) ### Data Observability * [Monte Carlo](/sources/data-observability/monte-carlo) ### Custom Integrations * [Custom Source Upload](/sources/custom-integrations/custom-integration) * [OpenLineage API](/sources/transformation-etl/openlineage-integration) ### Identity and Organization * [Okta](/sources/identity-and-organization/okta-integration) * [Google Workspace](/sources/identity-and-organization/google-workspace-integration) ### Gallery The Gallery section showcases the types of available source integrations.
### **Enabled Sources** The **Enabled Sources** section displays a list of configured sources with key information: * **Type**: The type of source (e.g., dbt Cloud, Looker). * **Health**: The current health status of the source. The source's health reflects the outcome of the last successful run of that source. * **Last Run**: The timestamp of the source's last run. * **Status**: Indicates whether the source is active or paused. When a source is paused. From this view, users can: * **Pause** an active source * **Edit** source details * Trigger a source run by clicking **Run Now** (via the horizontal ellipsis menu) * **Generate a Trigger URL** (via the horizontal ellipsis menu). This will create a secret and output an endpoint url for triggering a source run. * **Delete source**

Enabled Sources

### **Source Details** Clicking on a specific source directs the user to a detailed **Source Runs** page. This page contains a table listing all runs of the selected source, including the following fields: * **Timestamp**: The date and time of the run. * **Run Trigger**: What initiated the run (e.g., schedule, manual, automatic). * **Run Status**: The current or final status of the run (e.g., running, success, failed). * **Run Report**: A detailed summary of the run, including when it occurred, which configuration was used, and an aggregate overview of what was observed. From this screen, users can: * Pause the source * Trigger a new run * Edit the source’s configuration

Source Details

# Python Upload for Zip Artifacts Use this guide to programmatically upload a `.zip` file to any Euno source integration that accepts zip artifacts via the `prepare-upload` endpoint. This flow is shared by: * [dbt Core](/sources/transformation-etl/dbt-core) * [Cube Core](/sources/business-intelligence/cube-core-integration) * [Hex Semantic Project](/sources/analytics-notebooks/hex-semantic-project) * [Matillion DPC](/sources/transformation-etl/matillion) For Custom integration bulk uploads (`.json`, `.jsonl`, `.ndjson`), see [Handling High Volume of Observations](/sources/custom-integrations/custom-integration/handling-high-volume-of-observations). ## Prerequisites * Python 3.6+ * `requests` library (`pip install requests`) * A saved source with a **trigger secret** and **`prepare-upload` endpoint URL** from the post-save modal ## Prepare your zip file Create a single `.zip` file before running the script. What to include depends on your integration: | Integration | What to zip | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | **dbt Core** | `manifest.json`, `catalog.json`, `run_results.json`, and optional `semantic_manifest.json` from your dbt `target/` directory | | **Cube Core** | Your Cube `model/` directory (`model/**/*.yml` or `model/**/*.yaml`) | | **Hex Semantic Project** | `dbt_project.yml` plus dbt MetricFlow YAML files with Hex configuration | | **Matillion DPC** | Matillion export artifacts required by your source configuration | Example for Cube Core: ```bash cd /path/to/your/cube/project zip -r cube-model.zip model/ ``` ## Script ```python import json import os from pathlib import Path import requests # Configuration β€” replace with your values endpoint_url = ( "https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/" "integrations/YOUR_INTEGRATION_ID/prepare-upload" ) trigger_secret = "your_trigger_secret_here" zip_file_path = "artifacts.zip" headers = { "authorization": f"Bearer {trigger_secret}", "content-type": "application/json", } def get_signed_url(): """Request a signed upload URL for the zip file.""" body = {"filename": Path(zip_file_path).name} response = requests.post(endpoint_url, headers=headers, json=body, timeout=30) if response.status_code != 200: print(f"Failed to obtain signed URL (status {response.status_code}): {response.text}") return None payload = response.json() upload_url = payload.get("upload", {}).get("url") if not upload_url: print("Signed upload URL not found in response:", json.dumps(payload)[:500]) return None return upload_url, payload def upload_file(upload_url): """Upload the zip file to the signed URL.""" try: with open(zip_file_path, "rb") as file_handle: response = requests.put( upload_url, data=file_handle, headers={"content-type": "application/zip"}, timeout=120, ) if response.status_code not in (200, 201): print(f"Upload failed (status {response.status_code}): {response.text[:200]}") return False return True except OSError as exc: print("An error occurred during upload:", exc) return False def main(): print("Starting zip artifact upload...") if not os.path.exists(zip_file_path): print(f"Error: file '{zip_file_path}' not found.") return False print("Requesting signed URL...") result = get_signed_url() if not result: return False upload_url, meta = result print("Signed URL obtained β€” uploading archive...") if upload_file(upload_url): print("Upload succeeded! Euno will start processing automatically.") print(json.dumps(meta, indent=2)) return True print("Upload failed.") return False if __name__ == "__main__": main() ``` ## Usage 1. **Configure the script**: Set `endpoint_url`, `trigger_secret`, and `zip_file_path` 2. **Create the zip**: Package the files required by your integration (see [Prepare your zip file](#prepare-your-zip-file)) 3. **Install dependencies**: `pip install requests` 4. **Run the script**: Execute after your pipeline produces the artifacts 5. **Verify the run**: Open your source on the **Sources** page and review the latest operation and run report ## cURL alternative ### Step 1: Request a signed upload URL ```bash curl -X POST \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET" \ -H "Content-Type: application/json" \ -d '{"filename": "artifacts.zip"}' \ "https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload" ``` ### Step 2: Upload the zip to the signed URL Use the `url` from the `upload` object in the response: ```bash curl -X PUT \ -H "Content-Type: application/zip" \ --data-binary @artifacts.zip \ "SIGNED_URL_FROM_STEP_1" ``` When the upload succeeds, Euno automatically starts processing the file. No further API call is required. # Data Warehouses Connect your data warehouses to Euno to map your entire warehouse structure, track usage patterns, and optimize costs. ## Supported Warehouses * [**Snowflake**](/sources/data-warehouses/snowflake-integration) - Cloud data warehouse * [**BigQuery**](/sources/data-warehouses/bigquery-integration) - Google's serverless data warehouse * [**Databricks**](/sources/data-warehouses/databricks-integration) - Unified analytics platform * [**MySQL**](/sources/data-warehouses/mysql-integration) - Relational database * [**SQL Server**](/sources/data-warehouses/sql-server-integration) - Microsoft SQL Server and Azure SQL Database ## What You'll Get When you connect a data warehouse, Euno discovers: * **Tables and Views** - All tables, views, and their schemas * **Columns** - Column-level metadata and lineage * **Usage Data** - Query patterns, costs, and performance metrics * **Storage Information** - Table sizes and growth trends * **Relationships** - Dependencies between tables and views ## Common Use Cases * **Cost Optimization** - Identify expensive, unused tables * **Lineage Mapping** - Understand data flow across your warehouse * **Governance** - Track which tables are managed by dbt vs. ad-hoc * **Performance** - Find slow queries and optimization opportunities ## Getting Started 1. Choose your warehouse from the list above 2. Follow the setup guide for your platform 3. Grant Euno read access to metadata and query logs 4. Run your first sync **Setup Time:** 15-20 minutes per warehouse *** ## Next Steps * [View All Integrations](/sources) * [Learn About Usage Data](/using-euno/usage-optimization) * [Set Up Cost Optimization Workflows](/using-euno/metadata-activation/workflows) # Snowflake Euno's Snowflake integration connects to Snowflake metadata, query history, access history, and governance metadata to discover warehouse resources, usage, lineage, sensitive data classification, masking policies, semantic views, and Snowflake shares. Euno's Snowflake integration supports auto-discovery of: * Snowflake Views and Tables * Snowflake sensitive data classification and masking policy metadata * [Snowflake Table Activity and Tableau Usage](https://docs.euno.ai/guides/usage) * Hex Usage and Lineage Tracking ## Prerequisites / Requirements Before setting up the source, make sure you have: * A Snowflake user and role for Euno. * A Snowflake warehouse the role can use for metadata queries. * Access to Snowflake account usage views needed for the features you enable. * A private key for key-pair authentication, or credentials generated by Euno during setup. ## Stage 1: Configure Snowflake ### Step 1: Required Snowflake Permissions In order to integrate with Euno, create a custom role in Snowflake with the permissions listed below. In the code snippet below, we create a user `euno_user`, with a role `euno_role`. Euno discovers Snowflake resources by issuing SQL queries on Snowflake's [system database](https://docs.snowflake.com/en/sql-reference/snowflake-db) and [information schema](https://docs.snowflake.com/en/sql-reference/info-schema). The Snowflake role requires the Snowflake USAGE grant on a Snowflake warehouse. In the code snippet below, we use the warehouse `euno_dwh`. The RSA\_PUBLIC\_KEY is generated by the source integration (see step 2). ```sql CREATE OR ALTER ROLE euno_role; CREATE USER euno_user DEFAULT_ROLE='euno_role' RSA_PUBLIC_KEY= 'MIIBIjA...'; GRANT ROLE euno_role TO USER euno_user; GRANT USAGE ON WAREHOUSE euno_dwh TO ROLE euno_role; GRANT DATABASE ROLE SNOWFLAKE.OBJECT_VIEWER TO ROLE euno_role; GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE euno_role; GRANT DATABASE ROLE SNOWFLAKE.USAGE_VIEWER TO ROLE euno_role; ``` {% hint style="success" %} These grants do not provide the `euno_role` with access to your data, only to the metadata. To read more about these grants, see: * [`usage` privilege on Snowflake Warehouses](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#virtual-warehouse-privileges) * [`object_viewer` , `governance_viewer` and `usage_viewer` database roles](https://docs.snowflake.com/en/sql-reference/snowflake-db-roles) * [`references` privilege on Snowflake views](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#view-privileges) {% endhint %} The `SNOWFLAKE.GOVERNANCE_VIEWER` database role lets Euno observe Snowflake classification tags, object tags, and masking-policy metadata. If that metadata is unavailable, Euno can still discover core Snowflake resources, but sensitive-data fields may remain empty and object tags are not ingested (the run report notes the missing grant). In some environments, users may prefer to grant the Euno integration access to read the metadata of views defined in specific databases, rather than to all the databases in the Snowflake account. For more information, see [Control which views metadata Euno has access to](#control-which-views-metadata-euno-has-access-to). In some environments, users may prefer to limit the rows or columns in the Snowflake query history that Euno integration has access to. For more information, see [Using a non-default query history view](#using-a-non-default-query-history-view). ## Stage 2: Configure New Snowflake Source in Euno ### Step 1: Access the Sources Page 1. Go to the **Sources** page. 2. Click **Add New Source** and select **Snowflake** from the list. ### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name**\* | Enter a name for your Snowflake source (for example, `Snowflake - Data Warehouse`). | | **Host**\* | The Snowflake host associated with your Snowflake account, such as `mycompany.eu-central-1.snowflakecomputing.com` or `org-account.us-west-2.snowflakecomputing.com`. | | **User**\* | The Snowflake user to use for the integration. | | **Private Key**\* | The RSA private key used to authenticate to Snowflake. This value is stored securely and marked as a secret. You can provide your own private key or use one generated by Euno by clicking **Generate Credentials**. | | **Passphrase** | The passphrase used to decrypt the private key. This value is stored securely and marked as a secret. Leave it blank if the private key is not encrypted. | | **Generate Credentials** | Automatically generate an RSA key pair and passphrase. The public key will be shown in a modal for you to paste into your Snowflake user; the private key and passphrase will be filled into the form and stored securely. This is convenient for quick testing or POCs, but for production we recommend supplying keys generated and managed according to your security policies. | | **Role** | The Snowflake role to use. If the Snowflake user has a default role, you can keep this blank. | | **Warehouse** | The Snowflake warehouse to use for queries. If the Snowflake user has a default warehouse, you can keep this blank. | | **Query history table**\* | By default, Euno uses `snowflake.account_usage.query_history` to access Snowflake query history. If you prefer to use a different table or view, enter it here. See [Using a non-default query history view](#using-a-non-default-query-history-view). | | **Cost per credit (USD)** | The credit price paid to Snowflake. This is used to calculate table activity costs. | | **Storage cost per TB (USD)** | The monthly storage rate in USD per terabyte, used to estimate storage costs for Snowflake tables. | | **Observe warehouse information** | Enable this option to observe Snowflake warehouse information on the `snowflake_account` resource. | ### Step 3: Schedule 1. Toggle Schedule to activate updates. 2. Configure: 1. **Weekly**: Select the days and times for updates. 2. **Hourly**: Set the interval in hours (e.g., every 12 hours). ### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](https://docs.euno.ai/reference/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 5: Advanced Settings Click on the '**Advanced**' section to display these additional configurations. | Configuration | Description | | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Skip SSL verification** | Skip SSL verification. This is used in cases where Euno accesses Snowflake through a proxy. We recommend leaving this unchecked. | | **Auto discover views & tables metadata** | Enable auto-discovery of Snowflake views and tables metadata. | | **Auto discover materialized views refresh history** | Enable discovery of Snowflake materialized view refresh history. This feature is only available for Snowflake Enterprise edition. | | **Use Snowflake system database to collect views metadata** | Use `snowflake.account_usage.views` to discover Snowflake views. If you prefer not to grant metadata access for all databases, see [Control which views metadata Euno has access to](#control-which-views-metadata-euno-has-access-to). | | **Auto discover Hex usage** | Discover Hex projects that query Snowflake and extract Snowflake lineage for those projects. For more information, see [Hex Usage & Lineage Tracking](#hex-usage-and-lineage-tracking). | | **Auto discover Sigma usage** | Discover Sigma workbook, element, and data model usage from Snowflake query history. | | **Override Host URI** | Overrides the base URI used for generating resource URIs for databases, schemas, tables, views, semantic views, and columns. By default, Euno uses the host provided in the integration. | | **Database mapping** | Map Snowflake database names to alternative names in Euno. | | **Schema Pattern** | Use regular expressions to include or exclude specific `database.schema` combinations, such as `MY_DB\.PUBLIC`. | | **Observe Inbound Snowflake Shares** | Observe inbound Snowflake shares. This requires the `IMPORT SHARE` privilege. | | **Outbound Shares** | Configure outbound shares manually by share name and database name. | | **Extract column-level lineage from query history** | Extract column-level lineage from Snowflake `access_history`. | | **Extract column-level read usage** | Extract per-column read usage metrics from Snowflake `access_history`. This requires Snowflake Enterprise edition. | For Snowflake shares related configuration, see [Observing Snowflake Shares](/sources/data-warehouses/snowflake-integration/observing-snowflake-shares-and-cross-snowflake-account). #### Run report warning: database missing from metadata crawl If a database appears in access/query history but not in the metadata crawl, Euno can observe its tables by reference only. Table-level lineage may still appear, but field-level lineage will be missing. To fix this, either: * grant the Snowflake role used by Euno metadata visibility for the affected database * update the Snowflake integration settings so the database is included in the metadata crawl ### Step 6: Save Configuration Click **Test & Save** to complete the setup. ## What Euno Discovers * Snowflake account, databases, and schemas * Snowflake tables, views, materialized views, temporary tables, and columns * Snowflake sensitive data classification tags, privacy categories, and masking policies * Snowflake object tags on databases, schemas, tables, views, and columns * Snowflake semantic views, metrics, and dimensions * Snowflake inbound and outbound shares * Usage, cost, storage, lineage, and optional column usage metrics * Hex projects and lineage detected from Snowflake query history #### Sensitive data classification and masking metadata Euno observes Snowflake classification tags, including semantic and privacy categories, and masking policies attached directly to columns or through tag-based masking. These values can appear on table and column resources, in the Data Model's **Sensitive Data** tab, Data Model table columns, and Snowflake filters. For semi-structured columns, Euno also indicates when Snowflake reports classified paths. If Snowflake governance metadata is unavailable during a run, these fields may remain empty while the rest of the Snowflake crawl succeeds. #### Snowflake object tags Euno observes [Snowflake object tags](https://docs.snowflake.com/en/user-guide/object-tagging/introduction) on the account, databases, schemas, tables, views, materialized views, and columns, including propagated and inherited tags. Each assignment appears in the resource's **Tags** as `name=value`, where the tag name is the lowercased fully qualified tag (for example `governance.tags.cost_center=finance`) and the value keeps its Snowflake casing. When direct, propagated, and inherited assignments conflict, Euno follows Snowflake's precedence. Snowflake's built-in classification tags (`SNOWFLAKE.CORE.*`) are excluded here β€” they appear in the sensitive-data fields described above. Warehouse tags are not currently ingested because warehouse information is stored on the Snowflake account rather than modeled as a standalone Euno resource. Snowflake returns tag assignments only for objects the Euno role can access, so tags on objects the role has no grants on will not appear even though the objects themselves are discovered through the account-usage metadata. If tags are missing on specific objects, verify the Euno role's grants on those objects. Because a missing result can mean either "untagged" or "not visible to this role," Euno does not erase existing object tags based only on missing tag-reference rows; a removed assignment may remain until Euno can verify the deletion through a complete source. Snowflake permits quoted tag identifiers whose dots or letter casing can collapse to the same lowercased fully qualified name shown in Euno. If Euno detects two distinct tags that would have the same displayed name, it rejects that run's object-tag snapshot and reports the collision instead of attaching an ambiguous value or replacing previously observed tags. *** #### Using a non-default query history view By default, Euno uses the system view `snowflake.account_usage.query_history` to discover the query history. You might prefer, for security reasons, to limit the rows or columns that Euno integration has access to. In that case, you can manually create a Snowflake view to expose a subset of the query history. In order for Euno to use this non-default Snowflake view, the following columns, available in `snowflake.account_usage.query_history`, must be exposed by the view: * query\_tag * total\_elapsed\_time * bytes\_scanned * start\_time * end\_time * query\_type Make sure you grant Euno permissions to SELECT on this view. Below is an example of creating such a view, and excluding records that were initiated by the user `PRIVATE_USER` ```sql CREATE OR REPLACE VIEW user.public.query_history_for_euno AS ( SELECT query_tag, total_elapsed_time, start_time, end_time, bytes_scanned, query_type FROM snowflake.account_usage.query_history WHERE user_name != 'PRIVATE_USER' ); GRANT SELECT ON VIEW user.public.query_history_for_euno TO ROLE euno_role; ``` #### Control which views metadata Euno has access to By default, the Euno integration uses the system view `snowflake.account_usage.views` to discover all Snowflake views defined in the snowflake account. In some environments, users may prefer to grant the Euno integration access to read the metadata of views defined in specific databases, rather than to all the databases in the Snowflake account. To set up Euno this way, follow these steps: 1. Grant the Snowflake role used by Euno access to the metadata of views defined in the needed database. For example, to grant the Snowflake role `euno_role` access to the metadata of views defined in the database `example_database`, execute: ```sql GRANT USAGE ON DATABASE example_database TO ROLE euno_role; GRANT USAGE ON ALL SCHEMAS IN DATABASE example_database TO ROLE euno_role; GRANT REFERENCES ON ALL VIEWS IN DATABASE example_database TO ROLE euno_role; GRANT USAGE ON FUTURE SCHEMAS IN DATABASE example_database TO ROLE euno_role; GRANT REFERENCES ON FUTURE VIEWS IN DATABASE example_database TO ROLE euno_role; ``` {% hint style="success" %} These privileges do not provide the `euno_role` with access to the data in these views, only to the metadata. To read more about these privileges, see: * [`usage` privilege on Snowflake databases](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#database-privileges) * [`usage` privilege on Snowflake schema](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#schema-privileges) * [`references` privilege on Snowflake views](https://docs.snowflake.com/en/user-guide/security-access-control-privileges#view-privileges) {% endhint %} 2. Uncheck the box "Use Snowflake system database to query for Snowflake views" in the integration configuration. #### Granting Euno access to the definitions of Snowflake Semantic Views In order to get the complete definition of Snowflake Semantic Views, Euno requires the `REFERENCES` and `USAGE` privileges. This is a database-level permission, so it needs to be granted in each database where semantic views are defined. 
-- For every relevant database
GRANT REFERENCES ON ALL SEMANTIC VIEWS IN DATABASE <database_name> TO ROLE euno_role;
GRANT REFERENCES ON FUTURE SEMANTIC VIEWS IN DATABASE <database_name> TO ROLE euno_role;
GRANT USAGE ON DATABASE <database_name> TO ROLE euno_role;
GRANT USAGE ON ALL SCHEMAS IN DATABASE <database_name> TO ROLE euno_role;
GRANT USAGE ON FUTURE SCHEMAS IN DATABASE <database_name> TO ROLE euno_role;
If your setup has semantic views spanning multiple databases, you can consider using the following SQL script to grant these permissions in every database. This script will iterate over all the available databases (ignoring applications, imported databases and personal databases), and grant the above permissions in each one to `EUNO_ROLE`. Make sure you change the role name in the script.
```sql DECLARE v_db STRING; c1 CURSOR FOR ( SELECT DATABASE_NAME FROM INFORMATION_SCHEMA.DATABASES WHERE DATABASE_NAME NOT IN ('SNOWFLAKE', 'SNOWFLAKE_SAMPLE_DATA') ); BEGIN FOR rec IN c1 DO v_db := rec.DATABASE_NAME; BEGIN -- Existing semantic views EXECUTE IMMEDIATE 'GRANT REFERENCES ON ALL SEMANTIC VIEWS IN DATABASE ' || v_db || ' TO ROLE EUNO_ROLE'; -- Future semantic views EXECUTE IMMEDIATE 'GRANT REFERENCES ON FUTURE SEMANTIC VIEWS IN DATABASE ' || v_db || ' TO ROLE EUNO_ROLE'; -- Usage EXECUTE IMMEDIATE 'GRANT USAGE ON DATABASE ' || v_db || ' TO ROLE EUNO_ROLE'; EXECUTE IMMEDIATE 'GRANT USAGE ON ALL SCHEMAS IN DATABASE ' || v_db || ' TO ROLE EUNO_ROLE'; EXECUTE IMMEDIATE 'GRANT USAGE ON FUTURE SCHEMAS IN DATABASE ' || v_db || ' TO ROLE EUNO_ROLE'; EXCEPTION WHEN OTHER THEN -- Skip unsupported databases types IF (SQLERRM LIKE '%imported database%' OR SQLERRM LIKE '%IMPORTED PRIVILEGES%' OR SQLERRM LIKE '%personal database%' OR SQLERRM LIKE '%APPLICATION%' OR SQLERRM LIKE '%Operation not supported%') THEN NULL; ELSE -- Re-raise other errors RAISE; END IF; END; END FOR; END; ``` #### Hex Usage & Lineage Tracking When "Discover Hex usage" is enabled, the Snowflake integration automatically observes lineage data from Hex queries executed against Snowflake. **What is observed:** * **Hex Projects**: Discovery of Hex projects that execute queries against Snowflake * **Project Lineage**: Which Snowflake tables and columns are accessed by each Hex project **How it works:** Hex embeds metadata in SQL query comments when executing queries against Snowflake. This metadata includes: * Project ID and name * User email * Cell ID (for tracking specific notebook cells) The Snowflake integration parses these comments from the query history and correlates them with table access information to provide comprehensive lineage tracking of Hex activities. **Requirements:** * Hex queries must include metadata comments (this is automatic for standard Hex usage) * The Snowflake role must have access to `snowflake.account_usage.query_history` and `snowflake.account_usage.access_history` * The integration must have permissions to read query history data * The Snowflake instance needs to allow connections from Euno's public IP address in `35.224.29.172` This feature helps organizations understand which Hex analytics projects are accessing their Snowflake resources and track data lineage from Hex to Snowflake tables. # Snowflake Integration Discovered Resources ### Ingested Resources & Properties #### Snowflake account (`snowflake_account`) The Snowflake account is the top-level Snowflake resource observed by the integration. | Property | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------- | | `name` | Snowflake account identifier. | | `type` | Always `snowflake_account`. | | `subtype` | Always `snowflake_account`. | | `container_type` | Marks the account as a pure container. | | `snowflake_warehouses` | Warehouse descriptors from `SHOW WAREHOUSES`. Only present when **Observe warehouse information** is enabled. | | `snowflake_tableau_usage_raw` | Encoded raw Tableau usage records discovered from Snowflake query history. | | `last_integration_run_ended` | Timestamp of the last completed integration run. | When `snowflake_warehouses` is present, each warehouse object may include fields such as `name`, `size`, `state`, `type`, `min_cluster_count`, `max_cluster_count`, `started_clusters`, `running`, `queued`, `is_default`, `is_current`, `auto_suspend`, `auto_resume`, `owner`, `comment`, `created_on`, `resumed_on`, `updated_on`, `resource_monitor`, `scaling_policy`, and `uuid`. Other fields may be present depending on Snowflake's `SHOW WAREHOUSES` output. #### Database (`database`) Snowflake databases are observed as container resources. | Property | Description | | ------------------ | --------------------------------------- | | `name` | Database name. | | `type` | Always `database`. | | `subtype` | Always `database`. | | `container_type` | Marks the database as a pure container. | | `parent_container` | The parent `snowflake_account`. | #### Database schema (`database_schema`) Snowflake schemas are observed as container resources under databases. | Property | Description | | ------------------ | ------------------------------------- | | `name` | Schema name. | | `type` | Always `database_schema`. | | `subtype` | Always `database_schema`. | | `container_type` | Marks the schema as a pure container. | | `parent_container` | The parent `database`. | #### Table and view (`table`) Snowflake tables, views, materialized views, base tables, and temporary tables are represented as `table` resources. Euno uses the `subtype` property to distinguish the Snowflake object kind. | Property | Description | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `name` | Object name, excluding database and schema. | | `type` | Always `table`. | | `subtype` | Snowflake object kind, such as `table`, `base_table`, `view`, `materialized_view`, or `temporary table`. | | `description` | Snowflake comment, when present. | | `platform_uri_prefix` | Snowflake URI prefix used to generate resource URIs. | | `database_technology` | Always `snowflake`. | | `database_database` | Snowflake database name. | | `database_schema` | Snowflake schema name. | | `parent_container` | Parent `database_schema`. | | `native_owners` | Snowflake owner role or user references. | | `native_last_data_update` | Last observed data update time from write activity. | | `created_at` | Object creation timestamp. | | `updated_at` | Last altered timestamp. | | `row_count` | Current row count reported by Snowflake. | | `volume` | Current storage size in bytes. | | `projected_storage_cost` | Estimated monthly storage cost based on configured storage pricing. | | `table_properties` | Structured table or view details, including materialization state and view SQL when available. | | `table_schema` | Structured schema with column names, comments, and data types. | | `is_shared` | Indicates that the resource is part of a configured outbound share. | | `snowflake_query_history_upstream` | Upstream table URIs inferred from Snowflake access history. | | `upstream_temp_tables` | Details about temporary tables that participate in query-history lineage. | | `snowflake_privacy_category` | Snowflake privacy categories rolled up from classified columns in the table or view. | | `snowflake_masking_policy` | Snowflake masking policies found on columns in the table or view. | | `snowflake_contains_masked_columns` | Indicates that at least one column in the table or view has an effective masking policy. | | `snowflake_masked_column_count` | Number of columns in the table or view with an effective masking policy. | | `snowflake_sensitive_metadata_refreshed_at` | Timestamp when Euno last refreshed Snowflake classification or masking metadata for the resource. | Usage and cost optimization properties can also be set on table resources: | Property | Description | | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Number of read queries over the last 14, 30, or 60 days. | | `total_read_credits_14d`, `total_read_credits_30d`, `total_read_credits_60d` | Total read compute credits over the last 14, 30, or 60 days. | | `average_read_credits_14d`, `average_read_credits_30d`, `average_read_credits_60d` | Average read compute credits per query over the last 14, 30, or 60 days. | | `total_read_cost_14d`, `total_read_cost_30d`, `total_read_cost_60d` | Total read cost over the last 14, 30, or 60 days. | | `average_read_cost_14d`, `average_read_cost_30d`, `average_read_cost_60d` | Average read cost per query over the last 14, 30, or 60 days. | | `total_write_queries_14d`, `total_write_queries_30d`, `total_write_queries_60d` | Number of write operations over the last 14, 30, or 60 days. | | `total_write_credits_14d`, `total_write_credits_30d`, `total_write_credits_60d` | Total write compute credits over the last 14, 30, or 60 days. | | `average_write_credits_14d`, `average_write_credits_30d`, `average_write_credits_60d` | Average write compute credits per operation over the last 14, 30, or 60 days. | | `total_write_cost_14d`, `total_write_cost_30d`, `total_write_cost_60d` | Total write cost over the last 14, 30, or 60 days. | | `average_write_cost_14d`, `average_write_cost_30d`, `average_write_cost_60d` | Average write cost per operation over the last 14, 30, or 60 days. | | `total_write_runtime_14d`, `total_write_runtime_30d`, `total_write_runtime_60d` | Total runtime of write operations over the last 14, 30, or 60 days. | | `total_dml_queries_14d`, `total_dml_queries_30d`, `total_dml_queries_60d` | Deprecated DML query counts retained for compatibility. | | `total_dml_runtime_14d`, `total_dml_runtime_30d`, `total_dml_runtime_60d` | Deprecated DML runtime metrics retained for compatibility. | | `total_dml_credits_14d`, `total_dml_credits_30d`, `total_dml_credits_60d` | Deprecated DML credit metrics retained for compatibility. | | `total_dml_cost_14d`, `total_dml_cost_30d`, `total_dml_cost_60d` | Deprecated DML cost metrics retained for compatibility. | | `total_queries_14d`, `total_queries_30d`, `total_queries_60d` | Total query count over the last 14, 30, or 60 days. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Number of distinct users who queried the table over the last 14, 30, or 60 days. | {% hint style="warning" %} Snowflake usage metrics appear only when Euno can observe usable Snowflake query or access history. If that history is unavailable or yields no usable events, these fields may remain empty instead of showing `0`. {% endhint %} #### Column (`column`) Snowflake table and view columns are represented as `column` resources. | Property | Description | | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `name` | Column name. | | `type` | Always `column`. | | `subtype` | Always `column`. | | `description` | Snowflake column comment, when present. | | `parent_container` | Parent `table`. | | `parent_name` | Parent table or view name. | | `native_data_type` | Snowflake data type. | | `normalized_data_type` | Euno-normalized data type. | | `database_technology` | Always `snowflake`. | | `database_database` | Snowflake database name. | | `database_schema` | Snowflake schema name. | | `snowflake_semantic_category` | Snowflake semantic category values applied to the column through Snowflake classification tags. | | `snowflake_privacy_category` | Snowflake privacy category values applied to the column. | | `snowflake_classification_profile` | Classification profiles that produced observed classifications, when Snowflake provides them. | | `snowflake_classification_tag_values` | Values from Snowflake classification tags on the column, including user-defined classification tags. | | `snowflake_contains_classified_paths` | Indicates that Snowflake reported classified paths inside a semi-structured column. | | `snowflake_masking_policy` | Effective Snowflake masking policies attached to the column, either directly or through tag-based masking. | | `snowflake_masking_source` | How masking policies are attached to the column: `direct` or `tag_based`. | | `snowflake_is_masked` | Indicates that the column has an effective masking policy. | | `snowflake_sensitive_metadata_refreshed_at` | Timestamp when Euno last refreshed Snowflake classification or masking metadata for the resource. | | `snowflake_query_history_upstream_columns` | Upstream column URIs inferred from Snowflake access history. | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Column read query counts over the last 14, 30, or 60 days. Only present when column-level usage is enabled. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users who read the column over the last 14, 30, or 60 days. Only present when column-level usage is enabled. | Detailed classification and masking information may also appear in the Data Model's **Sensitive Data** tab, including classification tag names, tag values, classification profiles, masking-policy source, and tag-based masking context. #### Snowflake semantic view (`snowflake_semantic_view`) Snowflake semantic views are observed when semantic view discovery is enabled. | Property | Description | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | `name` | Semantic view name. | | `type` | Always `snowflake_semantic_view`. | | `description` | Semantic view description from Snowflake, when present. | | `parent_container` | Parent `database_schema`. | | `native_raw_object` | Parsed semantic view YAML definition. | | `raw_code` | Raw semantic view YAML definition. | | `database_database` | Snowflake database name. | | `database_schema` | Snowflake schema name. | | `database_technology` | Always `snowflake`. | | `platform_uri_prefix` | Snowflake URI prefix used to generate resource URIs. | | `created_at` | Semantic view creation timestamp. | | `updated_at` | Last altered timestamp. | | `native_owners` | Snowflake owner role reference. | | `table_dependencies` | Upstream Snowflake table URIs used by the semantic view. | | `native_id` | Native semantic view identifier. | | `external_links` | Link to open the semantic view in Snowflake, when host and object context are available. | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Semantic view read query counts over the last 14, 30, or 60 days. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users who queried the semantic view over the last 14, 30, or 60 days. | | `total_read_credits_14d`, `total_read_credits_30d`, `total_read_credits_60d` | Semantic view read credits over the last 14, 30, or 60 days. | #### Snowflake dimension (`snowflake_dimension`) Snowflake dimensions are child resources of a Snowflake semantic view. | Property | Description | | ---------------------- | ----------------------------------------------------------------- | | `name` | Dimension name. | | `type` | Always `snowflake_dimension`. | | `parent_container` | Parent `snowflake_semantic_view`. | | `upstream_fields` | Upstream Snowflake column URIs used by the dimension expression. | | `native_data_type` | Snowflake data type. | | `normalized_data_type` | Euno-normalized data type. | | `raw_code` | Dimension expression. | | `created_at` | Dimension creation timestamp, when present in Snowflake metadata. | | `updated_at` | Dimension update timestamp, when present in Snowflake metadata. | | `description` | Dimension description, when present. | | `synonyms` | Dimension synonyms from Snowflake metadata. | | `native_raw_object` | Raw dimension object from the semantic view definition. | | `native_id` | Native dimension identifier. | #### Snowflake metric (`snowflake_metric`) Snowflake metrics are child resources of a Snowflake semantic view. | Property | Description | | ---------------------- | -------------------------------------------------------------- | | `name` | Metric name. | | `type` | Always `snowflake_metric`. | | `subtype` | `derived` for metrics, or `base` when emitted from facts. | | `parent_container` | Parent `snowflake_semantic_view`. | | `upstream_fields` | Upstream Snowflake column URIs used by the metric expression. | | `native_data_type` | Snowflake data type. | | `normalized_data_type` | Euno-normalized data type. | | `raw_code` | Metric expression. | | `aggregation` | Metric aggregation, when provided by Snowflake. | | `created_at` | Metric creation timestamp, when present in Snowflake metadata. | | `updated_at` | Metric update timestamp, when present in Snowflake metadata. | | `description` | Metric description, when present. | | `synonyms` | Metric synonyms from Snowflake metadata. | | `native_raw_object` | Raw metric object from the semantic view definition. | | `native_id` | Native metric identifier. | #### Snowflake inbound share (`snowflake_inbound_share`) Inbound shares are observed when **Observe inbound Snowflake shares** is enabled. | Property | Description | | --------------------- | ------------------------------------------------------------------------------------------------ | | `name` | Share name. | | `type` | Always `snowflake_inbound_share`. | | `subtype` | Always `inbound_share`. | | `description` | Share comment from Snowflake, when present. | | `native_id` | Snowflake account and share identifier. | | `parent_container` | Parent `snowflake_account`. | | `database_technology` | Always `snowflake`. | | `native_raw_object` | Raw share metadata from Snowflake. | | `native_owners` | Share owner, when present. | | `defines` | Database URI defined by the share, when Euno can match it to an observed database. | | `table_dependencies` | Matching outbound share URI, when an inbound and outbound share with the same name are observed. | #### Snowflake outbound share (`snowflake_outbound_share`) Outbound shares are observed from the manually configured outbound shares list. | Property | Description | | ------------------ | ------------------------------------------------------------------------------------------ | | `name` | Share name. | | `type` | Always `snowflake_outbound_share`. | | `subtype` | Always `outbound_share`. | | `parent_container` | Parent `snowflake_account`. | | `defines` | Database URI shared by the outbound share, when Euno can match it to an observed database. | #### Hex project (`hex_project`) Hex projects can be observed from Snowflake query history when Hex usage or Hex lineage extraction is enabled. These observations enrich Hex project resources with Snowflake usage and lineage. | Property | Description | | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | `name` | Hex project name, when available. | | `type` | Always `hex_project`. | | `subtype` | Always `hex_project`. | | `native_id` | Hex project ID. | | `platform_uri_prefix` | Snowflake URI prefix used to resolve table and column lineage. | | `table_dependencies` | Snowflake table URIs used by the Hex project. | | `upstream_fields` | Snowflake column URIs used by the Hex project. | | `total_queries_14d`, `total_queries_30d`, `total_queries_60d` | Hex project query counts over the last 14, 30, or 60 days. | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Snowflake read query counts attributed to the Hex project. | | `total_read_credits_14d`, `total_read_credits_30d`, `total_read_credits_60d` | Snowflake read credits attributed to the Hex project. | | `total_read_cost_14d`, `total_read_cost_30d`, `total_read_cost_60d` | Snowflake read cost attributed to the Hex project. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct Hex users over the last 14, 30, or 60 days. | | `queries_breakdown_by_user_14d`, `queries_breakdown_by_user_30d`, `queries_breakdown_by_user_60d` | Query breakdown by Hex user. | | `used` | Users who used the Hex project, with recent activity counts. | | `hex_project_latest_sql_by_cell` | Latest observed SQL by Hex cell, used to derive lineage. | ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ----------------------------------------------------- | -------------------- | -------------------------- | ------------------------------------------------------------------------------------------------ | | `database` | has parent | `snowflake_account` | Databases are contained by the Snowflake account. | | `database_schema` | has parent | `database` | Schemas are contained by databases. | | `table` | has parent | `database_schema` | Tables, views, materialized views, and temporary tables are contained by schemas. | | `snowflake_semantic_view` | has parent | `database_schema` | Semantic views are contained by schemas. | | `column` | has parent | `table` | Columns are contained by tables and views. | | `snowflake_dimension`, `snowflake_metric` | has parent | `snowflake_semantic_view` | Dimensions and metrics are contained by semantic views. | | `snowflake_inbound_share`, `snowflake_outbound_share` | has parent | `snowflake_account` | Shares are contained by the Snowflake account. | | `table` | has upstream | `table` | Table-level lineage from view definitions and Snowflake access history. | | `column` | has upstream\_fields | `column` | Column-level lineage from Snowflake access history. | | `snowflake_semantic_view` | has upstream | `table` | Semantic views depend on their base Snowflake tables. | | `snowflake_dimension`, `snowflake_metric` | has upstream\_fields | `column` | Semantic view dimensions and metrics depend on Snowflake columns used in their expressions. | | `hex_project` | has upstream | `table` | Hex projects depend on Snowflake tables referenced by Hex SQL. | | `hex_project` | has upstream\_fields | `column` | Hex projects depend on Snowflake columns referenced by Hex SQL. | | `snowflake_inbound_share`, `snowflake_outbound_share` | defines | `database` | Shares define the databases they expose when Euno can match those databases. | | `snowflake_inbound_share` | has upstream | `snowflake_outbound_share` | Matching inbound and outbound shares with the same name establish cross-account sharing lineage. | {% hint style="info" %} For setup details related to inbound and outbound shares, see [Observing Snowflake Shares](/sources/data-warehouses/snowflake-integration/observing-snowflake-shares-and-cross-snowflake-account). {% endhint %} # Observing Snowflake Shares Euno observes inbound and outbound Snowflake shares as part of each Snowflake integration run. If Euno observes matching outbound and inbound Snowflake shares, it automatically creates lineage between objects in the inbound database and objects in the outbound database. This provides lineage across Snowflake accounts for teams with cross-account sharing setups. ## Observing Inbound Shares To observe inbound shares, check **Observe Inbound Snowflake Shares** under **Advanced** in the Snowflake source configuration screen. Make sure to grant the Snowflake role used by Euno the `IMPORT SHARE` privilege: ``` GRANT IMPORT SHARE ON ACCOUNT TO ROLE euno_role; ``` Then, for each of the imported databases, grant `IMPORTED PRIVILEGES`: ``` GRANT IMPORTED PRIVILEGES ON DATABASE TO ROLE euno_role; ``` ## Observing Outbound Shares Euno does not automatically discover outbound shares because Snowflake only allows outbound share owners and the `ACCOUNTADMIN` role to see these shares. To observe outbound shares, configure them in the source configuration screen. Run `SHOW SHARES` as `ACCOUNTADMIN` to get a complete list of outbound shares in the account. 1. Click **Advanced** to unfold the advanced configuration section. 2. Click **Outbound shares** to unfold the outbound shares table.
3. Edit the outbound shares table by adding, editing, and removing entries. Make sure to copy the exact names of the share and database.
4. Save the configuration. # BigQuery BigQuery is Google Cloud Platform's analytical data warehouse. Euno's BigQuery integration discovers resources from BigQuery metadata and query history so teams can understand warehouse structure, lineage, and usage. Euno's BigQuery integration supports auto-discovery of: * GCP projects * BigQuery databases and datasets * BigQuery tables, views, materialized views, external tables, snapshots, and clones * BigQuery columns, including nested STRUCT fields * Google Sheets referenced by BigQuery external tables * BigQuery usage, cost, and lineage properties from query history * Hex project usage and lineage from BigQuery query history * Optional Sigma usage from BigQuery query history ## Prerequisites Before you configure the integration, make sure you have: * A GCP project that Euno can use to run BigQuery metadata queries. This is the execution project. * The list of GCP projects Euno should discover. These are the discovered projects. * Permission to create a GCP service account and service account key. * Permission to grant IAM roles on every discovered project. * The BigQuery regions where Euno should scan query history, for example `us`, `eu`, or `us-central1`. To discover BigQuery resources, Euno queries system-defined, read-only [INFORMATION\_SCHEMA](https://cloud.google.com/bigquery/docs/information-schema-intro) views. A BigQuery query runs under one project and can access tables and views in other projects, so the execution project can be different from the discovered projects. ```mermaid graph LR exe[execution project] --> dis1[discovered project 1] exe --> dis2[discovered project 2] euno[Euno integration] -->|run queries| exe ``` ## Stage 1: Configure GCP ### Step 1: Create a GCP Service Account on the Execution Project 1. In GCP, choose the execution project. 2. Browse to the [Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) page. 3. Click **Create service account**. 4. Enter a service account name, for example `Euno BigQuery Integration`. 5. Copy the service account email address, for example `euno-bigquery-integration@my-project.iam.gserviceaccount.com`. 6. Click **Create and continue**. 7. Under **Permissions**, choose the **BigQuery User** role. The technical identifier is `roles/bigquery.user`. 8. Click **Continue**, then click **Done**. ### Step 2: Create a Service Account Key 1. Go to the service account page for the service account you created. 2. On the **Keys** tab, click **Add key**, then click **Create new key**. 3. Choose **JSON** as the key type. 4. Click **Create** and save the file. ### Step 3: Grant Access to Discovered Projects For each discovered project: 1. Browse to the project's [IAM configuration](https://console.cloud.google.com/iam-admin/iam). 2. Click **Grant access**. 3. Under **New principals**, enter the service account email address from Step 1. 4. Under **Assign roles**, choose **BigQuery Metadata Viewer**. The technical identifier is `roles/bigquery.metadataViewer`. 5. Click **Add another role** and choose **BigQuery Resource Viewer**. The technical identifier is `roles/bigquery.resourceViewer`. 6. Click **Save**. ## Stage 2: Configure New BigQuery Source in Euno ### Step 1: Access the Sources Page In Euno, open the **Sources** page and click **Add New Source**. Select **BigQuery**. ### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | **Service Account Key**\* | Copy the entire contents of the service account key JSON file and paste it here. | | **Query History Regions**\* | Comma-separated list of BigQuery regions to scan for query history, for example `us-east1,us-west1`. | | **Execution Project ID** | GCP project ID used as the execution project. If not specified, Euno uses the project where the service account is defined. | | **Query Location** | BigQuery location where Euno should run queries. | ### Step 3: Advanced Settings Click **Advanced** to display additional configurations. | Configuration | Description | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Discover dataset labels as meta** | Discover [BigQuery dataset labels](https://cloud.google.com/bigquery/docs/labels-intro) as `meta` values for BigQuery dataset resources. Defaults to disabled. | | **Auto discover Sigma usage from BigQuery query history** | Detect Sigma SQL footers in BigQuery query history and use them to populate Sigma usage. | | **Sigma usage lookback (days)** | Number of days to look back when detecting Sigma usage from BigQuery query history. Defaults to `7`. | | **Cost per slot per hour (USD)** | Cost used for BigQuery cost calculations. Defaults to `0.04`. | | **Project Discovery Pattern** | Use regular expressions to allow or exclude specific GCP projects. `.*` includes or excludes all projects. | | **Column Observation - Dataset Pattern** | Use regular expressions to allow or exclude specific datasets from column observation. `.*` includes or excludes all datasets. | | **Column Observation - Table Pattern** | Use regular expressions in `dataset.table` format to allow or exclude specific tables from column observation. For example, `bq_log\..*` excludes all tables in the `bq_log` dataset. | ### Step 4: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours, for example every 8 hours. ### Step 5: Resource Cleanup * **Immediate Cleanup**: Remove resources not detected in the most recent successful source run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. To keep your data relevant and free of outdated resources, Euno provides automatic **Resource Cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source run. For a detailed explanation, see [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). ### Step 6: Save Configuration Click **Test & Save** to complete the setup. ## What Euno Discovers Euno discovers BigQuery resources including GCP projects, databases, datasets, tables, columns, Google Sheets referenced by external tables, and Hex projects detected from BigQuery query history. For detailed information about discovered resources and their properties, see [BigQuery Integration Discovered Resources](/sources/data-warehouses/bigquery-integration/bigquery-integration-discovered-resources). # BigQuery Integration Discovered Resources ### Ingested Resources & Properties #### BigQuery Project (`gcp_project`) | Property | Description | | ----------- | -------------------- | | `name` | The GCP project ID | | `type` | Always `gcp_project` | | `native_id` | The GCP project ID | #### Database (`database`) In BigQuery, Euno represents each discovered GCP project as a `database` resource. | Property | Description | | ------------------ | ------------------------------------------------- | | `name` | The GCP project ID | | `type` | Always `database` | | `native_id` | The GCP project ID | | `subtype` | Always `database` | | `parent_container` | The `gcp_project` containing the BigQuery project | #### Dataset (`database_schema`) | Property | Description | | --------------------- | ------------------------------------------------------------------ | | `name` | The BigQuery dataset name | | `type` | Always `database_schema` | | `subtype` | Always `database_schema` | | `created_at` | Date and time when the dataset was created | | `updated_at` | Date and time when the dataset was last modified | | `parent_container` | The `database` resource containing the dataset | | `database_technology` | Always `bigquery` | | `location` | The BigQuery dataset location | | `meta` | Dataset labels when **Discover dataset labels as meta** is enabled | #### Table (`table`) | Property | Description | | ------------------------- | ------------------------------------------------------------------------------------------ | | `name` | The BigQuery table, view, materialized view, external table, snapshot, or clone name | | `type` | Always `table` | | `native_id` | Fully qualified BigQuery object ID in `project.dataset.table` format | | `subtype` | One of `base_table`, `view`, `materialized_view`, `external_table`, `snapshot`, or `clone` | | `created_at` | Date and time when the table was created | | `updated_at` | Date and time when table storage was last modified | | `native_last_data_update` | Last refresh time for materialized views, snapshots, and clones when available | | `parent_container` | The `database_schema` resource containing the table | | `database_technology` | Always `bigquery` | | `database_schema` | Parent **schema** (BigQuery dataset name) | | `meta` | Reserved for labels and metadata when available | #### Column (`column`) | Property | Description | | --------------------- | ------------------------------------------------------------------------------------------- | | `name` | The column name. Nested STRUCT fields use their full field path, for example `profile.city` | | `type` | Always `column` | | `subtype` | `struct` for root STRUCT columns | | `description` | Column description when available | | `parent_name` | The table name | | `parent_container` | The `table` resource containing the column | | `database_technology` | Always `bigquery` | | `database_schema` | Parent **schema** (BigQuery dataset name) | #### Google Sheet (`google_sheet`) Google Sheets are discovered when they are referenced as external tables in BigQuery with `format = 'GOOGLE_SHEETS'`. | Property | Description | | ---------------- | ----------------------------------- | | `name` | Always `Google Sheet` | | `type` | Always `google_sheet` | | `external_links` | Link to the referenced Google Sheet | #### Hex Project (`hex_project`) Hex projects are detected from BigQuery query history when Hex query metadata is present and the corresponding Hex project already exists in Euno. | Property | Description | | -------- | ------------------------------------------------------------ | | `name` | The Hex project name when present in BigQuery query metadata | | `type` | Always `hex_project` | {% hint style="info" %} Hex projects also receive internal BigQuery query metadata used for lineage. Which BigQuery tables and columns a Hex project depends on appears under **Lineage** and in the **Relationships** section below, not as named fields in resource details. {% endhint %} ### BigQuery Usage & Optimization Properties Usage and cost optimization properties are derived from BigQuery query history. For base tables and materialized views, read and write usage properties are derived from `INFORMATION_SCHEMA.JOBS` and `INFORMATION_SCHEMA.JOBS_BY_PROJECT`. For `subtype = 'view'`, `total_read_queries_{14d,30d,60d}` reflects queries that referenced the view. Other read optimization properties, such as bytes, slots, and runtime, reflect underlying table processing and may differ. #### Read Properties | Property | Description | | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Total number of read queries referencing the resource over the last 14, 30, or 60 days. Applies to tables and columns. | | `total_read_slots_14d`, `total_read_slots_30d`, `total_read_slots_60d` | Total slots consumed by read queries referencing the table over the last 14, 30, or 60 days. | | `average_read_slots_14d`, `average_read_slots_30d`, `average_read_slots_60d` | Average slots consumed per read query referencing the table over the last 14, 30, or 60 days. | | `total_read_bytes_processed_14d`, `total_read_bytes_processed_30d`, `total_read_bytes_processed_60d` | Total bytes processed by read queries referencing the table over the last 14, 30, or 60 days. | | `average_read_bytes_processed_14d`, `average_read_bytes_processed_30d`, `average_read_bytes_processed_60d` | Average bytes processed per read query referencing the table over the last 14, 30, or 60 days. | | `total_read_runtime_14d`, `total_read_runtime_30d`, `total_read_runtime_60d` | Total runtime of read queries referencing the table over the last 14, 30, or 60 days. | | `total_read_cost_14d`, `total_read_cost_30d`, `total_read_cost_60d` | Estimated read cost over the last 14, 30, or 60 days, based on the configured cost per slot hour. | | `average_read_cost_14d`, `average_read_cost_30d`, `average_read_cost_60d` | Estimated average read cost per query over the last 14, 30, or 60 days. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Number of distinct users who queried the table or column over the last 14, 30, or 60 days. | | `distinct_impressions_users_14d`, `distinct_impressions_users_30d`, `distinct_impressions_users_60d` | Number of distinct users from query-history based column impressions over the last 14, 30, or 60 days. | #### Write Properties | Property | Description | | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `total_write_queries_14d`, `total_write_queries_30d`, `total_write_queries_60d` | Total number of write operations involving the table over the last 14, 30, or 60 days. | | `total_write_slots_14d`, `total_write_slots_30d`, `total_write_slots_60d` | Total slots consumed by write operations involving the table over the last 14, 30, or 60 days. | | `average_write_slots_14d`, `average_write_slots_30d`, `average_write_slots_60d` | Average slots consumed per write operation involving the table over the last 14, 30, or 60 days. | | `total_write_bytes_processed_14d`, `total_write_bytes_processed_30d`, `total_write_bytes_processed_60d` | Total bytes processed by write operations involving the table over the last 14, 30, or 60 days. | | `average_write_bytes_processed_14d`, `average_write_bytes_processed_30d`, `average_write_bytes_processed_60d` | Average bytes processed per write operation involving the table over the last 14, 30, or 60 days. | | `total_write_runtime_14d`, `total_write_runtime_30d`, `total_write_runtime_60d` | Total runtime of write operations involving the table over the last 14, 30, or 60 days. | | `total_write_cost_14d`, `total_write_cost_30d`, `total_write_cost_60d` | Estimated write cost over the last 14, 30, or 60 days, based on the configured cost per slot hour. | #### Storage Properties Storage properties reflect the current size of the table as stored in BigQuery. | Property | Description | | ----------- | ---------------------------------------- | | `row_count` | Number of rows in the table. | | `volume` | Current logical storage volume in bytes. | ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ----------------- | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `gcp_project` | has child | `database` | The BigQuery project is represented as a `database` under its GCP project. | | `database` | has child | `database_schema` | BigQuery datasets are represented as `database_schema` resources under the BigQuery project. | | `database_schema` | has child | `table` | BigQuery tables, views, materialized views, external tables, snapshots, and clones are contained in datasets. | | `table` | has child | `column` | BigQuery columns and nested STRUCT fields are contained in tables. | | `table` | has upstream | `table` | For views and materialized views, lineage is based on SQL analysis. For base tables, lineage can be inferred from BigQuery query history. | | `column` | has upstream\_fields | `column` | Column-level lineage is based on SQL analysis and BigQuery query history. | | `table` | has upstream\_fields | `column` | Statement-level dependencies include fields used by clauses such as `JOIN`, `WHERE`, `GROUP BY`, `HAVING`, `QUALIFY`, and `ORDER BY`, even when those fields are not selected in the final projection. | | `table` | has upstream | `google_sheet` | External tables with `format = 'GOOGLE_SHEETS'` depend on the referenced Google Sheet. | | `hex_project` | has upstream | `table` | Hex project table lineage is inferred from BigQuery query history. | | `hex_project` | has upstream\_fields | `column` | Hex project field lineage is inferred from BigQuery query history. | ### What to expect from table-level field lineage For BigQuery views, table-level lineage in Euno may include columns that are used only by SQL logic and are not selected as output columns. Example: ```sql SELECT o.order_id, o.customer_id FROM `proj_sales.analytics.orders_raw` o WHERE o.order_status = 'COMPLETE' ``` In this case, table-level lineage for the view can include the `order_status` column from `orders_raw`, even though `order_status` is not part of the projection. ### Known Limitations * ARRAY of STRUCT paths through UNNEST are supported, but element-level instance or index lineage is not tracked. * Fields accessed via dynamic SQL or UDFs may not be captured. * Some complex UNNEST operations with multiple levels may not fully resolve. * Cross-project nested field lineage is most accurate when BigQuery metadata and query history are available for all referenced projects. # STRUCT Support ### Overview Euno provides **column-level lineage tracking for BigQuery STRUCT columns**, enabling you to trace data flow through nested and complex data structures. This enhancement allows you to understand exactly which nested fields within STRUCT columns are being used, transformed, and propagated throughout your data pipeline. ### Capabilities #### 1. Nested Field Discovery Euno automatically discovers and catalogs all nested fields within STRUCT columns, creating individual column resources for each field path. **Example Table Structure:** ```sql CREATE TABLE users ( user_id INT64, profile STRUCT< name STRING, email STRING, street STRING, city STRING, zip STRING > ) ``` **What You See in Euno:** In the Data Model screen, you'll see these columns: * `user_id` (INT64) * `profile` (STRUCT) - marked with `subtype: struct` * `profile.name` (STRING) * `profile.email` (STRING) * `profile.street` (STRING) * `profile.city` (STRING) * `profile.zip` (STRING) Each nested field appears as a separate, searchable column resource with its own URI, allowing you to track it independently. #### 2. Column-Level Lineage When queries reference specific fields within STRUCT columns, Euno traces lineage to the exact nested field, not just the parent STRUCT. **Example Query:** ```sql SELECT user_id, profile.name AS user_name, profile.city AS user_city FROM users ``` **Lineage Tracking:** * `user_name` column traces back to `users.profile.name` (not just `users.profile`) #### 3. Accurate Usage Metrics Column-level usage statistics now include nested STRUCT fields, showing you: * How many queries reference each nested field * Usage frequency for each field path **Example Usage Report:** | Column | Total Queries | Distinct Users | | ---------------- | ------------- | -------------- | | `profile.name` | 1,247 | 23 | | `profile.email` | 892 | 18 | | `profile.city` | 456 | 12 | | `profile.street` | 89 | 5 | | `profile.zip` | 67 | 4 | This data helps you: * **Optimize schemas:** Identify rarely-used nested fields that could be removed * **Understand access patterns:** See which parts of your STRUCT columns are most valuable * **Plan migrations:** Know which fields are critical before restructuring #### 4. Lineage Through Complex Transformations Euno tracks nested field lineage through complex SQL operations: **Example: Extracting and Transforming Nested Fields** ```sql CREATE TABLE customer_summary AS SELECT user_id, profile.name AS customer_name, UPPER(profile.city) AS city_upper, profile.zip AS postal_code FROM users WHERE profile.city IS NOT NULL ``` **Lineage Captured:** * `customer_summary.customer_name` traces back to `users.profile.name` * `customer_summary.city_upper` traces back to `users.profile.city` * `customer_summary.postal_code` traces back to `users.profile.zip` **Example: Aggregating Nested Fields** ```sql CREATE TABLE city_stats AS SELECT profile.city, COUNT(*) AS user_count, COUNT(DISTINCT profile.email) AS unique_emails FROM users GROUP BY profile.city ``` **Lineage Captured:** * `city_stats.city` traces back to `users.profile.city` * `city_stats.user_count` is an aggregation with no specific source column * `city_stats.unique_emails` traces back to `users.profile.email` with aggregation: COUNT DISTINCT ### Viewing STRUCT Columns in Euno #### In the Data Model Screen 1. **Navigate to a table** with STRUCT columns 2. **Click on the Columns tab** in the resource sidepane 3. **Identify STRUCT columns** by the `struct` badge 4. **Expand nested fields** to see all field paths 5. **Click on any nested field** to view its details and lineage #### Column Properties **Root STRUCT Column:** * **Type:** `column` * **Subtype:** `struct` * **Data Type:** `STRUCT<...>` (full schema definition) * **Normalized Type:** `object` **Nested Field:** * **Type:** `column` * **Name:** Full path (e.g., `profile.city`) * **Data Type:** The field's actual type (e.g., `STRING`, `INT64`) * **Parent Container:** The table URI #### Lineage View When viewing lineage for a nested field: 1. **Upstream:** Shows the source nested field (e.g., `source_table.profile.city`) 2. **Downstream:** Shows all columns and resources that depend on this nested field 3. **Usage:** Shows query count and user count for the field ### Limitations 1. **ARRAY of STRUCT:** Nested paths through UNNEST are supported, but element-level instance/index lineage is not tracked. 2. **Dynamic Field Access:** Fields accessed via dynamic SQL or UDFs may not be captured. 3. **Complex Unnesting:** Some complex UNNEST operations with multiple levels may not fully resolve. 4. **Cross-Project References:** Nested field lineage is most accurate when metadata and query history are available for all referenced projects. # Databricks Euno's Databricks integration connects to Unity Catalog metadata, Databricks system lineage and query history, and the Workspace API so you can discover catalogs, schemas, tables, columns, usage metrics, lineage, and notebooks in one source. ## What Euno Discovers Euno's Databricks integration supports auto-discovery of: * Databricks workspaces * Unity Catalog databases (catalogs) * Unity Catalog schemas * Unity Catalog tables, views, materialized views, and external tables * Table and column metadata (including tags) * Table usage metrics for 14/30/60 day windows * Column usage metrics for 14/30/60 day windows * Query-history-backed column lineage * Databricks notebooks and notebook-driven relationships For resource properties and relationships in detail, see [Databricks Integration Discovered Resources](/sources/data-warehouses/databricks-integration/databricks-integration-discovered-resources). ## Prerequisites / Requirements Before you add the source, make sure you have: * A Databricks workspace with Unity Catalog enabled * At least one running SQL warehouse and its HTTP path * The workspace hostname (for example `dbc-xxxxxxxx-xxxx.cloud.databricks.com`) * A Databricks personal access token for a user that can run metadata queries and access the objects you want to observe The token user should have: * Access to the Databricks workspace * Permission to use the configured SQL warehouse * `SELECT` access to the required `system.information_schema` and `system.access` / `system.query.history` objects used by the integration * Access to the catalogs, schemas, and tables you want to observe * Workspace API visibility to notebook paths you want to observe (under the default discovery roots) * To ingest column masking: the `MANAGE` privilege on, or ownership of, the observed schemas and tables whose ABAC policies must be read. The Unity Catalog policies API is queried on both levels so Euno can discover inherited catalog/schema policies and policies attached directly to a table. Column masking is not ingested for a scope whose policies cannot be read, and the run report identifies the affected scopes. ## Stage 1: Configure Databricks ### Step 1: SQL warehouse and hostname 1. Verify that Unity Catalog is enabled. 2. Ensure at least one SQL warehouse is available and running. 3. Note your workspace hostname (for example `dbc-xxxxxxxx-xxxx.cloud.databricks.com`). 4. Note the SQL warehouse HTTP path (for example `/sql/1.0/warehouses/warehouse-id`). ### Step 2: Personal access token 1. In Databricks, go to **Settings** β†’ **Developer** β†’ **Access tokens**. 2. Click **Generate new token**. 3. Set a comment (for example `Euno integration`). 4. Set token lifetime as needed. 5. Click **Generate**. 6. Copy and store the token securely. In Euno, the integration combines: * Unity Catalog metadata (`system.information_schema.*`) * Databricks system lineage and usage evidence (`system.access.*`, `system.query.history`) * Workspace notebook discovery (Workspace API) ## Stage 2: Configure New Databricks Source in Euno ### Step 1: Access the Sources Page 1. Open the **Sources** page. 2. Click **Add New Source** and choose **Databricks**. ### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------------- | ------------------------------------------------------------------------------------- | | **Server Hostname**\* | Databricks workspace hostname (for example `dbc-xxxxxxxx-xxxx.cloud.databricks.com`). | | **HTTP Path**\* | Databricks SQL warehouse HTTP path (for example `/sql/1.0/warehouses/warehouse-id`). | | **Access Token**\* | Databricks personal access token (stored as a secret). | | **Workspace Name** | Optional display name in Euno. If left blank, the hostname is used. | ### Step 3: Schedule * Enable scheduling. * Choose one option: 1. **Weekly** with a specific day and time. 2. **Hourly** with an interval. ### Step 4: Resource Cleanup * **Immediate Cleanup**: remove resources not detected in the latest successful run. * **No Cleanup**: keep resources even if they are no longer detected. For cleanup semantics, see [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). ### Step 5: Advanced Settings Open the **Advanced** section to configure optional filters. | Configuration | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Override Base URI** | Override the hostname used when Euno generates resource URIs. Leave blank to use **Server Hostname**. | | **Database Pattern** | Allow/deny regular expressions for catalog names. Defaults to include all catalogs; system catalogs such as `system` and `samples` are always excluded from metadata queries. | **Pattern examples** * `.*` β€” include all names that match. * `production_.*` β€” include only names starting with `production_`. * Allow `.*` and deny `test_.*` β€” include all except names starting with `test_`. ### Step 6: Save Click **Test & Save** to validate connectivity and save the source. ## Important behavior notes * Table read usage metrics are emitted for 14/30/60 day windows. * Databricks read DBU metrics are emitted as `total_read_dbu_14d`, `total_read_dbu_30d`, `total_read_dbu_60d` and `average_read_dbu_14d`, `average_read_dbu_30d`, `average_read_dbu_60d`. * Column usage metrics are emitted for 14/30/60 day windows. * Notebook execution evidence and notebook-derived relationships use `system.query.history`, `system.access.table_lineage`, and `system.access.column_lineage` with a 30-day lookback window. * Regular Databricks column lineage is read from `system.access.column_lineage` with a 30-day lookback; Euno represents it as column-level upstream field relationships after processing (see the discovered-resources page). * Notebook observation runs by default; scope follows the integration’s discovery rules described in the hint under **Advanced Settings**. # Databricks Integration Discovered Resources The Euno Databricks integration discovers and updates the following resource families: * Databricks workspace * Unity Catalog databases (catalogs) * Unity Catalog schemas * Databricks tables (including views, materialized views, and external tables) * Databricks table columns * Databricks notebooks * Usage and lineage properties across the resources above ## Discovery inputs The integration combines multiple Databricks data sources: * `system.information_schema.*` for metadata * `system.information_schema.column_masks` and the tag views (`catalog_tags`, `schema_tags`, `table_tags`, `column_tags`) for column masking * `system.query.history` for usage and notebook execution evidence * `system.access.table_lineage` for table lineage evidence * `system.access.column_lineage` for regular column-level lineage and notebook reporting evidence * Workspace API for notebook discovery * Unity Catalog policies API for ABAC column-mask policies ## Resource hierarchy The Databricks integration follows this container hierarchy: ``` Databricks Workspace └── Catalog (Database) └── Schema β”œβ”€β”€ Table / View / Materialized View / External Table β”‚ └── Column └── ... Databricks Workspace └── Notebook ``` ### Ingested Resources & Properties #### Databricks workspace (`databricks_workspace`) | Property | Description | | --------------------- | --------------------------------------------------------------------------------------------------- | | `name` | Workspace display name (from configuration; defaults to hostname if **Workspace Name** is not set). | | `type` | Always `databricks_workspace`. | | `subtype` | Always `databricks_workspace`. | | `description` | Short generated description that includes the workspace hostname. | | `database_technology` | Always `databricks`. | **URI pattern** ``` platform.databricks.databricks_workspace. ``` Example: `platform.databricks.databricks_workspace.dbc-50e7cad0-c2f9.cloud.databricks.com` #### Unity Catalog database (`database`) | Property | Description | | -------------------------- | ------------------------------------------------- | | `name` | Catalog name. | | `type` | Always `database`. | | `subtype` | Always `database`. | | `parent_container` | Parent workspace URI. | | `database_technology` | Always `databricks`. | | `description` | Catalog comment from Unity Catalog, when present. | | `native_owners` | Catalog owner from Unity Catalog metadata. | | `created_at`, `updated_at` | Catalog timestamps when available. | | `created_by`, `updated_by` | Creator and last modifier when available. | | `tags` | Tag names with empty or null values. | | `meta` | Tag key–value pairs when values are non-empty. | **URI pattern** ``` databricks.. ``` #### Unity Catalog schema (`database_schema`) | Property | Description | | -------------------------- | ---------------------------------------------- | | `name` | Schema name. | | `type` | Always `database_schema`. | | `subtype` | Always `database_schema`. | | `parent_container` | Parent catalog URI. | | `database_technology` | Always `databricks`. | | `database` | Catalog name. | | `description` | Schema comment when present. | | `native_owners` | Schema owner when present. | | `created_at`, `updated_at` | Schema timestamps when available. | | `created_by`, `updated_by` | Creator and last modifier when available. | | `tags` | Tag names with empty or null values. | | `meta` | Tag key–value pairs when values are non-empty. | **URI pattern** ``` databricks... ``` #### Databricks table (`table`) Databricks table resources use these subtypes: `base_table`, `external_table`, `view`, `materialized_view`. | Property | Description | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Table or view name. | | `type` | Always `table`. | | `subtype` | One of the Databricks subtypes above. | | `parent_container` | Parent schema URI. | | `database` | Catalog name. | | `database_schema` | Schema name. | | `database_technology` | Always `databricks`. | | `description` | Table or view comment when present. | | `native_owners` | Owner when present. | | `created_at`, `updated_at` | Object timestamps when available. | | `created_by`, `updated_by` | Creator and last modifier when available. | | `tags` | Tag names with empty or null values. | | `meta` | Tag key–value pairs when values are non-empty. | | `sql_dialect` | SQL dialect (`databricks`). | | `platform_uri_prefix` | URI prefix for this workspace (`databricks.`). | | `table_properties.materialized` | Whether the object is materialized, derived from Databricks table type and view metadata. | | `table_schema` | Structured column list for observed columns. | | `table_dependencies` | Upstream tables and views for lineage and dependencies. Values combine warehouse lineage from `system.access.table_lineage`, dependencies parsed from view SQL when applicable, and other merged inputs; Euno exposes the result as this resource’s upstream table relationships. | | `databricks_contains_masked_columns` | Indicates that at least one column in the table or view is protected by a column mask. | | `databricks_masked_column_count` | Number of columns in the table or view protected by a column mask. | | `databricks_masking_refreshed_at` | Timestamp when Euno last refreshed Databricks column-mask metadata for the resource. | **URI pattern** ``` databricks.... ``` **System exclusions** The crawler skips catalog discovery for `information_schema` and `hive_metastore`. Metadata queries for tables, schemas, and related objects exclude the `system` and `samples` catalogs and the `information_schema` schema. #### Table usage metrics Usage windows are emitted for `14d`, `30d`, and `60d` intervals. #### Read usage properties Applicable to table resources (`base_table`, `external_table`, `view`, `materialized_view`). | Property family | Description | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Number of read queries. | | `total_read_runtime_14d`, `total_read_runtime_30d`, `total_read_runtime_60d` | Total active compute runtime (ms) for read queries. | | `total_read_bytes_processed_14d`, `total_read_bytes_processed_30d`, `total_read_bytes_processed_60d` | Total read bytes. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users running read queries. | | `total_read_dbu_14d`, `total_read_dbu_30d`, `total_read_dbu_60d` | Total Databricks read DBU-seconds (compute-time proxy). | | `average_read_dbu_14d`, `average_read_dbu_30d`, `average_read_dbu_60d` | Average read DBU-seconds per read query. | Read DBU values are derived from query-history active compute time. #### Write usage properties Write applicability follows Databricks `table_type` classification: | Observed subtype | Databricks `table_type` values | Write usage applicable? | Behavior | | ------------------- | ------------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------- | | `view` | `VIEW` | No | Write metrics are emitted as zeros each crawl to clear stale values. | | `materialized_view` | `MATERIALIZED_VIEW` | Yes | Write metrics from write statements; zero-filled when there is no in-window activity. | | `base_table` | `MANAGED`, `STREAMING_TABLE`, `MANAGED_SHALLOW_CLONE` | Yes | Same as materialized view. | | `external_table` | `EXTERNAL`, `EXTERNAL_SHALLOW_CLONE`, `EXTERNAL_TABLE` | Yes | Same as materialized view. | | `external_table` | `FOREIGN` | No | Write metrics are not applicable. | | Property family | Meaning | | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | `total_write_queries_14d`, `total_write_queries_30d`, `total_write_queries_60d` | Number of write queries. | | `total_write_runtime_14d`, `total_write_runtime_30d`, `total_write_runtime_60d` | Total active compute runtime (ms) for write queries. | | `total_write_bytes_processed_14d`, `total_write_bytes_processed_30d`, `total_write_bytes_processed_60d` | Total bytes processed by write queries. | #### Databricks column (`column`) | Property | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Column name. | | `type` | Always `column`. | | `subtype` | Always `column`. | | `description` | Column comment when present. | | `parent_container` | Parent table URI. | | `database` | Catalog name. | | `database_schema` | Schema name. | | `database_technology` | Always `databricks`. | | `native_data_type` | Native Databricks data type. | | `normalized_data_type` | Euno-normalized data type. | | `upstream_fields` | Upstream column URIs for lineage. Values combine warehouse column lineage from `system.access.column_lineage` with other merged inputs; Euno exposes the result as this column’s upstream field relationships. | | `databricks_masking_policy` | Column masks protecting the column: the mask function for directly assigned masks, or the ABAC masking policy for tag-driven masks. | | `databricks_masking_source` | How the mask is attached: `direct` (assigned with `SET MASK`) or `abac_policy` (applied by an ABAC column-mask policy through a governed tag). | | `databricks_is_masked` | Indicates that the column is protected by a column mask. | | `databricks_masking_refreshed_at` | Timestamp when Euno last refreshed Databricks column-mask metadata for the resource. | Detailed masking information β€” the effective mask, the attachment source, the governed tag that triggered an ABAC mask, and any additional attachments β€” appears in the Data Model's shared **Masking Policy** section, which presents Databricks and Snowflake masking side by side. Column masks are discovered from `system.information_schema.column_masks` (direct masks) and from ABAC column-mask policies listed through the Unity Catalog policies API on each observed schema and table. Schema listing includes inherited catalog policies, while table listing captures policies attached directly to that table. ABAC policies are resolved to columns by evaluating their `MATCH COLUMNS` conditions against the governed tags set directly on each column (governed tags do not propagate to individual columns in Databricks); `WHEN` conditions are evaluated against table tags, including tags inherited from the parent catalog and schema. **URI pattern** ``` databricks.....column. ``` #### Column usage metrics Column usage is emitted for `14d`, `30d`, and `60d` windows. | Property family | Meaning | | ---------------------------------------------------------------------------- | ----------------------------------------- | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Distinct statements that read the column. | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users reading the column. | #### Databricks notebook (`databricks_notebook`) | Property | Description | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Notebook display name (path basename). | | `type` | Always `databricks_notebook`. | | `subtype` | Always `databricks_notebook`. | | `native_id` | Databricks notebook object ID. | | `parent_container` | Parent workspace URI. | | `created_at`, `updated_at` | Notebook object timestamps from the Workspace API. | | `description` | Notebook description when the API provides it. | | `native_last_data_update` | Latest observed execution time from query history evidence. | | `defines` | Table URIs the notebook is observed to define or update from table-lineage evidence (`defines` on the notebook). | | `table_dependencies` | Table URIs the notebook reads from (`system.access.table_lineage` with notebook context); stored on the notebook. | | `upstream_fields` | Column URIs referenced in notebook column-lineage evidence for reporting-style relationships; stored on the **notebook** (not on the column rows). | **URI pattern** ``` platform.databricks.databricks_workspace..notebook. ``` #### Statement-level column dependencies for views For Databricks views, Euno also captures statement-level column dependencies and reflects them in **table-level** `upstream_fields`. That means a view can show upstream field relationships to columns used only in SQL logic (for example filters), not only columns in the `SELECT` list. Included SQL logic clauses: * `JOIN` conditions * `WHERE` * `GROUP BY` * `HAVING` * `QUALIFY` * `ORDER BY` Example: for `SELECT order_id FROM orders_raw WHERE order_status = 'COMPLETE'`, the view can relate upstream to `orders_raw.column.order_status` even when `order_status` is not projected. ## Relationships | Source type(s) | Relationship | Target type(s) | Notes | | --------------------- | -------------------- | ---------------------- | --------------------------------------------------------------------------------------- | | `database` | has parent | `databricks_workspace` | `parent_container` on the catalog. | | `database_schema` | has parent | `database` | `parent_container` on the schema. | | `table` | has parent | `database_schema` | `parent_container` on the table. | | `column` | has parent | `table` | `parent_container` on the column. | | `databricks_notebook` | has parent | `databricks_workspace` | `parent_container` on the notebook. | | `table` | has upstream | `table` | Upstream warehouse tables and views (merged `table_dependencies` / lineage). | | `databricks_notebook` | has upstream | `table` | Tables the notebook reads (`table_dependencies` on the notebook). | | `column` | has upstream\_fields | `column` | Column-to-column lineage (merged `upstream_fields`). | | `databricks_notebook` | has upstream\_fields | `column` | Notebook reporting lineage: the notebook lists source column URIs in `upstream_fields`. | | `databricks_notebook` | defines | `table` | Tables the notebook is observed to define or update (`defines` on the notebook). | | `table` | has definer | `databricks_notebook` | Inverse of **defines** when the table is tied to a defining notebook. | ## Notes and caveats * Notebook execution evidence and notebook-derived relationships use `system.query.history`, `system.access.table_lineage`, and `system.access.column_lineage` with a 30-day lookback window. * Regular Databricks column lineage reads from `system.access.column_lineage` with a 30-day lookback; graph relationships use the merged column upstream field model described above. * Table and column usage metrics use `14d`, `30d`, and `60d` windows. * Usage and lineage relationships are emitted only for resources observed in the current integration scope. # MySQL Euno's MySQL integration supports auto-discovery of: * MySQL instances * MySQL schemas (databases) * MySQL tables and views * MySQL columns with data type mapping * View definitions with SQL code ## Setting up MySQL integration ### Overview MySQL is a popular open-source relational database management system. In MySQL, data is stored in *tables* and *views.* Each table or view belongs to one MySQL *schema* (also called database), and each *schema* belongs to a MySQL *instance*. A MySQL instance can have multiple schemas, and each schema can contain multiple tables and views. To discover MySQL resources, Euno connects directly to your MySQL instance and executes queries on the system-defined `INFORMATION_SCHEMA` views. These views provide metadata about all databases, tables, views, and columns in your MySQL instance. Euno automatically excludes system schemas (`mysql`, `sys`, `performance_schema`, `information_schema`) and allows you to configure which user schemas to discover using regular expression patterns. ### Step 1: Prepare MySQL Instance #### Ensure MySQL is accessible 1. Verify that your MySQL instance is running and accessible from Euno IP address `35.224.29.172` 2. Note the MySQL hostname/IP address and port (default: 3306) 3. Ensure your MySQL instance allows connections from Euno's network #### Create a dedicated MySQL user for Euno (Recommended) For security best practices, create a dedicated MySQL user with minimal required permissions: ```sql -- Create a dedicated user for Euno CREATE USER 'euno_reader'@'%' IDENTIFIED BY 'secure_password_here'; -- Grant necessary permissions to read metadata GRANT SELECT ON *.* TO 'euno_reader'@'%'; GRANT SHOW DATABASES ON *.* TO 'euno_reader'@'%'; -- Apply the changes FLUSH PRIVILEGES; ``` **Alternative**: You can use an existing MySQL user with read permissions, but ensure it has the following minimum privileges: * `SELECT` on all databases you want to discover * `SHOW DATABASES` privilege ### Step 2: Configure Euno's MySQL Integration | Configuration | Description | | -------------- | --------------------------------------------------- | | MySQL Hostname | The hostname or IP address of your MySQL server | | MySQL Port | The port number MySQL is running on (default: 3306) | | MySQL Username | Username for connecting to MySQL | | MySQL Password | Password for the MySQL user | ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). ### Step 4: Resource Cleanup * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). ### Step 5: Advanced Settings Click on the '**Advanced**' section to display these additional configurations. | Configuration | Description | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Override MySQL Instance Name | Custom name for the MySQL instance in Euno (defaults to hostname) | | Override MySQL Hostname for URIs | Custom hostname to use in resource URIs (defaults to actual hostname) | | Schema Pattern | Use a regular expression to allow or exclude specific schemas. ".\*" will include/exclude all schemas. System schemas (mysql, sys, performance\_schema, information\_schema) are always excluded regardless of this setting. | | Local Agent | Select a local agent to route traffic through when your MySQL instance is inside a private network. See [Local Agents](/setup-configuration/account-settings/local-agents) for setup instructions. | **Credentials from a local agent:** when a local agent is selected, the MySQL username and password can be fetched from your own secret store at connection time instead of being stored in Euno β€” pick a handle in the **from agent** dropdowns under Advanced. See [Local Agents β€” secret handles](/setup-configuration/account-settings/local-agents#fetching-credentials-from-your-vault-secret-handles). **Examples of Schema Patterns:** * `.*` - Include all user schemas * `production_.*` - Include only schemas starting with "production\_" * `.*` (allow) and `test_.*` (deny) - Include all schemas except those starting with "test\_" ### Step 6: Save Configuration Click the **Test & Save** button to complete the setup. Euno will validate the connection and permissions before saving. ## Discovered Resources The Euno-MySQL integration discovers various resources including MySQL instances, schemas, tables, views, and columns. For detailed information about discovered resources and their properties, see [mysql-integration-discovered-resources](/sources/data-warehouses/mysql-integration/mysql-integration-discovered-resources). # MySQL Integration Discovered Resources The Euno MySQL integration discovers and creates the following types of resources in your data catalog: ## Resource Discovery Overview The MySQL integration follows a hierarchical discovery approach: 1. **MySQL Instance** - The top-level MySQL server 2. **Schemas** - Individual databases within the MySQL instance 3. **Tables & Views** - Data objects within each schema 4. **Columns** - Individual fields within each table and view ## MySQL Instance **Resource Type**: `mysql_instance` Represents the top-level MySQL server instance. ### Properties | Property | Description | Example | | --------------------- | ----------------------------------------------------- | ---------------- | | `name` | Instance name (the mysql hostname or custom override) | `mysql-prod-01` | | `type` | Resource type | `mysql_instance` | | `subtype` | Resource subtype | `mysql_instance` | | `database_technology` | Database platform | `mysql` | ### URI Pattern ``` mysql. ``` **Example**: `mysql.mysql-prod-01.amazonaws.com` ## MySQL Schema (Database) **Resource Type**: `database_schema` Represents a MySQL database/schema within the instance. ### Properties | Property | Description | Example | | --------------------- | ----------------- | ----------------- | | `name` | Schema name | `ecommerce` | | `type` | Resource type | `database_schema` | | `subtype` | Resource subtype | `database_schema` | | `database_technology` | Database platform | `mysql` | ### URI Pattern ``` mysql.. ``` **Example**: `mysql.mysql-prod-01.amazonaws.com.ecommerce` ### System Schemas Excluded The following system schemas are automatically excluded from discovery: * `mysql` - MySQL system database * `sys` - MySQL sys schema * `performance_schema` - Performance monitoring schema * `information_schema` - Metadata schema ## MySQL Table **Resource Type**: `table` **Subtype**: `base_table` Represents a MySQL base table. ### Properties | Property | Description | Example | | ------------------------- | -------------------------------------- | ---------------------------- | | `name` | Table name | `customers` | | `type` | Resource type | `table` | | `subtype` | Resource subtype | `base_table` | | `description` | Table comment (if any) | `Customer information table` | | `database_technology` | Database platform | `mysql` | | `database_schema` | Schema name | `ecommerce` | | `created_at` | Table creation timestamp | `2023-01-15 10:30:00` | | `row_count` | Approximate number of rows | `125000` | | `volume` | Storage size in bytes (data + indexes) | `45829120` | | `native_last_data_update` | Last time data was modified | `2023-12-01 15:45:00` | ### URI Pattern ``` mysql... ``` **Example**: `mysql.mysql-prod-01.amazonaws.com.ecommerce.customers` ## MySQL View **Resource Type**: `table` **Subtype**: `view` Represents a MySQL view. ### Properties | Property | Description | Example | | --------------------- | ----------------------- | ------------------------------------ | | `name` | View name | `active_customers` | | `type` | Resource type | `table` | | `subtype` | Resource subtype | `view` | | `description` | View comment (if any) | `View of currently active customers` | | `database_technology` | Database platform | `mysql` | | `database_schema` | Schema name | `ecommerce` | | `created_at` | View creation timestamp | `2023-01-15 10:30:00` | ### URI Pattern ``` mysql... ``` **Example**: `mysql.mysql-prod-01.amazonaws.com.ecommerce.active_customers` ## MySQL Column **Resource Type**: `column` Represents a column within a MySQL table or view. ### Properties | Property | Description | Example | | ---------------------- | ---------------------------------- | ---------------------------- | | `name` | Column name | `customer_id` | | `type` | Resource type | `column` | | `subtype` | Resource subtype | `column` | | `description` | Column comment (if any) | `Unique customer identifier` | | `database_technology` | Database platform | `mysql` | | `database_schema` | Schema name | `ecommerce` | | `native_data_type` | Original MySQL data type with size | `VARCHAR(255)` | | `normalized_data_type` | Euno normalized data type | `string` | ### URI Pattern ``` mysql.... ``` **Example**: `mysql.mysql-prod-01.amazonaws.com.ecommerce.customers.customer_id` ## Data Type Mapping The MySQL integration maps MySQL native data types to Euno's normalized data types: ### String Types | MySQL Type | Normalized Type | | -------------------------------------------- | --------------- | | `CHAR`, `VARCHAR` | `string` | | `TINYTEXT`, `TEXT`, `MEDIUMTEXT`, `LONGTEXT` | `string` | | `ENUM`, `SET` | `string` | ### Binary Types | MySQL Type | Normalized Type | | -------------------------------------------- | --------------- | | `BINARY`, `VARBINARY` | `binary` | | `TINYBLOB`, `BLOB`, `MEDIUMBLOB`, `LONGBLOB` | `binary` | ### Numeric Types | MySQL Type | Normalized Type | | -------------------------------------------------------------- | --------------- | | `TINYINT`, `SMALLINT`, `MEDIUMINT`, `INT`, `INTEGER`, `BIGINT` | `number` | | `DECIMAL`, `NUMERIC`, `FLOAT`, `DOUBLE`, `REAL` | `number` | | `BIT`, `SERIAL` | `number` | ### Date/Time Types | MySQL Type | Normalized Type | | ----------------------------------------------- | --------------- | | `DATE`, `TIME`, `DATETIME`, `TIMESTAMP`, `YEAR` | `datetime` | ### Boolean Types | MySQL Type | Normalized Type | | ----------------- | --------------- | | `BOOLEAN`, `BOOL` | `boolean` | ### Complex Types | MySQL Type | Normalized Type | | ------------------------------------------ | --------------- | | `JSON` | `object` | | Geometry types (`GEOMETRY`, `POINT`, etc.) | `object` | ### Unknown Types Any data type not recognized in the mapping above will be assigned the normalized type `unknown`. ## Relationships The MySQL integration discovers the following relationships between resources: ### Parent-Child Relationships The MySQL integration creates a clear hierarchical structure: * **MySQL Instance β†’ Schemas**: Database schemas are considered children of their parent MySQL instance * **Schema β†’ Tables & Views**: Tables and views are considered children of their parent schema * **Table/View β†’ Columns**: Columns are considered children of their parent table or view ### Container Hierarchy The complete container hierarchy follows this pattern: ``` MySQL Instance └── Schema (Database) β”œβ”€β”€ Table β”‚ β”œβ”€β”€ Column 1 β”‚ β”œβ”€β”€ Column 2 β”‚ └── Column N └── View β”œβ”€β”€ Column 1 β”œβ”€β”€ Column 2 └── Column N ``` ### View Lineage | Source type(s) | Relationship | Target type(s) | Notes | | -------------- | ------------ | -------------- | --------------------------------------- | | `table` (view) | has upstream | `table` | Based on the SQL definition of the view | # SQL Server Euno's SQL Server integration connects to Microsoft SQL Server and Azure SQL Database to discover databases, schemas, tables, views, columns, lineage, and Query Store usage. The integration supports one source type for both: * On-premises Microsoft SQL Server 2016 and later * Azure SQL Database ## Setting up SQL Server integration ### Overview SQL Server stores data in databases. Each database contains schemas, and each schema contains tables, views, indexed views, and columns. Euno connects to SQL Server through Microsoft ODBC Driver 18 and reads system catalog views, `INFORMATION_SCHEMA`, and Query Store metadata. System databases (`master`, `tempdb`, `model`, and `msdb`) and system schemas are excluded from discovery. ### Step 1: Prepare SQL Server #### Ensure SQL Server is reachable 1. Confirm that Euno can reach the SQL Server host and port. The default port is `1433`. 2. For on-premises SQL Server, allow inbound traffic from the Euno runtime network or through your private connectivity setup. 3. For Azure SQL Database, allow the Euno runtime through the Azure SQL firewall or use a Private Endpoint/private networking path. 4. Note the hostname, port, and initial database. The default initial database is `master`. #### Choose an authentication method SQL Server authentication and Microsoft Entra client credentials are mutually exclusive. Euno selects the authentication method from the populated credential fields: * SQL authentication: provide **SQL Username** and **SQL Password**. * Entra client credentials: provide **Entra Tenant ID**, **Entra Client ID**, and **Entra Client Secret**. Do not populate both credential families for the same source. ### Step 2: Grant permissions Use a dedicated SQL Server login or Entra application for Euno. The account must be able to connect to each database you want Euno to discover. Minimum permissions depend on which features you enable: | Permission | Used for | | --------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Catalog visibility for databases, schemas, tables, views, and columns | Core metadata discovery | | `VIEW DEFINITION` | View definitions, computed column definitions, and definition-based lineage | | `VIEW DATABASE STATE` | Query Store usage and workload lineage | | Permissions to read size and row-count metadata | Table row counts and storage size metadata | For Query Store usage and workload lineage, each database must have Query Store enabled and readable by the Euno account. ### Step 3: Enable Query Store for usage and workload lineage Query Store is required for SQL Server usage metrics and workload lineage. If Query Store is disabled or read-only metadata is unavailable for a database, Euno still discovers catalog resources for that database, but it skips Query Store usage and workload lineage for that database. For on-premises SQL Server 2016-2019, Query Store is often disabled by default. Enable it per database: ```sql ALTER DATABASE [] SET QUERY_STORE = ON; ``` Azure SQL Database commonly has Query Store enabled, but you should verify it for the databases you want to observe. Query Store does not provide reliable per-user attribution for Euno's SQL Server usage model, so this integration does not emit per-user usage metrics. ### Step 4: Configure Euno's SQL Server source | Configuration | Description | | ------------------- | ----------------------------------------------------------------------------------------- | | SQL Server Hostname | SQL Server hostname, named instance host, or Azure SQL hostname. Do not include protocol. | | SQL Server Port | SQL Server TCP port. Defaults to `1433`. | | Initial Database | Initial database used for connection setup. Defaults to `master`. | | SQL Username | Username for SQL authentication. Required when using SQL authentication. | | SQL Password | Password for SQL authentication. Required when using SQL authentication. | | Entra Tenant ID | Microsoft Entra tenant ID. Required when using Entra client credentials. | | Entra Client ID | Microsoft Entra application/client ID. Required when using Entra client credentials. | | Entra Client Secret | Microsoft Entra client secret. Required when using Entra client credentials. | ### Step 5: Schedule 1. Enable the Schedule option. 2. Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours. ### Step 6: Resource Cleanup * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. For a detailed explanation on Euno's cleanup strategies, see [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). ### Step 7: Advanced Settings Click **Advanced** to display these additional settings. | Configuration | Description | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Encrypt Connection | Encrypt the SQL Server connection. Azure SQL usually requires this enabled. | | Trust Server Certificate | Trust the server certificate without normal certificate-chain validation. Use only when SQL Server presents a self-signed or private-CA certificate that Euno cannot otherwise validate. | | Override SQL Server Instance Name | Custom display name for the SQL Server instance in Euno. Defaults to the hostname. | | Override SQL Server Hostname for URIs | Stable hostname used in generated `mssql.` URIs. By default, Euno preserves the configured hostname and any named instance. Use this override when the connection hostname is not the canonical warehouse hostname or when matching existing external references that use a short host such as `sql01`. | | Extract Row Counts and Sizes | Read row-count and storage-size metadata. If this metadata is unavailable, Euno still discovers catalog resources and omits the row-count and size fields. | | Extract Query Store Usage | Read Query Store to emit table usage counts and last access/write timestamps. | | Extract Query Store Lineage | Read Query Store plans and SQL text to infer workload table lineage. | | Parse Query Store SQL with Eunolizer (Falls Back to Plan XML) | Runs Eunolizer first and, when enabled, uses plan XML for statements Eunolizer cannot parse. Enabling this sends Query Store SQL text, including literals, to Eunolizer, so it is opt-in and disabled by default in the UI. | | Query Store Lookback Days | Number of days to read from Query Store. Defaults to `60`. Euno clamps this to the database's Query Store retention and only publishes 14-, 30-, or 60-day totals when that full window is available. | | Max Query Store Queries Per Database | Per-database safety cap for Query Store workload processing. Defaults to `100000`. | | Database Pattern | Regular expressions to include or exclude user databases. System databases are always excluded. | | Schema Pattern | Regular expressions to include or exclude schemas. SQL Server system schemas are always excluded. | | Local Agent | Select a local agent to route traffic through when your SQL Server instance is inside a private network. See [Local Agents](/setup-configuration/account-settings/local-agents) for setup instructions. | **Credentials from a local agent:** when a local agent is selected, the SQL username, password, and Entra client secret can be fetched from your own secret store at connection time instead of being stored in Euno β€” pick a handle in the **from agent** dropdowns under Advanced. See [Local Agents β€” secret handles](/setup-configuration/account-settings/local-agents#fetching-credentials-from-your-vault-secret-handles). ### Step 8: Save Configuration Click **Test & Save** to validate the connection and permissions before saving. ## Discovered Resources For detailed information about discovered resources, properties, relationships, and URI patterns, see [SQL Server Integration Discovered Resources](/sources/data-warehouses/sql-server-integration/sql-server-integration-discovered-resources). # SQL Server Integration Discovered Resources The Euno SQL Server integration discovers and updates the following resource families: * SQL Server instances * SQL Server databases * SQL Server schemas * SQL Server tables, views, and indexed views * SQL Server table and view columns * SQL Server linked servers * Usage and workload lineage from Query Store when enabled By default, URI host identity preserves the configured hostname and any SQL Server named instance. For example, `SQL01.corp.example.com\DW` becomes `sql01.corp.example.com_sqlserver_instance_dw` in URI paths. Configure **Override SQL Server Hostname for URIs** when you want a different externally canonical URI host, such as `sql01`. ## Resource hierarchy ``` SQL Server Instance └── Database └── Schema β”œβ”€β”€ Table / View / Indexed View β”‚ └── Column └── ... SQL Server Instance └── Linked Server ``` ## Ingested resources ### SQL Server instance (`sqlserver_instance`) Represents the SQL Server or Azure SQL instance configured in the source. | Property | Description | | --------------------- | --------------------------------------------------------------------- | | `name` | Instance name. Defaults to the configured hostname unless overridden. | | `type` | Always `sqlserver_instance`. | | `subtype` | Always `sqlserver_instance`. | | `container_type` | SQL Server container classification. | | `database_technology` | Always `sqlserver`. | **URI pattern** ``` platform.sqlserver. ``` ### SQL Server database (`database`) Represents a user database within the SQL Server instance. | Property | Description | | --------------------- | ---------------------------------- | | `name` | Database name. | | `type` | Always `database`. | | `subtype` | Always `database`. | | `container_type` | Database container classification. | | `parent_container` | Parent SQL Server instance URI. | | `database_technology` | Always `sqlserver`. | **URI pattern** ``` mssql.. ``` ### SQL Server schema (`database_schema`) Represents a schema within a SQL Server database. | Property | Description | | --------------------- | -------------------------------- | | `name` | Schema name. | | `type` | Always `database_schema`. | | `subtype` | Always `database_schema`. | | `container_type` | Schema container classification. | | `parent_container` | Parent database URI. | | `database_database` | Database name. | | `database_technology` | Always `sqlserver`. | **URI pattern** ``` mssql... ``` ### SQL Server table (`table`, subtype `base_table`) Represents a SQL Server base table. | Property | Description | | ----------------------------------- | -------------------------------------------------------------------- | | `name` | Table name. | | `type` | Always `table`. | | `subtype` | `base_table`. | | `description` | Table description from extended properties, when present. | | `parent_container` | Parent schema URI. | | `database_database` | Database name. | | `database_schema` | Schema name. | | `database_technology` | Always `sqlserver`. | | `platform_uri_prefix` | SQL Server URI prefix for the source. | | `row_count` | Table row count when size extraction is enabled and permitted. | | `volume` | Storage size in bytes when size extraction is enabled and permitted. | | `table_properties` | Structured table metadata, including materialization information. | | `table_schema` | Structured column list for observed columns. | | `table_dependencies` | Upstream table dependencies from catalog and SQL parsing inputs. | | `sqlserver_definition_dependencies` | SQL Server catalog definition dependencies. | | `foreign_key_dependencies` | Foreign-key target table URIs. | | `native_raw_object` | Raw SQL Server metadata snapshot. | **URI pattern** ``` mssql.... ``` Example: `mssql.sql01.salesdb.dbo.customers` ### SQL Server view (`table`, subtype `view`) Represents a SQL Server view. View resources use the same URI pattern and common table properties as base tables. Euno also stores view SQL as code metadata and uses catalog dependencies and SQL parsing to populate table lineage. **URI pattern** ``` mssql.... ``` ### SQL Server indexed view (`table`, subtype `materialized_view`) Represents a SQL Server indexed view. Euno models indexed views as materialized views. Indexed view resources use the same URI pattern and common table properties as tables and views. **URI pattern** ``` mssql.... ``` ### SQL Server column (`column`) Represents a column within a SQL Server table, view, or indexed view. | Property | Description | | ------------------------------------------- | ---------------------------------------------------------- | | `name` | Column name. | | `type` | Always `column`. | | `subtype` | Always `column`. | | `description` | Column description from extended properties, when present. | | `parent_container` | Parent table/view URI. | | `database_database` | Database name. | | `database_schema` | Schema name. | | `database_technology` | Always `sqlserver`. | | `native_data_type` | Native SQL Server data type. | | `normalized_data_type` | Euno-normalized data type. | | `is_primary_key` | Whether the column participates in a primary key. | | `is_computed` | Whether the column is computed. | | `computed_definition` | Computed-column SQL expression, when available. | | `is_persisted` | Whether a computed column is persisted. | | `qualified_code_based_columns_dependencies` | Upstream column URIs from SQL parsing. | | `foreign_key_dependencies` | Foreign-key target column URIs. | **URI pattern** ``` mssql.....column. ``` Example: `mssql.sql01.salesdb.dbo.customers.column.customerid` ### SQL Server linked server (`sqlserver_linked_server`) Represents a linked server configured on the SQL Server instance. | Property | Description | | --------------------- | ---------------------------------------- | | `name` | Linked server name. | | `type` | Always `sqlserver_linked_server`. | | `subtype` | Always `sqlserver_linked_server`. | | `container_type` | Linked-server container classification. | | `parent_container` | Parent SQL Server instance URI. | | `database_technology` | Always `sqlserver`. | | `product` | Linked server product metadata. | | `provider` | Linked server provider metadata. | | `data_source` | Linked server data source. | | `default_catalog` | Linked server default catalog, when set. | **URI pattern** ``` mssql..linked_server. ``` Example: `mssql.sql01.linked_server.erp_link` ## Query Store usage and lineage When Query Store is enabled and readable, Euno can enrich table resources with workload metadata: | Property | Description | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `total_read_queries_14d`, `total_read_queries_30d`, `total_read_queries_60d` | Query Store read execution counts for each fully covered standard window. A property is unavailable when Query Store retention or the configured lookback is shorter than that window. | | `total_write_queries_14d`, `total_write_queries_30d`, `total_write_queries_60d` | Query Store write execution counts for each fully covered standard window. A property is unavailable when Query Store retention or the configured lookback is shorter than that window. | | `query_count_by_window_days` | Deprecated compatibility field containing read execution counts keyed by the effective Query Store lookback (for example, `{"30": 1234}`). Use `total_read_queries_14d/30d/60d` instead. | | `write_count_by_window_days` | Deprecated compatibility field containing write execution counts keyed by the effective Query Store lookback (for example, `{"30": 1234}`). Use `total_write_queries_14d/30d/60d` instead. | | `last_accessed_at` | Last observed read access time. | | `last_written_at` | Last observed write time. | | `popular_queries` | Representative popular Query Store statements for the table. | | `query_store_upstream` | Upstream table URIs inferred from Query Store workload lineage. | | `query_store_lineage_execution_count` | Number of Query Store executions supporting an inferred workload lineage edge. | Euno reads up to 60 days from Query Store by default and derives the standard windows from that one measurement. The effective lookback is clamped to Query Store retention. Only fully covered standard windows receive a number: for example, a 30-day effective lookback publishes 14- and 30-day totals and leaves the 60-day total unavailable. If individual Query Store statements cannot be attributed, the standard totals contain the resolved subset and may undercount; the run report keeps those parse failures visible. A capped workload leaves the standard totals unavailable. This preserves the distinction between measured history and history SQL Server did not provide, while matching the deprecated map fields' partial-attribution behavior. Query Store lineage also contributes to the canonical `table_dependencies` relationship. Query Store does not provide per-user attribution for this integration. The resource sidepane's **Usage** tab displays these read and write counts with the standard 14-, 30-, and 60-day selector, together with the latest access timestamps and the representative popular-query sample. EQL can use the same cross-warehouse fields, for example `total_read_queries_30d > 0`. ## Exact resource model The integration emits the following resource types and subtypes: ``` sqlserver_instance database database_schema table subtype base_table table subtype view table subtype materialized_view column sqlserver_linked_server ``` ## URI examples ``` mssql.sql01.salesdb.dbo.customers mssql.sql01.salesdb.dbo.customers.column.customerid mssql.sql01.linked_server.erp_link ``` # Transformation & ETL Connect your transformation and ETL tools to understand how data flows through your pipelines and track transformation logic. ## Supported Tools * [**dbt Core**](/sources/transformation-etl/dbt-core) - Open-source transformation tool * [**dbt Cloud**](/sources/transformation-etl/dbt-cloud) - Managed dbt service * [**dbt Source Project**](/sources/transformation-etl/dbt-source-project) - Connect your dbt repo; Euno builds the project and generates artifacts * [**Matillion DPC**](/sources/transformation-etl/matillion) - Data transformation platform * [**Fivetran**](/sources/transformation-etl/fivetran) - ETL/ELT platform * [**Stitch**](/sources/transformation-etl/stitch-integration) - ETL/ELT platform for replicated source data and destination warehouse table context * [**OpenLineage**](/sources/transformation-etl/openlineage-integration) - Open metadata standard * [**Apache Airflow**](/sources/transformation-etl/airflow-integration) - Workflow orchestration platform * [**Azure Data Factory**](/sources/transformation-etl/azure-data-factory) - Microsoft cloud data integration / ELT orchestration ## What You'll Get When you connect a transformation tool, Euno discovers: * **Models & Transformations** - All transformation logic and SQL * **Dependencies** - Upstream and downstream relationships * **Metrics & Semantic Models** - Business metrics and definitions * **Build Performance** - Execution times and success rates * **Lineage** - Complete data flow from sources to destinations ## Common Use Cases * **Impact Analysis** - Understand what breaks when you change a model * **Documentation** - Automatically catalog all transformations * **Performance Optimization** - Identify slow models and bottlenecks * **Governance** - Track which tables are managed vs. ad-hoc ## Getting Started 1. Choose your transformation tool from the list above 2. Follow the setup guide for your platform 3. Grant Euno access to your dbt project or transformation definitions 4. Run your first sync **Setup Time:** 15-20 minutes per tool *** ## Next Steps * [View All Integrations](/sources) * [Learn About Data Model Sync](/using-euno/metadata-activation/data-application-sync) * [Set Up Impact Analysis](/using-euno/impact-analysis) # dbt Core Euno's dbt Core integration supports auto-discovery of [dbt resources](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources). It automates processing of dbt uploaded artifacts from completed runs. This allows for seamless data synchronization and analysis using the latest available dbt job runs. ## How It Works The integration follows these steps: 1. **Outputs a url for uploading dbt artifacts**\ The following artifacts can then be uploaded to Euno's endpoint: * `run_results.json` * `manifest.json` * `semantic_manifest.json` * `catalog.json` 2. **Process the Artifacts** * The integration processes the uploaded artifacts to extract relevant information and adds the [discovered](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources) [resources](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources) to Euno's data model. ## Setting up Euno's dbt Core Integration ### Step 1: Access the Sources Page 1. Navigate to the **Sources** page in the Euno application. 2. Click on the **Add New Source** button. ### Step 2: General Configuration 1. **Name**: Enter a name for your dbt Core source (e.g., "dbt - Marketing Models"). 2. **Configuration Details**: 1. **Build target:** The default build target to use when observing dbt resources. Consult the table below depending on the warehouse technology your dbt is run against.
Snowflake The build target should follow the pattern: `...snowflakecomputing.com` For example: * foo-1234.us-east-1.aws.snowflakecomputing.com * bar-5678.us-west-2.gcp.snowflakecomputing.com
Trino The build target should follow the pattern `trino..` For example: * trino.us-west-2.aws * trino.us-east-2.gcp
Databricks The build target should be the hostname of the databrick workspace For example: * 4200747832468935.5.gcp.databricks.com * dbc-50e7cad0-c2f9.cloud.databricks.com * adb-5555555555555555.19.azuredatabricks.net
Bigquery The build target should just say `bigquery`
### Step 3: Resource Cleanup Options To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Time-Based Cleanup (default)**: Remove resources that were last detected X days before the most recent successful source integration run (user-defined X, default is 7 days). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 4: Advanced Settings (Optional) Click on the '**Advanced**' section to display these additional configurations.
ConfigurationDescription
Allow Processing Builds with a Partial Catalog

By default, Euno only observes dbt resources (e.g., models, sources, snapshots, and seeds) that have a corresponding entry in the catalog.json file. Checking this box will expand the scope to include all resources listed in the manifest.json file, even if they do not appear in the catalog.json file.


Note: Resources without a matching entry in the catalog.json file will not have schema information available, as this data is exclusively pulled from the catalog. By default, the integration processes only builds with a complete, error-free catalog.json.

Source Repository URLThe URL of the git repository where the dbt project is stored
Source Repository branchThe branch of the git repository where the dbt project is stored
Relative directory of the dbt projectSubdirectory within the git repository where the dbt project is stored
MappingEuno will ingest dbt resources to the database and schema stated in the manifest file, unless a database.schema mapping is added. In that case, the resource will be ingested to the database and schema stated in the value of the target.

example:
source: analytics_v2, target: analytics will map `analytics_v2.analytics` to `analytics.analytics`

source: r'/^(hive_metastore\..+)_spark$/, target: analytics will map hive_metastore.some_schema_spark to hive_metastore.some_schema.
### Step 5: Save Configuration Click the **Save** button, and Euno will generate a trigger key. Please copy the endpoint url and the key. The key will disappear after copying you click "done".\\ ## **Uploading artifacts to the integration** To run the integration, you need to upload dbt artifacts. Files to upload from your dbt build are: * `run_results.json` * `manifest.json` * `catalog.json` * `semantic_manifest.json` (optional) The easiest way to upload dbt artifacts to the integration is through Euno's UI. Browser to the integration and click "upload artifacts" and upload a .zip file with the above artifacts. To programatically upload the dbt artifacts after every successful run, which is recommended for production use-cases, see reference implementations below: #### **Python Script** Use a standalone Python script to zip and upload your dbt artifacts [β†’ View Python Upload Guide](/sources/zip-artifact-python-upload) #### **GitHub Actions** Automate your dbt builds and artifact uploads using GitHub Actions workflows [β†’ View GitHub Actions Guide](/sources/transformation-etl/dbt-core/github-actions-upload) #### **Prefect** Integrate with Prefect workflows for advanced orchestration and monitoring [β†’ View Prefect Integration Guide](/sources/transformation-etl/dbt-core/prefect-upload) **Generating a new trigger URL key:** If you need to create a new trigger URL key, go to the **Sources** page and click on the three-dot menu to the right of the source you want to create a new key for. In the dropdown menu click on **Generate trigger URL.** The generated URL will include the new key. # dbt Integration Discovered Resources ## Overview The dbt integration discovers the following resources: * dbt project * dbt Transformation Layer: * `dbt_source`, `dbt_seed`, `dbt_model`, `dbt_snapshot`, `dbt_exposure` * `dbt_column` β€” documented columns on those dbt resources (distinct from warehouse `column`) * dbt Semantic Layer: * `dbt_metric` * `dbt_measure` * `dbt_dimension` * `dbt_semantic_model` * warehouse resources (tables, schemas, and databases linked from dbt builds) * `column` * `table` * `database_schema` * `database` `` below is the warehouse URI prefix from the source **Build target** (for example a Snowflake or Databricks host mapped to Euno’s warehouse prefix). dbt transformation and semantic resources use the `dbt.` prefix unless **Override URI prefix** is set. ## dbt Project (`dbt_project`) The top-level container for a dbt project. | name | type | Description | | ------------- | ------ | ------------------------------------------------------- | | `type` | string | Always `dbt_project` | | `name` | string | dbt project name | | `dbt_version` | string | dbt version from the build | | `sql_dialect` | string | SQL dialect of the warehouse adapter used for the build | **URI pattern** ``` .dbt_project. ``` Example: `snowflake.xy12345.us-east-1.aws.dbt_project.jaffle_shop` ## dbt Model (`dbt_model`) A dbt model β€” SQL that is compiled and materialized in the warehouse. | name | type | Description | | ------------------------------------------ | ------------------- | ------------------------------------------------------------------------- | | `type` | string | Always `dbt_model` | | `name` | string | Model name (uses alias when set) | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this model | | `unique_id` | string | dbt unique id | | `package_name` | string | dbt package name | | `is_external_package` | boolean | Whether the model comes from an installed package | | `tags` | list | Tags from the model definition | | `meta` | object | Meta from the model definition | | `source_path` | string | Relative path of the defining file in the project | | `source_directory` | string | Directory of the defining file (from `source_path`) | | `database` | string | Target database | | `schema` | string | Target schema | | `sql_dialect` | string | SQL dialect of the warehouse adapter | | `dbt_version` | string | dbt version from the build | | `dbt_materialization_strategy` | string | Materialization (for example `table`, `view`, `incremental`, `ephemeral`) | | `dbt_model_access` | string | Model access level when set (`public`, `protected`, `private`) | | `dbt_model_group` | string | Model group when set | | `git_repo_url` | string | Source repository URL when configured on the Euno source | | `git_repo_branch` | string | Source repository branch when configured | | `dbt_last_run_pass` | boolean | Whether tests passed in the last observed build (when tests ran) | | `dbt_last_run_failing_tests` | list | Failing tests from the last observed build (when present) | | `last_dbt_build_runtime` | number | Duration of the last observed build | | `total_builds_14d` / `_30d` / `_60d` | number | Build counts over the lookback window | | `average_build_time_14d` / `_30d` / `_60d` | number | Average build duration over the lookback window | | `defines` | list (relationship) | Warehouse `table` this model defines | **URI pattern** ``` dbt.. ``` `` is dbt’s unique id (for example `model.jaffle_shop.customers`). Example: `dbt.jaffle_shop.model.jaffle_shop.customers` If **Override URI prefix** is set on the source, the pattern is `.` instead. ## dbt Snapshot (`dbt_snapshot`) A dbt snapshot β€” slowly changing dimension history materialized in the warehouse. | name | type | Description | | ------------------------------ | ------------------- | -------------------------------------------------------- | | `type` | string | Always `dbt_snapshot` | | `name` | string | Snapshot name (uses alias when set) | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this snapshot | | `unique_id` | string | dbt unique id | | `package_name` | string | dbt package name | | `is_external_package` | boolean | Whether the snapshot comes from an installed package | | `tags` | list | Tags from the snapshot definition | | `meta` | object | Meta from the snapshot definition | | `source_path` | string | Relative path of the defining file in the project | | `source_directory` | string | Directory of the defining file (from `source_path`) | | `database` | string | Target database | | `schema` | string | Target schema | | `sql_dialect` | string | SQL dialect of the warehouse adapter | | `dbt_version` | string | dbt version from the build | | `dbt_materialization_strategy` | string | Typically `snapshot` | | `git_repo_url` | string | Source repository URL when configured on the Euno source | | `git_repo_branch` | string | Source repository branch when configured | | `defines` | list (relationship) | Warehouse `table` this snapshot defines | **URI pattern** ``` dbt.. ``` Example: `dbt.jaffle_shop.snapshot.jaffle_shop.orders_snapshot` If **Override URI prefix** is set on the source, the pattern is `.` instead. ## dbt Source (`dbt_source`) A dbt source β€” a declared upstream warehouse table that models can reference. | name | type | Description | | ---------------------------- | ------------------- | ---------------------------------------------------------------- | | `type` | string | Always `dbt_source` | | `name` | string | Source table name (uses identifier when set) | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this source | | `unique_id` | string | dbt unique id | | `package_name` | string | dbt package name | | `is_external_package` | boolean | Whether the source comes from an installed package | | `tags` | list | Tags from the source definition | | `meta` | object | Meta from the source definition | | `source_path` | string | Relative path of the defining file in the project | | `source_directory` | string | Directory of the defining file (from `source_path`) | | `database` | string | Database of the declared source | | `schema` | string | Schema of the declared source | | `sql_dialect` | string | SQL dialect of the warehouse adapter | | `dbt_version` | string | dbt version from the build | | `git_repo_url` | string | Source repository URL when configured on the Euno source | | `git_repo_branch` | string | Source repository branch when configured | | `dbt_last_run_pass` | boolean | Whether tests passed in the last observed build (when tests ran) | | `dbt_last_run_failing_tests` | list | Failing tests from the last observed build (when present) | | `defines` | list (relationship) | Warehouse `table` this source defines | **URI pattern** ``` dbt.. ``` Example: `dbt.jaffle_shop.source.jaffle_shop.raw.orders` If **Override URI prefix** is set on the source, the pattern is `.` instead. ## dbt Seed (`dbt_seed`) A dbt seed β€” a CSV (or similar) file in the project loaded into the warehouse. | name | type | Description | | ------------------------------------------ | ------------------- | -------------------------------------------------------- | | `type` | string | Always `dbt_seed` | | `name` | string | Seed name | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this seed | | `unique_id` | string | dbt unique id | | `package_name` | string | dbt package name | | `is_external_package` | boolean | Whether the seed comes from an installed package | | `tags` | list | Tags from the seed definition | | `meta` | object | Meta from the seed definition | | `source_path` | string | Relative path of the defining file in the project | | `source_directory` | string | Directory of the defining file (from `source_path`) | | `database` | string | Target database | | `schema` | string | Target schema | | `sql_dialect` | string | SQL dialect of the warehouse adapter | | `dbt_version` | string | dbt version from the build | | `git_repo_url` | string | Source repository URL when configured on the Euno source | | `git_repo_branch` | string | Source repository branch when configured | | `total_builds_14d` / `_30d` / `_60d` | number | Build counts over the lookback window | | `average_build_time_14d` / `_30d` / `_60d` | number | Average build duration over the lookback window | | `defines` | list (relationship) | Warehouse `table` this seed defines | **URI pattern** ``` dbt.. ``` Example: `dbt.jaffle_shop.seed.jaffle_shop.raw_customers` If **Override URI prefix** is set on the source, the pattern is `.` instead. ## dbt Exposure (`dbt_exposure`) A dbt exposure β€” a documented downstream consumer of dbt resources (for example a Tableau dashboard or report). | name | type | Description | | ----------------------- | ------ | ------------------------------------------------------------ | | `type` | string | Always `dbt_exposure` | | `name` | string | Exposure name | | `label` | string | Display label when set | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this exposure | | `unique_id` | string | dbt unique id | | `subtype` | string | Exposure type from dbt (for example `dashboard`, `notebook`) | | `tags` | list | Tags from the exposure definition | | `meta` | object | Meta from the exposure definition | | `source_path` | string | Relative path of the defining file in the project | | `source_directory` | string | Directory of the defining file (from `source_path`) | | `owner` | string | Owner derived from dbt exposure owners when available | | `dbt_exposure_maturity` | string | Maturity from the exposure definition when set | **URI pattern** ``` dbt..exposure.. ``` Example: `dbt.jaffle_shop.exposure.jaffle_shop.weekly_revenue_dashboard` ## dbt Column (`dbt_column`) A `dbt_column` is the dbt-side documentation of a column on a `dbt_model`, `dbt_source`, `dbt_seed`, or `dbt_snapshot`. When the parent dbt resource maps to a warehouse table, Euno also observes a warehouse `column` on that table and links them with **defines** / **has definer**. | name | type | Description | | ---------------------------- | ------------------- | --------------------------------------------------------------------------------------- | | `type` | string | Always `dbt_column` | | `name` | string | Column name from the dbt schema YAML / catalog | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | dbt project that contains the parent resource | | `tags` | list | Tags from the dbt column definition | | `meta` | object | Meta from the dbt column definition | | `is_field` | boolean | `true` for field-level resources | | `dbt_last_run_pass` | boolean | Whether tests on this column passed in the last observed build (when tests ran) | | `dbt_last_run_failing_tests` | list | Failing tests from the last observed build (when present) | | `defines` | list (relationship) | Warehouse `column` this dbt column defines, when a matching warehouse column is present | **URI pattern** ``` .column. ``` Example: `dbt.jaffle_shop.model.jaffle_shop.customers.column.customer_id` {% hint style="info" %} Not every `dbt_column` **defines** a warehouse `column`. For example, ephemeral models or resources without dbt catalog entries may have no matching warehouse column. **Example EQL:** `type = 'dbt_column' AND has defined(type = 'column')` `type = 'dbt_column' AND has parent(type = 'dbt_model')` {% endhint %} ## dbt Semantic Model (`dbt_semantic_model`) A Metricflow semantic model that groups related metrics, dimensions, and measures. | name | type | Description | | --------------------- | ------- | ---------------------------------------------------------- | | `type` | string | Always `dbt_semantic_model` | | `name` | string | Semantic model name | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this semantic model | | `unique_id` | string | dbt unique id | | `package_name` | string | dbt package name | | `is_external_package` | boolean | Whether the semantic model comes from an installed package | **URI pattern** ``` dbt.. ``` Example: `dbt.jaffle_shop.semantic_model.jaffle_shop.customers` If **Override URI prefix** is set on the source, the pattern is `.` instead. ## dbt Metric (`dbt_metric`) A Metricflow metric defined over semantic models. | name | type | Description | | ------------------ | ------- | --------------------------------------------------- | | `type` | string | Always `dbt_metric` | | `name` | string | Metric name | | `label` | string | Display label when set | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this metric | | `unique_id` | string | dbt unique id | | `metric_type` | string | Metricflow metric type | | `aggregation` | string | Aggregation when applicable | | `has_filter` | boolean | Whether the metric has filters | | `tags` | list | Tags from the metric definition | | `meta` | object | Meta from the metric definition | | `source_path` | string | Relative path of the defining file in the project | | `source_directory` | string | Directory of the defining file (from `source_path`) | **URI pattern** ``` metric.. ``` Example: `metric.jaffle_shop.total_revenue` ## dbt Dimension (`dbt_dimension`) A dimension on a Metricflow semantic model (or related model). | name | type | Description | | ---------------- | ------- | ------------------------------------ | | `type` | string | Always `dbt_dimension` | | `name` | string | Dimension name | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this dimension | | `dimension_type` | string | Dimension type from Metricflow | | `is_field` | boolean | `true` for field-level resources | **URI pattern** ``` .dimension. ``` Example: `dbt.jaffle_shop.model.jaffle_shop.customers.dimension.customer_id` ## dbt Measure (`dbt_measure`) A measure on a Metricflow semantic model (or related model). | name | type | Description | | ------------------ | ------- | ----------------------------------- | | `type` | string | Always `dbt_measure` | | `name` | string | Measure name | | `description` | string | Description from dbt, when provided | | `dbt_project` | string | Project that contains this measure | | `dbt_measure_type` | string | Aggregation type for the measure | | `is_field` | boolean | `true` for field-level resources | **URI pattern** ``` .measure. ``` Example: `dbt.jaffle_shop.model.jaffle_shop.customers.measure.total_revenue` ## Warehouse observed resources ## Database (`database`) A warehouse database linked from dbt build targets. | name | type | Description | | --------------------- | ------ | ------------------------------- | | `type` | string | Always `database` | | `name` | string | Database name | | `database_technology` | string | Warehouse technology when known | **URI pattern** ``` . ``` Example: `snowflake.xy12345.us-east-1.aws.analytics` ## Schema (`database_schema`) A warehouse schema linked from dbt build targets. | name | type | Description | | --------------------- | ------ | ------------------------------- | | `type` | string | Always `database_schema` | | `name` | string | Schema name | | `database` | string | Parent database name | | `database_technology` | string | Warehouse technology when known | **URI pattern** ``` .. ``` Example: `snowflake.xy12345.us-east-1.aws.analytics.public` ## Table (`table`) A warehouse table linked from dbt (materialized output or declared source). | name | type | Description | | --------------------- | ------- | -------------------------------------------------------------------------- | | `type` | string | Always `table` | | `name` | string | Table name (uses dbt alias or identifier when set) | | `description` | string | Description when provided (including from dbt) | | `database` | string | Database name | | `schema` | string | Schema name | | `sql_dialect` | string | SQL dialect from the dbt build adapter | | `database_technology` | string | Warehouse technology when known | | `dbt_last_run_pass` | boolean | Whether related dbt tests passed in the last observed build (when present) | **URI pattern** ``` ... ``` Example: `snowflake.xy12345.us-east-1.aws.analytics.public.customers` ## Column (`column`) A warehouse column on a table linked from dbt. | name | type | Description | | ---------------------------- | ------- | -------------------------------------------------------------- | | `type` | string | Always `column` | | `name` | string | Column name | | `native_data_type` | string | Native warehouse data type when available | | `dbt_last_run_pass` | boolean | Whether tests passed in the last observed build (when present) | | `dbt_last_run_failing_tests` | list | Failing tests from the last observed build (when present) | **URI pattern** ``` ....column. ``` Example: `snowflake.xy12345.us-east-1.aws.analytics.public.customers.column.customer_id` ## Relationships For lineage, **Source** is the downstream resource and **Target** is an upstream dependency. Each row is one source type β†’ one target type. ### Hierarchical (`has parent` / `has child`) | Source type | Relationship | Target type | Notes | | -------------------- | ------------ | -------------------- | ----- | | `dbt_model` | has parent | `dbt_project` | | | `dbt_seed` | has parent | `dbt_project` | | | `dbt_source` | has parent | `dbt_project` | | | `dbt_snapshot` | has parent | `dbt_project` | | | `dbt_exposure` | has parent | `dbt_project` | | | `dbt_metric` | has parent | `dbt_project` | | | `dbt_semantic_model` | has parent | `dbt_project` | | | `dbt_column` | has parent | `dbt_model` | | | `dbt_column` | has parent | `dbt_source` | | | `dbt_column` | has parent | `dbt_seed` | | | `dbt_column` | has parent | `dbt_snapshot` | | | `dbt_dimension` | has parent | `dbt_model` | | | `dbt_dimension` | has parent | `dbt_semantic_model` | | | `dbt_measure` | has parent | `dbt_model` | | | `dbt_measure` | has parent | `dbt_semantic_model` | | | `column` | has parent | `table` | | | `table` | has parent | `database_schema` | | | `database_schema` | has parent | `database` | | ### Lineage (`has upstream` / `has downstream`) | Source type | Relationship | Target type | Notes | | -------------------- | ------------ | -------------------- | -------------------------------------------------- | | `dbt_model` | has upstream | `dbt_model` | From dbt `depends_on` | | `dbt_model` | has upstream | `dbt_seed` | From dbt `depends_on` | | `dbt_model` | has upstream | `dbt_source` | From dbt `depends_on` | | `dbt_model` | has upstream | `dbt_snapshot` | From dbt `depends_on` | | `dbt_model` | has upstream | `dbt_metric` | When metrics are referenced (for example via meta) | | `dbt_model` | has upstream | `table` | Warehouse tables referenced by the model | | `dbt_snapshot` | has upstream | `dbt_model` | From dbt `depends_on` | | `dbt_snapshot` | has upstream | `dbt_seed` | From dbt `depends_on` | | `dbt_snapshot` | has upstream | `dbt_source` | From dbt `depends_on` | | `dbt_snapshot` | has upstream | `dbt_snapshot` | From dbt `depends_on` | | `dbt_snapshot` | has upstream | `table` | Warehouse tables referenced by the snapshot | | `dbt_source` | has upstream | `table` | Declared warehouse table | | `dbt_seed` | has upstream | `table` | Loaded warehouse table | | `dbt_exposure` | has upstream | `dbt_model` | From exposure dependencies | | `dbt_exposure` | has upstream | `dbt_source` | From exposure dependencies | | `dbt_semantic_model` | has upstream | `dbt_model` | From semantic model `depends_on` | | `dbt_semantic_model` | has upstream | `dbt_source` | From semantic model `depends_on` | | `dbt_semantic_model` | has upstream | `dbt_seed` | From semantic model `depends_on` | | `dbt_semantic_model` | has upstream | `dbt_snapshot` | From semantic model `depends_on` | | `dbt_semantic_model` | has upstream | `table` | When a dependency resolves to a warehouse table | | `dbt_metric` | has upstream | `dbt_semantic_model` | Metricflow metric dependencies | | `dbt_metric` | has upstream | `dbt_model` | Metricflow metric dependencies | | `dbt_metric` | has upstream | `dbt_metric` | Derived / ratio metrics | | `dbt_metric` | has upstream | `dbt_measure` | Metricflow metric dependencies | | `dbt_metric` | has upstream | `table` | When a dependency resolves to a warehouse table | | `table` | has upstream | `table` | From compiled dbt SQL | ### Field-level lineage (`has upstream_fields` / `has downstream_fields`) | Source type | Relationship | Target type | Notes | | ------------ | -------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `column` | has upstream\_fields | `column` | Column lineage from compiled SQL | | `dbt_column` | has upstream\_fields | `column` | Field lineage onto warehouse columns | | `dbt_metric` | has upstream\_fields | `dbt_measure` | Metric β†’ measure field lineage | | `table` | has upstream\_fields | `column` | May include columns used only in query logic (`JOIN`, `WHERE`, `GROUP BY`, `HAVING`, `QUALIFY`, `ORDER BY`), not only selected columns | ### Defined-by (`defines` / `has definer`) | Source type | Relationship | Target type | Notes | | -------------- | ------------ | -------------- | ------------------------------------------- | | `dbt_model` | defines | `table` | Warehouse table the model materializes | | `dbt_seed` | defines | `table` | Warehouse table the seed loads | | `dbt_source` | defines | `table` | Warehouse table the source declares | | `dbt_snapshot` | defines | `table` | Warehouse table the snapshot materializes | | `dbt_column` | defines | `column` | When a matching warehouse column is present | | `table` | has definer | `dbt_model` | Inverse of `dbt_model` **defines** | | `table` | has definer | `dbt_seed` | Inverse of `dbt_seed` **defines** | | `table` | has definer | `dbt_source` | Inverse of `dbt_source` **defines** | | `table` | has definer | `dbt_snapshot` | Inverse of `dbt_snapshot` **defines** | | `column` | has definer | `dbt_column` | Inverse of `dbt_column` **defines** | {% hint style="info" %} Use EQL relationship predicates such as `has upstream`, `has upstream_fields`, `has defined`, and `has definer`. {% endhint %} # GitHub Actions Upload This GitHub Actions workflow is an example of automatically build your dbt project and upload the artifacts to Euno after successful completion. ## Prerequisites * GitHub repository with your dbt project * Euno **trigger secret** for your dbt source, stored as a GitHub secret ## Setup ### 1. Add GitHub Secrets In your GitHub repository, add the following secrets (Settings β†’ Secrets and variables β†’ Actions): * `EUNO_INTEGRATION_KEY`: Your dbt source **trigger secret** (Bearer token for uploads) * `EUNO_ENDPOINT_URL`: Your Euno endpoint URL (e.g., `https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload`) ### 2. Create Workflow File Create `.github/workflows/dbt-euno-upload.yml` in your repository: 
name: dbt Build and Upload to Euno

on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
  workflow_dispatch: # Allow manual trigger

env:
  DBT_PROFILES_DIR: /tmp/profiles
  DBT_PROJECT_DIR: .

jobs:
  dbt-build-and-upload:
    runs-on: ubuntu-latest
    
    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.9'

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install dbt-core dbt-snowflake # Replace dbt-snowflake with your adapter

    - name: Set up dbt profile
      run: |
        mkdir -p $DBT_PROFILES_DIR
        cat << EOF > $DBT_PROFILES_DIR/profiles.yml
        # Your dbt profiles configuration here
        # This is a template - replace with your actual configuration
        your_project_name:
          target: prod
          outputs:
            prod:
              type: snowflake # or your warehouse type
              account: ${{ secrets.DBT_SNOWFLAKE_ACCOUNT }}
              user: ${{ secrets.DBT_SNOWFLAKE_USER }}
              password: ${{ secrets.DBT_SNOWFLAKE_PASSWORD }}
              role: ${{ secrets.DBT_SNOWFLAKE_ROLE }}
              database: ${{ secrets.DBT_SNOWFLAKE_DATABASE }}
              warehouse: ${{ secrets.DBT_SNOWFLAKE_WAREHOUSE }}
              schema: ${{ secrets.DBT_SNOWFLAKE_SCHEMA }}
              threads: 4
              keepalives_idle: 240
        EOF

    - name: Run dbt build
      run: |
        dbt deps
        dbt build
        dbt docs generate

    - name: Create and upload artifacts zip
      run: |
        cd target
        
        # Check if required files exist
        for file in catalog.json manifest.json run_results.json; do
          if [ ! -f "$file" ]; then
            echo "Error: Required file $file not found"
            exit 1
          fi
        done
        
        # Create zip file with artifacts
        echo "Creating zip file with dbt artifacts..."
        zip -r dbt_artifacts.zip catalog.json manifest.json run_results.json
        
        # Add semantic_manifest.json if it exists
        if [ -f "semantic_manifest.json" ]; then
          zip dbt_artifacts.zip semantic_manifest.json
          echo "Added semantic_manifest.json to zip"
        fi
        
        # Step 1: Get signed upload URL from Euno
        echo "Requesting signed upload URL from Euno..."
        prepare_response=$(curl -s -w "\n%{http_code}" \
          -X POST \
          -H "Authorization: Bearer ${{ secrets.EUNO_INTEGRATION_KEY }}" \
          -H "Content-Type: application/json" \
          -d '{"filename": "dbt_artifacts.zip"}' \
          "${{ secrets.EUNO_ENDPOINT_URL }}")
        
        # Extract status code and response body
        prepare_status_code=$(echo "$prepare_response" | tail -n1)
        prepare_response_body=$(echo "$prepare_response" | head -n -1)
        
        echo "Prepare-upload response: $prepare_response_body"
        echo "Prepare-upload status code: $prepare_status_code"
        
        # Check if prepare-upload was successful
        if [ "$prepare_status_code" -ne 200 ]; then
          echo "❌ Failed to get signed URL with status code: $prepare_status_code"
          exit 1
        fi
        
        # Extract upload URL from response (handle both response formats)
        upload_url=$(echo "$prepare_response_body" | jq -r '.body.upload.url // .upload.url // empty')
        
        if [ -z "$upload_url" ] || [ "$upload_url" = "null" ]; then
          echo "❌ Failed to extract upload URL from response"
          echo "Response: $prepare_response_body"
          exit 1
        fi
        
        echo "βœ“ Signed URL obtained: ${upload_url:0:50}..."
        
        # Step 2: Upload file to signed URL
        echo "Uploading artifacts to signed URL..."
        upload_response=$(curl -s -w "\n%{http_code}" \
          -X PUT \
          -H "Content-Type: application/zip" \
          --data-binary @dbt_artifacts.zip \
          "$upload_url")
        
        # Extract status code and response body
        upload_status_code=$(echo "$upload_response" | tail -n1)
        upload_response_body=$(echo "$upload_response" | head -n -1)
        
        echo "Upload response: $upload_response_body"
        echo "Upload status code: $upload_status_code"
        
        # Check if upload was successful
        if [ "$upload_status_code" -eq 200 ] || [ "$upload_status_code" -eq 201 ]; then
          echo "βœ“ Artifacts uploaded successfully!"
        else
          echo "❌ Upload failed with status code: $upload_status_code"
          exit 1
        fi

    - name: Upload artifacts on failure
      if: failure()
      uses: actions/upload-artifact@v4
      with:
        name: dbt-artifacts
        path: target/
        retention-days: 3
## Configuration Notes ### Database Adapter Update the workflow to install the correct dbt adapter for your data warehouse: * **Snowflake**: `pip install dbt-snowflake` * **BigQuery**: `pip install dbt-bigquery` * **Redshift**: `pip install dbt-redshift` * **Databricks**: `pip install dbt-databricks` ### Profiles Configuration Replace the `profiles.yml` template with your actual dbt configuration. You'll need to add the necessary secrets to your GitHub repository for database credentials. ### Required GitHub Secrets **For all warehouses:** * `EUNO_INTEGRATION_KEY` β€” your dbt source **trigger secret** (Bearer token) * `EUNO_ENDPOINT_URL` ## Workflow Triggers The workflow runs on: * **Push to main/master branch**: Automatically uploads production artifacts * **Pull requests**: Validates dbt build but can be configured to upload test artifacts * **Manual trigger**: Use the "Actions" tab to run manually ## Customization * **Conditional upload**: Add conditions to only upload on specific branches * **Slack/Teams notifications**: Add notification steps for success/failure * **Artifact retention**: Adjust the retention period for failed build artifacts * **Parallel jobs**: Split into separate build and upload jobs for complex workflows # Prefect Upload This Prefect flow automates the process of running your dbt project, collecting artifacts, and uploading them to Euno. ## Prerequisites * Python 3.8+ * Prefect 2.1+ * `prefect-dbt` package * `requests` library ## Installation ```bash pip install prefect prefect-dbt requests ``` ## Configuration Before running the flow, update these configuration variables: ```python WEBHOOK_URL = "https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload" EUNO_TOKEN = "your_trigger_secret_here" # Trigger secret from your dbt source (Bearer token) DBT_PROJECT_PATH = "/path/to/your/dbt/project" DBT_PROFILES_PATH = "/path/to/your/dbt/profiles" ``` ### With Prefect Blocks ```bash # Create secret blocks prefect block register -m prefect.blocks.system # Set your secrets prefect config set PREFECT_API_URL="your-prefect-server-url" ``` ```python from prefect.blocks.system import Secret # Create secrets (run once) Secret(value="your_webhook_url").save("euno-webhook-url") Secret(value="your_integration_token").save("euno-integration-token") Secret(value="/path/to/your/dbt/project").save("dbt-project-path") Secret(value="/path/to/your/dbt/profiles").save("dbt-profiles-path") ``` ## Prefect Flow The following example is basd on `prefect-dbt` 0.7.0 and later. For earlier versions see: [prefect-dbt documentation](https://docs.prefect.io/integrations/prefect-dbt#prefect-dbt-0-6-6-and-earlier) 
from prefect import flow, task, get_run_logger
from prefect_dbt import PrefectDbtRunner, PrefectDbtSettings
from prefect.blocks.system import Secret
import zipfile
import tempfile
import requests
import os
import json
from pathlib import Path
from typing import List, Optional, Tuple, Dict, Any

DBT_PROJECT_PATH = "/path/to/your/dbt/project"
DBT_PROFILES_PATH = "/path/to/your/dbt/profiles"

# Configuration using Prefect Blocks for secrets. 
# Can be replaced by other secret managers or env variables
@task
def get_euno_integration_credentials():
    """Retrieve Euno Integration credentials from Prefect Secret Blocks."""
    return {
        "webhook_url": Secret.load("euno-webhook-url").get(),
        "token": Secret.load("euno-integration-token").get(),
    }

@task
def dbt_build(project_dir: str, profiles_dir: str) -> Path:
    """Run dbt build and docs generate commands."""
    logger = get_run_logger()

    try:
        logger.info(f"Running dbt commands in {project_dir}")
        
        # Run dbt build
        PrefectDbtRunner(
            settings=PrefectDbtSettings(
                project_dir=project_dir,
                profiles_dir=profiles_dir,
            )
        ).invoke(["build"])
        
       # Run dbt docs generate
        PrefectDbtRunner(
            settings=PrefectDbtSettings(
                project_dir=project_dir,
                profiles_dir=profiles_dir,
            )
        ).invoke(["docs", "generate"])

        target_path = Path(project_dir, "target")
        logger.info(f"dbt commands completed successfully. Target path: {target_path}")
        return target_path

    except Exception as e:
        logger.error(f"dbt build failed: {str(e)}")
        raise


@task
def collect_and_zip_artifacts(path: Path) -> Optional[str]:
    """Collect dbt artifacts and create a zip file with validation."""
    logger = get_run_logger()

    # List of required artifact files
    artifact_files = ["manifest", "catalog", "run_results", "semantic_manifest"]

    # Check if all required files exist
    missing_files = []
    for f in artifact_files:
        file_path = path / f"{f}.json"
        if not file_path.exists():
            missing_files.append(f"{f}.json")

    if missing_files:
        logger.warning(f"Missing artifact files: {missing_files}")

    # Check if we have at least manifest and catalog (minimum required)
    required_files = ["manifest", "catalog"]
    missing_required = [f for f in required_files if f"{f}.json" in missing_files]

    if missing_required:
        logger.error(f"Missing required files: {missing_required}")
        raise FileNotFoundError(f"Required artifact files missing: {missing_required}")

    temp_file_name = tempfile.mktemp(suffix=".zip")
    logger.info(f"Creating zip file: {temp_file_name}")

    try:
        with zipfile.ZipFile(
            temp_file_name, mode="w", compression=zipfile.ZIP_DEFLATED
        ) as archive:
            for f in artifact_files:
                file_path = path / f"{f}.json"
                if file_path.exists():
                    archive.write(file_path, arcname=f"{f}.json")
                    logger.info(f"Added {f}.json to zip")
                else:
                    logger.warning(f"Skipping {f}.json - file not found")

        # Get file size for logging
        file_size = os.path.getsize(temp_file_name)
        logger.info(f"Zip file created successfully. Size: {file_size} bytes")

        return temp_file_name

    except Exception as e:
        # Clean up on error
        logger.error(f"Error creating zip file: {str(e)}")
        if os.path.exists(temp_file_name):
            os.unlink(temp_file_name)
        raise


@task
def get_signed_url(webhook_url: str, token: str, filename: str) -> Tuple[str, Dict[Any, Any]]:
    """Request a signed upload URL for the zip file."""
    logger = get_run_logger()
    headers = {"Authorization": f"Bearer {token}"}
    body = {"filename": filename}
    
    logger.info(f"Requesting signed upload URL from: {webhook_url}")
    response = requests.post(webhook_url, headers=headers, json=body, timeout=30)
    
    logger.info(f"Signed URL request status code: {response.status_code}")
    
    if response.status_code != 200:
        logger.error(f"Failed to obtain signed URL (status {response.status_code}): {response.text}")
        response.raise_for_status()
    
    payload = response.json()
    upload_url = payload.get("upload", {}).get("url")
    
    if not upload_url:
        logger.error(f"Signed upload URL not found in response: {json.dumps(payload)[:500]}")
        raise ValueError("Signed upload URL not found in response")
    
    logger.info("Signed URL obtained successfully")
    return upload_url, payload


@task
def upload_file(zip_file_path: str, upload_url: str) -> bool:
    """Upload the zip file to the signed URL."""
    logger = get_run_logger()
    
    logger.info("Uploading ZIP file to signed URL")
    try:
        with open(zip_file_path, "rb") as fp:
            response = requests.put(
                upload_url, 
                data=fp, 
                headers={"content-type": "application/zip"}, 
                timeout=300  # 5 minute timeout for large files
            )
        
        logger.info(f"Upload status code: {response.status_code}")
        
        if response.status_code not in (200, 201):
            logger.error(f"Upload failed (status {response.status_code}): {response.text[:200]}")
            response.raise_for_status()
        
        logger.info("ZIP file uploaded successfully")
        return True
        
    except Exception as e:
        logger.error(f"An error occurred during upload: {e}")
        raise


@flow(log_prints=True)
def dbt_euno_upload_flow():
    """Enhanced Prefect flow to run dbt and upload artifacts to Euno."""
    logger = get_run_logger()

    # Get secrets from Prefect Blocks (recommended for production)
    euno_integration_credentials = get_euno_integration_credentials()
    dbt_project_path = Secret.load("dbt-project-path").get() # or use DBT_PROJECT_PATH
    dbt_profiles_path = Secret.load("dbt-profiles-path").get() # or use DBT_PROFILES_PATH

    logger.info("Starting enhanced dbt build and upload flow...")

    try:
        # Run dbt build and docs generate
        target_path = dbt_build(
            project_dir=dbt_project_path,
            profiles_dir=dbt_profiles_path,
        )

        # Collect artifacts and create zip
        artifacts_zip_path = collect_and_zip_artifacts(target_path)

        if artifacts_zip_path:
            try:
                # Get signed URL
                upload_url, meta = get_signed_url(
                    euno_integration_credentials["webhook_url"],
                    euno_integration_credentials["token"],
                    Path(artifacts_zip_path).name
                )
                
                # Upload file
                success = upload_file(artifacts_zip_path, upload_url)
                
                if success:
                    logger.info("Upload succeeded! Prepare-upload response:")
                    logger.info(json.dumps(meta, indent=2))
                    return True
                else:
                    logger.error("Upload failed.")
                    return False
                
            finally:
                # Clean up temporary zip file after upload attempt
                if os.path.exists(artifacts_zip_path):
                    try:
                        os.unlink(artifacts_zip_path)
                        logger.info(f"Cleaned up temporary file: {artifacts_zip_path}")
                    except Exception as e:
                        logger.warning(f"Failed to clean up temporary file: {str(e)}")
        else:
            logger.error("No artifacts collected, skipping upload")
            return None

    except Exception as e:
        logger.error(f"Flow failed: {str(e)}")
        raise


if __name__ == "__main__":
    dbt_euno_upload_flow()
## Notes * The artifact files `manifest.json` and `catalog.json` are required. Other missing artifacts won't fail the flow * dbt `build` can be replaced with dbt `compile` ## Usage ### Option 1: Direct Execution ```bash python your_flow_file.py ``` ### Option 2: Prefect Deployment #### Prefect 2.10+ ```bash # Create a deployment prefect deployment build your_flow_file.py:dbt_euno_upload_flow -n "dbt-euno-upload" # Apply the deployment prefect deployment apply dbt_euno_upload_flow-deployment.yaml # Run the deployment prefect deployment run dbt-euno-upload-flow/dbt-euno-upload ``` #### Prefect 3.x+ ```bash # Create a work pool (one-time setup, required in Prefect 3.x) prefect work-pool create dbt-pool --type process # Deploy the flow prefect deploy your_flow_file.py:dbt_euno_upload_flow --name dbt-euno-upload --pool dbt-pool # Run the deployment prefect deployment run dbt-euno-upload-flow/dbt-euno-upload ``` ### Option 3: Scheduled Runs ```python from prefect.deployments import Deployment from prefect.server.schemas.schedules import CronSchedule deployment = Deployment.build_from_flow( flow=dbt_euno_upload_flow, name="dbt-euno-daily", schedule=CronSchedule(cron="0 6 * * *"), # Daily at 6 AM ) deployment.apply() ``` ## Monitoring and Alerts Add notification blocks for monitoring: ```python from prefect.blocks.notifications import SlackWebhook @flow(log_prints=True) def dbt_euno_upload_flow(): try: # ... flow logic ... return status_code except Exception as e: # Send failure notification slack_webhook = SlackWebhook.load("your-slack-block") slack_webhook.notify(f"dbt Euno upload flow failed: {str(e)}") raise ``` # Dagster Upload This guide shows how to run your dbt Core project with [Dagster](https://dagster.io/) (using the `dagster-dbt` integration) and automatically upload the resulting artifacts to Euno after every successful build. Euno's dbt Core integration is **orchestrator-agnostic**: it only needs your dbt artifacts (`manifest.json`, `catalog.json`, `run_results.json`, and optionally `semantic_manifest.json`) delivered through the two-step `prepare-upload` flow. Dagster is therefore fully compatible β€” it is responsible only for running dbt and ordering the upload after the build. The same upload logic used in the Python, GitHub Actions, and Prefect guides applies here. ### Prerequisites * Python 3.9+ * A dbt Core project you already run with Dagster (`dagster` + `dagster-dbt`) * Euno **trigger secret** and **endpoint URL** for your dbt source ```bash pip install "dagster" "dagster-dbt" "dbt-core" "dbt-" "requests" ``` Replace `` with your warehouse adapter, e.g. `dbt-snowflake`, `dbt-bigquery`, `dbt-redshift`, or `dbt-databricks`. ### Configuration Provide these as environment variables (recommended) so secrets stay out of code: ```bash export EUNO_ENDPOINT_URL="https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload" export EUNO_TOKEN="your_trigger_secret_here" # Bearer token from your dbt source settings export DBT_PROJECT_DIR="/path/to/your/dbt/project" export DBT_PROFILES_DIR="/path/to/your/dbt/profiles" # optional ``` ### How it works The example defines three things: 1. **`euno_dbt_assets`** β€” a `@dbt_assets` definition that runs `dbt build` (producing `manifest.json` and `run_results.json`) and then `dbt docs generate` (producing `catalog.json`, which Euno requires). 2. **`euno_artifacts_upload`** β€” an asset that `deps` on the dbt assets, so it runs **only after a successful build**. It zips the artifacts from `target/` and uploads them to Euno. 3. **A job + daily schedule** wiring the two together. The upload is the same two steps used everywhere in Euno's dbt Core integration: * **Step 1 – prepare-upload:** `POST` the zip filename to your endpoint with the Bearer token and receive a short-lived **signed URL**. * **Step 2 – upload:** `PUT` the zip to that signed URL. > The helper handles both `prepare-upload` response shapes Euno may return (`{"body": {"upload": {"url": ...}}}` and `{"upload": {"url": ...}}`). ### Example A complete, ready-to-adapt module is provided here: `dagster_euno_upload.py`. The key parts: ```python from dagster import AssetExecutionContext, AssetKey, Definitions, ScheduleDefinition, asset, define_asset_job from dagster_dbt import DbtCliResource, DbtProject, dbt_assets dbt_project = DbtProject(project_dir=DBT_PROJECT_DIR) dbt_project.prepare_if_dev() @dbt_assets(manifest=dbt_project.manifest_path) def euno_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() # manifest.json + run_results.json dbt.cli(["docs", "generate"], context=context).wait() # catalog.json @asset(deps=[euno_dbt_assets]) def euno_artifacts_upload(context: AssetExecutionContext) -> None: target_dir = Path(dbt_project.project_dir) / "target" zip_path = _zip_artifacts(target_dir) # zips the 3-4 artifact files upload_url, _ = _get_signed_url(endpoint_url, token, Path(zip_path).name) # step 1 _put_file(zip_path, upload_url) # step 2 ``` ### Integrating with an existing dagster-dbt project If you already have a `@dbt_assets` definition, you don't need to rewrite it. Just: 1. Add a `dbt docs generate` call after your `dbt build`/`dbt run` so `catalog.json` is produced (Euno needs it for schema information). 2. Copy the upload helpers and the `euno_artifacts_upload` asset from the example, pointing `deps` at **your** dbt assets definition. 3. Add `euno_artifacts_upload` to your job/`Definitions` and select it alongside your dbt assets. ### Notes * `manifest.json` and `catalog.json` are required; `run_results.json` and `semantic_manifest.json` are included when present. * `dbt build` can be replaced with `dbt run` (or `dbt compile`) depending on your pipeline β€” just make sure `dbt docs generate` still runs so `catalog.json` exists. * To run on a schedule, keep the `ScheduleDefinition` (defaults to daily at 06:00). To upload only on demand, remove the schedule and trigger the job manually or from a Dagster sensor. * If you split dbt models across multiple Dagster assets/jobs, run `dbt docs generate` and upload once per full build so Euno receives a complete, consistent set of artifacts. # dbt Cloud Euno's integration with dbt Cloud supports auto-discovery of [dbt resources](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources). It automates the retrieval and processing of dbt artifacts from completed runs. This allows for seamless data synchronization and analysis using the latest available dbt job runs. ### How It Works The integration follows these steps: 1. **Retrieve Jobs from dbt Cloud** * Using a dbt Cloud service token, the integration queries the dbt Cloud API to fetch all jobs associated with the single dbt Cloud project specified by the **dbt Cloud Project ID** in the integration configuration. The integration does not iterate over other projects that may be accessible via the service token; to cover multiple projects, create a separate Euno integration per project. 2. **Find the Latest Unprocessed Run for Each Job** * For each job, the integration identifies the latest run that has completed successfully and has not yet been processed by Euno. 3. **Retrieve Artifacts from the Run** * Once an eligible job run is found, the integration retrieves its associated artifacts: * `run_results.json` * `manifest.json` * `semantic_manifest.json` * `catalog.json` 4. **Process the Artifacts** * The integration processes the retrieved artifacts to extract relevant information and adds the [discovered](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources) [resources](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources) to Euno's data model. Runs are started by a **schedule**, **Run now** in the Euno UI, or an on-demand **Trigger URL** (for example, wired to a dbt Cloud `job.run.completed` webhook). See [Triggering integration runs](#triggering-integration-runs). ### Advanced Configuration #### Filtering Jobs It's possible to define patterns to include or exclude jobs based on: * Job names * Environments * Branch names **Without Advanced Filtering** If no filtering patterns are set, the integration will process all jobs retrieved from dbt Cloud. This means every available job in the configured project will be considered. **With Advanced Filtering** By defining inclusion/exclusion patterns, users can: * Focus on specific jobs that match certain naming conventions. * Exclude jobs from non-production environments. * Restrict processing to specific Git branches (e.g., `main` or `production`). This provides better control over which jobs are processed, reducing unnecessary artifact downloads and processing time. ### Prerequisites * A dbt Cloud **service token** with sufficient `Read-only` permission. See dbt Cloud's documentation on [how to create service tokens](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens). * The service token should grant access to the dbt project with the job run(s) to retrieve. A single token with access to multiple projects can be reused across separate Euno integrations. ## Setting up Euno's dbt Cloud Integration ### Step 1: Configure New dbt Cloud Source in Euno #### Step 1: Access the Sources Page 1. Navigate to the **Sources** page in the Euno application. 2. Click on the **Add New Source** button. #### Step 2: General Configuration Asterik (\*) means a mandatory field.
ConfigurationDescription
Name*Enter a name for your dbt Cloud source (e.g., "dbt - Marketing Models")
dbt Cloud Account ID*Enter you dbt Cloud Account ID. It can be found in the URL, e.g. https://cloud.getdbt.com/settings/accounts/{account_id}/pages/projects/{project_id}
dbt Cloud Project ID*Enter you dbt Cloud Project ID. It can be found in the URL,
e.g. https://cloud.getdbt.com/settings/accounts/{account_id}/pages/projects/{project_id}
dbt Cloud Api Token*Enter the service token created in the previous section
dbt Cloud Job ID (optional)Specific dbt Cloud job ID to process. If not specified, all jobs under the dbt project associated with this integration will be processed.
#### **Step 3: Resource Cleanup Options** To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno.](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno) * **Time-Based Cleanup (default)**: Remove resources that were last detected X days before the most recent successful source integration run (user-defined X, default is 7 days). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. #### Step 4: Scheduling Updates 1. Enable the Schedule option. 2. Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). #### Regenerating the Trigger URL Euno shows the dbt Cloud Trigger URL once when the source is created. If you need to recover or rotate it, go to **Sources**, open the dbt Cloud source's menu (**β‹―**), and select **Generate Trigger URL**. You can also generate a new Trigger URL from the source details page. Generating a new URL immediately invalidates the previous URL, so update any dbt Cloud webhooks to use the new value. #### Step 5: Advanced Settings (Optional) Click on the '**Advanced**' section to display these additional configurations.
ConfigurationDescription
Job NameDefine patterns to include or exclude jobs based on their names.
Job EnvironmentDefine patterns to include or exclude jobs based on their environment.
Job BranchDefine patterns to include or exclude jobs based on their branch name.
dbt Cloud URLIf you have a dedicated URL to access dbt Cloud enter it here. Defaults to "https://cloud.getdbt.com"
Override Build TargetOverride the auto-detected warehouse URI prefix. Use when the warehouse account identifier reported by dbt Cloud differs from the one used by the warehouse integration (e.g. Snowflake account locator vs organization account name). Accepts a Snowflake host (e.g. account.snowflakecomputing.com), a Databricks host, or an already-canonized prefix (e.g. snowflake.myorg-myaccount).
#### Step 6: Save Configuration Click **Test & Save** (or **Save**). Euno generates a **Trigger URL** for the new source. Copy and store it securely β€” it is shown only once. You will need it if you want dbt Cloud (or another orchestrator) to start an Euno integration run immediately after a dbt job finishes. ## Triggering integration runs Euno can pick up new dbt Cloud runs in three ways: 1. **Schedule** β€” Enable scheduling in Step 4 so Euno polls dbt Cloud on a recurring interval. 2. **Run now** β€” From the **Sources** page, open the source menu (β‹―) and choose **Run now**, or use **Run now** on the source details page. 3. **Trigger URL** β€” Send an HTTP **POST** to the Trigger URL to start a crawl on demand. This is the URL to configure as the target of a [dbt Cloud webhook](https://docs.getdbt.com/docs/deploy/webhooks) when you want Euno to sync right after a job completes. ### Trigger URL format When you create the source, Euno returns a URL in this form: ``` https://{euno-api-host}/accounts/{account_id}/integrations/{integration_id}/crawl?trigger_secret={secret} ``` Replace `{euno-api-host}` with your Euno API hostname (`api.app.euno.ai` in production). This is the API host, not the Euno web app (`app.euno.ai`). `{account_id}` and `{integration_id}` are the numeric IDs from the URL path when you view the source in Euno. `{secret}` is a one-time secret embedded in the query string. Example: ``` https://api.app.euno.ai/accounts/42/integrations/17/crawl?trigger_secret=a1b2c3d4e5f6789012345678901234ab ``` To trigger a run, send **POST** to that URL. Euno ignores the request body (including JSON payloads from dbt Cloud webhooks) and authenticates using `trigger_secret`. Alternative authentication (without the query parameter): ```bash curl -X POST "https://api.app.euno.ai/accounts/42/integrations/17/crawl" \ -H "X-Trigger-Secret: YOUR_SECRET" ``` If you lose the Trigger URL, you can still use **Run now** in the Euno UI. Rotating the secret requires generating a new Trigger URL through your Euno account administrator. ### Configure a dbt Cloud webhook To have dbt Cloud notify Euno when a job finishes, create a webhook subscription in dbt Cloud that **POST**s to your Euno Trigger URL. 1. In dbt Cloud, go to **Account Settings β†’ Webhooks** (or use the [Webhooks API](https://docs.getdbt.com/docs/deploy/webhooks)). 2. Create a subscription with: * **Event type:** `job.run.completed` β€” use this event (not `job.run.errored`) so job metadata and artifacts are available in the dbt Cloud API before Euno runs. * **Client URL:** your full Euno Trigger URL, including the `trigger_secret` query parameter. * **Job IDs:** limit to the jobs this Euno source should process, or leave empty to fire on every job in the account (Euno still only processes jobs in the configured dbt Cloud project and filters). 3. Save the subscription. dbt Cloud sends a **POST** request to the Client URL after each matching run completes. Example subscription payload (API): ```json { "name": "Euno sync", "description": "Trigger Euno dbt Cloud source after job completion", "client_url": "https://api.app.euno.ai/accounts/42/integrations/17/crawl?trigger_secret=YOUR_SECRET", "event_types": ["job.run.completed"], "active": true, "job_ids": [123456] } ``` Each webhook call starts an Euno crawl. The crawl fetches the latest unprocessed run per configured job, downloads artifacts from dbt Cloud, and updates Euno's data model. For production, combine webhooks (near-real-time sync after builds) with a schedule (fallback if a webhook delivery fails). # dbt Source Project Euno's dbt Source Project integration supports auto-discovery of [dbt resources](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources). Unlike **dbt Cloud** (where Euno fetches artifacts from the dbt Cloud API) or **dbt Core** (where you upload pre-built artifacts), the dbt Source Project integration **builds your dbt project inside Euno's infrastructure** and generates the JSON artifacts for you. You do not need to run dbt yourself or upload any files. ## How It Differs from Other dbt Integrations | Integration | How artifacts are obtained | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **dbt Cloud** | Euno fetches artifacts from the dbt Cloud API after jobs run in dbt Cloud. | | **dbt Core** | You upload pre-built artifacts (manifest, catalog, run\_results, etc.) to Euno. | | **dbt Source Project** | Euno clones your git repository, runs the dbt project in a secure, managed environment, and generates the JSON artifacts; then Euno processes them the same way as dbt Core. | ## How It Works 1. **Clone the repository** – Euno clones your dbt project from GitHub or GitLab using the deploy key you provide (and optional branch or subdirectory). 2. **Build in Euno's environment** – Euno prepares the project and runs dbt compile (including dependency resolution) in a managed container, using the warehouse target you configured (e.g. BigQuery project and dataset). 3. **Collect artifacts** – Euno downloads the generated artifacts (manifest.json, catalog.json, run\_results.json, and semantic\_manifest.json when present). 4. **Process the artifacts** – Euno processes these artifacts and adds the same [discovered dbt resources](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources) to the data model as with dbt Core. ## Prerequisites * A **git repository** (HTTPS URL) containing your dbt project. GitHub and GitLab are supported. * An **SSH deploy key** with read access to the repository. The deploy key is used by Euno to clone the repo. * **Warehouse credentials**: The credentials and target details required depend on the **target environment** you select. Currently, **BigQuery** is the only supported target environment. See the configuration section below for details. ## Setting Up Euno's dbt Source Project Integration ### Step 1: Access the Sources Page 1. Navigate to the **Sources** page in the Euno application. 2. Click on the **Add New Source** button. 3. Select **dbt Source Project**. ### Step 2: General Configuration An asterisk (\*) indicates a required field. **Common settings** (all target environments):
ConfigurationDescription
Name*Enter a name for your dbt Source Project source (e.g., "dbt - Marketing Models").
Repository URL*The HTTPS URL of the git repository containing the dbt project (e.g., https://github.com/your-org/your-dbt-repo).
Repository subdirectorySubdirectory within the repository where the dbt project lives. Defaults to / (repository root).
Repository branchGit branch to use. If not specified, the repository's default branch is used.
Deploy key*SSH private key for repository access. Euno uses this to clone the repository. Ensure the corresponding public key is added to the repo as a deploy key with read access. See GitHub's Managing deploy keys for how to add a deploy key to a GitHub repository.
Target environment*The warehouse to use for dbt compilation. Select your warehouse below and configure the required fields for that target.
**Target environment – warehouse-specific configuration** Currently, **BigQuery** is the only supported target environment. Select **BigQuery** in the integration form and configure the required fields below.
BigQuery When **BigQuery** is selected as the target environment, configure:
ConfigurationDescription
Service account JSON*GCP service account JSON key. The service account must have access to the BigQuery project and dataset used for compilation. Minimum privileges: BigQuery Job User (to run compile jobs) and BigQuery Data Viewer or BigQuery Data Editor on the target project/dataset so dbt can read metadata and generate the catalog. See Create and delete service account keys and BigQuery: Use service accounts for how to create and configure the key.
Target GCP project*The GCP project ID containing the BigQuery dataset.
Target BigQuery dataset*The BigQuery dataset used for dbt compile (e.g., for metadata/catalog generation).
#### Future support The following warehouse targets are not yet available but are planned for a future release.
Snowflake (planned) Support for **Snowflake** as a build target is planned. Configuration and required fields will be documented here when available.
Databricks (planned) Support for **Databricks** as a build target is planned. Configuration and required fields will be documented here when available.
### Step 3: Resource Cleanup Options To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation, see [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Time-Based Cleanup (default)**: Remove resources that were last detected X days before the most recent successful source integration run (user-defined X, default is 7 days). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 4: Advanced Settings (Optional) Click on the **Advanced** section to display these additional configurations.
ConfigurationDescription
Schema aliasesA mapping of database.schema combinations. Euno will ingest dbt resources to the database and schema stated in the manifest file, unless a mapping is defined. Same behavior as [dbt Core](../dbt-core/README.md) mapping.
Allow resources with no catalog entryIf enabled, Euno will include dbt resources that do not have a corresponding entry in the catalog (e.g., from manifest only). By default, only resources with catalog entries are observed.
Override URI prefixOptional prefix to override the URI of ingested resources. If not set, Euno uses dbt.<dbt project name>.
### Step 5: Save and Run 1. Click **Save**. Euno validates the configuration (including repository access and dbt project structure). 2. After saving, you can **run the integration** on a schedule or manually via **Run now**. There is no artifact upload stepβ€”each run clones the repo, builds the project, and processes the generated artifacts. ## Running the Integration * **Scheduled runs**: Configure a schedule (e.g., daily or weekly) in the source settings. Euno will clone the repository, run dbt compile, and process the artifacts on each run. * **Manual runs**: Use **Run now** on the source page to trigger a run on demand. Each run uses the current state of the configured branch (or default branch) and subdirectory. ## Logs and Artifacts After each run, the integration provides: * **Run report** – Includes repository URL, branch, commit SHA, Cloud Run execution status, and dbt compile duration. * **Logs** – Execution logs from the dbt compile step, available from the integration run details in the UI. * **Artifacts** – The generated artifacts (e.g., manifest.json, catalog.json, run\_results.json) are stored and can be downloaded from the run details for debugging if needed. ## Troubleshooting | Issue | What to check | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Clone or authentication failed** | Verify the deploy key has read access to the repository. Ensure the repository URL is HTTPS and correct. For GitHub/GitLab, confirm the deploy key is added to the repo (or org) with read access. | | **Branch not found** | Ensure the specified branch exists in the repository. If left blank, the default branch is used. | | **dbt project not found** | Verify the repository subdirectory points to the folder containing `dbt_project.yml`. | | **Cloud Run / compile failed** | Ensure the credentials and target settings for your chosen warehouse are correct (e.g. for BigQuery: GCP service account, project, and dataset). Check that the dbt project is valid (e.g., `dbt deps` and `dbt compile` would succeed locally with the same target). | | **Missing artifacts** | The run may have failed during compile. Review the run logs and run report for errors. | ## Related Documentation * [dbt resources discovered by Euno](/sources/transformation-etl/dbt-core/dbt-integration-discovered-resources) – Types of resources (models, sources, metrics, etc.) added to the data model. * [dbt Core](/sources/transformation-etl/dbt-core) – Upload pre-built artifacts instead of having Euno build the project. * [dbt Cloud](/sources/transformation-etl/dbt-cloud) – Use dbt Cloud jobs and have Euno fetch artifacts from the dbt Cloud API. * [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno) – How cleanup and sponsorship work for source integrations. # Matillion DPC Euno's Matillion Data Productivity Cloud (DPC) integration supports auto-discovery of the following Matillion resources: * Matillion Projects * Matillion Pipelines (Transformations and Orchestrations) * Database Tables and Schemas {% hint style="info" %} This integration is designed for Matillion Data Productivity Cloud (DPC), not Matillion ETL (Matillion's legacy product). {% endhint %} ## Setting up Euno's Matillion Integration ### Step 1: Configure New Matillion Source in Euno #### Step 1: Access the Sources Page 1. Go to the **Sources** page. 2. Click on **Add New Source** and select **Matillion** from the list of supported platforms. #### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | ------------------------ | ---------------------------------------------------------------------------------------------------------------- | | Name\* | Enter a name for your Matillion source (e.g., "Matillion") | | Data Platform\* | Select your target data platform: BigQuery, Snowflake, or Databricks | | Account Identifier\* | If Snowflake or Databricks is selected, this is the account URL (e.g. abc12345.us-east-1.snowflakecomputing.com) | | Default Database\* | The default database name for your data warehouse, as configured in Matillion | | Default Schema\* | The default schema name for your data warehouse, as configured in Matillion | | Matillion Project Name\* | The name of your Matillion project | #### Step 3: Optional Matillion Configuration | Configuration | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Matillion Project ID | The unique identifier for your Matillion project | | Source Git Repository | The Git repository URL containing your Matillion pipeline definitions | | Source Git Branch | The Git branch to reference (e.g., "main" or "develop") | | Default Environment ID | The default environment ID for your Matillion project (e.g., d3779338-264f-1f3-aeb7-8493cdcca9d4) | | Default Agent ID | The default agent ID for pipeline execution | | Project Variables | To have Euno substitute project variables for their values, see [Configure Matillion Project variables](/sources/transformation-etl/matillion/configure-matillion-project-variables) | **How Optional Configuration Affects Discovery:** * **Project Creation**: When both **Matillion Project Name** and **Matillion Project ID** are provided, Euno will create a Matillion project resource that serves as the parent container for all discovered pipelines. * **Git Source Links**: When **Source Git Repository** and **Source Git Branch** are configured, Euno will generate external links from each pipeline directly to its source file in the Git repository, enabling easy navigation to pipeline definitions. * **Matillion Designer Links**: When **all four fields** (Project ID, Source Git Branch, Default Environment ID, and Default Agent ID) are provided, Euno will create direct links to edit each pipeline in the Matillion Designer interface. These links follow the format: ``` https://app.matillion.com/designer/project/{project_id}/branch/{branch}?environmentId={environment_id}&agentId={agent_id} ``` #### Step 4: File Pattern Configuration | Configuration | Description | | ---------------- | --------------------------------------------------------------------------------- | | Include Patterns | File patterns to include (defaults to files ending with .tran.yaml or .orch.yaml) | | Exclude Patterns | File patterns to exclude from processing | #### Step 5: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. #### Step 7: Save Configuration Click the **Save** button to complete the setup. ## How to Use the Integration ### Step 1: Prepare Your Matillion DPC Artifacts The Matillion integration works by processing DPC artifacts that you upload to Euno. For a one-time ingestion of Matillion pipelines, you can upload the DPC artifact directly through the Euno UI. To upload artifacts programmatically after each build or export, use the shared [Python Upload for Zip Artifacts](/sources/zip-artifact-python-upload) guide with your Matillion source's `prepare-upload` endpoint and trigger secret. ## Features ### Project and Pipeline Discovery The Matillion integration automatically discovers: * **Matillion Projects**: When project name and ID are configured * **Transformation Pipelines**: From `.tran.yaml` files * **Orchestration Pipelines**: From `.orch.yaml` files * **Pipeline Metadata**: Including names, types, and hierarchical relationships ### Database Resource Discovery The integration maps your data infrastructure by discovering: * **Databases**: Target databases referenced in your pipelines * **Schemas**: Database schemas used by your transformations * **Tables**: Input and output tables with full lineage tracking ### Lineage and Dependencies Advanced dependency tracking includes: * **Table Dependencies**: Automatic discovery of input and output tables in each pipeline, including the lineage between them. This means that for each output table, we will show the input tables it depends on. * **Cross-Platform Integration**: Works with Euno's existing dependency resolution for comprehensive lineage ### External Links When Git repository information is provided, the integration creates direct links to: * **Source Code**: Links to pipeline definitions in your Git repository * **Matillion Designer**: Direct links to pipeline editing in Matillion DPC (when all required IDs are configured) # Matillion Integration Discovered Resources ## Overview The Matillion integration discovers the following resources: * Matillion Project * Matillion Pipeline * Database * Database Schema * Database Table ## Matillion Project A single resource of type "matillion\_project" is observed when both project name and project ID are configured in the integration settings. ### Properties | Property | Value | | ---------- | ----------------------------------------------------------------------------------- | | name | The name of the Matillion project (from configuration) | | type | always `matillion_project` | | subtype | always `matillion_project` | | native\_id | The Matillion project ID (from the configuration, if not set we won't observe this) | ### External Links When Git repository information is configured, the project includes links to: * Source repository for pipeline definitions ## Matillion Pipeline Pipelines are discovered from `.tran.yaml` (transformation) and `.orch.yaml` (orchestration) files in the uploaded DPC artifacts. ### Properties | Property | Value | | ---------- | ------------------------------------------------------------------------- | | name | The filename without the `.yaml`, `.tran.yaml`, or `.orch.yaml` extension | | type | always `matillion_pipeline` | | subtype | either `transformation` or `orchestration` | | native\_id | Derived from the filename and file path | ### Relationships * **Parent**: If a Matillion project is configured, pipelines are children of the project * **Defines**: Pipelines have "defines" relationships with their output tables ### External Links Pipelines can include links to: * **Git Source**: Direct links to the pipeline definition files in the source repository (when Git repository and branch are configured) * **Matillion Designer**: Direct links to edit the pipeline in Matillion DPC when all required configuration is provided: * Project ID * Source Git branch * Default environment ID * Default agent ID Example Designer URL format: ``` https://app.matillion.com/designer/project/{project_id}/branch/{branch}?environmentId={environment_id}&agentId={agent_id} ``` ## Lineage and Dependencies ### Input vs Output Tables For the Matillion integration, let's define: * **Input Tables**: Discovered from components that read from tables * **Output Tables**: Discovered from components that create or write to tables The Matillion integration observes lineage between each output table and its input tables. The lineage is used only for materialized tables. For `views`, Euno will use the lineage analysis of the SQL definition of the view, if available. # Configure Matillion Project variables If your Matillion project make use of [Project Variables](https://docs.matillion.com/data-productivity-cloud/designer/docs/variables/), consider configuring this entry in the Euno Matillion Integration. By configuring project variables, Euno will substitue any reference to `${PROJECT_VARIABLE}` with it's configured value. For example, if you are using Project Variables to store the target schema for a Matillion pipeline, and you are making use of that Project Variable in your "Write Table" Matillion nodes, then Euno will only be able to observe the target table and it's relationship with the pipline if you configure the value of Project Variable in Euno. This configure is done in the Matillion integration configuration screen, under "advanced". The configuration expects a JSON object, where each entry is a project variable. For example, if your project variables are called `foo_db` and `foo_schema` and their values are `bar_db` and `bar_schema` respectively, enter: ``` { "foo_db": "bar_db", "foo_schema": "bar_schema" } ``` ## Using a pipeline to extract all Project Variables In case you have many Project Variables, it might not be feasible to manually configure each one. Instead, follow these steps to extract them all: 1. In Matillion Designer, create a new "Orchestration" pipeline 2. Add a single node of type "python script". 3. Under the "script", configure: ``` SYSTEM_VARIBLES={'__name__', '__doc__', '__package__', '__loader__', '__spec__', '__builtins__', '__file__', '__cached__', 'io', 'os', 'redirect_stdout', 'traceback', 'json', 'SimpleNamespace', '__loadObject', 'ScriptContext', 'processDPCUser', 'context', 'sysvar'} local_variables = [x for x in globals().keys() if x not in SYSTEM_VARIBLES] local_vars_with_value = {x: str(globals()[x]) for x in local_variables} print(json.dumps(local_vars_with_value)) ``` 4. Run the pipeline 5. Open the run, and expand the message. The result is a properly parsed JSON with variable configuration, which you can then copy into the Euno configuration field. # Fivetran Euno's Fivetran integration observed the following resource types: * [Fivetran destination](https://fivetran.com/docs/using-fivetran/fivetran-dashboard/destination) (f.k.a Fivetran Group) * [Fivetran connector](https://fivetran.com/docs/getting-started/glossary#connector) {% hint style="warning" %} In order to use the Fivetran integration, you must be on the "Standard" tier or higher, as the lower tiers do not support REST API access. {% endhint %} ## Setting up Euno's Fivetran Integration ### Step 1: Creating a Fivetran API key 1. Login to your Fivetran account. 2. Click your user avatar, and select API Key 3. Click "**Generate new API key**" 4. Note the **API key** and **API secret**. You'll need them in the next step. ### Step 2 - Configuring Euno's Fivetran Integration #### Step 1: Access the Sources Page 1. Navigate to the **Sources** page and click on the **Gallery** Tab. 2. Click on the **Connect** button in the **Fivetran** Tile. #### Step 2: General Configuration 1. **Name**: Enter a name for your Fivetran source (e.g., "Fivetran - Core"). 2. **Configuration Details**: 1. **API Key:** Enter your Fivetran API key. 2. **API Secret:** Enter your Fivetran API secret. #### Step 3: Advanced configuration (optional) Under **Advanced**, you can configure: | Field | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Base URL** | Fivetran API base URL. Default: `https://api.fivetran.com/v1`. Change only if your account uses a non-default API endpoint. | | **Database mapping** | Optional key/value map from Fivetran destination hostname to the Snowflake host or account identifier Euno uses for the same warehouse. Required when Fivetran's destination host does not match the host observed by your Snowflake integration, so connector `defines` table URIs resolve to the correct warehouse resources. | #### Step 4: Scheduling Updates 1. Enable the Schedule toggle. 2. Choose from the following: 1. **Weekly**: Specify the days and times for updates. 2. **Hourly**: Enter an interval in hours (e.g., every 6 hours). #### Step 5: Resource Cleanup * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). #### Step 6: Save Configuration Click the **Save** button to complete the setup. # Fivetran Integration Discovered Resources The Euno-Fivetran integration discovers the following resources: * Fivetran destination * Fivetran connector * Warehouse tables referenced by each connector's enabled destination schemas (non-authoritative stubs when the table is not already observed by a warehouse integration) Supported destination services for connector-to-table lineage: **Snowflake** and **BigQuery**. ## Fivetran Destination | Property | Value | | ------------------- | ------------------------------------------------------------------------ | | `type` | `fivetran_destination` | | `name` | Destination name in Fivetran | | `created_at` | Destination creation timestamp | | `native_id` | Destination (group) ID in the Fivetran account | | `is_active` | `true` when destination `setup_status` is `connected`; otherwise `false` | | `container_type` | `pure_container` | | `native_raw_object` | Combined Fivetran group and destination API payloads | ## Fivetran Connector | Property | Value | | ------------------- | ------------------------------------------------------------------------------------------ | | `type` | `fivetran_connector` | | `name` | Connector schema name in Fivetran | | `created_at` | Connector creation timestamp | | `parent_container` | URI of the Fivetran destination for this connector | | `native_id` | Connector ID in the Fivetran account | | `is_active` | `true` when the connector is not paused and has succeeded at least once; otherwise `false` | | `native_raw_object` | Full connector payload from the Fivetran API, including enabled destination schemas | | `defines` | Warehouse table URIs the connector loads into its destination (see Relationships) | Connectors with no enabled tables in their Fivetran schema configuration may have an empty `defines` list (for example paused connectors or connectors with empty `schemas` in the API). ## Destination Tables (by reference) For each enabled table in a connector's destination schema, the crawler may emit a non-authoritative `table` observation so Euno can resolve `defines` URIs before the warehouse integration observes the physical table. Lifecycle and rich table metadata remain owned by the warehouse source (for example Snowflake). *** ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ---------------------- | ------------ | -------------------- | ------------------------------------------------------------------------------------ | | `fivetran_destination` | has child | `fivetran_connector` | Connectors belong to their destination | | `fivetran_connector` | defines | `table` | Destination tables the connector syncs into; set from enabled Fivetran schema tables | | `table` | has definer | `fivetran_connector` | Inverse of **defines** | {% hint style="info" %} Connector-to-warehouse-table lineage uses the **defined-by** relationship (`defines` / `has defined`), not dependency lineage (`has downstream`). **Example EQL:** find connectors that define tables: `type = 'fivetran_connector' AND has defined(type = 'table')` Find tables defined by Fivetran: `type = 'table' AND has definer(type = 'fivetran_connector')` {% endhint %} # Stitch Euno's Stitch integration observes replication metadata from Stitch and connects it to the warehouse tables Stitch writes. The integration discovers: * The Stitch account * Stitch sources * Stitch streams * Destination warehouse table references for supported warehouses {% hint style="info" %} Stitch table resolution is supported for Snowflake, BigQuery, SQL Server, and Databricks destinations. Destination configuration is shown as redacted context on the Stitch account rather than as separate resources. Other destination types can appear in the account details, but Euno does not create warehouse table links for them. The account's destination resolver context contains only allow-listed, non-secret identifiers reported by Stitch before **Database mapping** is applied. These values help with support and mapping configuration; they are not canonical mapped warehouse identities. Each run resolves tables from the current typed Stitch destination payload, not from stored account context. {% endhint %} ## Setting up Euno's Stitch Integration ### Step 1: Create a Stitch API access token 1. Log in to your Stitch account. 2. Open **Account Settings**. 3. Go to **API Access Keys**. 4. Create or copy an API access token for the Stitch account you want Euno to observe. 5. Keep the token available for the Euno source setup. ### Step 2: Configure Euno's Stitch source #### Step 1: Access the Sources Page 1. Go to the **Sources** page. 2. Click **Add New Source** or open the source gallery. 3. Select **Stitch**. #### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | ------------- | ----------------------------------------------------------------------- | | Name\* | Enter a name for your Stitch source, for example "Stitch - Production". | | API token\* | Enter the Stitch API access token generated in Stitch. | #### Step 3: Advanced Configuration Under **Advanced**, you can configure: | Configuration | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Base URL | Optional Stitch API base URL. Leave blank to use `https://api.stitchdata.com/v4`. | | Observe extraction and load history | When enabled, Euno collects bounded latest successful extraction and load timestamps for Stitch sources. | | History lookback days | Number of days of Stitch extraction/load history to inspect. The supported range is 1-60 days. Default: 30. | | Observe streams and warehouse lineage | Enabled by default. Euno creates `stitch_stream` resources and resolves selected streams to warehouse tables when possible. When disabled, Euno observes only the Stitch account and sources; streams, warehouse table mappings, and Stitch-derived lineage are omitted. | | Source filter | Optional allow/deny regular expressions matched against Stitch source names. The default includes all Stitch sources. | | Database mapping | Maps Stitch-reported destination identifiers to matching Euno warehouse identities. It is optional for Snowflake, BigQuery, and SQL Server when both systems already use the same identifier, and required for Databricks. | {% hint style="info" %} Use these key/value shapes for **Database mapping**: * **Snowflake:** Stitch destination `host` β†’ the Snowflake host or account identifier Euno observes. * **BigQuery:** Stitch destination `project_id` β†’ the BigQuery project ID Euno observes. * **SQL Server:** Stitch destination `host` β†’ the SQL Server hostname Euno observes. * **Databricks:** hostname from the Stitch `jdbc:spark://[:port]...` URL β†’ `databricks..`. When no mapping entry exists, Snowflake, BigQuery, and SQL Server use the Stitch identifier directly. Databricks always requires a mapping because the JDBC URL does not identify the Unity Catalog catalog. Snowflake and SQL Server host keys and BigQuery project-ID keys must exactly match the values reported by Stitch. Only Databricks key lookup trims and compares keys without case; multiple Databricks keys that normalize to the same JDBC hostname fail as ambiguous. {% endhint %} {% hint style="info" %} Stitch always constructs BigQuery table references with the default `bigquery` URI prefix, and **Database mapping** changes only the project ID. A native BigQuery source configured with a custom `uri_base` emits a different URI prefix, so its tables do not automatically join the Stitch table references. Use the default `bigquery` URI base when automatic matching is required. {% endhint %} #### Step 4: Scheduling Updates 1. Enable the Schedule toggle. 2. Choose from the following: 1. **Weekly**: Specify the days and times for updates. 2. **Hourly**: Enter an interval in hours, for example every 6 hours. #### Step 5: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. #### Step 6: Save Configuration Click the **Save** button to complete the setup. ## What Euno Discovers The Stitch integration discovers the Stitch account, its sources and streams, and supported destination warehouse table references. For the full resource and relationship reference, see [Stitch Integration Discovered Resources](/sources/transformation-etl/stitch-integration/stitch-integration-discovered-resources). ## Destination Warehouse Table Links Euno can link selected Stitch streams to destination warehouse tables when the configured destination is Snowflake, BigQuery, SQL Server, or Databricks and its metadata contains the required warehouse identifiers. For the best table matching results, connect the corresponding Snowflake, BigQuery, SQL Server, or Databricks source in Euno as well. Stitch contributes replication context, but the warehouse integration remains the source of truth for the physical table lifecycle and rich table metadata. Each landed table is defined by the selected Stitch stream that writes it. Stitch sources group streams but do not define tables themselves. Euno uses the stable Stitch source name as the destination schema or dataset and the Stitch stream name as the destination table name. For Databricks, the catalog comes only from the required canonical database mapping; Euno does not infer it from the JDBC URL. If Stitch returns missing, blank, or incorrectly typed identity metadata for a selected supported destination, the run fails closed. Non-string **Database mapping** keys and values are rejected when the source configuration is validated. During a crawl, a blank used mapping, missing required Databricks mapping, normalized Databricks-key collision, or semantically invalid mapped string also fails without treating the incomplete result as proof that an existing mapping was removed. Unsupported destination types are reported and omitted without preventing supported sibling destinations from resolving. ## Salesforce Lineage For Stitch Salesforce sources, Euno can connect replicated warehouse tables back to Salesforce objects and fields. This requires: * A Stitch source whose source type is Salesforce. * A Salesforce source in Euno that has crawled the Salesforce org metadata. * A Snowflake, BigQuery, SQL Server, or Databricks source in Euno that has crawled the destination tables and columns. * **Observe streams and warehouse lineage** enabled for the Stitch source, which is the default. Euno avoids guessing when Salesforce objects or fields are missing or ambiguous. Non-Salesforce Stitch sources are visible as Stitch metadata, but they do not produce source-system lineage in this version. ## Validation After Setup After the first Stitch run completes: 1. Confirm the Stitch source run is successful. 2. Open the data model and search for the Stitch account, sources, and streams. 3. Check that selected Stitch streams define the expected destination warehouse tables. 4. For Salesforce lineage, confirm the related Salesforce and warehouse sources have also run successfully. 5. If table links are missing, review the destination type and database mapping configuration. ## Troubleshooting | Symptom | Possible cause | What to check | | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Stitch setup fails during validation | Invalid or expired Stitch API token | Create a fresh Stitch API access token and update the Euno source. | | A successful run has no destination table links | Stream observation is disabled, the destination type is unsupported, or the selected source has no mapped destination | Enable **Observe streams and warehouse lineage**, then confirm the selected source is mapped to a supported Snowflake, BigQuery, SQL Server, or Databricks destination. | | The run fails because destination inventory or required metadata is unavailable | A mapped destination was not returned to Euno, or a required Stitch-reported identity value is missing, blank, or incorrectly typed | Revalidate the source and confirm that the API token can read the required destination configuration. | | The run fails with a Database mapping instruction | A used mapping is blank, a Databricks hostname mapping is missing or ambiguous, or a mapped string does not satisfy the destination identity contract | Follow the run's destination-specific remediation and correct the exact mapping key/value shape documented above. | | Destination tables do not match existing warehouse resources | Stitch and Euno use different warehouse identifiers | Add or adjust **Database mapping** using the destination-specific key/value shapes above. | | Databricks reports a missing or invalid mapping | The JDBC hostname has no mapping, more than one key normalizes to that hostname, or the value is not a canonical warehouse-and-catalog identity | Keep exactly one mapping for the JDBC hostname and set its value to `databricks..`. Do not use the JDBC path as the catalog. | | Salesforce lineage is missing | Required Salesforce or warehouse metadata is missing | Run the Salesforce source and the destination warehouse source, then rerun Stitch. | | Stream resources or selected-field details are missing | Stream observation is disabled or Stitch stream details were unavailable | Enable **Observe streams and warehouse lineage** and rerun the source. | | The run cannot determine the Stitch account | The token scope does not return a client ID, or saved and current API data report conflicting client IDs | Revalidate the source and confirm that the API token belongs to one Stitch account. | | Latest extraction/load timestamps are missing | History observation is disabled or Stitch history data is unavailable in the lookback window | Enable **Observe extraction and load history** and adjust **History lookback days** if needed. | ## Known Limitations * Destination warehouse table resolution is supported for Snowflake, BigQuery, SQL Server, and Databricks. Other Stitch destination types remain unsupported for table links. * Stitch destinations are account context, not separate data-model resources. * Stitch does not own destination warehouse tables in Euno; the warehouse integration owns the physical table lifecycle. * Databricks destination resolution requires an explicit JDBC-hostname-to-warehouse-and-catalog mapping. * Stitch BigQuery references always use the `bigquery` URI prefix. They do not automatically join native BigQuery resources emitted under a custom `uri_base`; **Database mapping** changes only the project ID. * Source-system lineage is currently supported only for Stitch Salesforce sources. * Row-count metrics, rolling success-rate metrics, and rolling error-count metrics are not currently supported. # Stitch Integration Discovered Resources The Euno-Stitch integration discovers the following resources: * The Stitch account * Stitch sources * Stitch streams * Warehouse tables referenced by selected Stitch streams Supported destination services for Stitch-to-warehouse table links: **Snowflake**, **BigQuery**, **SQL Server**, and **Databricks**. {% hint style="info" %} Stitch warehouse tables are observed by reference. A selected Stitch stream can define that it writes a table, but the corresponding Snowflake, BigQuery, SQL Server, or Databricks integration remains the source of truth for the physical table lifecycle and rich table metadata. {% endhint %} ## Stitch Account A Stitch account represents the client/account scope available to the configured Stitch API token. Destination configuration is summarized on the account for table resolution and support; individual destinations are not data-model resources. | Property | Value | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Stitch client/account ID | | `type` | `stitch_account` | | `native_id` | Stitch client/account ID | | `container_type` | `pure_container` | | `stitch_destination_count` | Number of destinations configured in the Stitch account | | `stitch_active_destination_count` | Number of active destinations configured in the Stitch account | | `stitch_destination_types` | Sorted destination types found in the Stitch account | | `stitch_destination_context` | Minimal account details for each destination, including its name, type, active state, sorted list of mapped source IDs, and allow-listed non-secret Stitch-reported pre-mapping identifiers | | `external_links` | Link to the account in Stitch | | `native_raw_object` | Recursively redacted destination payloads retained as account context | The account sidepane shows at most five destination summaries and the number of additional destinations. Resolver details are limited to the Snowflake or SQL Server host/database and the BigQuery project identifier reported by Stitch. These values help support and **Database mapping** configuration; they are not canonical post-mapping warehouse identities. The crawler resolves tables from each run's current typed destination payload, not from stored account context or graph state. Databricks JDBC URLs and their derived hostnames are not included in account context. ## Stitch Source A Stitch source represents a Stitch integration that extracts data from an upstream SaaS, database, or other source system. | Property | Value | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `name` | Source display name in Stitch, when available; otherwise the Stitch source name | | `type` | `stitch_source` | | `subtype` | Source type reported by Stitch | | `native_id` | Source ID in the Stitch account | | `parent_container` | URI of the Stitch account that contains this source | | `parent_container_name` | Stitch client/account ID | | `is_active` | `true` when the Stitch source is active; otherwise `false` | | `stitch_source_type` | Stitch source type, such as `platform.salesforce` | | `created_at` | Source creation timestamp, when Stitch provides it | | `updated_at` | Source update timestamp, when Stitch provides it | | `description` | Source description, when available | | `external_links` | Link to the source in Stitch, when available | | `stitch_schedule_interval` | Replication schedule interval, when Stitch provides it | | `stitch_schedule_unit` | Replication schedule unit, when Stitch provides it | | `stitch_replication_schedule_type` | Replication schedule type, when Stitch provides it | | `stitch_last_successful_extraction` | Latest successful extraction timestamp in the configured lookback window, when history observation is enabled and Stitch provides the data | | `stitch_last_successful_load` | Latest successful load timestamp in the configured lookback window, when history observation is enabled and Stitch provides the data | | `native_raw_object` | Redacted Stitch source payload | Sources group their child streams and retain connection-level schedule and status details. A source does not define warehouse tables; each selected stream owns its own table mapping. ## Stitch Stream A Stitch stream represents stream or table metadata inside a Stitch source. Euno observes streams when **Observe streams and warehouse lineage** is enabled, which is the default, and Stitch stream metadata is available. | Property | Value | | --------------------------- | ---------------------------------------------------------------------------------------- | | `name` | Stream name in Stitch | | `type` | `stitch_stream` | | `subtype` | Replication method, when Stitch provides it | | `native_id` | Stream ID in Stitch | | `parent_container` | URI of the Stitch source that contains this stream | | `parent_container_name` | Name of the Stitch source that contains this stream | | `is_active` | `true` when the Stitch stream is active; otherwise `false` | | `stitch_is_selected` | `true` when the stream is selected for replication in Stitch | | `stitch_tap_stream_id` | Tap stream ID reported by Stitch, when available | | `stitch_replication_method` | Replication method reported by Stitch, when available | | `stitch_replication_key` | Replication key reported by Stitch, when available | | `stitch_selected_fields` | Selected field names, when Stitch stream detail is available | | `defines` | Warehouse table URIs written by this selected stream, when table resolution is available | | `native_raw_object` | Redacted Stitch stream payload | ## Destination Tables (by reference) For each selected Stitch stream with a supported destination, Euno may observe a non-authoritative `table` reference so it can resolve stream-to-table **defines** relationships before or alongside the warehouse integration. | Property | Value | | --------------------- | ----------------------------------------------------------------------------------------- | | `name` | Destination table name | | `type` | `table` | | `subtype` | `base_table` | | `database_schema` | Destination schema or dataset name derived from the Stitch source name | | `database_database` | Snowflake or SQL Server database name, BigQuery project identifier, or Databricks catalog | | `database_technology` | `snowflake`, `bigquery`, `mssql`, or `databricks` | | `platform_uri_prefix` | Warehouse platform URI prefix used by Euno | | `utl` | Euno URI for the destination warehouse table | Stitch does not write the shared, single-valued `defined_by` property on destination tables. The exact Stitch provenance is represented by the stream's **defines** relationship and the table's inverse **has definer** relationship, which can represent every current defining stream. ### Destination Table Resolution Euno resolves Stitch destination tables for supported destination types: * **Snowflake**: Euno uses the Stitch destination host and database. If Stitch and Euno use different identifiers for the same Snowflake account, configure **Database mapping** from the Stitch host to the Snowflake identifier Euno observes. * **BigQuery**: Euno uses the Stitch `project_id`. The official Stitch destination type is `bigquery_v2`; the compatibility types `bigquery` and `big_query` use the same identity contract. If Stitch and Euno use different project identifiers, configure **Database mapping** from the Stitch project ID to the BigQuery project ID Euno observes. * **SQL Server**: Euno uses the Stitch destination host and database and applies the same canonical hostname rules as the native SQL Server integration. If Stitch and Euno use different hostnames, configure **Database mapping** from the Stitch host to the SQL Server hostname Euno observes. * **Databricks**: Euno parses only the hostname from the Stitch `jdbc:spark://[:port]...` URL. **Database mapping** is required from that hostname to `databricks..`. Euno does not infer the catalog from the JDBC path or a default catalog. Snowflake and SQL Server host keys and BigQuery project-ID keys must exactly match the identifiers reported by Stitch. Only Databricks mapping-key lookup trims and compares keys without case; multiple keys that normalize to the selected JDBC hostname fail as ambiguous. Stitch always constructs BigQuery references with the default `bigquery` URI prefix, and its **Database mapping** changes only the project identifier. A native BigQuery source configured with a custom `uri_base` emits a different URI prefix and therefore does not automatically join those Stitch references. This is a limitation of the pre-existing configurable BigQuery identity, not a change to the native crawler's output. Other destination types can appear in Stitch account context, but destination warehouse table references are not currently supported for them. A selected stream with no mapped destinations remains visible without a **defines** relationship. If a mapped destination is missing from the API response, or a supported destination has missing, blank, incorrectly typed, or malformed Stitch-reported resolver input, the run fails instead of treating the incomplete result as proof that an existing table mapping was removed. Non-string **Database mapping** keys and values are rejected when configuration is validated. Crawl-time mapping failures are limited to a blank used mapping, a missing or ambiguous required Databricks mapping, or a semantically invalid mapped string. Failure details contain safe source, stream, destination, type, reason, and remediation information; credential-bearing resolver or mapping values are not included. ## Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ---------------- | -------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `stitch_account` | has child | `stitch_source` | Sources belong to the Stitch account available to the configured API token | | `stitch_source` | has child | `stitch_stream` | Streams belong to their Stitch source when stream observation is enabled | | `stitch_stream` | defines | `table` | Destination warehouse tables written by selected streams | | `table` | has definer | `stitch_stream` | Inverse of **defines**; the stream is the mapping owner, while the warehouse integration owns the physical table lifecycle | | `table` | has upstream | `salesforce_object` | Only for Stitch Salesforce sources when Salesforce metadata and authoritative warehouse table metadata are present | | `column` | has upstream\_fields | `salesforce_field` | Only when Stitch selected-field evidence, Salesforce field metadata, and authoritative warehouse column metadata are present | {% hint style="info" %} Definition relationships answer which selected Stitch stream writes a warehouse table. They use **defines** and **has definer**. Dependency lineage answers which upstream data object a downstream table or column depends on. Stitch dependency lineage is currently documented only for Salesforce sources, where replicated warehouse tables can have upstream Salesforce objects and replicated warehouse columns can have upstream Salesforce fields. {% endhint %} ## Salesforce Lineage For Stitch Salesforce sources, Euno can connect replicated warehouse tables back to Salesforce objects and fields. This lineage requires: * A Stitch source whose source type is Salesforce. * **Observe streams and warehouse lineage** enabled for the Stitch source, which is the default. * Selected Stitch streams with destination table links. * A Salesforce source in Euno that has crawled the Salesforce org metadata. * A Snowflake, BigQuery, SQL Server, or Databricks source in Euno that has crawled the destination tables and columns. * A destination table name that matches the Salesforce object represented by the Stitch stream. * Selected-field evidence from Stitch for column-level lineage. When all required evidence is present, Euno can show: * `table` **has upstream** `salesforce_object` * `column` **has upstream\_fields** `salesforce_field` Euno avoids guessing when Salesforce objects or fields are missing or ambiguous. Non-Salesforce Stitch sources are visible as Stitch metadata, but they do not produce source-system lineage in this version. ## Known Limitations * Destination warehouse table resolution is supported for Snowflake, BigQuery, SQL Server, and Databricks. Other Stitch destination types do not create table links. * Stitch destinations are account context, not separate data-model resources. * Stitch does not create or own `database`, `schema`, or `column` placeholders for the destination warehouse. * Stitch does not own destination warehouse table lifecycle in Euno. * Databricks destination resolution requires an explicit JDBC-hostname-to-warehouse-and-catalog mapping. * Stitch BigQuery references always use the `bigquery` URI prefix and do not automatically join a native BigQuery source configured with a custom `uri_base`. * Source-system lineage is currently supported only for Stitch Salesforce sources. * Row-count metrics, rolling success-rate metrics, and rolling error-count metrics are not currently supported. # OpenLineage Euno's OpenLineage integration enables seamless ingestion of data lineage events from any system that produces [OpenLineage](https://openlineage.io/) compliant events. This integration automatically processes lineage metadata, table information, and tags to build a comprehensive view of your data pipeline dependencies and transformations. ### How It Works The integration follows these steps: 1. **Provides a secure endpoint** Euno generates a unique trigger secret and endpoint URL for receiving OpenLineage events 2. **Accepts OpenLineage Events** The integration accepts both single events and arrays of events in standard OpenLineage format via HTTP POST 3. **Processes Lineage and Metadata** * Extracts table information from input and output datasets * Creates lineage relationships between input and output tables * Processes tags and metadata from dataset facets * Validates naming conventions for supported data warehouses ### Supported Data Warehouses Currently, the Euno's OpenLineage integration supports: * **Snowflake** - with naming convention: * namespace: `snowflake://org-account` * name: `database.schema.table` * **BigQuery** - with naming convention: * namespace: `bigquery` * name: `project.dataset.table` For detailed naming conventions, see the [OpenLineage Naming Specification](https://openlineage.io/docs/spec/naming). ## Setting up Euno's OpenLineage Integration ### Step 1: Configure New OpenLineage Source in Euno #### Access the Sources Page 1. Navigate to the **Sources** page in the Euno application 2. Click on the **Add New Source** button 3. Select **OpenLineage** from the available integrations ### Step 2: General Configuration 1. **Name**: Enter a descriptive name for your OpenLineage source (e.g., "Data Pipeline Lineage") 2. **Configuration Details**: * OpenLineage integration requires minimal configuration as it's a push-based integration * No schedule configuration is needed since events are pushed in real-time ### Step 3: Resource Cleanup Options Configure automatic **resource cleanup** options to manage outdated resources: * **Time-Based Cleanup (default)**: Remove resources not detected for X days (default: 7 days) * **Immediate Cleanup**: Remove resources not detected in the most recent run * **No Cleanup**: Keep all resources indefinitely ### Step 4: Save Configuration Click the **Save** button, and Euno will generate a trigger secret. **Copy and save this secret securely** as it will not be displayed again. ### Step 5: Get the Upload Endpoint 1. Click **"Reset Trigger Key"** to get the endpoint URL 2. Copy the provided endpoint URL where you'll send OpenLineage events 3. Use the trigger secret from Step 4 as the Bearer token in your Authorization header ## Sending OpenLineage Events ### Example: Single Event with Lineage and Tags Here's a complete example of an OpenLineage event with input/output lineage and tags: ```json { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": { "runId": "my-etl-run-12345" }, "job": { "namespace": "production-pipeline", "name": "customer-analytics-etl" }, "inputs": [ { "namespace": "snowflake://myorg-account123", "name": "raw_data.public.customer_events" }, { "namespace": "bigquery", "name": "external_data.staging.product_catalog" } ], "outputs": [ { "namespace": "snowflake://myorg-account123", "name": "analytics.public.customer_analytics", "facets": { "tags": [ { "key": "environment", "value": "production" }, { "key": "team", "value": "data-engineering" }, { "key": "contains_pii" }, { "key": "data_classification", "value": "sensitive" } ], "schema": { "fields": [ { "name": "customer_id", "type": "BIGINT", "description": "Unique customer identifier" }, { "name": "total_purchases", "type": "DECIMAL(10,2)", "description": "Total purchase amount" } ] } } } ], "producer": "https://my-etl-system.com/v1.2.0", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } ``` ### cURL Command Examples #### Single Event Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '{ "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "simple-etl-123"}, "job": {"namespace": "my-pipeline", "name": "daily-aggregation"}, "inputs": [{ "namespace": "snowflake://myorg-account123", "name": "raw.public.events" }], "outputs": [{ "namespace": "snowflake://myorg-account123", "name": "analytics.public.daily_stats", "facets": { "tags": [ {"key": "environment", "value": "prod"}, {"key": "automated"} ] } }], "producer": "my-pipeline-v1.0", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" }' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Multiple Events Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '[ { "eventType": "START", "eventTime": "2024-01-15T10:00:00.001Z", "run": {"runId": "batch-job-456"}, "job": {"namespace": "etl", "name": "batch-processor"}, "producer": "scheduler-v2.1", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" }, { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:15:00.001Z", "run": {"runId": "batch-job-456"}, "job": {"namespace": "etl", "name": "batch-processor"}, "inputs": [{"namespace": "bigquery", "name": "raw.events.user_actions"}], "outputs": [{"namespace": "bigquery", "name": "processed.analytics.user_metrics"}], "producer": "scheduler-v2.1", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } ]' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Upload from File ```bash # Save your event to a file cat > event.json << 'EOF' { "eventType": "COMPLETE", "eventTime": "2024-01-15T10:30:00.001Z", "run": {"runId": "file-upload-test"}, "job": {"namespace": "testing", "name": "file-upload"}, "outputs": [{ "namespace": "snowflake://myorg-account123", "name": "test.public.sample_table" }], "producer": "test-script", "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent" } EOF # Upload the file curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d @event.json \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` ## What Gets Observed in Euno When OpenLineage events are processed, Euno observes: ### Table Resources * **Tables** from both inputs and outputs with properties: * `name`: Table name * `database`: Database name * `schema`: Schema name * `database_technology`: `snowflake` or `bigquery` * `type`: `table` * `meta` : see below * `tags` : see below ### Lineage Relationships * **Lineage**: Output tables get lineage pointing to input tables ### Tags and Metadata * **Tags with values** (e.g., `{"key": "environment", "value": "prod"}`) become **meta properties** * **Tags without values** (e.g., `{"key": "pii"}`) become **simple tags** **Generating a new trigger secret:** If you need to rotate the secret, go to the **Sources** page and click on the three-dot menu next to your OpenLineage source. Select **"Reset Trigger Key"** to generate a new trigger secret and endpoint URL. # Apache Airflow Euno connects to Apache Airflow to discover workflow orchestration metadata, including Airflow instances, DAGs, tasks, and Airflow datasets or Airflow 3 assets. Euno's Airflow integration supports auto-discovery of: * Airflow instances * DAGs * Tasks * Datasets and assets ## Prerequisites * Airflow 2 with the stable REST API, or Airflow 3 with REST API v2. * An Airflow API token, or an Airflow username and password. * Network access from Euno to the Airflow webserver. * The configured identity must be able to read DAGs and tasks. Dataset or asset permissions are needed to discover dataset-aware scheduling metadata. Euno detects the supported Airflow API version automatically during validation. If the dataset or asset endpoint is not present or the identity cannot read it, Euno continues without those resources. ## Stage 1: Configure Airflow ### Step 1: Create an API token or service account Euno supports two authentication methods: a bearer token (recommended) or username/password basic auth. **Option A β€” API token (recommended)** 1. Log in to your Airflow webserver and go to **Admin β†’ Users**. 2. Open or create the user you want Euno to authenticate as. 3. Scroll to the **Extra** section and generate a new token, or use the Airflow CLI: ```bash airflow users create --username euno-service --role Viewer \ --email euno@example.com --firstname Euno --lastname Service --password ``` 4. Copy the token β€” it will be shown only once. The Airflow user must have at minimum the built-in **Viewer** role so it can read DAGs, tasks, and dataset metadata. If you also want Euno to collect Airflow connections for lineage resolution (the **Resolve warehouse lineage** option), the user additionally needs the **Op** role or a custom role that includes `can_read on Connections`. {% hint style="info" %} Airflow 2 requires the stable REST API to be enabled. In `airflow.cfg`, set `[api] auth_backends = airflow.api.auth.backend.basic_auth` (or `jwt_auth` for token-based auth). Airflow 3 enables the REST API v2 by default. {% endhint %} **Option B β€” Username and password** If your Airflow deployment does not support token authentication, prepare a username and password for a dedicated service account with the permissions described above. Euno will use HTTP Basic auth. ### Step 2: Verify network access Ensure Euno's servers can reach the Airflow webserver URL over HTTPS. If your Airflow instance uses a private CA or self-signed certificate, disable SSL verification in the source configuration (see Stage 2). {% hint style="warning" %} Do not expose your Airflow webserver directly to the public internet if it is not already. Prefer network-level controls (VPN, VPC peering, IP allowlist) to allow only Euno's egress IPs to reach your Airflow instance. {% endhint %} ## Stage 2: Configure New Airflow Source in Euno ### Step 1: Access the Sources Page Navigate to the **Sources** page in Euno and click **Add New Source**. Select **Airflow** from the integration list. ### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | ----------------------------- | ---------------------------------------------------------------------------------------------------- | | **Base URL**\* | Airflow webserver URL. Host-only values are canonicalized to HTTPS. | | **API token** | Bearer token for Airflow API access. If present, Euno uses token auth. | | **Username** | Basic auth username. Used only when no API token is configured. | | **Password** | Basic auth password. Used only when no API token is configured. | | **Verify SSL certificates** | Keep enabled for production. Disable only for local labs or private certificates. Default: enabled. | | **DAG pattern** | Optional allow/deny regex pattern for DAG IDs. Use this to limit discovery to selected DAGs. | | **Connection pattern** | Optional allow/deny regex pattern for Airflow connection IDs to include in lineage resolution. | | **Observe execution history** | Collect bounded DAG run and task instance events for metrics and operator lineage. Default: enabled. | | **Execution history days** | Lookback window for execution history. Range: 1–60 days. Default: 30. | | **Observe datasets** | Collect Airflow datasets or Airflow 3 assets when the endpoint is available. Default: enabled. | | **Resolve warehouse lineage** | Collect raw connection and operator evidence for warehouse lineage resolution. Default: enabled. | | **Connection mapping** | Optional manual mapping from Airflow connection IDs to Euno resource URIs. | Large Airflow deployments are protected by internal safety limits; if a limit is reached, the run report explains what stopped or was truncated. ### Step 3: Schedule Enable the **Schedule** option and choose how often Euno crawls the Airflow source: * **Hourly**: Set the interval in hours (e.g., every 4 hours). Recommended for active pipelines where DAG and task metadata changes frequently. * **Weekly**: Set specific days and times for a lighter crawl cadence. {% hint style="info" %} **Recommended**: Schedule the Airflow integration to run every 1–4 hours if you have execution history collection enabled. This keeps DAG run metrics and task lineage close to real time. Manual runs are also supported at any time. {% endhint %} ### Step 4: Resource Cleanup Configure the cleanup policy to control how Euno handles resources that disappear from Airflow: * **Immediate Cleanup**: Resources not detected in the most recent successful run are removed immediately. Use this to keep the catalog tightly in sync with your Airflow deployment. * **TTL-based Cleanup**: Resources are retained for a configurable number of days after they were last seen, then removed. Useful when DAGs are temporarily disabled or when Airflow is redeployed. * **No Cleanup**: Resources are retained indefinitely even if no longer detected by Airflow. {% hint style="info" %} **Recommended**: Use **Immediate Cleanup** for most deployments. This ensures that retired or deleted DAGs and tasks are promptly removed from the catalog. {% endhint %} ## What Euno Discovers * Airflow instances * DAGs * Tasks * Datasets and assets Airflow dataset and asset payloads are kept as raw source evidence. When a dataset or asset URI is a supported warehouse table URI, Euno also emits non-authoritative database, schema, and table observations from that explicit Airflow metadata. Global processors then derive the DAG and task lineage from the same evidence. When Airflow defines an output table, Euno also applies the defining DAG or task upstream table lineage to that output table. When the lineage processor reprocesses a previous Airflow-defined output and sees that Airflow no longer defines it, stale Airflow-owned output lineage is cleared. Operator SQL lineage is more heuristic. When execution history and connection metadata provide a complete, supported input or output target, Euno emits non-authoritative warehouse database, schema, and table observations for that SQL or operator table. Temporary tables created inside the SQL are skipped as warehouse observations. MySQL and Postgres operator lineage still enriches only warehouse tables that already exist in Euno. When a task loads data into Salesforce via the `SalesforceBulkOperator`, Euno records that the task (and its DAG) defines the target Salesforce object, so the object shows the Airflow task as upstream. The target object name from the operator is matched, case-insensitively, against Salesforce objects already crawled from your connected Salesforce org; if the same object name exists in more than one Salesforce org the link is skipped rather than guessed, and reported as diagnostic evidence. Any warehouse tables the loading task itself reads are surfaced on the Salesforce object as upstream tables. This requires the Salesforce integration to be connected and crawled; if Salesforce is crawled after Airflow, the link is resolved on the next Airflow lineage run. When execution history collection is enabled, Euno stores Airflow DAG runs and task instances as DAMA events: * `airflow_dag_run` stores the raw Airflow DAG run payload on the DAG URI. * `airflow_task_instance` stores the raw Airflow task instance payload on the task URI. Euno derives DAG and task execution metrics from these events. These include latest run status, failing task IDs, run counts, success rates, duration averages, retry rates, and failure streaks. Operator lineage processors also use task instance events when rendered operator fields are needed. Current operator lineage heuristics cover generic SQL operators, Snowflake operators, `MySqlOperator`, `PostgresOperator`, `DatabricksSubmitRunOperator`, `S3ToSnowflakeOperator`, `BigQueryInsertJobOperator`, `BigQueryExecuteQueryOperator`, and `SalesforceBulkOperator` (for Airflow-to-Salesforce lineage). Unsupported operators are reported separately rather than guessed. SQL operator lineage uses Airflow connection metadata or Connection mapping to resolve warehouse URI namespaces. If Airflow cannot read connections, or if an input or output target is partial or ambiguous, Euno does not create warehouse observations from that SQL evidence. MySQL, Postgres, and Databricks need a single matching mapped warehouse prefix when rendered fields omit `conn_id`. Databricks submit-run lineage reads only SQL task `query_text` from rendered JSON. Missing rendered SQL is reported as diagnostic evidence and is separate from final unresolved lineage when another task instance provides resolvable SQL. For the full list of discovered resource types, properties, and relationships, see [Airflow Integration Discovered Resources](/sources/transformation-etl/airflow-integration/airflow-integration-discovered-resources). # Airflow Integration Discovered Resources ### Ingested Resources & Properties #### Airflow Instance (`airflow_instance`) The top-level resource representing a single Airflow deployment. All DAGs and datasets discovered from one webserver are children of this instance. | Property | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `name` | Hostname extracted from the Airflow webserver URL | | `type` | Always `airflow_instance` | | `native_id` | Same as `name`; the hostname used as the stable identifier | | `subtype` | Always `airflow_instance` | | `airflow_base_url` | Canonical Airflow webserver URL | | `airflow_api_version` | Detected REST API version (`v1` for Airflow 2, `v2` for Airflow 3) | | `airflow_version` | Airflow version string reported by the health endpoint | | `airflow_connections` | List of Airflow connections collected for lineage resolution (only present when **Resolve warehouse lineage** is enabled) | | `is_active` | `true` when all Airflow health components report a healthy status | | `external_links` | Link to the Airflow UI | #### DAG (`airflow_dag`) An Airflow DAG observed from the Airflow REST API. | Property | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `name` | DAG display name, or the DAG ID when no display name is set | | `type` | Always `airflow_dag` | | `native_id` | DAG ID | | `subtype` | Always `airflow_dag` | | `parent_container` | Parent **`airflow_instance`** URI | | `is_active` | `true` when the Airflow scheduler still sees the DAG file; independent of `is_paused`. For Airflow 3, this is derived from `is_stale` | | `description` | DAG description (when available) | | `source_path` | Relative or absolute path to the DAG file | | `tags` | List of DAG tags | | `native_owners` | List of DAG owners; each entry contains `name`, `display_name`, and `email` | | `external_links` | Link to the DAG in the Airflow UI | | `airflow_last_run_pass` | `true` if the most recent DAG run succeeded, `false` if it failed, `null` if unknown | | `airflow_last_run_id` | Run ID of the most recent DAG run | | `airflow_last_run_duration` | Duration of the most recent DAG run in seconds | | `airflow_last_run_failing_tasks` | List of task IDs that failed in the most recent run | | `airflow_last_successful_run` | Timestamp of the most recent successful run | | `airflow_failure_streak` | Number of consecutive failed runs (resets to 0 on success) | | `airflow_total_runs_14d` | Total DAG runs in the last 14 days | | `airflow_total_runs_30d` | Total DAG runs in the last 30 days | | `airflow_total_runs_60d` | Total DAG runs in the last 60 days | | `airflow_success_rate_14d` | Fraction of successful runs (0–1) over the last 14 days | | `airflow_success_rate_30d` | Fraction of successful runs (0–1) over the last 30 days | | `airflow_success_rate_60d` | Fraction of successful runs (0–1) over the last 60 days | | `airflow_avg_duration_14d` | Average DAG run duration in seconds over the last 14 days | | `airflow_avg_duration_30d` | Average DAG run duration in seconds over the last 30 days | | `airflow_avg_duration_60d` | Average DAG run duration in seconds over the last 60 days | {% hint style="info" %} Execution metrics (`airflow_last_run_*`, `airflow_total_runs_*`, `airflow_success_rate_*`, `airflow_avg_duration_*`, `airflow_failure_streak`) are derived from stored DAG run events and require **Observe execution history** to be enabled. {% endhint %} #### Task (`airflow_task`) An individual task within an Airflow DAG. | Property | Description | | ------------------------------- | --------------------------------------------------------------------------------- | | `name` | Task display name, or the task ID when no display name is set | | `type` | Always `airflow_task` | | `native_id` | Task ID | | `subtype` | Operator class name (e.g. `BashOperator`, `PythonOperator`, `SnowflakeOperator`) | | `parent_container` | Parent **`airflow_dag`** URI | | `parent_container_name` | Display name of the parent DAG | | `airflow_task_avg_duration_14d` | Average task instance duration in seconds over the last 14 days | | `airflow_task_avg_duration_30d` | Average task instance duration in seconds over the last 30 days | | `airflow_task_retry_rate_14d` | Fraction of task instances that required at least one retry over the last 14 days | | `airflow_task_retry_rate_30d` | Fraction of task instances that required at least one retry over the last 30 days | {% hint style="info" %} Task execution metrics (`airflow_task_avg_duration_*`, `airflow_task_retry_rate_*`) are derived from stored task instance events and require **Observe execution history** to be enabled. {% endhint %} #### Dataset / Asset (`airflow_dataset`) An Airflow dataset (Airflow 2) or asset (Airflow 3) observed from the Airflow REST API. Dataset and asset payloads are kept as raw source evidence for lineage resolution. | Property | Description | | ------------------ | ------------------------------------------------------------------------------------------------------------------ | | `name` | Dataset name, or the URI string when no explicit name is set | | `type` | Always `airflow_dataset` | | `native_id` | Dataset URI (Airflow 2) or asset URI (Airflow 3) | | `subtype` | `dataset` for Airflow 2 resources, `asset` for Airflow 3 resources, or the dataset group name when one is assigned | | `parent_container` | Parent **`airflow_instance`** URI | {% hint style="info" %} `airflow_dataset` resources are only discovered when **Observe datasets** is enabled. {% endhint %} #### Database (`database`) β€” non-authoritative A warehouse database inferred from an Airflow dataset/asset URI or from a supported SQL/operator input or output target. These observations are non-authoritative and are enriched or overridden by the primary warehouse integration when one is configured. | Property | Description | | --------------------- | -------------------------------------------------------------- | | `name` | Database name extracted from the dataset URI | | `database_technology` | Warehouse technology (e.g. `snowflake`, `bigquery`) | | `platform_uri_prefix` | Warehouse platform prefix (e.g. `snowflake://`, `bigquery://`) | #### Schema (`database_schema`) β€” non-authoritative A warehouse schema inferred from an Airflow dataset/asset URI or from a supported SQL/operator input or output target. | Property | Description | | --------------------- | --------------------------------------------------- | | `name` | Schema name extracted from the dataset URI | | `database_database` | Database name used for warehouse hierarchy matching | | `database_technology` | Warehouse technology (e.g. `snowflake`, `bigquery`) | | `platform_uri_prefix` | Warehouse platform prefix | #### Table (`table`) β€” non-authoritative A warehouse table inferred from an Airflow dataset/asset URI or from a supported SQL/operator input or output target. Temporary tables created inside operator SQL are skipped. These observations are non-authoritative; final lineage and property resolution is handled by global processors. | Property | Description | | --------------------- | ------------------------------------------------------------------------------------------------------- | | `name` | Table name extracted from the dataset URI | | `database_database` | Database name | | `database_schema` | Schema name | | `database_technology` | Warehouse technology | | `defined_by` | Set to `airflow` when the table is currently defined by an Airflow DAG or task | | `platform_uri_prefix` | Warehouse platform prefix | | `table_dependencies` | Derived upstream warehouse tables inherited from the Airflow DAG or task that defines this output table | {% hint style="info" %} When the lineage global processor reprocesses a previous Airflow-defined output and sees that Airflow no longer defines it, stale Airflow-owned output lineage is cleared and final table dependencies are recomputed. {% endhint %} *** ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ------------------- | ------------ | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `airflow_instance` | has child | `airflow_dag` | DAGs are direct children of the Airflow instance | | `airflow_instance` | has child | `airflow_dataset` | Datasets and assets are direct children of the Airflow instance | | `airflow_dag` | has child | `airflow_task` | Tasks are grouped under their parent DAG | | `airflow_dag` | has upstream | `table`, `airflow_dataset` | Tables and datasets the DAG reads from; derived by the lineage global processor from dataset metadata and operator SQL heuristics | | `airflow_dag` | defines | `table`, `airflow_dataset`, `salesforce_object` | Tables, datasets, and Salesforce objects the DAG writes to; derived by the lineage global processor | | `airflow_task` | has upstream | `airflow_task` | Upstream tasks within the same DAG, derived from Airflow `downstream_task_ids` metadata (`upstream_task_ids`) | | `airflow_task` | has upstream | `table`, `airflow_dataset` | Tables and datasets the task reads from; derived by the lineage global processor | | `airflow_task` | defines | `table`, `airflow_dataset`, `salesforce_object` | Tables, datasets, and Salesforce objects the task writes to (e.g. via `SalesforceBulkOperator`); derived by the lineage global processor | | `airflow_dataset` | defines | `table` | Only when the dataset or asset URI maps to a supported warehouse table URI (e.g. Snowflake or BigQuery table path) | | `table` | has upstream | `table` | Airflow-defined output tables inherit upstream warehouse table lineage from the defining DAG or task | | `salesforce_object` | has upstream | `table` | Warehouse tables an Airflow loading task reads, surfaced on the Salesforce object it loads into (`airflow_upstream_tables`, merged into the object's upstream lineage) | | `database` | has child | `database_schema` | Non-authoritative warehouse hierarchy emitted from dataset URIs and supported SQL/operator targets | | `database_schema` | has child | `table` | Non-authoritative warehouse hierarchy emitted from dataset URIs and supported SQL/operator targets | # Azure Data Factory Euno's Azure Data Factory (ADF) integration auto-discovers your data factories and resolves the lineage of the tables your pipelines and data flows read and write. It discovers the following resources: * Data factories * Pipelines * Datasets * Mapping Data Flows * Power Query (Wrangling) Data Flows * The warehouse tables and columns your pipelines and data flows read and write {% hint style="info" %} Euno reads factory metadata through the Azure Resource Manager API only. It never modifies your factories, and it never runs pipelines. {% endhint %} {% hint style="info" %} This integration supports **Azure Data Factory** only. Azure Synapse Pipelines / Synapse workspaces are not supported. {% endhint %} ## Prerequisites To connect a source, you need: * A **Microsoft Entra (Azure AD) application** (service principal) with a **client secret**. You can enter the secret directly in Euno or select a Secret Handle served by a local agent. * The built-in **Reader** role at a scope that covers the factories you want Euno to discover β€” the subscription, a resource group, or an individual factory. A single assignment is enough; Euno lists factories and reads their pipelines, datasets, linked services, and data flows through that one grant. * One **Subscription ID** per Euno source (you cannot list multiple subscriptions on one source) * For the Secret Handle path, a connected local agent whose host has a managed identity with **Key Vault Secrets User** on the vault that contains the client secret ## Stage 1: Configure Microsoft Entra and Azure RBAC ### Step 1: Create an Entra app registration 1. Open [Microsoft Entra](https://entra.microsoft.com/) and sign in. 2. In the sidebar, pick **App registrations**. 3. Click **New registration**. 4. Fill in the name (for example, "Euno ADF"). 5. Under **Supported account types**, choose **Single tenant only** (your directory name appears on the option, for example **Single tenant only - Contoso**). This is the single-tenant choice β€” older Entra UIs labeled it **Accounts in this organizational directory only**. 6. Leave **Redirect URI** empty. 7. Click **Register**. 8. On the Overview page, copy the **Application (client) ID** and the **Directory (tenant) ID**. You will need these when configuring the source in Euno. 9. In the sidebar, pick **Certificates & secrets**. 10. Under **Client secrets**, click **New client secret**. 11. Set an expiry and optional description, then click **Add**. Copy the secret **Value** immediately. It is shown only once. Enter it directly in Euno, or store it in Azure Key Vault for the local-agent Secret Handle path. ### Step 2: Grant RBAC to the service principal Assign the built-in **Reader** role to the app from Step 1 at a scope that covers the factories you want to crawl (subscription, resource group, or individual factory). Azure scopes nest like this: ``` Subscription └── Resource group (e.g. rg-analytics) └── Data Factory (e.g. adf-prod) β”œβ”€β”€ Pipelines β”œβ”€β”€ Datasets └── Data flows ``` Grant **Reader** at the **subscription** (simplest), a **resource group**, or a single **factory** (most least-privilege). A broader assignment covers everything under it. | Role | Scope | Purpose | | ---------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------- | | **Reader** | Subscription, resource group, **or** individual factory | List factories and read their pipelines, datasets, linked services, and data flows | {% hint style="info" %} Azure’s ADF-specific built-in role is **Data Factory Contributor** (includes write). Euno only needs read access β€” use **Reader**, not Contributor. There is no built-in β€œData Factory Reader” role. {% endhint %} #### Option A β€” Azure portal 1. Open the [Azure portal](https://portal.azure.com) and sign in. 2. Open the scope where you want to grant access: * **Subscription** β€” search for **Subscriptions**, then open the subscription that contains your factories; or * **Resource group** β€” open the resource group that contains the factories; or * **Factory** β€” open the specific Azure Data Factory resource. 3. In the left menu, select **Access control (IAM)**. 4. Click **Add** β†’ **Add role assignment**. 5. On the **Role** tab, search for and select **Reader**, then click **Next**. 6. On the **Members** tab, for **Assign access to**, select **User, group, or service principal**. 7. Click **Select members**, search for the app name from Step 1 (for example, "Euno ADF"), select it, then click **Select**. 8. Click **Review + assign**, confirm the assignment, then click **Review + assign** again. {% hint style="info" %} App registrations do not always appear in the member list until you search by name. If nothing shows up, type the exact app display name from Step 1. {% endhint %} Repeat at a narrower scope (for example, a single factory) only if you did not already grant **Reader** at a broader scope that includes that factory. #### Option B β€” Azure CLI ```bash # Reader at subscription scope (lists factories and can read factory children) az role assignment create \ --assignee \ --role "Reader" \ --scope /subscriptions/ # Or Reader at a single factory when you prefer least privilege az role assignment create \ --assignee \ --role "Reader" \ --scope /subscriptions//resourceGroups//providers/Microsoft.DataFactory/factories/ ``` {% hint style="warning" %} Rotate the client secret per your security policy; expired secrets cause crawl failures. If you enter the secret directly, update the source in Euno after rotation. If you use a Secret Handle, update the Key Vault secret instead. A handle without a pinned Key Vault version reads the latest value. {% endhint %} ### Azure identity and role summary | Identity | Required role | What it does | | ----------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | ADF service principal | **Reader** on the subscription, resource group, or factory | Authenticates to Microsoft Entra and reads ADF metadata through Azure Resource Manager. | | Agent host's managed identity | **Key Vault Secrets User** on the vault or client-secret entry | Retrieves the service principal's client secret from Key Vault. | The managed identity only retrieves the client secret. It does not replace the ADF service principal, authenticate the ADF crawl, or read factory metadata. ## Stage 2: Configure a new Azure Data Factory source in Euno ### Step 1: Access the Sources page 1. Go to the **Sources** page. 2. Click **Add New Source** and select **Azure Data Factory** from the list of supported platforms. ### Step 2: General configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------- | | Name\* | A name for your Azure Data Factory source (e.g., "Azure Data Factory") | | Tenant ID\* | Your Microsoft Entra tenant identifier | | Client ID\* | The Azure AD application (service principal) client identifier | | Client Secret\* | The application secret, entered directly or selected from a local-agent Secret Handle. | | Subscription ID\* | The Azure subscription ID that contains the factories to crawl. Each source crawls **one** subscription only. | {% hint style="info" %} **Multiple Azure subscriptions?** Create a separate Azure Data Factory source for each subscription. You can reuse the same Entra app registration (tenant ID, client ID, and client secret) if that app has **Reader** on every subscription you configure. {% endhint %} ### Step 3: Advanced configuration (optional) In the UI, the two allow/deny blocks are each titled **Pattern** (distinguished by their subtitles), and runtime parameter overrides appear under **JSON Mapping**. | Configuration | Description | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Local Agent | Select a local agent to route Microsoft Entra token and Azure Resource Manager requests through its network. | | Client Secret | Select the Secret Handle that resolves to the ADF service principal's client secret. Manual entry in the general configuration remains available. | | Crawl Data Flows | Whether to discover Mapping Data Flows and Power Query (Wrangling) Data Flows and resolve their lineage. Enabled by default. | | Pattern (resource groups) | Allow/deny regular expressions on resource group names. `.*` matches everything; add a deny pattern to exclude specific resource groups. | | Pattern (factory names) | Allow/deny regular expressions on factory names within the filtered resource groups. | | Linked Service to Platform Instance | Manual mapping from a Linked Service to the warehouse location prefix Euno uses to match tables, when Azure does not provide enough information automatically (see below). | | JSON Mapping (runtime parameter overrides) | Per-pipeline overrides for runtime-only parameters whose defaults aren't representative, as a nested object: `{ "pipelineName": { "paramName": "value" } }`. Keys may be factory-scoped as `factoryName/pipelineName` when the same pipeline name exists in more than one factory. | #### Use a client-secret handle from Azure Key Vault Follow [Fetching credentials from your vault (secret handles)](/setup-configuration/account-settings/local-agents#fetching-credentials-from-your-vault-secret-handles) to grant the agent host's managed identity access to Key Vault, configure the canonical `/etc/euno/secrets.yaml` file, mount it, and restart the agent. An ADF entry can be as small as: ```yaml version: 1 secrets: - handle: adf-client-secret type: azure_kv directions: vault_uri: https://.vault.azure.net/ secret_name: ``` In Euno, select the agent under **Local Agent**, then select `adf-client-secret` under the advanced **Client Secret** field. The agent retrieves the value with its managed identity. Euno uses that value with the ADF service principal's tenant ID and client ID, and routes both the Microsoft Entra token request and all Azure Resource Manager metadata requests through the selected agent. To keep using an inline secret, leave **Local Agent** unselected and enter the value in the general **Client Secret** field. You can also select an agent while keeping an inline secret when you need agent routing but do not need Key Vault retrieval. {% hint style="info" %} **When is a Linked Service mapping needed?** For most warehouses, Euno can tell which database or project a dataset points at from the Linked Service alone. You only need a mapping when Azure does not give Euno enough information β€” most often **Azure Blob Storage** (the storage account name is hidden) or **Azure Databricks Delta Lake** (the dataset does not include the catalog). Enter only the missing location prefix; Euno adds the database, schema, and table from the dataset. Use the Linked Service name to apply the mapping across all factories, or `factoryName/linkedServiceName` when two factories share the same Linked Service name. {% endhint %} #### Supported Linked Services (where tables are found) | Linked Service type | Setup | Notes | | ----------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AzureSqlDatabase`, `AzureSqlMI` | Automatic | Azure SQL Database / Managed Instance | | `AzureSqlDW` | Automatic | Azure Synapse dedicated SQL pool | | `SnowflakeV2` | Automatic | Snowflake **V2** connector only | | `GoogleBigQueryV2` | Automatic | BigQuery **V2** connector only | | `AzureBlobFS` | Automatic | ADLS Gen2 (Blob FS) via the storage account URL | | `AzureBlobStorage` | **Mapping required** | Storage account name is hidden by Azure β€” see [Azure Blob Storage mapping](#azure-blob-storage-mapping) | | `AzureDatabricksDeltaLake` | **Mapping required** | Dataset does not carry the catalog; supply a prefix such as `databricks..` (use the workspace host the same way your Databricks source is identified in Euno) | | `Snowflake` / `GoogleBigQuery` (V1) | Not supported | Use the V2 connectors. A Linked Service mapping does not enable V1 β€” Euno only reads table locations from V2 dataset types | | `AzureDataLakeStoreGen2` | Not supported | Distinct from `AzureBlobFS` | #### Azure Blob Storage mapping When a Linked Service uses **Azure Blob Storage**, Azure hides the storage account name from Euno (and often shows it as `**********` in the ADF Linked Service editor). Without that name, Euno cannot connect ADF datasets on that Linked Service to the right place in your data model. Add a one-time mapping under **Linked Service to Platform Instance**. **Example** Your factory has a Linked Service named `SalesBlob` that points at the Azure storage account `contosoanalytics`. In Euno, add: | Field | Value | | -------------------- | -------------------------------- | | Linked Service name | `SalesBlob` | | Warehouse URI prefix | `azure_storage.contosoanalytics` | After you save and re-run the source, Euno can tie **table-style** datasets that use `SalesBlob` to that storage account (it adds database, schema, and table names from each dataset). **File** datasets on the same Linked Service (CSV, Parquet, JSON, and similar) still appear in Euno as ADF datasets, but Euno does not build table-level lineage for them. If the same Linked Service name exists in more than one factory, scope the key, for example `my-factory/SalesBlob`. **How to find the storage account name** Do not use the masked **Storage account name** field on an **Azure Blob Storage** Linked Service β€” Azure hides it there. Prefer one of these: **Option A β€” From an ADLS Gen2 Linked Service in Azure Data Factory Studio (often easiest)** If the factory also has an [Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage) Linked Service for the **same** storage account, the account name appears in the URL (it is not masked the way Blob Storage is). 1. In the [Azure portal](https://portal.azure.com), open [Data factories](https://portal.azure.com/#browse/Microsoft.DataFactory%2FdataFactories) and select your factory. 2. On the factory Overview page, select **Launch studio** to open [Azure Data Factory Studio](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory#create-a-data-factory-in-the-azure-portal). 3. In Studio, open the **Manage** hub (toolbox icon in the left rail) β†’ **Linked services**. See [Management hub](https://learn.microsoft.com/en-us/azure/data-factory/author-management-hub). 4. Open the **Azure Data Lake Storage Gen2** Linked Service (not the Blob Storage one). 5. Copy the account from the **URL** field. Example: `https://eunolake.dfs.core.windows.net` β†’ use `eunolake`. Only use this when Blob and ADLS Gen2 point at the same account. If they differ, use Option B for the Blob account. **Option B β€” From the Azure portal Storage accounts list** 1. Open [Storage accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts) (or search for **Storage accounts**). 2. Prefer accounts in the same subscription (and resource group, when possible) as your data factory. 3. Open a candidate account β†’ **Containers** (blob containers β€” not Tables) and confirm it holds the paths your ADF datasets use (for example a `raw` container). 4. Copy **Name** from the account Overview. Guide: [Get storage account configuration information](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-get-info). **Option C β€” From your own docs** Use the short account name your team already records for that Linked Service β€” not a connection string or key. Then enter `azure_storage.` followed by that name in the Euno mapping (as in the example above). For the URL in Option A, that would be `azure_storage.eunolake`. ### Step 4: Schedule Set how often Euno re-crawls the source to keep discovered resources and lineage current. ### Step 5: Resource cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource is removed if it is no longer detected by a source run. For a detailed explanation, see [Resource cleanup in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 6: Test and save the configuration Click **Test & Save**. Euno resolves the Secret Handle when one is selected, requests an Azure token with the ADF service principal, and lists the factories available through Azure Resource Manager before saving. If the selected agent is disconnected or the handle cannot be resolved, the test fails without falling back to a direct connection. ## What Euno discovers Once configured, the integration discovers, for each factory it can access: * **Factories, pipelines, datasets, and data flows** β€” the full inventory, with each pipeline and data flow linked to its parent factory. * **Table-level lineage** β€” for every table a pipeline or data flow writes, the source tables it reads. Euno resolves this from Copy activities, Script activities, Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded native SQL queries. * **Column-level lineage** β€” where the transformation names its columns (for example a Power Query rename, or a Copy activity with a column mapping), the source column each output column derives from. * **Which ADF resource writes a table** β€” the pipeline or data flow that produces a table is linked as its **definer**, so you can navigate from a table to the ADF resource that writes it. Warehouse tables and columns discovered from ADF are matched to the same tables from your warehouse integration (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse, Azure Storage), so ADF lineage appears alongside the rest of your graph. {% hint style="info" %} **Current limitations** * **Mapping Data Flows** β€” table-level lineage only (no column-level lineage). Flowlets are inventory only. * **Copy** β€” column lineage only when the activity has explicit `translator.mappings` **and** a single resolved source and single resolved sink (fan-in/out Copies keep table-level lineage only). Only mapped columns are emitted. * **Script** β€” table lineage from analyzed SQL when the Linked Service is a supported SQL warehouse. Column lineage only when the SQL names destination columns (for example `INSERT` with an explicit column list, or CTAS). Bare `INSERT … SELECT`, MERGE, and multi-statement scripts are limited or table-level only. * **Power Query (Wrangling)** β€” table and column lineage for supported M connectors (`Snowflake.Databases`, `GoogleBigQuery.Database`, `DatabricksMultiCloud.Catalogs`, `Sql.Database`), sources that combine those connectors with a Linked Service, and `Value.NativeQuery` SQL. Other M connectors may leave a sink without resolved sources. * **Stored Procedure** β€” ADF does not expose the procedure body, so Euno cannot resolve the tables it writes. No table or column lineage is produced. * **No run history** β€” design-time factory metadata only; pipeline run and activity execution metrics are not collected. * **Other activities** (Lookup, Databricks Notebook, Azure Function, Web, and similar) β€” discovered in the pipeline inventory only; Euno does not resolve lineage for them. The run report lists them under `tier_2_activities_skipped`. * Activities inside **ForEach**, **IfCondition**, **Until**, and **Switch** **are** walked for lineage and attributed to the outer pipeline. {% endhint %} For the full list of discovered resource types, their properties, and their relationships, see [Azure Data Factory Integration Discovered Resources](/sources/transformation-etl/azure-data-factory/azure-data-factory-discovered-resources). ## Understanding the run report After each crawl, the run report helps you interpret coverage and gaps: | Field | Meaning | | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `datasets_resolved` / `datasets_opaque` | How many datasets resolved to a warehouse table versus remained unresolved | | `unresolved_linked_service_types` | Linked Service types that blocked warehouse resolution | | `linked_service_types_seen` | Distribution of every Linked Service type observed (supported and not) | | `unsupported_dataset_types` | Non-tabular datasets (DelimitedText, Parquet, and similar) β€” expected for file-style datasets | | `mapping_dataflows_total` / `flowlets_total` / `power_queries_total` | Counts of Mapping Data Flows, Flowlets, and Power Query (Wrangling) Data Flows | | `dataflow_inline_endpoints_resolved` / `dataflow_inline_endpoints_unsupported` / `dataflow_inline_endpoints_unresolved` | Inline Mapping Data Flow source/sink endpoints (no Dataset reference) | | `tier_2_activities_skipped` | Activity types seen in pipelines for which Euno does not resolve lineage (Lookup, Azure Function, and similar) | | `tier1_activities_seen` / `tier1_activity_counts` | Lineage-supported activities encountered (Copy, Script, Execute Pipeline, Execute Data Flow, Execute Wrangling Data Flow, Stored Procedure) versus those that produced resolved lineage, by type | | `unresolved_parameter_references` | Design-time expressions that could not be evaluated (system variables, missing defaults, unsupported functions) | | `pipelines_with_lineage` / `pipelines_with_no_lineage` | Pipelines that produced at least one lineage outcome (Copy, Script, Execute\*, or Stored Procedure note) versus none | | `copy_activities_total` / `copy_activities_with_column_mappings` / `copy_columns_with_lineage` | Copy column-lineage coverage (activities seen, activities with mappings, columns emitted) | | `scripts_with_lineage` / `script_columns_with_lineage` / `scripts_without_destination` / `scripts_unsupported_linked_service` | Script activity SQL analysis outcomes | | `eunolizer_errors_total` | Script or Power Query native SQL that Euno could not analyze | | `power_query_source_edges` / `power_query_sinks_without_source` / `power_query_columns_with_lineage` | Power Query table and column lineage coverage | {% hint style="info" %} Mapping Data Flow lineage outcomes (`dataflows_with_lineage`, `dataflows_with_no_lineage`, `unresolved_dataflow_dataset_references`) are computed after the crawl completes, so they appear on a separate run report rather than the crawler report above. {% endhint %} High `datasets_opaque` with Blob Storage or Databricks Delta Lake types usually means adding a **Linked Service to Platform Instance** mapping. High `tier_2_activities_skipped` means those activity types are inventory-only today. ## Troubleshooting | Symptom | Check | | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Test & Save** reports that the agent is disconnected | Start the selected agent and confirm that its status is **Connected** under **Account Settings β†’ Local Agents**. ADF does not bypass a selected agent when it is unavailable. | | Secret resolution returns `provider_auth_failed` or `provider_denied` | Check that the agent host's managed identity has **Key Vault Secrets User** and that the configured secret name exists. This role belongs to the managed identity, not the ADF service principal. | | Microsoft Entra authentication fails | Check the tenant ID, client ID, and current service-principal client secret. For a Secret Handle, confirm that the Key Vault value matches the active client secret. | | A crawl succeeds but discovers no factories | Confirm that the ADF service principal has **Reader** on a factory, its resource group, or the subscription, and that the **Subscription ID** contains that factory. Azure can return an empty list when no factories are visible. New role assignments can take a few minutes to take effect. | # Azure Data Factory Discovered Resources ## Overview The Azure Data Factory integration discovers the following resources: * Data Factory (`adf_factory`) * Pipeline (`adf_pipeline`) * Dataset (`adf_dataset`) * Mapping Data Flow (`adf_dataflow`) * Power Query / Wrangling Data Flow (`adf_power_query`) * Table (`table`) and Column (`column`) β€” the warehouse resources ADF reads and writes ### Ingested Resources & Properties #### Data Factory (`adf_factory`) The top-level resource representing a single Azure Data Factory. All pipelines, datasets, and data flows discovered from one factory are children of it. | Property | Description | | ----------- | ----------------------------------------------- | | `name` | The data factory name | | `type` | Always `adf_factory` | | `native_id` | The factory's Azure Resource Manager identifier | #### Pipeline (`adf_pipeline`) An ADF pipeline (an orchestration of activities such as Copy, Data Flow, Execute Pipeline, and Stored Procedure). | Property | Description | | ------------------ | ------------------------------------------------ | | `name` | The pipeline name | | `type` | Always `adf_pipeline` | | `native_id` | The pipeline's Azure Resource Manager identifier | | `parent_container` | Parent **`adf_factory`** URI | #### Dataset (`adf_dataset`) An ADF dataset β€” a named pointer to a warehouse table or file. | Property | Description | | ------------------ | -------------------------------------------------------------------------------------------------- | | `name` | The dataset name | | `type` | Always `adf_dataset` | | `native_id` | The dataset's Azure Resource Manager identifier | | `subtype` | The ADF dataset type (e.g. `AzureSqlTable`, `SnowflakeV2Table`, `AzureDatabricksDeltaLakeDataset`) | | `parent_container` | Parent **`adf_factory`** URI | #### Mapping Data Flow (`adf_dataflow`) An ADF Mapping Data Flow or Flowlet (`subtype`). A Mapping Data Flow resolves the tables its sinks write; Flowlets are inventory only. | Property | Description | | ------------------ | ------------------------------------------------- | | `name` | The data flow name | | `type` | Always `adf_dataflow` | | `subtype` | `MappingDataFlow` or `Flowlet` | | `native_id` | The data flow's Azure Resource Manager identifier | | `parent_container` | Parent **`adf_factory`** URI | #### Power Query / Wrangling Data Flow (`adf_power_query`) An ADF Power Query (Wrangling Data Flow), whose transformations are authored in Power Query M. Euno parses the M to resolve the warehouse tables and columns its queries read and write, including embedded native SQL queries. | Property | Description | | ------------------ | ------------------------------------------------- | | `name` | The Power Query data flow name | | `type` | Always `adf_power_query` | | `subtype` | Always `WranglingDataFlow` | | `native_id` | The data flow's Azure Resource Manager identifier | | `parent_container` | Parent **`adf_factory`** URI | #### Table (`table`) A warehouse table referenced by an ADF dataset, data flow, or activity. When the source resolves to a supported warehouse (Snowflake, BigQuery, Databricks, Azure SQL, Azure Synapse, Azure Storage), Euno matches it to the same table discovered by your warehouse integration. | Property | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | `name` | The table name | | `type` | Always `table` | | `database_database` | The database the table belongs to | | `database_schema` | The schema the table belongs to | | `database_technology` | The warehouse technology (e.g. `snowflake`, `bigquery`, `databricks`, `mssql`, `azure_synapse`, `azure_storage`) | | `defined_by` | Set to `adf` when the table is currently written (defined) by an ADF pipeline or data flow | #### Column (`column`) A warehouse column discovered to carry column-level lineage. `parent_container` links it to its table. | Property | Description | | ------------------ | ---------------------- | | `name` | The column name | | `type` | Always `column` | | `parent_container` | Parent **`table`** URI | *** ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ----------------- | -------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `adf_factory` | has child | `adf_pipeline`, `adf_dataset`, `adf_dataflow`, `adf_power_query` | Pipelines, datasets, and data flows are direct children of their factory | | `adf_pipeline` | defines | `table` | Tables the pipeline's Copy and Script activities write | | `adf_pipeline` | has upstream | `adf_pipeline`, `adf_dataflow`, `adf_power_query` | The pipelines and data flows the pipeline executes (Execute Pipeline, Execute Data Flow, Execute Wrangling Data Flow) | | `adf_dataflow` | defines | `table` | Sink tables a Mapping Data Flow writes (Flowlets do not resolve lineage) | | `adf_power_query` | defines | `table` | Sink tables a Power Query (Wrangling) Data Flow writes | | `column` | has parent | `table` | Columns are grouped under their table | | `table` | has definer | `adf_pipeline`, `adf_dataflow`, `adf_power_query` | The pipeline or data flow that writes the table (`defined_by = adf`) | | `table` | has upstream | `table` | The source tables an ADF-written table reads, resolved from Copy, Script, Mapping Data Flow, and Power Query lineage (including native SQL) | | `column` | has upstream\_fields | `column` | Source columns a written column derives from when named by the transformation (Copy `translator.mappings` on single-source/single-sink Copies; Script SQL where destination columns are named; Power Query M-named columns and native-query columns) | ### Lineage Euno resolves ADF lineage at two levels so ADF-written tables and columns appear alongside the rest of your lineage: * **Table level** β€” for each table an ADF pipeline or data flow writes, Euno records the source tables it reads (`has upstream`) and the pipeline or data flow that produces it (`has definer`). Sources are resolved from Copy activities, Script activities (SQL analysis), Mapping Data Flows, and Power Query (Wrangling) Data Flows, including embedded `Value.NativeQuery` SQL. A transform that joins or appends several sources attributes the written table to all of them. * **Column level** β€” where the transformation names its output columns, Euno records the source column each output column derives from (`has upstream_fields`). Because warehouse tables and columns from ADF are matched to the same resources from your warehouse integration, a single table can show both its ADF-resolved lineage and any lineage from your warehouse or other tools. # Business Intelligence Connect your BI tools to understand dashboard usage, identify optimization opportunities, and track data consumption. ## Supported BI Platforms * [**Tableau**](/sources/business-intelligence/tableau-integration) - Visual analytics platform * [**Looker**](/sources/business-intelligence/looker-integration) - Business intelligence platform * [**Power BI**](/sources/business-intelligence/powerbi-integration) - Microsoft's BI platform * [**Thoughtspot**](/sources/business-intelligence/thoughtspot-integration) - Search-driven analytics * [**Sigma**](/sources/business-intelligence/sigma-integration) - Spreadsheet-based analytics * [**Sigma Impressions**](/sources/business-intelligence/sigma-impressions-integration) - Usage and impression data from Sigma CSV exports * [**Omni**](/sources/business-intelligence/omni-integration) - Analytics and BI platform * [**Cube Core**](/sources/business-intelligence/cube-core-integration) - Self-hosted Cube semantic models * [**Salesforce**](/sources/business-intelligence/salesforce-integration) - CRM with built-in reports and dashboards ## What You'll Get When you connect a BI tool, Euno discovers: * **Dashboards & Reports** - All dashboards, workbooks, and reports * **Data Sources** - Connections to underlying data * **Usage Metrics** - View counts, user engagement, last accessed * **Lineage** - Which tables and models feed each dashboard * **Fields & Measures** - Column-level usage in visualizations ## Common Use Cases * **Usage Optimization** - Identify unused dashboards for cleanup * **Cost Management** - Optimize refresh schedules based on usage * **Impact Analysis** - Understand what breaks when upstream data changes * **Adoption Tracking** - Measure which dashboards are actually used ## Getting Started 1. Choose your BI platform from the list above 2. Follow the setup guide for your platform 3. Grant Euno API access and enable usage tracking 4. Run your first sync **Setup Time:** 20-25 minutes per platform *** ## Next Steps * [View All Integrations](/sources) * [Learn About Usage Data](/using-euno/usage-optimization) * [Set Up Data Model Sync](/using-euno/metadata-activation/data-application-sync) (for Looker) # Tableau Euno integrates with Tableau, ingesting its metadata to enhance data observability and support proactive measures such as shift-left practices. By providing valuable insights into the data structure and usage patterns within your Tableau environment, Euno enables more informed decision-making and efficient data management. Euno's Tableau integration supports auto-discovery of the following Tableau resources and sub-resources: * Tableau workbook * Tableau data sources * Published data source * Embedded data source * Tableau Custom SQL * Tableau Views * Tableau Dashboards * Tableau Fields * Measures * Dimensions * Tables * Users ## Tableau Requirements To ingest metadata from Tableau, you will need: 1. **Tableau Server Version 2021.1.10 and newer.** 2. **Tableau Metadata API** (if already enabled please skip to 3): 1. Ensure the Tableau Metadata API is enabled. 2. This is enabled by default on Tableau Cloud. 3. For Tableau Server, follow [this guide ](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html#enable-the-tableau-metadata-api-for-tableau-server)by Tableau. 3. **Tableau Credentials:** Euno authenticates as a Tableau individual user. The Euno user would require a [Personal Access Token](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) with the [Site Administrator Explorer](https://help.tableau.com/current/server/en-us/users_site_roles.htm#general-capabilities-allowed-with-each-site-role) role or higher - an already existing user with same or higher permissions can be used. ## Setting up Euno's Tableau Integration ### Stage 1: Create Tableau Personal Access Token In Tableau, generate a new personal access token with a [Personal Access Token](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm) (PAT) with the [Site Administrator Explorer](https://help.tableau.com/current/server/en-us/users_site_roles.htm#general-capabilities-allowed-with-each-site-role). Once the token has been created, copy the secret. #### ### Stage 2: Configure New Tableau Source in Euno #### Step 1: Access the Sources Page 1. Open the **Sources** page. 2. Select Add New Source and choose **Tableau** from the available options. #### Step 2: General Configuration
ConfigurationDescription
Name*Enter a name for your Tableau source (e.g., "Tableau - Financial Reports").
Connection URL*Provide the Tableau server URL
Site*Specify the Tableau site name
Token Name*Enter the name of your Tableau access token.
Token Value*Input the secret value of your token.
#### Step 3: Scheduling Updates 1. Enable the Schedule option. 2. Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). #### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. #### Step 5: Advanced Settings (Optional) Click on the '**Advanced**' section to display these additional configurations. | Configuration | Description | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Override URL for Link | Add a custom URL if required. | | Skip SSL Verification | Enable this option to skip SSL checks | | Observe Tableau Pulse | Check this to have Euno observe Tableau Pulse Metrics and Metrics Definitions | | Observe Impression data | Check this to have Euno observe impressions data for Tableau. See [this link](/sources/business-intelligence/tableau-integration/tableau-cloud-impression-data-setup) | | Project pattern | Use a regular expression to allow or exclude specific projects, for example, ".\*" will include/exclude all projects. | #### Step 6: Save Configuration Click the **Save** button to complete the setup. ## What's next It is recommended to setup observing Tableau Impression data. Follow this guide [tableau-cloud-impression-data-setup.md](/sources/business-intelligence/tableau-integration/tableau-cloud-impression-data-setup). # Tableau Integration Discovered Resources ## Tableau Integration Discovered Resources The Tableau integration discovers the following resources: * Tableau workbooks * Tableau data sources * Published * Embedded * Tableau custom sql * Tableau view * Tableau dashboards * Tableau Fields * Fields of type measure * Fields of type dimension * Table #### Tableau Workbooks | Property | Value | | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | | name | The name of the workbook. | | description | Stores the description of the workbook from the native application. | | native\_id | The ID of the folder in Tableau's API | | parent | The parent folder of this folder. Null if the folder has no parent folder. | | created at | Stores the date the resource was created within the native application. | | created by | Stores the email of the user who created the resource within the native application. | | tableau\_extract\_refresh\_frequency | The scheduled refresh frequency when there are extracts (\*see Limitations section regarding suspended extracts). | #### Tableau Custom SQL Although custom SQL in Tableau is part of the data source, Euno observes it as an individual resource with both upstream and downstream connections to the data source it belongs to. | Property | Value | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | The name of the resource within the native application. | | type | `tableau_custom_sql` | | parent\_container | The data source the custom sql is found within. | | is\_identity\_transformation | If the custom SQL is a basic `SELECT *` statement, the property is set to `true`; otherwise, it is set to `false`. | | total\_impressions\_14d/30d/60d |

The number of impressions over the last 14/30/60 days. This data is aggregated from the impressions of fields within the Custom SQL, ensuring counts are not duplicated when multiple fields are viewed together.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| #### Tableau Data Sources | Property | Value | Applicable to | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | name | The name of the resource within the native application. | published data source, embedded data source | | description | Stores the description of the resource from the native application. | published data source, embedded data source | | owner | Stores the users email who is designated the owner of the resource within the native application | published data source, embedded data source | | type | `tableau_published_datasource`, `tableau_embedded_datasource` | tableau\_published\_data, tableau\_embedded\_datasource, tableau\_views, tableau\_dashboards | | native\_id | The ID of the resource within Tableau | published data source, embedded data source | | browse\_path | Stores the entire path of the resource of where the resource is located in the source, such as folders within folders. | published data source, embedded data source | | parent\_container | For an embedded data source, the parent is the Tableau workbook; for a published data source, the parent is the Tableau project. | published data source, embedded data source | | tableau\_project | The property stores the project information for the tableau project the resource is directly found in. | published data source, embedded data source | | tableau\_has\_extracts | The property stores whether the resource has an extract or not. | published data source, embedded data source | | tableau\_extract\_refresh\_frequency | The scheduled refresh frequency when there are extracts (\*see Limitations section regarding suspended extracts). | published data source | | total\_queries\_14d/30d/60d | The total number of queries the resource found within over the last 14/30/60 days. This information is taken from the query log of snowflake (see [Snowflake](/sources/data-warehouses/snowflake-integration) integration). In addition, this information is only available for LIVE data sources, as extracts, including the reload of the extract, are not tagged by Tableau. | published data source, embedded data source | | distinct\_users\_14d/30d/60d |

The number of distinct users who triggered the query the resource is found within over the last 14/30/60 days. This information is taken from the query log of snowflake. Only available for LIVE data sources.

Note: Unlike the distinct-users property available in BI tools (see impressions) with native usage tracking, this distinct-user count from the warehouse does not provide a user breakdown.

| published data source, embedded data source | | total\_impressions\_14d/30d/60d | The number of impressions the resource is found within over the last 14/30/60 days. This information is taken from the Tableau impression data pulled directly from the application. Unlike usage this data is found for both LIVE and EXTRACTED data sources. | published data source, embedded data source | | distinct\_impressions\_users\_14d/30d/60d |

The number of distinct users who triggered the impressions of the resource over the last 14/30/60 days. This information is taken from the Tableau impression data. Available for both LIVE and EXTRACTED data sources.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| published data source, embedded data source | | tableau\_workbook | The property stores the tableau workbook the resource is found within. | embedded data source | | is\_proxy\_to\_published\_datasource | Identifies if the resource proxies an upstream published data source. | embedded data source | | native\_last\_data\_update | The timestamp indicating the most recent update to the data source. | published data source, embedded data source | | luid | The Tableau LUID (Locally Unique Identifier) | published data source, | #### Tableau Visualizations | Property | Value | Applicable to | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | name | The name of the resource within the native application | views, dashboards | | description | Stores the description of the resource from the native application. | views, dashboards | | owner | Stores the users email who is designated the owner of the resource within the native application | views, dashboards | | type | `tableau_view`, `tableau_dashboard` | views, dashboards | | native\_id | The ID of the resource within Tableau | views, dashboards | | browse\_path | Stores the entire path of the resource of where the resource is located in the source. | views, dashboards | | parent\_container | For views and dashboards, the parent is the Tableau workbook. | views, dashboards | | tableau\_project | The property stores the project information for the tableau project the resource is directly found in. | views, dashboards | | total\_queries\_14d/30d/60d | The total number of queries over the last 14/30/60 days. Only available for LIVE data sources. | views, dashboards | | distinct\_users\_14d/30d/60d |

The number of distinct users who triggered queries over the last 14/30/60 days. Only available for LIVE data sources.

Note: Unlike the distinct-users property available in BI tools (see impressions) with native usage tracking, this distinct-user count from the warehouse does not provide a user breakdown.

| views, dashboards | | total\_impressions\_14d/30d/60d | The number of impressions over the last 14/30/60 days. Available for both LIVE and EXTRACTED data sources. | views, dashboards | | distinct\_impressions\_users\_14d/30d/60d |

The number of distinct users who triggered impressions over the last 14/30/60 days. Available for both LIVE and EXTRACTED data sources.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| views, dashboards | | tableau\_workbook | The property stores the tableau workbook the resource is found within. | views, dashboards | | hidden\_sheet | A boolean value of whether or not the sheet is hidden | views, dashboards | | luid | The Tableau LUID (Locally Unique Identifier) | dashboards, workbooks | #### Tableau Fields For the fields found within the above resources, Euno discovers the following properties. | Property | Value | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | The name of the field | | type | `tableau_field` | | parent\_container | The URI of the visualization or data source where the custom field is defined | | parent\_container\_name | The name of the visualization or data source where the custom field is defined | | parent\_type | The parent resource's type | | is\_calculated | If the field is a calculated field then `true` otherwise `false` | | is\_identity\_transformation | If the field is a calculated field then `false` otherwise `true` | | total\_queries\_14d/30d/60d | The total number of queries over the last 14/30/60 days. Only available for LIVE data sources. | | distinct\_users\_14d/30d/60d |

The number of distinct users who triggered queries over the last 14/30/60 days. Only available for LIVE data sources.

Note: Unlike the distinct-users property available in BI tools (see impressions) with native usage tracking, this distinct-user count from the warehouse does not provide a user breakdown.

| | total\_impressions\_14d/30d/60d | The number of impressions over the last 14/30/60 days. Available for both LIVE and EXTRACTED data sources. | | distinct\_impressions\_users\_14d/30d/60d |

The number of distinct users who triggered impressions over the last 14/30/60 days. Available for both LIVE and EXTRACTED data sources.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| | tableau\_workbook | The property stores the tableau workbook the resource is found within. | ## Tableau Prep Flows Tableau Prep Flows are data preparation workflows that can be used to clean and transform data before analysis. The Tableau integration discovers flows and their dependencies. | Property | Value | | ------------------- | ------------------------------------------------------------------------- | | name | The name of the flow | | description | The description of the flow | | created\_at | The timestamp when the flow was created | | updated\_at | The timestamp when the flow was last modified | | native\_owners | List of users who own the flow, containing their email addresses | | table\_dependencies | List of URIs representing the input tables used by the flow | | external\_links | Contains a link to open the flow in Tableau | | thumbnail | Base64 encoded PNG image of the flow diagram (if available and under 1MB) | #### Flow Dependencies The integration discovers both input and output dependencies of flows: **Input Tables** Input tables are discovered from LoadSql nodes in the flow. **Output Resources** Flows can output to: 1. Database Tables 2. Published Data Sources The integration maintains the lineage between flows and their inputs/outputs, allowing you to track data transformations through your Tableau Prep workflows. ## Tableau Pulse Metric Definitions Tableau Pulse Metric Definitions are high-level metric definitions discovered from Tableau Pulse. Each definition may have one or more associated metrics. | Property | Value | | --------------- | ----------------------------------------------------------------------------------- | | name | The name of the metric definition as defined in Tableau Pulse. | | description | The description of the metric definition. | | type | `tableau_metric_definition` | | native\_id | The ID of the metric definition in Tableau Pulse. | | external\_links | List of links to open the metric definition in Tableau Pulse and any related links. | | updated\_at | The timestamp when the metric definition was last modified. | | updated\_by | The username of the user who last modified the metric definition, if available. | **Lineage**: Each `metric definition` has an `upstream` relationship with the Published Datasource it depends on, and an `upstream_fields` relationship with each of the fields it is using in it. ## Tableau Pulse Metrics Tableau Pulse Metrics are the concrete metrics associated with a metric definition in Tableau Pulse. | Property | Value | | --------------------- | --------------------------------------------------------- | | name | The name of the metric (currently: 'metric based on '). | | type | `tableau_metric` | | native\_id | The ID of the metric in Tableau Pulse. | | native\_raw\_object | The full raw object as returned by the Tableau Pulse API. | | number\_of\_followers | The number of users following this metric | **Lineage**: Each `metric` has an `upstream` relationship with the metric definition it is based on ## Warehouse Tables and Columns #### Table The Tableau Integration will discover resources of type Table when it identifies a published data source, flow or Tableau extract referring to a database table. For example, if a data source is using a table from the data warehouse, which is not connected to dbt, the Tableau source will discover this table. To determine the exact location of the referred data source or extract, the Tableau Integration utilizes the project and data source properties specified in the Tableau metadata API. In the case of cloud-hosted extracts, the Tableau Integration may encounter limitations in handling credentials or region-specific configurations. For instance, API calls might lack sufficient detail to automatically distinguish between regions or accounts, requiring manual intervention to map these attributes correctly. In this case, Euno defaults to using the aws account id default\_aws\_account.us\_central1 . If you have multiple AWS accounts and you need to distinguish between them, set up manual "connection mapping". | Property | Value | | -------------------- | ---------------------------------------------------------------------- | | name | The name of the table. | | type | table | | database\_technology | The technology of the database, based on the dialect of the connection | | database | The name of the database | | schema | The name of the schema | #### Column The Tableau integration will discover resources of type `column` when it identifies a dataset referring to a database column. | Property | Description | | -------------------- | ---------------------------------------------------------------------- | | name | The name of the column. | | type | column | | database\_technology | The technology of the database, based on the dialect of the connection | | database | The name of the database | | schema | The name of the schema | ## Limitations ### Extract Refresh Status Due to Tableau API limitations, the `tableau_extract_refresh_frequency` property may not reflect correctly if an extract refresh has been suspended due to inactivity. Tableau automatically suspends extract refreshes for workbooks and data sources after a period of inactivity. When a schedule is suspended, Euno still considers it active and reflects the original frequency in this field. For more information about this Tableau feature, see the [Tableau documentation on automatic extract suspension](https://help.tableau.com/current/online/en-us/extract_auto_suspend.htm). # Tableau Impression Data Setup Euno can observe information about how Tableau Users are using the different data products in their Tableau Cloud Instance. This information is useful for data governance to identify popular dashboards, unused dashboards, dashboards used by specific teams, etc. Read more about impressions data in [tableau-usage-data.md](/using-euno/usage-optimization/tableau-usage-data) {% hint style="info" %} Tableau updates access information at regular intervals, which can cause delays in displaying the latest impression data. This means the data shown in Tableau might not always reflect real-time updates. See [Tableau documentation.](https://help.tableau.com/current/online/en-us/adminview_insights_manage.htm?utm_source=chatgpt.com#about-data-freshness) for more information. {% endhint %} To set up the Tableau Integration to observe impression data, Complete the following steps: 1. Find (Tableau Cloud users) or create (Tableau Server users) the TS Events datasource 2. Create and publish a Euno-compatible Tableau Access Log View 3. Provide Euno access to the published view ### Step 1 - TS Events Datasource **Tableau Cloud:** 1. Go to β€œExplore” 2. Under the built-in Tableau Project β€œAdmin Insights”, look for a datasource called β€œTS Events”. This is a built-in datasource that collects all the Tableau access events (or other events). **Tableau Server:** Follow the instructions [here](https://github.com/tableau/community-tableau-server-insights) to download and publish the TS Events datasource. ### Step 2 - Create and Publish a Euno-compatible Tableau Access Log View 1. Create a new Workbook from this datasource 2. Under β€œRows”, drag: 1. Actor User Id 2. Actor User Name 3. Event Type 4. Item Type 5. Item LUID 6. Event Date 3. The β€œEvent Date” field could already be set to β€œexact date”. If it's not, set it to β€œexact date”. Your TS event screen should look like this: 4. Filter only for β€œaccess” events: 1. Click β€œEvent Type”->Filter. Under β€œGeneral”, Select β€œAccess” \*ONLY\*. Euno is only interested in β€œAccess” events. 5. Filter only for β€œaccess” events from a tracked resource: 1. Click β€œItem LUID”->Filter. Under β€œGeneral”, deselect all options except β€œnull”, and check the box that says β€œExclude selected values” 6. Filter only for events from the last 60 days: 1. Click "Evnet Date"-> Filter. Select a "Relative Date" filter, and set it to past 60 days. 7. Hit OK -> The view is ready. 8. To see the csv contents, Click the β€œdownload” icon, and choose β€œdata”. Download the result. The csv should look like this: 9. At the top right corner of the screen, go to β€œPublish As”: 10. Name the workbook β€œevents for Euno”. Make sure to check the box β€œEmbed password for data source”. ### Step 3 - Provide Euno Access to the Published View 1. Find the published workbook in the Explore screen 2. Click β€œPermissions…” for the workbook 3. The Euno-Tableau integration is using a Personal Access Token. Make sure that the user has β€œDownload Full Data” permissions to this workbooks. 4. If the user is not there, add it ## Troubleshooting If impression data is not collected, open the integration run report and review **impressions parsing errors sample**. Missing-column errors identify the configured view and workbook and list both the required columns that are missing and the columns Euno received. Invalid-date errors show the configured date format and the value that could not be parsed. Correct the published access log view to match the fields and date setup in Step 2, then run the integration again. # Looker Euno's Looker integration supports auto-discovery of the following Looker resources and sub-resources: * Modeling layer resources: * [LookML Explores](https://cloud.google.com/looker/docs/lookml-terms-and-concepts#explore), [LookML Views](https://cloud.google.com/looker/docs/lookml-terms-and-concepts#view), [LookML Dimensions](https://cloud.google.com/looker/docs/reference/param-field-dimension), and [LookML Measures](https://cloud.google.com/looker/docs/reference/param-field-measure) * Application layer resources: * [Looker Tiles](https://cloud.google.com/looker/docs/viewing-dashboards#dashboard_tiles), [Looker Looks](https://cloud.google.com/looker/docs/viewing-looks) and [Looker Dashboards](https://cloud.google.com/looker/docs/viewing-dashboards) Setting up Euno's Looker integration consists of three steps: 1. Creating a [Looker API key](https://cloud.google.com/looker/docs/admin-panel-users-users?hl=en#api_keys) to be used by Euno to access Looker's API 2. Configuring Euno's Looker integration 3. Providing Euno Access to your LookML Git Repo ## Setting up Euno's Looker Integration ### Step 1: Creating a Looker API key to be used by Euno to access Looker's API Euno uses Looker API keys to access Looker's API and auto-discover Looker resources. We recommend creating a dedicated API Key for Euno's Looker integration.\ To ensure that the API Key has the required permissions , create the API Key for a user with the [Admin ](https://cloud.google.com/looker/docs/admin-panel-users-roles?hl=en#default_roles)role.\ For more granular control over the permissions provided to the API Key, Follow the steps in [Using a custom Looker Role](/sources/business-intelligence/looker-integration/using-a-custom-looker-role), and come back here once a dedicated Euno user is created. 1. Log in to Looker with a user with "Admin" role 2. Go to "admin" found in the left sidebar 3. Scroll down to find "Users" in the left sidebar 4. Click on a user to create the API key for. Make sure this user has the "Admin" role 5. Find "API Keys" and click the button "Edit Keys" located next to it 6. Click "New API Key". You should now see the new key's Client ID and Client Secret. Make note of the two values (or leave the browser tab open). You'll need both values to configure Euno's Looker integration in the next stage. ### Step 2: Configuring New Looker Source in Euno #### Step 1: Access the Sources Page 1. Go to the **Sources** page. 2. Click on **Add New Source** and select Looker from the list of supported platforms. #### Step 2: General Configuration An asterisk (\*) means a mandatory field.
ConfigurationDescription
Name*Enter a name for your Looker source (e.g., "Looker - Sales Data")
Host*Enter the Looker host URL (e.g., https://<your-looker-domain>.cloud.looker.com).
Client ID*Provide the Client ID from your Looker API credentials.
Client Secret*Enter the Client Secret from your Looker API credentials.
#### Step 3: Scheduling Updates 1. Enable the Schedule toggle. 2. Choose from the following: 1. **Weekly**: Specify the days and times for updates. 2. **Hourly**: Enter an interval in hours (e.g., every 6 hours). #### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. #### Step 5: Advanced Settings (Optional) Click on the '**Advanced**' section to display these additional configurations. | Configuration | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **LookML Project Includes** | Comma-separated list of LookML project names or wildcard patterns (for example, `project_a, project_b` or `mart*`). Leave blank or set to `*` to process all LookML projects. | | **LookML Model Includes** | Comma-separated list of LookML model names or wildcard patterns (for example, `model_a, model_b` or `sales*`). Leave blank or set to `*` to process all LookML models. | | **Folder filter** | Allow/deny regex patterns matched against canonical full Looker folder paths (for example, `/marketing/dev/` or `/marketing/dev/my dashboard`). Use `.*` to match all folders. | | **Crawl LookML Only** | Enable this if you want to crawl LookML only (no Looks, Tiles, or Dashboards). | | **Observe users** | Enabled by default. Observe Looker users and their metadata. Disable to skip user observation. | | **Observe groups** | Enabled by default. Observe Looker groups and their memberships. Disable to skip group observation. | #### Mapping Euno will ingest looker resources using the connections defined in the application unless a mapping is added. Each connection can be mapped to a target data source. This can be useful, for example, when Looker is connected to a copy of production database, and we want to trace lineage to the original instead of the copy. If mapping is configured, the resource will be ingested using the mapped value of the target.
ConfigurationDescription
connection_nameThe name of the connection as it appears in your Looker instance. This should match exactly what you see in Looker's connection settings, for example: snowflake_prod, bigquery_analytics, postgres_main
host

The target database server hostname to map to.

Examples:

  • Snowflake: mycompany.snowflakecomputing.com
  • BigQuery: bigquery.googleapis.com
  • PostgreSQL: db.example.com
  • Redshift: redshift-cluster.amazonaws.com
port (optional)

The target database port number. Defaults to 0 if not specified.

Examples:

  • PostgreSQL: 5432
  • MySQL: 3306
  • SQL Server: 1433
  • Snowflake: 443
database

The specific database name that contains your tables.

Examples:

  • Snowflake: ANALYTICS_DB, PROD_DW
  • BigQuery: my-project-id (the Google Cloud project ID)
  • PostgreSQL: production, analytics
dialectThe database type, i.e. Snowflake, BigQuery etc.
db_schema (optional)The default schema within the database. Leave empty if not applicable.
aws_account (optional)AWS account ID, required only for Athena connections.
#### Step 6: Save Configuration Click the **Save** button to complete the setup. {% hint style="warning" %} If the integration is not successful, refer to [Looker Integration Troubleshooting Guide](/sources/business-intelligence/looker-integration/troubleshooting-looker-integration) {% endhint %} ### Step 3: Providing Euno Access to your LookML Git Repo To finish configuring Looker in Euno, we need access to the Git repository that stores your LookML project. Please link your Git hosting account: * [Github](/setup-configuration/linked-accounts/github-integration)​ * [GitLab](/setup-configuration/linked-accounts/gitlab-integration)​ During the setup, make sure the linked account has permission to read the repositories that contain your LookML code (your LookML repos). {% hint style="info" %} If Euno is not provided access to your LookML Git repository, only visualization-layer assets (such as Looks, Tiles, and Dashboards) will be discovered. In this case, Euno will not be able to establish lineage or relationships to the underlying data layer. {% endhint %} ### Step 4: Users and Groups (optional) By default, **Observe users** and **Observe groups** are enabled in Advanced Settings. Euno observes Looker users and groups (with memberships) on each run unless you disable these options. For additional information, see [Users as a Resource](/developer-reference/technical-concepts/users-as-a-resource). ## What Euno Discovers Euno discovers LookML modeling resources (views, explores, dimensions, measures), application-layer content (folders, looks, tiles, dashboards), warehouse tables referenced in LookML, andβ€”when enabledβ€”users and groups. Usage metrics (queries, impressions, PDT builds) are attached to supported resource types. For the full resource and property reference, see [Looker Integration Discovered Resources](/sources/business-intelligence/looker-integration/looker-integration-discovered-resources). For usage signals and how to interpret them, see [Looker Usage Data](/using-euno/usage-optimization/looker-usage-data). # Using a custom Looker role Some teams prefer to use custom [Looker roles](https://cloud.google.com/looker/docs/admin-panel-users-roles) to configure Euno's Looker integration. Consider using a custom Looker Role for Euno's Looker integration to: 1. Further lock down the permissions of Euno's Looker integration 2. Avoid breaking Euno's Looker integration when configuring permissions for other Looker users ## Setting up a Custom Role for Euno's Looker Integration To set up a custom Looker Role for Euno's Looker integration complete the following: 1. Create a Looker Model Set 2. Create a Permission Set with the minimal permissions required 3. Create a Looker Role 4. Create a Looker User and associate it with the Looker Role ### Stage 1: Create a Looker Model Set 1. Browse to the "admin" section in your Looker instance and find the "roles" menu item

Looker: Admin Settings - Roles

2. Click "New Model Set"

Looker: New Model Set

3. Name the new Model Set `euno_model_set`. Do not check any of the LookML models. Euno's Looker integration does not require access to any LookML model. 4. Click "New Model Set" to save the new Model Set.

Looker: Save New Model Set

5. Once saved, you'll be redirected back to the "Roles" screen. Verify that the model set was created by scrolling down to the "Model Sets" table. You should see the new Model Set there.

Looker: Model Sets Table

### Stage 2: Create a Permission Set 1. In the "Roles" screen, click "New Permission Set"
2. Name the new Permission Set `euno_permission_set` and check the following permissions: * access\_data * manage\_models * see\_users * see\_system\_activity `see_users` is required when **Observe users** or **Observe groups** are enabled in Euno (they are enabled by default). `see_system_activity` is required for source validation and usage observation. 3. Click "New Permission Set" to save the new Permission Set

Looker: Save Permission Set

4. Verify that the new Permission Set was created in the Permission Sets table in the "roles" screen.

Looker: Permission Sets Table

### Stage 3: Create a Looker Role 1. Back in the "roles" screen, click "New Role" 2. Name the role `euno_role`, pick `euno_permission_set` and `euno_model_set`, and click "New Role"

Looker: Create a Looker Role

3. Verify that the role is created in the "Roles" tables

Looker: Roles

### Stage 4: Create a Looker User and assign it the Looker Role 1. Find the "Users" option in the "admin" sidebar

Looker: Admin Settings - Users

2. Click "Add Users"
3. Give the user an email address, assign the role `euno_role`, then click "Save"

Looker: Assigning Role to User

You're done. Go back to [stage 1 in Setting up Euno's Looker Integration](/sources/business-intelligence/looker-integration#stage-1-create-a-looker-api-key-to-be-used-by-euno-to-access-lookers-api) and follow the steps there. # Looker Integration Discovered Resources ## Overview The Looker integration discovers the following resources: * Looker Instance * Looker Model * LookML Explore * LookML View * Looker Dimension * Looker Measure * Looker Folder * Looker Look * Looker Tile * Looker Dashboard * Table * User (when **Observe users** is enabled) * User Group (when **Observe groups** is enabled) Note: We don't observe built-in looker models `i__looker` and `system__activity` ## Looker Instance The top-level container for a Looker deployment. | Property | Description | | -------------------- | ------------------------------------------------------------------------------------------------------ | | `name` | The Looker host URL for the instance. | | `type` | `looker_instance` | | `looker_connections` | Connection metadata discovered from the Looker API (used for table resolution and connection mapping). | ## Looker Model A LookML model defines the database connection and the explores it exposes. | Property | Description | | ------------------ | ----------------------------------------- | | `name` | The name of the LookML model. | | `type` | `looker_model` | | `parent_container` | The Looker instance containing the model. | ## Looker Folder ### Properties | Property | Description | | ------------------ | ----------------------------------------------------------- | | `name` | The name of the folder. | | `type` | `looker_folder` | | `native_id` | The ID of the folder in Looker's API. | | `parent_container` | The parent folder, or the Looker instance for root folders. | | `created_at` | The date the folder was created in Looker. | | `created_by` | The email of the user that created the folder. | ## LookML: Views & Explores | Property | Description | Applicable to | | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `name` | The title of the LookML file. | View, Explore | | `type` | `looker_view` or `looker_explore`. | View, Explore | | `description` | The resource description from LookML. | View, Explore | | `sql_dialect` | The SQL dialect of the query powering the visualization. | View, Explore | | `looker_project_name` | The name of the LookML Git project containing the model. | View, Explore | | `looker_model` | The name of the LookML model (display name; the parent resource type is `looker_model`). | View, Explore | | `native_id` | The native identifier for the view or explore in Looker. | View, Explore | | `parent_container` | The LookML model (`looker_model`) containing the view or explore. | View, Explore | | `is_derived` | Whether the view is derived. | View | | `derived_type` | The type of derived view: `sql_derived_view` or `native_derived_view`. Null for non-derived views. | View | | `has_refinements` | Whether the view includes refinements. | View | | `looker_refinement_chain` | File locations of refinements applied to the view. | View | | `pdt_builds_last_30d` | Total number of PDT builds in the last 30 days. | View | | `pdt_total_build_time_30d` | Cumulative build time for PDTs in the last 30 days. | View | | `lookml_view_persistency` | Persistency triggers for PDTs (`sql_trigger`, `datagroup_trigger`, or `interval_trigger`). | View | | `git_repo_url` | The URL of the associated Git repository. | View, Explore | | `git_repo_branch` | The branch of the Git repository being used. | View, Explore | | `total_queries_14d`, `total_queries_30d`, `total_queries_60d` | Query counts over the last 14/30/60 days from the Looker API (includes cached queries). | View, Explore | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users querying the resource over the last 14/30/60 days. User-level breakdown is available in the side panel and Euno Assistant, but not via EQL. | View, Explore | | `source_path` | Full file path (for example, `models/sales/orders.view.lkml`). | View, Explore | | `source_directory` | Directory path of the resource in your Git repository. | View, Explore | | `table_dependencies` | Upstream resources (tables, views, or explores) referenced by this resource. | View, Explore | ## LookML Refinements Euno discovers refinement files; however, unlike regular views, refinement files do not constitute resources themselves. Instead, Euno includes the refinement information in the view that it refines. For views with refinements, the property `has_refinements` will be set to `true`. Additionally, in the [About tab](/using-euno/data-model-screen#unri9kmp2usw-1) of the refined resource's [side panel](/using-euno/data-model-screen#xgdhrrj8g7h9), the property `looker_refinement_chain` is displayed, indicating the file locations of the refinement files that refine the resource. When viewing the code of the refined resource, it includes the dimensions and measures from the refinement file as if it were compiled. {% hint style="info" %} Currently Euno does not support the discovery of refinement files for LookML Explores. {% endhint %} ## Looker data visualizations: Looker Look, Looker Tile and Looker Dashboards The term "Looker Tile" is used to describe a visualization in a Looker Dashboard that is not a reference to a Looker Look. {% hint style="info" %} In Looker, a dashboard is made up of *dashboard elements*. Some *dashboard elements* are data visualizations, which can either reference a Looker Look, or a visualization that "lives" inside the dashboard. Looker documentation uses the terms *lookless* and *tile* to describe the latter type of visualization. Euno uses the more commonly used term *tile*. {% endhint %} | Property | Description | Applicable to | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | --------------------- | | `name` | The title of the visualization. | Look, Tile, Dashboard | | `type` | `looker_look`, `looker_tile`, or `looker_dashboard`. | Look, Tile, Dashboard | | `description` | The description of the resource. | Look, Dashboard | | `sql_dialect` | The SQL dialect of the query powering the visualization. | Look, Tile | | `looker_model` | The name of the LookML model powering the visualization. | Look, Tile | | `looker_project_name` | The name of the LookML project containing the model. | Look, Tile | | `looker_folder` | The name of the Looker folder the item is found within. | Look, Dashboard | | `created_at` | The date the resource was created in Looker. | Look, Tile, Dashboard | | `updated_at` | For tiles, includes updates to the tile or the dashboard containing it. | Look, Tile, Dashboard | | `native_id` | The ID of the look, tile, or dashboard in Looker. | Look, Tile, Dashboard | | `created_by` | The display name of the Looker user that created the Look. | Look | | `updated_by` | The display name of the Looker user that last updated the Look. | Look | | `owner` | The display name of the dashboard owner (derived from `native_owners`). | Dashboard | | `parent_container` | For Looks, the containing folder. For tiles, the containing dashboard. | Look, Tile | | `last_7d_views`, `last_30d_views` | Impression counts over the last 7 or 30 days. | Look, Dashboard | | `last_viewed_at` | The last impression time. | Look, Dashboard | | `total_queries_14d`, `total_queries_30d`, `total_queries_60d` | Query counts over the last 14/30/60 days from the Looker API (includes cached queries). | Look, Tile, Dashboard | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users over the last 14/30/60 days. User-level breakdown is available in the side panel and Euno Assistant, but not via EQL. | Look, Tile, Dashboard | | `table_dependencies` | Upstream explores (and related resources) referenced by the look or tile query. | Look, Tile | ## LookML and Looker Measures and Dimensions Euno observes fields in both LookML files (views and explores) and visualizations (looks and tiles). For visualizations, Looker may make use of [Custom Fields](https://cloud.google.com/looker/docs/custom-fields). Euno observes each custom field as a resource in the same manner as a LookML measure or dimension. A custom measure, including a filtered custom measure, is observed with the type `looker_measure`. A custom dimension is observed with the type `looker_dimension`. Looker dimension groups are also observed as dimensions with the type `looker_dimension`. | Property | Description | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `name` | The name of the Looker measure or dimension. | | `type` | `looker_measure` or `looker_dimension`. | | `parent_container` | The URI of the view, explore, look, or tile where the field is defined. | | `is_calculated` | `true` for custom fields defined in a visualization. | | `is_identity_transformation` | `true` when the field directly references an existing warehouse column; `false` for custom fields. | | `total_queries_14d`, `total_queries_30d`, `total_queries_60d` | Query counts over the last 14/30/60 days (includes cached queries). | | `distinct_users_14d`, `distinct_users_30d`, `distinct_users_60d` | Distinct users over the last 14/30/60 days. User-level breakdown is available in the side panel and Euno Assistant, but not via EQL. | | `tags` | Tags associated with the measure or dimension, as defined in LookML. | | `source_directory` | Directory path of the resource in your Git repository. | | `source_path` | Full file path (for example, `models/sales/orders.view.lkml`). | | `upstream_fields` | Upstream fields (`column`, `looker_dimension`, `looker_measure`, or `dbt_metric`) referenced by this field. | ## Users and Groups When **Observe users** is enabled, Euno observes Looker users as `user` resources. When **Observe groups** is enabled, Euno observes Looker groups as `user_group` resources with `members` pointing to observed users. For property details and membership relationships, see [Users as a Resource](/developer-reference/technical-concepts/users-as-a-resource). ## Table The Looker Integration will discover resources of type Table when it discovers a LookML view that refers to a database table. For example, if a LookML view is using the [`sql_table_name`](https://cloud.google.com/looker/docs/reference/param-view-sql-table-name) parameter, Euno will observe the referred table. To determine the exact location of the referred table, the Looker Integration looks up the correct [Looker Connection](https://cloud.google.com/looker/docs/admin-panel-database-connections) using the LookML Model's [connection parameter](https://cloud.google.com/looker/docs/reference/param-model-connection). The Looker Connection usually has all the metadata required for Euno to uniquely determine the URI of the referred table. In the case of a connection with dialect `athena`, The Looker Integration is not able to fetch from Looker's API the term "AWS account id" which is required to uniquely identify the referred tables. In this case, Euno defaults to using the aws account id `default_aws_account.us_central1` . If you have multiple AWS accounts and you need to distinguish between them, set up manual "connection mapping". | Property | Description | | --------------------- | --------------------------------------------------------- | | `name` | The name of the table. | | `database_technology` | The database technology, based on the connection dialect. | | `database` | The name of the database. | | `schema` | The name of the schema. | ## Relationships ### Containment (`has parent` / `has child`) | Source type(s) | Relationship | Target type(s) | Notes | | ------------------------------------ | ------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `looker_dimension`, `looker_measure` | has parent | `looker_view`, `looker_explore`, `looker_look`, `looker_tile` | | | `looker_view`, `looker_explore` | has parent | `looker_model` | | | `looker_model` | has parent | `looker_instance` | | | `looker_folder` | has parent | `looker_instance`, `looker_folder` | Root folders parent to the instance; nested folders parent to a parent folder. | | `looker_look`, `looker_dashboard` | has parent | `looker_folder` | | | `looker_tile` | has parent | `looker_dashboard` | | ### Resource lineage (`has upstream` / `has downstream`) Downstream resources list their upstream dependencies via `table_dependencies`. For example, `looker_view` **has upstream** `table` when the view references a warehouse table. | Source type(s) | Relationship | Target type(s) | Notes | | ------------------ | ------------ | ------------------------------------------------------ | ----- | | `looker_view` | has upstream | `table`, `looker_view`, `looker_explore`, `dbt_metric` | | | `looker_explore` | has upstream | `table`, `looker_view`, `looker_explore`, `dbt_metric` | | | `looker_look` | has upstream | `table`, `looker_explore`, `looker_view` | | | `looker_tile` | has upstream | `table`, `looker_explore`, `looker_view` | | | `looker_dashboard` | has upstream | `looker_tile`, `looker_look` | | ### Column-level lineage (`has upstream_fields` / `has downstream_fields`) Downstream fields list upstream fields via `upstream_fields`. For example, `looker_measure` **has upstream\_fields** `column` when the measure references a warehouse column. | Source type(s) | Relationship | Target type(s) | Notes | | ------------------------------------ | -------------------- | ------------------------------------------------------------ | ----- | | `looker_dimension`, `looker_measure` | has upstream\_fields | `column`, `looker_dimension`, `looker_measure`, `dbt_metric` | | ### Defined-by (`defines` / `has definer`) | Source type(s) | Relationship | Target type(s) | Notes | | ------------------ | ------------ | ------------------------------------ | --------------------------------------------- | | `dbt_column` | defines | `looker_dimension`, `looker_measure` | Via `euno.ai/based_on_column` tags in LookML. | | `dbt_metric` | defines | `looker_measure` | Via `euno.ai/based_on_metric` tags in LookML. | | `looker_dimension` | has definer | `dbt_column` | Inverse of `dbt_column` **defines**. | | `looker_measure` | has definer | `dbt_column`, `dbt_metric` | Inverse of dbt **defines**. | {% hint style="info" %} Euno does not currently support column-level lineage for Looker derived views when the referenced upstream tables or columns are not observed through an authoritative source integration (such as dbt, Snowflake, BigQuery or Databricks). If the columns are only referenced in the Looker SQL and not part of an observed schema, they will not appear in lineage. {% endhint %} # Troubleshooting Looker integration ### "Invalid Looker Credential" when trying to connect Looker To troubleshoot this issue, follow the steps: 1. Verify you correctly typed the Client ID and Client Secret in the "Configure Looker Integration" pop-up window. 2. If you're using IP whitelisting on your Looker instance, make sure Euno's public IP address is one of the rules in the whitelist. Follow Looker's guide [here](https://cloud.google.com/looker/docs/admin-panel-server-ip-allowlist#adding_a_new_rule). Euno's public IP address is `35.224.29.172`. Add a rule with: * **Type**: IP Address * **Value**: `35.224.29.172` 3. Confirm the API key user has permission to query Looker's `system__activity` model (required during source validation). The Admin role satisfies this requirement; for custom roles, include the `see_system_activity` permission. See [Using a custom Looker role](/sources/business-intelligence/looker-integration/using-a-custom-looker-role). ### Integration saves but LookML resources are missing If Looks, Tiles, and Dashboards appear but views, explores, and lineage are missing: 1. Link a Git account with read access to your LookML repositories. See [GitHub](/setup-configuration/linked-accounts/github-integration) or [GitLab](/setup-configuration/linked-accounts/gitlab-integration). 2. Check **LookML Project Includes** and **LookML Model Includes** in Advanced Settingsβ€”overly narrow patterns can exclude projects or models. 3. If **Crawl LookML Only** is enabled, application-layer content (Looks, Tiles, Dashboards) is intentionally skipped. ### Users or groups are not observed 1. Confirm **Observe users** and **Observe groups** are enabled in Advanced Settings (they are enabled by default). 2. For custom Looker roles, ensure the permission set includes `see_users` (required for user and group observation). See [Using a custom Looker role](/sources/business-intelligence/looker-integration/using-a-custom-looker-role). # Power BI ## Power BI (via Microsoft Fabric) Euno integrates with Microsoft Fabric / Power BI, ingesting its metadata to enhance data observability and support proactive, shift‑left practices. By surfacing structure, lineage, and usage patterns across your Power BI environment, Euno enables more informed decision‑making and efficient data management. Euno's Power BI integration supports auto‑discovery of the following resources and sub‑resources: * Power BI workspace * Power BI dataflows * Power BI semantic models (Formally known as "datasets") * Power BI tables (within semantic model) * Power BI reports * Power BI pages (within reports) * Power BI dashboards * Power BI tiles (on dashboards) * Power BI fields * Measures * Columns * Fabric Lakehouse database * Fabric Mirrored database * Users (when "Observe users" is enabled) * User groups (when "Observe groups" is enabled) If the Microsoft Purview source is also configured, Euno can enrich matching Power BI resources with Purview metadata such as tags, classifications, descriptions, and DLP data policy names. Power BI resources must first be discovered by the Power BI source; Microsoft Purview adds metadata to those existing resources. ## Power BI Requirements To ingest metadata from Power BI (Fabric), you will need: * Microsoft Entra application / Service Principal with access to target workspaces. * Fabric Admin settings: * Create a Power BI application for Euno to use - "Euno Power BI application" * Power BI admin: * Enable API access to various resources for Euno to be able to discover * Power BI app: * Grant Euno's Power BI application access to the connection and gateway metadata ## Stage 1: Configure Azure and Power BI ### Step 1: Create an Euno Power BI application in Microsoft Entra 1. Open [Microsoft Entra](https://entra.microsoft.com/) and sign in 2. In the sidebar, pick "App registrations" 3. Click "New registration" 4. Fill in the name ("Euno", for example), leave the "Supported account types" as default ("Accounts in this organizational directory only") and leave the "Redirect URI" empty 5. Click "Register" 6. You'll be redirected back to the Overview page. Make sure to note and copy the Application ID (Also called Client ID) and the Tenant ID (Also called Directory ID). You'll need these in the next steps. * Skip to step 14 if you don't want Euno to observe Entra Users as part of this integration 7. Next, using the sidebar, pick "API permissions" 8. Click "+ Add a permission" 9. Select "Microsoft Graph" 10. Select "Application permissions" 11. Add the following permissions: * `User.Read.All` - Read all users' full profiles (required if "Observe users" is enabled) * `Group.Read.All` - Read all groups (required if "Observe users" or "Observe groups" is enabled) * `GroupMember.Read.All` - Read all group memberships (required if "Observe users" or "Observe groups" is enabled) 12. Click "Add permissions" 13. Click "Grant admin consent for \[your organization]" to grant admin consent for these permissions {% hint style="info" %} These Microsoft Graph API permissions are required if you enable "Observe users" or "Observe groups" in the Power BI integration configuration. If you don't plan to observe users or groups, you can skip steps 7–13. {% endhint %} 14. Next, using the sidebar, pick "Certificates & secrets" 15. Under "Client secrets", click "+ New client secret" 16. Fill in the expiry time, and optionally a description, and click "Add". This creates a secret for Euno to use to authenticate as the Euno app. Make sure to note and copy the **secret Value**. ### Step 2: Create a security group for the Euno application Note: This step is not strictly required, but highly recommended. As you'll see in the next steps, some privileges of Power BI can be either granted to the entire organization, or to specific security groups. If you don't configure a security group for the Euno application, you'll need to enable these privileges to the entire organization. 1. Open the [Azure portal](https://ms.portal.azure.com/#allservices) and sign in 2. Find the service "Microsoft Entra ID" 3. In the sidebar, under "Manage", pick "Groups" 4. Click "+ Add" and "Group" * Group type: Security * Group name: any name (i.e "Euno application") * Group description is optional 5. Once the group is created, click the link under "Members" to open a modal to add users. Add the Euno application and click "Select" to add the Euno application to the new security group ### Step 3: Configure access to Power BI APIs 1. Open the [Power BI admin portal](https://app.powerbi.com/admin-portal/) and sign in 2. Pick "Tenant settings" from the list of admin pages 3. Scroll to "Developer settings" section find "Service principals can call Fabric public APIs" and expand it. Switch to "Enable" and grant access to the security group we created in Step 2. 4. Scroll to "Admin API settings" and repeat the process for: * Service principals can access read-only admin APIs * Enhance admin APIs responses with detailed metadata * Enhance admin APIs responses with DAX and mashup expressions ### Step 4: Add the Euno Power BI application to your workspaces Note: this should be done for every Power BI workspace > Many workspaces? To grant the Euno app Admin access programmatically, use a separate bootstrap app and the device-code script in [Grant Euno’s Power BI app access to many workspaces (programmatic)](/sources/business-intelligence/powerbi-integration/powerbi-workspace-access-bootstrap). 1. Open [Power BI](https://app.powerbi.com/) and sign in 2. Open "Workspaces" and find the workspace you want to enable. Click the `...` next to the name and pick "Workspace access" 3. Click "+ Add people or groups" 4. Add the Euno Power BI application you created in step 1 above, and select it. Make sure to assign the "Admin" role. ### Step 5: Add the Euno Power BI application to your gateways and connection 1. Open [Fabric Dashboard](https://app.fabric.microsoft.com/) 2. On the top right, click the "Settings" icon (gear icon) and pick "Manage connections and gateways" 3. For each connection under "Connections", click the `...` next to the name, and select "manage users", search the security group we created in step 2, and add it to the members of that connection {% hint style="warning" %} Rotate the client secret per your security policy; expired secrets will cause ingestion failures. {% endhint %} ## Stage 2: Configure New Power BI Source in Euno ### Step 1: Access the Sources Page * Open Sources. * Select Add New Source and choose Power BI. ### Step 2: General Configuration *Asterisk (\*) means a mandatory field.* | Configuration | Description | | --------------------------- | ------------------------------------------------------------------------- | | **Name**\* | Enter a name for your source (e.g., "Power BI – Finance"). | | **Tenant ID**\* | Your Microsoft Entra tenant ID. | | **Client ID**\* | The application (client) ID of your SPN. | | **Client Secret**\* | The client secret for the app registration. | | **Power BI URL (optional)** | Base URL override for deep links (defaults to `https://app.powerbi.com`). | ### Step 3: Schedule * Weekly: set specific days and times. * Hourly: define the interval (e.g., every 8 hours). ### Step 4: Resource Cleanup Keep your catalog current by choosing a cleanup strategy (see Resource Sponsorship in Euno): * Immediate Cleanup: remove resources not detected in the most recent successful run. * No Cleanup: keep resources indefinitely, even if no longer detected. ### Step 5: Advanced Settings (Optional) | Configuration | Description | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Workspace/Object pattern** | Regular expressions to allow/exclude specific workspaces or objects (e.g., `.*` to match all). | | **Observe users** | When enabled, Euno will observe users from Entra ID and track which users access Power BI resources. Requires Microsoft Graph API permissions: User.Read.All, Group.Read.All, and GroupMember.Read.All (see Step 1 above). | | **Observe groups** | When enabled, Euno will observe security groups from Entra ID and their memberships. Requires Microsoft Graph API permissions: Group.Read.All and GroupMember.Read.All (see Step 1 above). | | **Observe content** | When enabled (default), Euno fetches report content (page/tile definitions) for richer lineage. Disable to speed up crawls at the cost of tile-level lineage. | | **Observe usage data** | When enabled (default), Euno ingests Power BI activity log events to populate impression and distinct-user metrics. | | **Observe folders** | When enabled (default), Euno discovers Fabric workspace folders and uses them as the parent container for reports, dashboards, and other items. Requires Fabric Core endpoint access on the workspace. | | **Observe dataflows** | When enabled (default), Euno discovers Power BI dataflows and their lineage to upstream warehouse tables. | ## What Euno Discovers * Workspaces (with member information when available) * Datasets (semantic models) * Reports and Pages * Dashboards and Tiles * Apps * Model fields: Columns and Measures * Users (when "Observe users" is enabled) * User groups (when "Observe groups" is enabled) * Microsoft Purview metadata on matching Power BI resources, when a Microsoft Purview source is configured # Power BI Integration Discovered Resources ### Ingested Resources & Properties #### Account (`powerbi_account`) The Power BI integration discovers the top-level tenant as a single `powerbi_account` resource. All workspaces are children of this account. | Property | Description | | ---------------- | ------------------------------------ | | `name` | Tenant identifier (tenant ID string) | | `native_id` | Microsoft Entra tenant ID (GUID) | | `subtype` | Always `powerbi_account` | | `external_links` | URL to the Power BI tenant portal | #### Workspaces (`powerbi_workspace`) | Property | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Display name of the workspace | | `description` | Workspace description (when available) | | `native_id` | Workspace object ID (GUID) | | `subtype` | Type of workspace: `workspace` for shared workspaces, `personal` for personal workspaces, `admin_workspace` for the admin monitoring workspace | | `native_owners` | List of admin users (primary first); each entry contains `email`, `display_name`, and other user-reference fields | | `members` | List of users and groups that are members of the workspace, including their roles (e.g., Admin, Member, Contributor). Each entry contains `uri` and `role`. | | `external_links` | URL to open the workspace in the service | #### Folders (`powerbi_folder`) Fabric workspace folders are discovered when **Observe folders** is enabled and the workspace has Fabric Core endpoint data. Items (reports, dashboards, semantic models, etc.) are placed as children of the folder rather than directly of the workspace. | Property | Description | | ---------------- | ---------------------------------------------------------------- | | `name` | Folder display name | | `native_id` | Fabric folder ID (GUID) | | `subtype` | Always `powerbi_folder` | | `external_links` | Deep link to the folder in the Power BI portal (when resolvable) | #### Semantic Models (`powerbi_semantic_model`) | Property | Description | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Display name | | `native_id` | Semantic model object ID (GUID) | | `subtype` | Always `powerbi_semantic_model` | | `native_owners` | List of owners (primary first); each entry contains `email`, `display_name`, and other user-reference fields | | `members` | Users and groups with direct access to the semantic model. Each entry includes `uri` and may include `role`. | | `tags` | User-defined tags | | `created_at` | Timestamp for creation date | | `updated_at` | Timestamp last modification | | `total_impressions_14d/30d/60d` | The number of impressions the resource is found within over the last 14/30/60 days. | | `distinct_impressions_users_14d/30d/60d` |

The number of distinct users who triggered the impressions of the resource over the last 14/30/60 days.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| | `external_links` | Deep link to open the semantic model | #### Table (`powerbi_table`) | Property | Description | | -------- | ------------ | | `name` | Display name | #### Dataflows (`powerbi_dataflow`) A **dataflow** is a Power BI ETL artifact that loads and transforms data and exposes **output entities** (tables) that can be consumed by semantic models (e.g. via the `PowerBI.Dataflows` connector). In Euno, a dataflow is observed as a resource of type **powerbi\_dataflow**. The `powerbi_table` resources that consume a dataflow have a **has upstream** relationship to it; conversely, the dataflow itself has a **has upstream** relationship to the warehouse tables it reads from (resolved from its Power Query M expression). | Property | Description | | ---------------- | ------------------------------------------------------------------------------------------------------ | | `name` | Dataflow display name | | `subtype` | Always `powerbi_dataflow` | | `native_id` | Dataflow object ID (GUID) | | `tags` | User-defined tags | | `created_at` | Timestamp for creation date (when available from scan/API) | | `updated_at` | Timestamp last modification (when available) | | `members` | Users and groups with direct access to the dataflow. Each entry includes `uri` and may include `role`. | | `external_links` | Deep link to the dataflow details page | #### Reports (`powerbi_report`) | Property | Description | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Display name | | `subtype` | Always `powerbi_report`; the report kind (e.g., Power BI vs. paginated) is stored in the native payload (`reportType`) | | `native_id` | Report object ID (GUID) | | `native_owners` | List of owners (primary first); each entry contains `email`, `display_name`, and other user-reference fields | | `members` | Users and groups with direct access to the report. Each entry includes `uri` and may include `role`. | | `tags` | User-defined tags | | `created_at` | Timestamp for creation date | | `updated_at` | Timestamp last modification | | `total_impressions_14d/30d/60d` | The number of impressions the resource is found within over the last 14/30/60 days. | | `distinct_impressions_users_14d/30d/60d` |

The number of distinct users who triggered the impressions of the resource over the last 14/30/60 days.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| | `external_links` | Deep link to the report | #### Pages (`powerbi_page`) | Property | Description | | ---------------- | ---------------------------------------- | | `name` | Page display name (from report metadata) | | `native_id` | Page identifier (when exposed) | | `folder` | Page grouping/order (if present) | | `created_at` | Timestamp for creation date | | `updated_at` | Timestamp last modification | | `external_links` | Deep link to page (when resolvable) | #### Dashboards (`powerbi_dashboard`) | Property | Description | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Display name | | `native_id` | Dashboard object ID (GUID) | | `native_owners` | List of owners (primary first); each entry contains `email`, `display_name`, and other user-reference fields | | `members` | Users and groups with direct access to the dashboard. Each entry includes `uri` and may include `role`. | | `tags` | User-defined tags | | `created_at` | Timestamp for creation date | | `updated_at` | Timestamp last modification | | `total_impressions_14d/30d/60d` | The number of impressions the resource is found within over the last 14/30/60 days. | | `distinct_impressions_users_14d/30d/60d` |

The number of distinct users who triggered the impressions of the resource over the last 14/30/60 days.

Note: The distinct user count does support a user-level breakdown, which you can access via the Euno Assistant or from the resource side panel. However, this breakdown is not available via an EQL query.

| | `external_links` | Deep link to the dashboard | #### Tiles (`powerbi_tile`) | Property | Description | | ------------------- | --------------------------------------------------------------- | | `name` | Visual identifier (internal name from the report layout config) | | `parent_container` | URI of the parent page | | `native_raw_object` | Raw visual container config from the report layout (JSON) | | `raw_code` | Pretty-printed version of the visual container config | | `external_links` | Deep link to the page that contains the tile | #### Apps (`powerbi_app`) | Property | Description | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | App name | | `native_id` | App object ID (GUID) | | `members` | Users and groups with direct access to the stored app-backed report row. This does not include published app audience membership. Each entry includes `uri` and may include `role`. | | `tags` | User-defined tags | | `created_at` | Timestamp for creation date | | `updated_at` | Timestamp last modification | | `external_links` | Deep link to the app | #### Power BI Columns & Measures (`powerbi_column` & `powerbi_measure`) | Property | Description | | ---------------------- | ---------------------------------------------------------------------- | | `name` | Field name (column or measure) | | `native_data_type` | Original data type (e.g., whole number, decimal, text, date) | | `normalized_data_type` | Standardized type (string, number, date, boolean) | | `is_calculated` | `true` if DAX (calculated column/measure) or M‑authored transformation | | `raw_code` | DAX expression or M code when available | #### Table (`table`) The Power BI integration will discover resources of type `table` when it identifies a dataset or dataflow referring to a database table. For example, if a data source is using a table from the data warehouse, which is not connected to dbt, or an integrated DW. | Property | Description | | --------------------- | ---------------------------------------------------------------------- | | `name` | The name of the table. | | `database_technology` | The technology of the database, based on the dialect of the connection | | `database` | The name of the database | | `schema` | The name of the schema | #### Column (`column`) The Power BI integration will discover resources of type `column` when it identifies a dataset or dataflow referring to a database column. | Property | Description | | --------------------- | ---------------------------------------------------------------------- | | `name` | The name of the column. | | `database_technology` | The technology of the database, based on the dialect of the connection | | `database` | The name of the database | | `schema` | The name of the schema | #### Lakehouse (`database`, with optional `database_schema`, `table`, `column`) A **lakehouse** is a Microsoft Fabric managed data asset stored in OneLake and queryable from Fabric. In Euno it is observed as a resource of type **database** with **database\_technology** = `lakehouse`. The list of lakehouses comes from the Power BI admin scan (`workspace.Lakehouse[]`); no separate Fabric API call is required for the list. * **When the lakehouse has a SQL analytics endpoint**, the integration also observes **schema**, **table**, and **column** by querying the SQL endpoint (INFORMATION\_SCHEMA). Schema/table/column use the same property conventions as other database-backed resources. | Property | Description | | --------------------- | ------------------------------------ | | `name` | Lakehouse display name from the scan | | `database_technology` | Always `lakehouse` | | `native_id` | Fabric item id of the lakehouse | Child resources (when SQL endpoint is available): **Schema (`database_schema`)** | Property | Description | | --------------------- | ---------------------------------------------------------------- | | `name` | Schema name | | `database_technology` | Same as the parent database (`lakehouse` or `mirrored_database`) | **Table** (`table`) and **Column** (`column`) β€” same property conventions as other database-backed resources (see `table` and `column` sections below). #### Mirrored database (`database`, with optional `database_schema`, `table`, `column`) A **mirrored database** is a Microsoft Fabric item (type MirroredDatabase) that replicates data from SQL Server or other sources into OneLake. It is observed as a resource of type **database** with **database\_technology** = `mirrored_database`. * **When the mirrored database has a SQL analytics endpoint** (provisioningStatus = success), the integration also observes **schema**, **table**, and **column** via the same SQL endpoint pattern as lakehouse (INFORMATION\_SCHEMA). | Property | Description | | --------------------- | ------------------------------------------ | | `name` | Mirrored database name from the scan | | `database_technology` | Always `mirrored_database` | | `parent_container` | Workspace URI | | `native_id` | Fabric item id of the mirrored database | | `native_raw_object` | Admin scan entry for the mirrored database | Child resources (when SQL endpoint is available): **schema** (`database_schema`), **table**, **column**, with the same property conventions. #### Users (`user`) Users are discovered when "Observe users" is enabled in the Power BI integration configuration. Users are fetched from Microsoft Entra ID via the Microsoft Graph API. | Property | Description | | --------------- | --------------------------------------------- | | `name` | User's email address (used as the identifier) | | `email_address` | User's email address | | `label` | User's display name from Entra ID | #### User Groups (`user_group`) User groups (security groups) are discovered when "Observe groups" is enabled in the Power BI integration configuration. Groups are fetched from Microsoft Entra ID via the Microsoft Graph API. | Property | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Display name of the group | | `native_id` | Group ID from Entra ID | | `subtype` | Always `entra` for groups discovered from Entra ID | | `members` | List of users and groups that are members of this group. Each member entry includes the URI. Groups can contain other groups, creating nested hierarchies. | ### Microsoft Purview Metadata When the Microsoft Purview source is configured with Data Map access, Euno can add Purview metadata to existing Power BI resources. This enrichment can apply to `powerbi_report`, `powerbi_dashboard`, `powerbi_semantic_model`, `powerbi_app`, `powerbi_dataflow`, and `powerbi_column` resources when Microsoft Purview exposes a matching Data Map asset. Microsoft Purview does not create Power BI resources in Euno. The matching Power BI resource must already exist from the Power BI source. | Property | Description | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `description` | Purview description for the matching Data Map asset. Euno fills this only when the Power BI resource does not already have a description, or when the existing description was previously filled from Purview. | | `purview_tags` | Microsoft Purview tags applied to the matching Power BI asset in Purview Data Map. | | `purview_classifications` | Microsoft Purview classifications applied to the matching Power BI asset in Purview Data Map. | | `purview_data_policy_names` | Names of Microsoft Purview DLP data policies that apply to the matching Power BI resource. | {% hint style="info" %} `powerbi_column` resources can receive Purview metadata fields, but they are not currently linked to `purview_dlp_policy` resources through regulator relationships. {% endhint %} ### Microsoft Purview DLP Policy Relationships When the Microsoft Purview source is configured, supported Power BI item resources can be linked to `purview_dlp_policy` resources through policy regulation relationships. The Power BI resource must already exist in Euno; Microsoft Purview adds policy context, not the Power BI resource itself. ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | --------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `powerbi_workspace` | has child | `powerbi_semantic_model`, `powerbi_report`, `powerbi_dashboard`, `powerbi_dataflow`, `powerbi_app`, `powerbi_folder`, lakehouse (`database`), mirrored database (`database`) | Direct children via Admin/Fabric APIs. Lakehouse and mirrored database from admin scan (`workspace.Lakehouse[]`, `workspace.MirroredDatabase[]`). Folders only when **Observe folders** is enabled and Fabric Core endpoint data is available. | | `powerbi_semantic_model` | has child | `powerbi_table` | Tables belong to a specific semantic model. | | `powerbi_table` | has upstream | `powerbi_dataflow` | When a table's M query source is a Power BI dataflow. | | `powerbi_table` | has upstream | `table` | DirectQuery/Import sources resolved to a warehouse table (e.g., Snowflake, BigQuery, Databricks). | | `powerbi_table` | has child | `powerbi_column`, `powerbi_measure` | Columns/measures come from Scanner (datasetSchema/expressions) or XMLA. | | `powerbi_dataflow` | has child | `powerbi_column` | Output entity columns (CDM attributes) emitted when the group dataflow GET document is available. | | `powerbi_dataflow` | has upstream | `powerbi_dataflow` | A dataflow can consume another dataflow's output entities as its source (dataflow chaining). | | `powerbi_dataflow` | has upstream | `table` | Upstream warehouse tables resolved from the dataflow's Power Query M expression. | | `powerbi_report` | has child | `powerbi_page` | Pages enumerated via /reports/{id}/pages. | | `powerbi_report` | has upstream | `powerbi_table` | When the report has no tile children, all tables in the upstream semantic model become the report's upstreams. | | `powerbi_report` | has upstream | `powerbi_tile` | When the report has tile children, each tile becomes the report's direct upstream (mutually exclusive with the `powerbi_table` row above). | | `powerbi_page` | has child | `powerbi_tile` | Visuals on a page. | | `powerbi_dashboard` | has upstream | `powerbi_report` | Reports whose visuals are pinned on the dashboard. | | `powerbi_tile` | has upstream | `powerbi_table` | Tables referenced by Q\&A tiles and streaming dataset tiles. Static tiles (text/image) have no lineage and are excluded. | | `powerbi_app` | has upstream | `powerbi_report` | The source report that the app distributes to its audience (the original report object the app was built from). | | `powerbi_column` | has upstream\_fields | `column`, `powerbi_column` | Source columns β†’ table columns β†’ dataflow columns β†’ Power BI table columns; derived from the semantic model's Power Query M expressions and DAX. | | `powerbi_measure` | has upstream\_fields | `powerbi_column`, `powerbi_measure` | DAX references to columns and other measures. | | `powerbi_tile` | has upstream\_fields | `powerbi_column`, `powerbi_measure` | Fields used in the tile's visualization. | | `powerbi_report`, `powerbi_dashboard`, `powerbi_semantic_model`, `powerbi_app`, `powerbi_dataflow` | has regulator | `purview_dlp_policy` | Added by the Microsoft Purview source when a Purview DLP policy applies to the Power BI item. | | `powerbi_workspace` | has group | `user`, `user_group` | Workspace members (users and groups) with role information (e.g., Admin, Member, Contributor). | | `powerbi_report`, `powerbi_dashboard`, `powerbi_semantic_model`, `powerbi_app`, `powerbi_dataflow` | has group | `user`, `user_group` | Direct item access for users and groups, with role information when available. | | `user_group` | has group | `user`, `user_group` | Group members from Entra ID. Groups can contain users and other groups, creating nested hierarchies. | | `user`, `user_group` | has member | `user_group`, `powerbi_workspace`, Power BI item resources | Reverse relationship: resources that a user or group is a member of. | | Lakehouse, mirrored database (`database` with `database_technology` = `lakehouse` or `mirrored_database`) | has child | `database_schema` | When the item has a SQL analytics endpoint; schema discovered via INFORMATION\_SCHEMA on that endpoint. | | `database_schema` | has child | `table` | Tables within the schema, discovered via INFORMATION\_SCHEMA. | | `table` | has child | `column` | Columns within the table, discovered via INFORMATION\_SCHEMA. | # Power BI workspace access bootstrap This guide walks through adding the **main Euno Power BI application** (the service principal you configure in Euno as **Client ID** / **Client secret**) as a **workspace Admin** across **many** workspaces, without using the Power BI UI for each workspace. It complements [Step 4 in the main Power BI setup](/sources/business-intelligence/powerbi-integration#step-4-add-the-euno-powerbi-application-to-your-workspaces). ## Prerequisites (tenant) Complete the main Power BI guide through **Step 3** (tenant settings, security groups, admin API toggles). In particular, for the bootstrap flow you need a **Fabric / Power BI admin** who can complete device-code sign-in. ## Step A β€” Create the bootstrap app registration 1. In [**Microsoft Entra admin center**](https://entra.microsoft.com/) β†’ **App registrations** β†’ **New registration**. 2. **Name:** e.g. `Euno Power BI workspace bootstrap`. 3. **Supported account types:** *Accounts in this organizational directory only*. 4. **Redirect URI:** leave empty (device code flow). 5. **Register** and note the **Application (client) ID** and **Directory (tenant) ID**. ### API permissions (bootstrap app) 1. **API permissions** β†’ **Add a permission** β†’ **Power BI Service** β†’ **Delegated permissions**. 2. Add **`Tenant.ReadWrite.All`**. 3. **Grant admin consent** for the tenant (an administrator must consent for all users). Do **not** add Power BI **Application** `Tenant.ReadWrite.All` on this app if you want to avoid the same service-principal admin API conflicts you can hit on the main appβ€”**delegated** is enough for this script. ### Public client (device code) 1. **Authentication** β†’ **Advanced settings** β†’ **Allow public client flows** = **Yes**. 2. **Save**. ## Step B β€” Find the main Euno app’s **service principal object ID** 1. **Entra** β†’ **Enterprise applications** β†’ search for your main Euno app by name or **Application ID**. 2. Open it and copy **Object ID** (a UUID). You will pass this to the script as `--principal-object-id`. ## Step C β€” Run the script Download the script here. {% file src="/files/x6pVSgXoPzj3jfhaGq85" %} {% file src="/files/BWllDh5EiMf66XdVwZGj" %} ```bash python3 scripts/powerbi/grant_euno_workspace_access.py --help # or: chmod +x scripts/powerbi/grant_euno_workspace_access.py && ./scripts/powerbi/grant_euno_workspace_access.py --help ``` The `grant-euno-workspace-access.sh` wrapper runs Python with **`-u`** (unbuffered I/O) so the device user code appears immediately when stdout is not a terminal (CI, log capture, etc.). Pass **exactly one** workspace selection flag (see table below), plus optional `--skip`, `--limit`, `--dry-run`, `--role`, `--poll-interval`, or `--access-token-file`. Use **exactly one** workspace selection mode: | Mode | Flags | | -------------------------- | ----------------------------------- | | Explicit UUIDs | `--workspace-ids 'uuid1,uuid2,...'` | | Name substring | `--name-contains 'Finance'` | | Name regex (Python) | `--name-regex '^prod-.+'` | | All workspaces (inventory) | `--all` | **`--skip N`:** after filtering, skip the first **N** workspaces, then apply **`--limit`** (stable order follows the Admin API listing). **`--limit N`:** after filtering and **`--skip`**, process at most **N** workspaces. **`--dry-run`:** print targets only; **no** `POST`. If you use only `--workspace-ids`, no sign-in is required. For `--name-*` / `--all`, dry-run needs a token: pass **`--access-token-file`** whose first line is a delegated Power BI access token (advanced). **`--role`:** defaults to `Admin` (matches the main guide’s Step 4). Other values follow [GroupUser](https://learn.microsoft.com/en-us/rest/api/power-bi/admin/groups-add-user-as-admin) documentation. ### What the script does 1. Starts [**OAuth 2.0 device code**](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-device-code) against your tenant for the bootstrap app, with scope `Tenant.ReadWrite.All` on Power BI (plus `offline_access` for refresh token issuance if Microsoft returns one). 2. Prints a **user code** and opens a browser to **`https://login.microsoft.com/device`** (override with env `DEVICE_VERIFY_URL` if needed). Complete sign-in as a **Fabric admin**. 3. Polls the token endpoint every **5 seconds** (override with `--poll-interval` or env `POLL_INTERVAL`) until tokens are issued. 4. Unless you only passed `--workspace-ids`, calls **`GET /admin/groups`** (paginated) and filters to `type == "Workspace"`. 5. For each target workspace, calls **`POST /admin/groups/{workspaceId}/users`** with `principalType: App` and `identifier` = your main app’s **service principal object ID**. ### Example: two workspaces by ID (dry run, no sign-in) ```bash python3 scripts/powerbi/grant_euno_workspace_access.py \ --tenant "" \ --bootstrap-client-id "" \ --principal-object-id "" \ --workspace-ids "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee,11111111-2222-3333-4444-555555555555" \ --dry-run ``` ### Example: first 10 workspaces whose names contain β€œtest” ```bash python3 scripts/powerbi/grant_euno_workspace_access.py \ --tenant "" \ --bootstrap-client-id "" \ --principal-object-id "" \ --name-contains "test" \ --limit 10 ``` ### Example: all workspaces (use with care) ```bash python3 scripts/powerbi/grant_euno_workspace_access.py \ --tenant "" \ --bootstrap-client-id "" \ --principal-object-id "" \ --all ``` ## Security and operations * The **bootstrap app** is powerful in combination with a **Fabric admin** user. Restrict who can run the script; rotate bootstrap app settings if credentials leak. Consider removing the app after the bootstrap is successful. * **Refresh tokens** returned by the device-code flow are sensitive; this script does **not** print them. Run from a secure workstation. * **Re-running** the script for a workspace that already has the app may return a non-2xx response from Microsoft; treat as idempotent β€œalready added” and verify in the Power BI UI if unsure. ## Related links * [Power BI integration (main setup)](/sources/business-intelligence/powerbi-integration) * [Add User As Admin (Microsoft Learn)](https://learn.microsoft.com/en-us/rest/api/power-bi/admin/groups-add-user-as-admin) * [Device code flow](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-device-code) * [Enable service principal authentication for admin APIs](https://learn.microsoft.com/en-us/fabric/admin/enable-service-principal-admin-apis) ## Q\&A ### Why is automation awkward? (Two different Microsoft models) **What you want:** Call the Power BI Admin API [**Groups - Add User As Admin**](https://learn.microsoft.com/en-us/rest/api/power-bi/admin/groups-add-user-as-admin) so a **Fabric / Power BI administrator** can add your Euno app to each workspace in one pass. **What Microsoft requires:** The same API documentation states: * The caller must be a [**Fabric administrator**](https://learn.microsoft.com/en-us/rest/api/power-bi/admin/groups-add-user-as-admin#permissions) (directory role on the identity making the call). * The token must include scope **`Tenant.ReadWrite.All`** for the Power BI resource. At the same time, Microsoft’s guidance for **service principals** using **read-only** Power BI admin APIs says the app **must not** have certain **Power BI *****Application*** permissions configured, or service-principal admin calls can fail in practice. See [Enable service principal authentication for admin APIs](https://learn.microsoft.com/en-us/fabric/admin/enable-service-principal-admin-apis). **Net effect for many customers:** * The **Euno integration app** (client credentials, no interactive user) should stay **clean**: no Power BI **Application** permission like `Tenant.ReadWrite.All`, so it can keep using **read-only admin** APIs and tenant settings as documented in the main guide. * **`AddUserAsAdmin`** is reliably exercised with a **delegated (user) token** from a **Fabric admin** account, obtained through a **small separate β€œbootstrap” app** that is **not** the integration. That is why this page describes a **second Entra application** used only for this bootstrap script. # Thoughtspot Euno's Thoughtspot integration supports auto-discovery of: * Thoughtspot Liveboards * Thoughtspot Answers * Thoughtspot Models * Thoughtspot Views * Thoughtspot Tables * Thoughtspot Formulas ## Setting up Thoughtspot integration ### Prerequisites To integrate Thoughtspot with Euno, you will need: 1. A Thoughtspot user account with appropriate permissions 2. The user's credentials (username and password) ### Step 1: Create a Thoughtspot User 1. Log in to your Thoughtspot instance as an administrator 2. Create a new user account that will be used for Euno integration 3. Ensure the user has sufficient permissions to access the resources you want to discover ### Step 2: Configure Euno's Thoughtspot Integration #### Step 1: Access the Sources Page 1. Navigate to the **Sources** page and click on the **Gallery** Tab 2. Click on the **Connect** button in the **Thoughtspot** Tile #### Step 2: General Configuration 1. **Name**: Enter a name for your Thoughtspot source (e.g., "Thoughtspot - Analytics") 2. **Configuration Details**: 1. **Connection URL**: Enter your Thoughtspot instance URL 2. **Username**: Enter the Thoughtspot user's username 3. **Password**: Enter the Thoughtspot user's password #### Step 3: Scheduling Updates 1. Enable the Schedule toggle 2. Choose from the following: 1. **Weekly**: Specify the days and times for updates 2. **Daily**: Enter an interval in hours (e.g., every 6 hours) #### Step 4: Resource Cleanup * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). #### Step 5: Save Configuration Click the **Save** button to complete the setup. ## Discovered Resources For detailed information about discovered resources and their properties, see [discovered resources](/sources/business-intelligence/thoughtspot-integration/thoughtspot-integration-discovered-resources). # Thoughtspot Integration Discovered Resources The Euno-Thoughtspot integration discovers the following resources: ### Thoughtspot Liveboard | Property | Value | | ----------- | ---------------------------------------------- | | name | The name of the Liveboard | | description | The description of the Liveboard, if available | | created\_at | Timestamp when the Liveboard was created | | updated\_at | Timestamp when the Liveboard was last modified | | native\_id | The ID of the Liveboard in Thoughtspot | | owner | The owner of the Liveboard | | type | always `thoughtspot_liveboard` | ### Thoughtspot Answer | Property | Value | | ----------- | ------------------------------------------- | | name | The name of the Answer | | description | The description of the Answer, if available | | created\_at | Timestamp when the Answer was created | | updated\_at | Timestamp when the Answer was last modified | | native\_id | The ID of the Answer in Thoughtspot | | owner | The owner of the Answer | | type | always `thoughtspot_answer` | ### Thoughtspot Model | Property | Value | | ----------- | ------------------------------------------ | | name | The name of the Model | | description | The description of the Model, if available | | created\_at | Timestamp when the Model was created | | updated\_at | Timestamp when the Model was last modified | | native\_id | The ID of the Model in Thoughtspot | | owner | The owner of the Model | | type | always `thoughtspot_model` | ### Thoughtspot View | Property | Value | | ----------- | ----------------------------------------- | | name | The name of the View | | description | The description of the View, if available | | created\_at | Timestamp when the View was created | | updated\_at | Timestamp when the View was last modified | | native\_id | The ID of the View in Thoughtspot | | owner | The owner of the View | | type | always `thoughtspot_view` | ### Thoughtspot Table | Property | Value | | ----------- | ------------------------------------------ | | name | The name of the Table | | description | The description of the Table, if available | | created\_at | Timestamp when the Table was created | | updated\_at | Timestamp when the Table was last modified | | native\_id | The ID of the Table in Thoughtspot | | owner | The owner of the Table | | type | always `thoughtspot_table` | ### Thoughtspot Formula | Property | Value | | ------------------- | -------------------------------------------- | | name | The name of the Formula | | description | The description of the Formula, if available | | created\_at | Timestamp when the Formula was created | | updated\_at | Timestamp when the Formula was last modified | | native\_id | The ID of the Formula in Thoughtspot | | owner | The owner of the Formula | | type | always `thoughtspot_formula` | | formula\_definition | The SQL or formula definition | # Sigma Euno's Sigma integration supports auto-discovery of: * Sigma accounts (organizations) * Sigma folders and workspaces * Sigma data models * Sigma workbooks * Workbook lineage to source tables {% hint style="info" %} **Impression data**: To observe workbook and element impressions from Sigma-exported CSV files, also set up the [Sigma Impressions integration](/sources/business-intelligence/sigma-impressions-integration). It is a separate integration from the Sigma metadata crawler. {% endhint %} ## Setting up Sigma integration ### Overview Sigma is a cloud-native analytics and business intelligence platform that enables users to explore, analyze, and visualize data directly in their cloud data warehouse. In Sigma, data is organized into *workbooks* and *data models*. Workbooks contain visualizations and analyses, while data models provide reusable semantic layers over your data warehouse tables. To discover Sigma resources, Euno connects to the Sigma API using OAuth client credentials. The integration automatically discovers your Sigma organization structure, including workspaces, folders, data models, and workbooks, along with their lineage to underlying data warehouse tables. ### Step 1: Create Sigma API Credentials #### Generate OAuth Client Credentials 1. Log in to your Sigma account as an administrator 2. Navigate to **Administration** β†’ **Developer Access** 3. Click **Create New** to generate a new OAuth client 4. Provide a name for the client (e.g., "Euno Integration") 5. Check **Rest API** scope 6. Copy the **Client ID** and **Client Secret** - you'll need these for Euno configuration 7. Note your Sigma API server URL (e.g., `https://aws-api.sigmacomputing.com`) {% hint style="warning" %} **Important**: The Client Secret is only shown once. Store it securely - you won't be able to retrieve it again. {% endhint %} ### Step 2: Configure New Sigma Source in Euno #### Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click on **Add New Source** and select **Sigma** from the list of supported platforms. #### General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Name\* | Enter a name for your Sigma source (e.g., "Sigma - Analytics") | | API Server\* | The Sigma API server URL. Defaults to `https://aws-api.sigmacomputing.com`. Use your region-specific URL if different (e.g., `https://eu-api.sigmacomputing.com` for EU). | | Client ID\* | The OAuth Client ID from Step 1 | | Client Secret\* | The OAuth Client Secret from Step 1 | ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). {% hint style="info" %} **Recommended**: Schedule Sigma integration to run daily or every 12 hours to keep workbook lineage and metadata up to date. {% endhint %} ### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 5: Save Configuration Click the **Test & Save** button to complete the setup. Euno will: 1. Validate the API Server URL is accessible 2. Authenticate using the provided Client ID and Client Secret 3. Verify access to the Sigma organization 4. Save the configuration if all validations pass If the test fails, review the error message and verify your credentials and API server URL are correct. ## Workbook lineage and Sigma API errors Euno loads each workbook’s **lineage** from Sigma’s `GET /v2/workbooks/{workbookId}/lineage` API (used to relate workbook elements to warehouse tables). In rare cases **Sigma’s service** may return **HTTP 5xx** for that endpoint for a specific workbook while other workbooks and other API calls succeed. That condition originates on **Sigma’s side**, not in Euno. When that happens, Euno **does not fail the entire crawl**: the run continues, workbook and element metadata are still ingested where possible, and **lineage for that workbook may be omitted**. Details are recorded on the workbook resource’s stored snapshot (including an **`errors`** list noting the lineage endpoint and HTTP status) so operators and support can see that lineage was skipped for that workbook. Retries on later runs may succeed if Sigma resolves the issue. # Sigma Integration Discovered Resources ## Overview The Sigma integration discovers the following resources: * Sigma Account * Sigma Folder (Workspaces and Folders) * Sigma Data Model * Sigma Workbook * Sigma Workbook Element * Sigma Data Model Element * Data Warehouse Tables (referenced by workbooks) ## Sigma Account A single resource of type `sigma_account` is observed for each configured Sigma integration, representing the top-level Sigma organization. ### Properties | Property | Value | | --------------- | -------------------------------------------------------------------------------------------- | | name | The organization name, or `"Sigma Account ({organization_id})"` if the name is not available | | type | always `sigma_account` | | subtype | always `sigma_account` | | native\_id | The Sigma organization ID | | container\_type | always `pure_container` | | external\_links | List of links to open this resource in Sigma. See [External links](#external-links) below. | ### External links Many Sigma resources expose an **external\_links** property so API and UI consumers can offer β€œopen in Sigma”–style actions. This property is a list of link objects. Each object has: | Field | Type | Description | | ----- | ------ | ----------------------------------------------------------------------- | | label | string | Display label for the link (e.g. `"Open in Sigma"`, `"Sigma Account"`). | | url | string | URL that opens the resource in the Sigma web application. | **Where it appears:** * **sigma\_account**: Typically one link with label `"Sigma Account"` pointing to the organization in Sigma (e.g. `https://app.sigmacomputing.com/{organization_name}`). * **sigma\_folder**: Link with label `"Open in Sigma"` to the folder in the Sigma UI (when a URL is available). * **sigma\_data\_model**: Link with label `"Open in Sigma"` to the data model. * **sigma\_workbook**: Link with label `"Open in Sigma"` to the workbook. * **sigma\_workbook\_element**: Link with label `"Open in Sigma"` to the element within its workbook. * **sigma\_data\_model\_element**: Link with label `"Open in Sigma"` to the element within its workbook. If the Sigma API does not provide a URL for the resource, **external\_links** may be an empty list. ## Sigma Folders and workspaces Folders and workspaces are organizational containers in Sigma. Workspaces are top-level folders, while folders can be nested within workspaces or other folders. ### Properties | Property | Value | | --------------- | ------------------------------------------------------------------------- | | name | The folder or workspace name | | type | always `sigma_folder` | | subtype | `sigma_folder` or `sigma_workspace` | | native\_id | The unique folder/workspace ID in Sigma | | created\_at | The timestamp when the folder was created | | updated\_at | The timestamp when the folder was last modified | | created\_by | The email address of the user who created the folder | | updated\_by | The email address of the user who last updated the folder | | owner | The owner of the folder | | external\_links | Links to open the folder in Sigma. See [External links](#external-links). | ## Sigma Data Model Data models are reusable semantic layers in Sigma that provide business-friendly views of data warehouse tables. ### Properties | Property | Value | | --------------- | ----------------------------------------------------------------------------- | | name | The data model name | | type | always `sigma_data_model` | | subtype | always `sigma_data_model` | | native\_id | The unique data model ID in Sigma | | created\_at | The timestamp when the data model was created | | updated\_at | The timestamp when the data model was last modified | | created\_by | The email address of the user who created the data model | | updated\_by | The email address of the user who last updated the data model | | owner | The owner of the data model | | tags | List of version tag names associated with the data model | | external\_links | Links to open the data model in Sigma. See [External links](#external-links). | ## Sigma Workbook Workbooks are analytics documents in Sigma containing visualizations, tables, and analyses. ### Properties | Property | Value | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | name | The workbook name | | type | always `sigma_workbook` | | subtype | always `sigma_workbook` | | native\_id | The unique workbook ID in Sigma | | description | The workbook description, if provided | | created\_at | The timestamp when the workbook was created | | updated\_at | The timestamp when the workbook was last modified | | created\_by | The email address of the user who created the workbook | | updated\_by | The email address of the user who last updated the workbook | | owner | The owner of the workbook | | tags | List of version tag names associated with the workbook | | external\_links | Links to open the workbook in Sigma. See [External links](#external-links). | | total\_read\_queries\_14/30/60d | When integrated with a supported warehouse, the total read queries for the timeframe are propagated to the resource. | | total\_read\_cost\_14/30/60d | When integrated with Snowflake warehouse, the total read cost for the timeframe propagated to the resource. | | total\_read\_credits\_14/30/60d | When integrated with Snowflake, the total read credits for the timeframe propagated to the resource. | ## Sigma Element Elements are individual components within a Sigma workbook or a data model, such as tables, visualizations, or calculations. Each element may contain SQL queries and can reference other elements or data warehouse tables. ### Properties | Property | Value | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | name | The element name | | type | `sigma_workbook_element` or `sigma_data_model_element` | | subtype | `sigma_workbook_element` or `sigma_data_model_element` | | native\_id |

The element ID in the format {workbook\_uri}/{element\_id}
or {semantic\_model\_uri}/{element\_id}

| | external\_links | Links to open the element in Sigma. See [External links](#external-links). | | total\_read\_queries\_14/30/60d | When integrated with a supported warehouse, the total read queries for the timeframe are propagated to the resource. | | total\_read\_cost\_14/30/60d | When integrated with Snowflake warehouse, the total read cost for the timeframe propagated to the resource. | | total\_read\_credits\_14/30/60d | When integrated with Snowflake, the total read credits for the timeframe propagated to the resource. | ## Data Warehouse Tables When a Sigma workbook references a data warehouse table, Euno automatically observes that table if it hasn't been discovered yet. ### Properties | Property | Value | | -------------------- | ----------------------- | | name | The table name | | type | always `table` | | subtype | always `table` | | database\_technology | The data warehouse type | | database | The database name | | schema | The schema name | ## Relationships ### Container Relationships * **Sigma Account β†’ Workspaces**: Workspaces are top-level folders under the account * **Workspaces β†’ Folders**: Folders can be nested within workspaces * **Folders β†’ Subfolders**: Folders can contain other folders * **Folders β†’ Workbooks**: Workbooks belong to folders * **Sigma Account β†’ Data Models**: Data models are direct children of the account * **Workbooks β†’ Workbook Elements**: Workbook Elements belong to workbooks * **Data Model β†’ Data Model Elements**: Data Model Elements belong to workbooks ### Lineage Relationships | Source type(s) | Relationship | Target type(s) | Notes | | ------------------------ | ------------ | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `sigma_workbook` | has upstream | `sigma_workbook_element` | Workbook depends on its elements | | `sigma_workbook_element` | has upstream | `table`, `sigma_workbook_element`, `sigma_data_model_element` | Root elements reference warehouse tables; non-root reference other elements or data model elements | ### Column level lineage | Source type(s) | Relationship | Target type(s) | Notes | | -------------------------- | -------------------- | -------------- | ------------------------ | | `sigma_workbook_element` | has upstream\_fields | `column` | Root workbook elements | | `sigma_data_model_element` | has upstream\_fields | `column` | Root data model elements | # Sigma Impressions Euno can observe impression data for Sigma workbooks and elements by reading CSV exports that Sigma writes to cloud storage. This helps you identify popular workbooks, find underused content, and understand who is actively using Sigma. {% hint style="info" %} **Prerequisite**: Sigma Impressions requires the [Sigma integration](/sources/business-intelligence/sigma-integration) to also be set up. The Sigma integration discovers workbooks and elements in Euno; the Sigma Impressions integration then associates exported impression data with those existing resources. {% endhint %} For details on the usage properties observed (for example `total_impressions_14d` and `used`), see [Sigma Impressions Data](/using-euno/usage-optimization/sigma-impressions-data). ## Overview Sigma Impressions is an integration that periodically reads CSV files from a configured `gs://`, `s3://`, or `azure://` prefix, converts each row to an impression event, and aggregates those events onto Sigma workbooks and elements. This integration is intentionally separate from the main Sigma integration: * **Sigma** discovers the metadata and lineage for workbooks, data models, and elements. * **Sigma Impressions** reads usage exports and enriches the discovered Sigma resources with impression metrics. ## Step 1: Set up the Sigma integration in Euno If you have not already done so, follow the [Sigma integration setup guide](/sources/business-intelligence/sigma-integration) first. {% hint style="warning" %} Without the Sigma integration, Euno has no `sigma_workbook`or `sigma_workbook_element` resources to attach the impression data to. {% endhint %} ## Step 2: Configure Sigma to export usage data Sigma's audit-log export feature writes CSV files to cloud storage. Only a Sigma Admin can configure this. High-level workflow in Sigma: 1. Open the **Sigma Audit Logs** connection. 2. Save the relevant audit-log table or workbook page element as a workbook. 3. Configure a scheduled export to **Cloud Storage**. 4. Point that export at a dedicated prefix in **GCS**, **S3**, or **Azure Blob Storage**. 5. Make sure Sigma creates a new file on each run instead of overwriting previous files. Sigma documentation for this export flow: * [Export audit log data to cloud storage](https://help.sigmacomputing.com/docs/export-audit-log-data-to-cloud-storage) * [Manage an audit logs storage integration](https://help.sigmacomputing.com/docs/manage-an-audit-logs-storage-integration) * [Audit log events and metadata](https://help.sigmacomputing.com/docs/audit-log-events-and-metadata) ## Step 3: Prepare an Euno-compatible export Euno expects the exported CSV to contain the following columns: | Column | Required | Description | | ----------------- | -------- | ------------------------------------------------------------------------ | | `request_time` | Yes | Timestamp of the impression event | | `user_email` | Yes | Email of the Sigma user who viewed the resource | | `workbook_url_id` | Yes | Sigma workbook URL ID | | `element_id` | Optional | Sigma element ID, if the row should map to a specific element | | `count` | Optional | Number of impressions represented by the row. Defaults to `1` if omitted | Euno matches headers case-insensitively and also accepts a few common aliases, but using the exact names above is strongly recommended. ### Recommended workbook/export shape Create a Sigma workbook or view that produces one row per impression, with: * a timestamp column * a user email column * the workbook URL ID * the element ID when available * an optional count column if your export pre-aggregates rows If your export mixes workbook-level and element-level rows: * rows **with** `element_id` are attached to `sigma_element` * rows **without** `element_id` are attached to `sigma_workbook` ## Step 4: Grant Euno read access to the export location ### Option A: Google Cloud Storage Recommended approach: 1. Create or choose a dedicated bucket or prefix for Sigma audit-log exports. 2. Configure Sigma's audit-log storage integration to write to that location. 3. Grant **read access** on that bucket/prefix to Euno's runtime service account. At minimum, Euno needs permission to: * list objects in the configured prefix * read the exported CSV objects ### Option B: Amazon S3 Recommended approach: 1. Create or choose a dedicated bucket/prefix for the exports. 2. Configure Sigma's audit-log storage integration to write to that location. 3. Create a dedicated read-only IAM principal for Euno. 4. Give it least-privilege access to: * `s3:ListBucket` on the bucket (scoped to the export prefix) * `s3:GetObject` on the exported files 5. Store the credentials in the Euno Sigma Impressions integration. {% hint style="info" %} You can also use temporary AWS credentials (`aws_access_key_id`, `aws_secret_access_key`, `aws_session_token`) if that fits your security model better. {% endhint %} ### Option C: Azure Blob Storage Recommended approach: 1. Create or choose a dedicated container/prefix for the exports, and configure Sigma's audit-log storage integration to write to that location. 2. Give Euno read access using **one** of the following methods: **Method 1 β€” Container-scoped SAS token** (simplest): * In the Azure portal, open the container β†’ **Shared access tokens**. * Grant **Read** and **List** permissions, set a start/expiry, and click **Generate SAS token and URL**. * Copy the **Blob SAS token** (the `sv=...&sig=...` query string) β€” this is what you paste into Euno. {% hint style="info" %} A SAS token is a bearer credential with an expiry. To make it revocable without rotating the account key, generate it against a container **stored access policy**. {% endhint %} **Method 2 β€” Microsoft Entra service principal** (RBAC-based, no shared token): * In **Microsoft Entra ID β†’ App registrations**, register an app and create a **client secret** (copy the secret **Value**, shown only once). * Assign the app **Storage Blob Data Reader** on the container: container β†’ **Access Control (IAM) β†’ Add role assignment**. Creating role assignments requires **Owner** or **User Access Administrator** on the scope; role propagation can take a few minutes. * From the app's **Overview**, you'll need the **Directory (tenant) ID**, the **Application (client) ID**, and the client **secret Value**. At minimum, Euno needs permission to **list** objects in the configured prefix and **read** the exported CSV objects. ## Step 5: Add Sigma Impressions in Euno 1. Go to the **Sources** page in Euno. 2. Click **Add New Source** and select **Sigma Impressions**. 3. Configure: | Configuration | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | **Name** | A name for this integration (for example `Sigma Impressions - Production`) | | **Bucket URI** | Prefix where Sigma writes the CSV files: `gs://...`, `s3://...`, or `azure://.blob.core.windows.net//` | | **Storage provider** | Selects which credential fields are shown; inferred from the Bucket URI scheme | | **Days back to look** | How many days of files Euno should scan on each run. Maximum: `6` | | **AWS Access Key ID** | Optional. Needed only for `s3://` if Euno cannot use ambient AWS credentials | | **AWS Secret Access Key** | Optional. Needed only for `s3://` if Euno cannot use ambient AWS credentials | | **AWS Session Token** | Optional. Use when providing temporary AWS credentials | | **AWS Region** | Optional region override for S3 access | | **Azure auth method** | For `azure://`: choose **SAS token** or **Service principal (Microsoft Entra ID)** | | **Azure SAS token** | For Azure SAS auth: a container-scoped SAS token with read and list permissions | | **Azure tenant ID** | For Azure service principal: the app's **Directory (tenant) ID** | | **Azure client ID** | For Azure service principal: the app's **Application (client) ID** | | **Azure client secret** | For Azure service principal: the app's client secret **Value** | 4. Click **Test & Save**. Euno validates the bucket URI format and normalizes the storage prefix before saving. ## Step 6: Schedule the integration Sigma audit log exports are not real-time, and Sigma notes that fresh event data may take time to appear in the logs. Recommended schedule: * Run the Sigma export on a regular cadence. * Run **Sigma Impressions** shortly after the export is expected to land. * Keep **Days back to look** small (for example `2` or `3`) unless your export cadence requires more history. {% hint style="info" %} The integration enforces a maximum lookback of **6 days**. {% endhint %} ## Testing the setup For a simple validation: 1. Configure the export and send one test export from Sigma. 2. Confirm that new CSV files appear in the configured bucket prefix. 3. Run the Sigma Impressions integration in Euno. 4. Open a known Sigma workbook or element in Euno and confirm that impression properties appear. Good test data: * one workbook-level row * one element-level row * at least two distinct users * one row with `count > 1` if you plan to export pre-aggregated data ## Troubleshooting ### No impression data appears in Euno Check: * the main [Sigma integration](/sources/business-intelligence/sigma-integration) is active * the exported CSV includes `workbook_url_id` * the Sigma workbook exists in Euno * the bucket URI points to the correct prefix * Euno can list and read the exported files ### Workbook metrics appear, but element metrics do not Check: * the export includes `element_id` * the `element_id` matches Sigma element IDs discovered by the Sigma integration ### Euno cannot save the integration Check: * the bucket URI starts with `gs://`, `s3://`, or `azure://` * for S3, either both access key and secret key are provided, or both are left blank * for Azure, provide either a SAS token **or** a full service principal (tenant ID, client ID, and client secret); if both are supplied, the service principal is used ### Azure authentication fails Check: * **SAS token**: it has both **Read** and **List** permissions (List is required to enumerate the export files) and has not expired * **Service principal**: the **Azure tenant ID** is the app's **Directory (tenant) ID** and the **Azure client ID** is its **Application (client) ID** β€” these are easy to swap * **Service principal**: the app has the **Storage Blob Data Reader** role on the container (role changes can take a few minutes to take effect) ## Related * [Sigma integration](/sources/business-intelligence/sigma-integration) - Discovers Sigma workbooks, data models, and elements. * [Sigma Usage Data](/using-euno/usage-optimization/sigma-usage-data) - Warehouse-query usage from supported warehouses. * [Sigma Impressions Data](/using-euno/usage-optimization/sigma-impressions-data) - Impression properties observed by this integration. # Omni Euno's Omni integration supports auto-discovery of the following Omni resources: * Omni accounts (organizations) * Omni folders * Omni models (semantic layer containers for views and topics) * Omni views and topics * Omni dimensions and measures * Omni workbooks * Omni queries (tabs in workbooks) * Omni dashboards * Lineage to source tables and models * Users (when "Observe users" is enabled) * User groups (when "Observe groups" is enabled) {% hint style="info" %} **Usage data**: To observe usage (impressions, views) on Omni workbooks and dashboards, you must also set up the [Omni Usage integration](/sources/business-intelligence/omni-integration/omni-usage-setup). It is a separate push integration that receives usage data from your Omni instance. {% endhint %} ## What Euno discovers The Omni source discovers organization structure, semantic layer resources, workbooks and dashboards, and (optionally) users and groups. Lineage includes Omni-to-Omni dependencies and, when configured, warehouse table lineage for Snowflake-backed models. For property and relationship details, see [Omni Integration Discovered Resources](/sources/business-intelligence/omni-integration/omni-integration-discovered-resources). ## Setting up Omni integration ### Overview Omni is an analytics and business intelligence platform that enables users to explore, analyze, and visualize data directly in their cloud data warehouse. In Omni, data is organized into *workbooks* (with query tabs), *dashboards*, and *models*. Models contain views and topics that provide a semantic layer over your data warehouse tables. To discover Omni resources, Euno connects to the Omni API using Organization API keys. The integration automatically discovers your Omni organization structure, including folders, models, workbooks, and their lineage to underlying data warehouse tables. ### Step 1: Create Omni API Credentials #### Generate Organization API Key 1. Log in to your Omni instance as an **Organization Admin** 2. Navigate to **Settings** β†’ **API Access** β†’ **Organization keys** 3. Click **Generate new key** 4. In the modal, enter a descriptive name for the API key (e.g., "Euno Integration") 5. Click **Generate** 6. Copy the key immediately. **Note**: API keys are only displayed once when created. {% hint style="warning" %} **Important**: The API key is only shown once. Store it securelyβ€”you won't be able to retrieve it again. {% endhint %} For more details, see [Omni API authentication](https://docs.omni.co/api/authentication#creating-organization-api-keys). ### Step 2: Configure New Omni Source in Euno #### Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click on **Add New Source** and select **Omni** from the list of supported platforms. #### General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Name\* | Enter a name for your Omni source (e.g., "Omni - Analytics") | | Instance\* | Your Omni instance subdomain. For example, if your Omni URL is `https://mycompany.omniapp.co`, enter `mycompany`. | | API Key\* | The Organization API key from Step 1 | | Default Snowflake Account Identifier | The Snowflake account identifier (e.g., `wh51692.eu-central-1`) used to link Omni views to warehouse `table` resources for all Snowflake connections in this instance. You can also paste the full Snowflake host (e.g., `wh51692.eu-central-1.snowflakecomputing.com`); it is normalized to the account identifier. If your organization or account name contains underscores, enter the account identifier directly β€” Snowflake hostnames replace underscores with hyphens, so the identifier cannot be recovered from the host. Covers the common single-warehouse case. Leave empty for non-Snowflake instances; use per-connection overrides (see Advanced) when a connection uses a different warehouse. Requires a separate Snowflake source that observes those tables. | ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). {% hint style="info" %} **Recommended**: Schedule Omni integration to run daily or every 12 hours to keep workbook lineage and metadata up to date. {% endhint %} ### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 5: Advanced Settings (Optional) Click on the '**Advanced**' section to display these additional configurations. | Configuration | Description | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Observe users** | When enabled, Euno observes users from Omni via the SCIM API (`/api/scim/v2/users`). Default: enabled. Requires an Organization API key. | | **Observe groups** | When enabled, Euno observes user groups and memberships via the SCIM API (`/api/scim/v2/groups`). Default: disabled. Requires an Organization API key. | | **Ingest topic ai\_context as glossary terms** | When enabled, each Omni topic with non-empty `ai_context` creates a linked glossary term in the Universal domain. Euno refreshes its description and related resources on each crawl. Default: disabled. | | **Path filter** | Allow/deny regex patterns for Omni folder paths. Affects which folders and workbooks are discovered. Default deny: `omni-examples.*`. | | **Model filter** | Allow/deny regex patterns for Omni shared model names. Default deny: `Ecommerce Demo Data`. | | **Connection overrides** | Optional per-connection overrides of the Default Snowflake Account Identifier, for Omni connections that use a different Snowflake warehouse than the default. Enter the Omni connection id (listed under **Snowflake connections discovered** in each run report) and its Snowflake account identifier. Connections without an override fall back to the default target. Requires a separate warehouse source (for example Snowflake) that observes those tables. | For additional information regarding Users & Groups see [Users as a Resource](/developer-reference/technical-concepts/users-as-a-resource). #### Linked glossary terms from topic AI context Linked terms are marked **Managed by Omni** in the Data Glossary. Their descriptions come from the topic's `ai_context` and cannot be edited in Euno. You can still edit the term name, owner, domain, synonyms, tags, external links, and your related resources. Euno keeps the Omni-managed related-resource set separate, so a crawl refresh does not replace manual selections or a defining EQL. Euno links the topic, its dimensions and measures, and downstream Omni queries and dashboards. If a topic's `ai_context` becomes empty, the topic disappears, or you turn the setting off, Euno removes the linked term on the next crawl. If the topic name conflicts with a manually created term in the Universal domain, Euno keeps the manual term unchanged and reports the conflict in the crawl processing results. ### Step 6: Save Configuration Click the **Test & Save** button to complete the setup. Euno will: 1. Validate the instance URL is accessible 2. Authenticate using the provided API key 3. Verify access to the Omni organization 4. Save the configuration if all validations pass If the test fails, review the error message and verify your API key and instance subdomain are correct. # Omni Integration Discovered Resources ## Omni Integration Discovered Resources ### Overview The Omni integration discovers the following resources: * Omni Account * Omni Folder (including structural **Hub** and per-user **personal folder** containers) * Omni Model * Omni View * Omni Topic * Omni Dimension * Omni Measure * Omni Workbook * Omni Query * Omni Dashboard * Users (when "Observe users" is enabled) * User groups (when "Observe groups" is enabled) ### Omni Account A single resource of type `omni_account` is observed for each configured Omni source, representing the top-level Omni organization. #### Properties | Property | Description | | ---------------- | --------------------------------- | | `name` | The organization or instance name | | `type` | Always `omni_account` | | `subtype` | Always `omni_account` | | `external_links` | Link to open the Omni instance | ### Omni Folder Folders are organizational containers in Omni for organizing workbooks. Folders can be nested to create a hierarchy. Euno also observes structural **Hub** and per-user **personal folder** containers under the account so organization and restricted folder trees have a consistent parent in Euno. #### Properties | Property | Description | | ---------------- | ------------------------------------------------------------------------------------- | | `name` | The folder name | | `type` | Always `omni_folder` | | `subtype` | Always `omni_folder` | | `created_by` | The name of the user who created the folder | | `external_links` | Link to open the folder in Omni | | `members` | Users and/or user groups that have access to the folder (with optional role metadata) | ## Omni Semantic Layer ### Omni Model Models are semantic layers in Omni that provide business-friendly views of your data warehouse. They contain views and topics, and serve as the data source for workbooks and dashboards. #### Properties | Property | Description | | --------------------- | -------------------------------------------------------------- | | `name` | The model name | | `type` | Always `omni_model` | | `subtype` | Omni model kind (for example `SHARED`) | | `created_at` | The timestamp when the model was created | | `updated_at` | The timestamp when the model was last modified | | `database_technology` | Warehouse technology for the model connection (when available) | ### Omni View Views are defined within Omni models or workbooks and represent tables or logical structures with fields. They can reference other views or database tables. #### Properties | Property | Description | | --------- | ------------------ | | `name` | The view name | | `type` | Always `omni_view` | | `subtype` | Always `omni_view` | ### Omni Topic Topics are business-friendly abstractions within Omni models or workbooks that group related views and fields for analysis. #### Properties | Property | Description | | --------- | ------------------- | | `name` | The topic name | | `type` | Always `omni_topic` | | `subtype` | Always `omni_topic` | ### Omni Dimension Dimensions are dimensional fields within Omni views or topics, typically used for grouping and filtering in queries. #### Properties | Property | Description | | ------------- | ------------------------------------------------------------------ | | `name` | The dimension name (label when provided, otherwise the field name) | | `type` | Always `omni_dimension` | | `subtype` | Always `omni_dimension` | | `label` | Display label from Omni (when provided) | | `description` | Field description from Omni (when provided) | ### Omni Measure Measures are aggregatable fields within Omni views or topics, typically used for numeric summarization in queries. #### Properties | Property | Description | | ------------- | ---------------------------------------------------------------- | | `name` | The measure name (label when provided, otherwise the field name) | | `type` | Always `omni_measure` | | `subtype` | Always `omni_measure` | | `label` | Display label from Omni (when provided) | | `description` | Field description from Omni (when provided) | ## Omni Data Analysis Layer ### Omni Workbooks In Omni, workbooks are the main organization unit for queries. A workbook can either have a dashboard or not. In Euno, workbooks are modeled as containers whose children are query tabs and, at most, one dashboard. #### Properties | Property | Description | | ---------------- | ------------------------------------------------------------ | | `name` | The workbook name | | `type` | Always `omni_workbook` | | `subtype` | Always `omni_workbook` | | `description` | The workbook description (when provided) | | `updated_at` | The timestamp when the workbook was last modified | | `created_by` | Display name of the creating user (from Omni owner metadata) | | `external_links` | Link to open the workbook in Omni | | `tags` | Omni labels, excluding verified/certification labels | | `meta` | Certification metadata when Omni verified labels are loaded | ### Omni Query Queries are individual tabs within an Omni workbook. Each query tab represents a pivot-table or SQL-based analysis. #### Properties | Property | Description | | ---------------- | ------------------------------ | | `name` | The query tab name | | `type` | Always `omni_query` | | `subtype` | Always `omni_query` | | `external_links` | Link to open the query in Omni | ### Omni Dashboard Dashboards are visualizations in Omni that display charts, tables, and KPIs. They are observed as children of workbooks when the workbook has a dashboard. #### Properties | Property | Description | | ---------------- | ----------------------------------------------------------- | | `name` | The dashboard name | | `type` | Always `omni_dashboard` | | `subtype` | Always `omni_dashboard` | | `description` | The dashboard description (when provided) | | `external_links` | Link to open the dashboard in Omni | | `tags` | Omni labels, excluding verified/certification labels | | `meta` | Certification metadata when Omni verified labels are loaded | {% hint style="warning" %} Dashboard lineage (dashboard β†’ query) relies on an unstable Omni API endpoint (`/api/unstable/documents/{id}/export`). This endpoint may change or be removed by Omni in the future, which could temporarily affect dashboard lineage until an alternative is available. {% endhint %} ### Users (`user`) Users are discovered when "Observe users" is enabled in the Omni source configuration. | Property | Description | | --------------- | --------------------------------------------- | | `name` | User's email address (used as the identifier) | | `email_address` | User's email address | | `label` | User's display name from Omni | ### User Groups (`user_group`) User groups are discovered when "Observe groups" is enabled in the Omni source configuration. | Property | Description | | ----------- | --------------------------------------------- | | `name` | Display name of the group | | `native_id` | Group ID from Omni | | `subtype` | Always `omni` for groups discovered from Omni | ### Relationships #### Containment (`has parent` / `has child`) | Source type(s) | Relationship | Target type(s) | Notes | | -------------------------------- | ------------ | ------------------------- | ---------------------------------------------------- | | `omni_folder`, `omni_model` | has parent | `omni_account` | Includes structural **Hub** folder under the account | | `omni_folder`, `omni_workbook` | has parent | `omni_folder` | Organization and personal (restricted) folder trees | | `omni_view`, `omni_topic` | has parent | `omni_model` | Shared-model semantic resources | | `omni_view`, `omni_topic` | has parent | `omni_workbook` | Workbook-scoped semantic resources | | `omni_dimension`, `omni_measure` | has parent | `omni_view`, `omni_topic` | | | `omni_dimension` | has parent | `omni_query` | Query-level calculated fields | | `omni_query`, `omni_dashboard` | has parent | `omni_workbook` | A workbook has at most one dashboard child | #### Resource lineage (`has upstream` / `has downstream`) | Source type(s) | Relationship | Target type(s) | Notes | | ---------------- | ------------ | ------------------------- | ----------------------------------------------------------------------------------------------------------- | | `omni_model` | has upstream | `table` | When the model's warehouse tables are observed and Snowflake connection mapping is configured | | `omni_view` | has upstream | `table` | View SQL references warehouse tables | | `omni_workbook` | has upstream | `omni_model` | When linked to a backing shared model | | `omni_query` | has upstream | `omni_view`, `omni_topic` | Query tabs depend on semantic resources; warehouse tables are reached through the referenced views' lineage | | `omni_dashboard` | has upstream | `omni_query` | Dashboard tiles depend on query tabs | | `omni_topic` | has upstream | `omni_view` | Topics include joined/base views | Direct query-to-model lineage (`omni_query` β†’ `omni_model`) is not observed. #### Column-level lineage (`has upstream_fields` / `has downstream_fields`) | Source type(s) | Relationship | Target type(s) | Notes | | -------------------------------- | -------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | `omni_dimension`, `omni_measure` | has upstream\_fields | `column`, `omni_dimension`, `omni_measure` | View fields point at warehouse columns when observed; topic-scoped fields point at their definition-view fields | | `omni_query` | has upstream\_fields | `omni_dimension`, `omni_measure` | Including query-level calculated fields | #### User and group relationships (`has group` / `has member`) | Source type(s) | Relationship | Target type(s) | Notes | | -------------- | ------------ | -------------------- | -------------------------------------------------------------- | | `omni_folder` | has group | `user`, `user_group` | From folder permission metadata | | `user_group` | has member | `user` | From SCIM group memberships when **Observe groups** is enabled | {% hint style="info" %} The Omni crawler does **not** currently observe model-level user or group memberships. {% endhint %} {% hint style="info" %} `is used by` relationships on `omni_workbook` and `omni_dashboard` come from the separate [Omni Usage integration](/sources/business-intelligence/omni-integration/omni-usage-setup), not from the Omni crawler itself. See [Omni Usage Data](/using-euno/usage-optimization/omni-usage-data). {% endhint %} # Omni Usage Setup Euno can observe usage data for Omni workbooks and dashboardsβ€”how often they are viewed and by whom. This information helps you identify popular resources, find unused dashboards to archive, and understand adoption patterns. {% hint style="info" %} **Prerequisite**: Omni Usage requires the [Omni integration](/sources/business-intelligence/omni-integration) to also be set up. The Omni integration discovers workbooks and dashboards in Euno; the Omni Usage integration then associates usage data with those resources. Without the Omni integration, there are no resources to attach usage to. {% endhint %} For details on the usage properties observed (e.g. `total_impressions_14d`, `used`), see [Omni Usage Data](/using-euno/usage-optimization/omni-usage-data). ## Overview Omni Usage is a **push integration**: your Omni instance (or an external system) sends usage data to Euno via a webhook. Euno stores the events and a global processor aggregates them into usage observations on workbooks and dashboards. ## Step 1: Add Omni Usage Integration in Euno 1. Go to the **Sources** page in Euno. 2. Click **Add New Source** and select **Omni Usage**. 3. Configure: Asterisk (\*) means a mandatory field. | Configuration | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name\*** | A name for this source (e.g. "Omni Usage - Analytics"). | | **Instance\*** | Your Omni instance subdomain. Must match the [Omni source](/sources/business-intelligence/omni-integration) (e.g. `mycompany` for `mycompany.omniapp.co`). | 4. Click **Test & Save**. After the source is created, Euno shows a **webhook URL** with an embedded secret. Copy this URL β€” you will configure it as the Omni Analytics delivery destination. ## Step 2: Configure Omni to Push Usage Data Configure your Omni instance or an external export process to send usage data to the **webhook URL** from Step 1. Only an Omni Account Admin can complete these steps. {% stepper %} {% step %} #### Open Omni Analytics embedded interface Login to your Omni instance and locate the Analytics interface.\ ![](/files/lmZdfLjMXZqWFkjTK2kG) {% endstep %} {% step %} #### Create a new workbook in Omni Analytics embedded interface Navigate to the Omni Analytics embedded interface side menu
Pick "Hub". This is the Hub of the Omni Analytics embedded interface. It's distinct from the "Hub" of your Omni instance.
Click "New" to create a new Omni workbook.
{% endstep %} {% step %} #### Create the Usage query to be pushed to Euno Pick the `Content Curated` Omni Topic. This will be the topic used by the Query.
Pick the following fields to be included in the Query, the order of the fields doesn't matter: | Omni View | Field | | ----------------- | ----------------------- | | Document Views | Timestamp -> Date | | Document Views | Dashboard or Workbook? | | Document Views | Document View Count | | Document Consumer | Display Name (Consumer) | | Workbooks | Identifier | If you picked the right fields, your table should look like:
Sort the table by Timestamp Date, in descending order:
{% endstep %} {% step %} #### Save the workbook Click File -> Save... Pick any folder and name to save the workbook in. Remember that the folder structure is not your organization standard folder structure. This is the Omni Analytics embedded interface. To keep things organized, we recommend selecting your organization under the "Hub", and naming the workbook "Omni usage data for Euno integration"
{% endstep %} {% step %} #### Create a Dashboard for this workbook
{% endstep %} {% step %} #### Publish the workbook
{% endstep %} {% step %} #### Configure Omni Analytics to send usage information to Euno on a schedule Click File -> "Deliveries & Alerts"
This will open a side menu to configure a scheduled delivery of the query we configured to Euno. Pick, from top to bottom: Delivery: Schedule Send: Query Destination: Webhook Delivery name: query
Configure The Schedule of delivery. We recommend keeping the defaults.
Under "Chart", pick Format: JSON Make sure to mark "Send all possible results"
Under "Webhook", set: URL: the webhook URL you copied from Step 1 File Name: "query"
To test, click "Test now". To save, click "Save" {% endstep %} {% endstepper %} ## Related * [Omni integration](/sources/business-intelligence/omni-integration) – Set up the Omni integration to discover workbooks and dashboards. * [Omni Usage Data](/using-euno/usage-optimization/omni-usage-data) – Properties and metrics observed for Omni resources. # Cube Core Euno's Cube Core integration ingests a zip of Cube schema YAML from self-hosted Cube Core deployments. Euno discovers cubes, views, measures, and dimensions, and enriches them with lineage to warehouse tables and columns. {% hint style="info" %} This integration is push-only: upload your Cube `model/` tree as a zip file. Euno does not call Cube APIs. {% endhint %} ## Limitations Euno currently supports Cube Core deployments connected to **Snowflake** only. Other warehouses, Cube Cloud, and JavaScript schema files are not supported. ## What Euno discovers * **Cube deployment** β€” optional container when **Instance slug** is set * **Cubes** β€” semantic models over warehouse tables * **Views** β€” consumer-facing projections over cubes * **Dimensions and measures** β€” under cubes or views * **Lineage** β€” warehouse table and column dependencies, cube-to-cube joins, and view member aliases For property and relationship details, see [Cube Core Integration Discovered Resources](/sources/business-intelligence/cube-core-integration/cube-core-integration-discovered-resources). ## Prerequisites Before setting up a Cube Core source in Euno: 1. **Cube schema YAML** β€” Export or package your Cube `model/` directory as a `.zip` file. Files must be under `model/**/*.yml` or `model/**/*.yaml` and contain `cubes:` and/or `views:` at the root of each document. 2. **Jinja stubs (if needed)** β€” If your `sql_table` values use `COMPILE_CONTEXT` Jinja, configure **Security context** and **Custom context** in the source so Euno can render table names offline. {% hint style="info" %} Upload the complete `model/` tree when your deployment has joins or views that reference cubes defined in other YAML files. Partial uploads produce incomplete graphs. {% endhint %} ## Setting up a Cube Core source in Euno ### Step 1: Access the Sources page 1. Navigate to the **Sources** page in Euno. 2. Click **Add New Source** and select **Cube Core**. ### Step 2: General configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name**\* | Display name for this source (for example, `Cube Core - Production`). | | **Warehouse technology**\* | Warehouse type for the default Cube data source. | | **Snowflake host**\* | Warehouse host or account locator for the default data source (for example, `JGB69604.us-west-2.aws` or `account.region.snowflakecomputing.com`). | | **Default database** | Default database for unqualified `sql_table` names on the default Cube data source. | | **Default schema** | Default schema for unqualified table names on the default Cube data source. | | **Instance slug** | URI namespace for cube resources. When set, Euno also creates a `cube_deployment` container for this instance. | | **Instance display name** | Optional label for the deployment container. Defaults to the instance slug. | ### Step 3: Resource cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation, see [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Time-Based Cleanup (default)**: Remove resources that were last detected a number of days before the most recent successful source integration run (default is 7 days). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 4: Advanced settings (optional) Click **Advanced** to configure additional options: | Configuration | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Data source mapping** | Map named Cube `data_source` values to warehouse hosts and default database/schema. Required for every non-default `data_source` referenced in your schema. | | **Cube pattern** | Include or exclude cubes by name using regular expressions. Defaults to including all cubes. | | **View pattern** | Include or exclude views by name using regular expressions. Defaults to including all views. | | **Security context** | Stub values for offline Jinja rendering of `COMPILE_CONTEXT.securityContext`. Supported keys: `tenant`, `deployment_id`, `team`, `environment`. | | **Custom context** | Additional top-level `COMPILE_CONTEXT` keys for offline Jinja rendering (string values). | Each **data source mapping** entry includes: | Field | Description | | ------------------------ | ------------------------------------------------------------------------- | | **Warehouse technology** | Warehouse type for this named data source. | | **Snowflake host** | Warehouse host for cubes that use this named data source. | | **Default database** | Default database for unqualified `sql_table` names on this data source. | | **Default schema** | Default schema for unqualified table names on this data source. | | **Security context** | Optional override of the top-level security context for this data source. | | **Custom context** | Optional override of the top-level custom context for this data source. | ### Step 5: Save configuration Click **Save**. Euno opens a modal with: * Your **trigger secret** β€” copy and store it securely; it is shown only once * The **`prepare-upload` endpoint URL** β€” for programmatic zip uploads Use the trigger secret as a Bearer token in the `Authorization` header for `prepare-upload` calls. **Rotating your trigger secret:** Open your Cube Core source on the **Sources** page and click **Reset Trigger Key** to generate a new secret. This invalidates the previous secret immediately. ## Uploading Cube schema YAML ### Preparing your upload Package your Cube model tree as a single `.zip` file: * Include YAML files under `model/**/*.yml` or `model/**/*.yaml` * Each file must parse as YAML with a mapping root and contain a `cubes:` and/or `views:` section * Upload exactly one `.zip` file per run ### Upload methods #### Manual upload via UI The easiest way to upload Cube schema YAML is through Euno's UI: 1. Open your Cube Core source on the **Sources** page. 2. Click **Upload Artifacts**. 3. Select your `.zip` file and start the run. Euno parses the schema, emits Cube resources, and runs lineage enrichment to warehouse tables and columns. Check the run report for inventory counts, skipped files, and Jinja render results. #### Programmatic upload (recommended for production) For production workflows, upload your zip after Cube schema changes using the **`prepare-upload` endpoint**: 1. **Request a signed upload URL** β€” `POST` to `.../integrations/YOUR_INTEGRATION_ID/prepare-upload` with the zip filename and `Authorization: Bearer YOUR_TRIGGER_SECRET` 2. **Upload the zip** β€” `PUT` the file to the signed URL from the response 3. **Processing starts automatically** β€” no further API call is required See reference implementations below: **Python script** Use a standalone Python script to zip and upload your Cube model tree [β†’ View Python Upload Guide](/sources/zip-artifact-python-upload) **GitHub Actions** Automate uploads when Cube schema YAML changes in your repository [β†’ View GitHub Actions Guide](/sources/business-intelligence/cube-core-integration/github-actions-upload) {% hint style="info" %} Euno performs offline Jinja2 rendering for `sql_table` values that contain `COMPILE_CONTEXT` references. Rendering uses only `COMPILE_CONTEXT` assembled from **Security context** and **Custom context**β€”control-flow constructs such as `{% if %}` work when they depend on those stub values. Cube runtime helpers and macros (for example `env_var()` or functions from `model/globals.py`) are not available, and query-compilation variables such as `{CUBE}` are not evaluated. {% endhint %} # Cube Core Integration Discovered Resources ## Overview The Cube Core integration discovers the following resources: * Cube deployment (when **Instance slug** is configured) * Cube * Cube view * Cube dimension * Cube measure ## Cube deployment A single resource of type `cube_deployment` is observed when **Instance slug** is set on the source. It groups all cubes and views for that Cube instance. ### Properties | Property | Value | | ----------- | ----------------------------------------------------------------------- | | `type` | always `cube_deployment` | | `name` | Instance display name, or the instance slug when no display name is set | | `native_id` | Configured instance slug | ## Cube Cubes are semantic models defined in Cube schema YAML. Each cube typically maps to a warehouse table through `sql_table`. ### Properties | Property | Value | | ------------------------- | --------------------------------------------------------------------- | | `type` | always `cube_cube` | | `name` | Cube title from schema, or cube name when no title is set | | `native_id` | Cube name from schema | | `description` | Cube description when present in schema | | `raw_code` | YAML fragment for this cube | | `database_technology` | Resolved warehouse technology | | `database_database` | Default database used to qualify unqualified `sql_table` names | | `database_schema` | Default schema used to qualify unqualified table names | | `sql_dialect` | SQL dialect for the resolved warehouse | | `no_compiled_code_reason` | Reason warehouse table lineage could not be resolved, when applicable | Lineage to warehouse tables and joined cubes appears under **Lineage** in the resource sidepane, not as named fields in resource details. ### Relationships * **Parent**: `cube_deployment` (only when **Instance slug** is configured) ## Cube view Views expose subsets or aliases of cube members to consumers. View-scoped dimensions and measures are emitted under the view, not the underlying cube. ### Properties | Property | Value | | ------------- | --------------------------------------------------------- | | `type` | always `cube_view` | | `name` | View title from schema, or view name when no title is set | | `native_id` | View name from schema | | `description` | View description when present in schema | | `raw_code` | YAML fragment for this view | Lineage to underlying cubes appears under **Lineage** in the resource sidepane. ### Relationships * **Parent**: `cube_deployment` (only when **Instance slug** is configured) ## Cube dimension Dimensions are observed under their parent cube or view. ### Properties | Property | Value | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `type` | always `cube_dimension` | | `name` | Dimension title from schema, or dimension name when no title is set | | `native_id` | Fully qualified member id (`{cube_name}.{dimension_name}` for cube-scoped members, or view-qualified id for view-scoped members) | | `description` | Dimension description when present in schema | | `raw_code` | YAML fragment for this dimension | | `no_compiled_code_reason` | Reason column-level lineage could not be resolved, when applicable | Field-level lineage to warehouse columns and other cube members appears under **Lineage** in the resource sidepane. ### Relationships * **Parent**: `cube_cube` or `cube_view` ## Cube measure Measures are observed under their parent cube or view. ### Properties | Property | Value | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `type` | always `cube_measure` | | `name` | Measure title from schema, or measure name when no title is set | | `native_id` | Fully qualified member id (`{cube_name}.{measure_name}` for cube-scoped members, or view-qualified id for view-scoped members) | | `description` | Measure description when present in schema | | `raw_code` | YAML fragment for this measure | | `no_compiled_code_reason` | Reason column-level lineage could not be resolved, when applicable | Field-level lineage to warehouse columns and other cube members appears under **Lineage** in the resource sidepane. ### Relationships * **Parent**: `cube_cube` or `cube_view` ## Relationships ### Parent / child | Source type(s) | Relationship | Target type(s) | Notes | | -------------------------------- | ------------ | ------------------------ | -------------------------------------------------------- | | `cube_cube`, `cube_view` | has parent | `cube_deployment` | Only when **Instance slug** is configured on the source. | | `cube_dimension`, `cube_measure` | has parent | `cube_cube`, `cube_view` | Members are children of their cube or view. | ### Lineage | Source type(s) | Relationship | Target type(s) | Notes | | -------------------------------- | -------------------- | -------------------------------- | --------------------------------------------------------------------------------------- | | `cube_cube` | has upstream | `table` | Warehouse table from `sql_table` on the cube. | | `cube_cube` | has upstream | `cube_cube` | Joined cubes from `joins` in cube schema. | | `cube_view` | has upstream | `cube_cube` | Underlying cubes referenced by the view. | | `cube_dimension`, `cube_measure` | has upstream\_fields | `column` | Warehouse columns referenced in member SQL, filters, or case branches. | | `cube_dimension`, `cube_measure` | has upstream\_fields | `cube_dimension`, `cube_measure` | Same-cube member references in member SQL, or underlying cube members for view aliases. | {% hint style="info" %} When **Instance slug** is not set, cubes and views use the `default` URI namespace and do not have a `cube_deployment` parent. {% endhint %} # GitHub Actions Upload This workflow zips your Cube `model/` tree and uploads it to Euno after schema changes land in your repository. ## Prerequisites * GitHub repository containing your Cube Core schema YAML under `model/` * Euno **trigger secret** for your Cube Core source, stored as a GitHub secret ## Setup ### 1. Add GitHub Secrets In your GitHub repository, add the following secrets (Settings β†’ Secrets and variables β†’ Actions): * `EUNO_INTEGRATION_KEY`: Your Cube Core source **trigger secret** (Bearer token for uploads) * `EUNO_ENDPOINT_URL`: Your Euno `prepare-upload` endpoint URL (for example, `https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload`) ### 2. Create Workflow File Create `.github/workflows/cube-core-euno-upload.yml` in your repository: ```yaml name: Cube Core Upload to Euno on: push: branches: [ main, master ] paths: - 'model/**' workflow_dispatch: jobs: upload-cube-schema: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Create and upload Cube model zip run: | set -euo pipefail if [ ! -d model ]; then echo "Error: model/ directory not found" exit 1 fi shopt -s globstar nullglob yaml_files=(model/**/*.yml model/**/*.yaml) if [ ${#yaml_files[@]} -eq 0 ]; then echo "Error: no YAML files found under model/" exit 1 fi echo "Creating zip file with Cube schema YAML..." zip -r cube-model.zip model/ echo "Requesting signed upload URL from Euno..." prepare_response=$(curl -s -w "\n%{http_code}" \ -X POST \ -H "Authorization: Bearer ${{ secrets.EUNO_INTEGRATION_KEY }}" \ -H "Content-Type: application/json" \ -d '{"filename": "cube-model.zip"}' \ "${{ secrets.EUNO_ENDPOINT_URL }}") prepare_status_code=$(echo "$prepare_response" | tail -n1) prepare_response_body=$(echo "$prepare_response" | head -n -1) echo "Prepare-upload status code: $prepare_status_code" if [ "$prepare_status_code" -ne 200 ]; then echo "Failed to get signed URL with status code: $prepare_status_code" echo "Response: $prepare_response_body" exit 1 fi upload_url=$(echo "$prepare_response_body" | jq -r '.upload.url // empty') if [ -z "$upload_url" ] || [ "$upload_url" = "null" ]; then echo "Failed to extract upload URL from response" echo "Response: $prepare_response_body" exit 1 fi echo "Signed URL obtained" echo "Uploading Cube model zip..." upload_response=$(curl -s -w "\n%{http_code}" \ -X PUT \ -H "Content-Type: application/zip" \ --data-binary @cube-model.zip \ "$upload_url") upload_status_code=$(echo "$upload_response" | tail -n1) upload_response_body=$(echo "$upload_response" | head -n -1) echo "Upload status code: $upload_status_code" if [ "$upload_status_code" -eq 200 ] || [ "$upload_status_code" -eq 201 ]; then echo "Cube model uploaded successfully" else echo "Upload failed with status code: $upload_status_code" echo "Response: $upload_response_body" exit 1 fi - name: Upload zip on failure if: failure() uses: actions/upload-artifact@v4 with: name: cube-model-zip path: cube-model.zip retention-days: 3 ``` ## Configuration Notes ### Model directory layout Euno expects YAML files under `model/**/*.yml` or `model/**/*.yaml`. Upload the complete `model/` tree when your deployment has joins or views that reference cubes defined in other files. ### Path filters The example workflow runs only when files under `model/` change. Remove the `paths` filter if you want every push to main to upload, or adjust it to match your repository layout. ### Required GitHub Secrets * `EUNO_INTEGRATION_KEY` β€” your Cube Core source **trigger secret** (Bearer token) * `EUNO_ENDPOINT_URL` β€” your Cube Core source `prepare-upload` endpoint URL ## Workflow Triggers The workflow runs on: * **Push to main/master** when `model/**` changes * **Manual trigger** β€” use the **Actions** tab to run on demand ## Related guides * [Cube Core integration setup](/sources/business-intelligence/cube-core-integration) * [Python Upload for Zip Artifacts](/sources/zip-artifact-python-upload) β€” standalone script using the same `prepare-upload` flow # Salesforce Salesforce CRM is a cloud platform for sales, service, and marketing. The underlying metadata (objects, fields, folders, reports, dashboards, and flows) is what teams build analytics and integrations onβ€”so observing that structure helps you understand impact, ownership, and relationships across the org. Euno integrates with Salesforce by **inventorying metadata and lineage**, not by running data-quality monitors against table rows (such as freshness or volume anomaly detection on queried data). This page describes **what Euno captures** and how to **connect using OAuth 2.0 client credentials** via Salesforce’s **External Client App** model (consumer key, consumer secret, and My Domain). Euno's Salesforce integration supports auto-discovery of: * Salesforce Organization (`salesforce_org`) * **`user`** resources (when **Observe users & groups** is enabled) * **`user_group`** resources (when **Observe users & groups** is enabled) * Salesforce Folders (reports and dashboards) * Salesforce Objects (standard and custom, subject to your patterns) * Salesforce Fields (standard, custom, formula, relationships) * Salesforce Reports * Salesforce Dashboards * Salesforce Flows (when **Include flows** is enabled) ## Capability overview Euno’s Salesforce source emphasizes **discovery and graph relationships**, not running monitors against live Salesforce records: | Area | What Euno does today | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | **Metadata** | Discovers organizations, folders, objects, fields, reports, dashboards, and flows | | **Lineage-style relationships** | Links dashboards β†’ reports; reports and Flows β†’ objects/fields; folders β†’ contents; objects β†’ fields; optional users/groups β†’ ownership | | **Data monitors** | Not applicableβ€”Euno does not run freshness/volume/count monitors on Salesforce data by default | {% hint style="info" %} For **sandbox** vs **production**, use the **My Domain URL** that Salesforce shows for that org (sandbox hostnames typically differ from prod). Paste the full host or the prefix described below; Euno normalizes common formats. {% endhint %} ## Prerequisites / Requirements You need: 1. **An External Client App** in Salesforce with **OAuth 2.0 client credentials flow** enabled and an **execution user** (Salesforce acts on behalf of that user). 2. **Consumer key**, **consumer secret**, and **My Domain** hostname (see setup below). 3. For the execution user profile/permission sets: rights sufficient to describe metadata and read report/dashboard structure over the REST and Analytics APIs, and (**if you enable Flow discovery**) to query **Flow** metadata over the **Tooling API**. ### Recommended permissions on the execution user Grant the Salesforce user designated as **execution user** (or equivalent profile) at least: * **API Enabled** β€” required for REST access * **View Setup and Configuration** β€” metadata for objects/setup * **Run Reports** β€” report metadata/analytics endpoints as needed by your org * **View All Data** *(recommended)* β€” consistent visibility across objects in large orgs; tighten only if policy requires a scoped integration account * **Tooling API** β€” for Flow definition reads; omit if **Include flows (Tooling API)** is turned off in Euno. {% hint style="info" %} Many teams create a dedicated **integration / service** user whose only job is powering API clients, assign it the External Client App’s **client credentials execution user**, and grant the minimum workable permissions. {% endhint %} ## Stage 1: Configure Salesforce for API access For many Salesforce orgs, long-lived integrations should use machine-to-machine access instead of **username**, **password**, and **security token** against the legacy login APIs. **Euno uses OAuth client credentials** with an External Client App, as Salesforce documents for secured API access patterns. Consult [Salesforce Help](https://help.salesforce.com/) for your edition’s latest steps (β€œExternal Client App,” β€œOAuth,” β€œclient credentials,” β€œexecution user”)β€”menu names vary by release. ### Step 1: Create an External Client App Adapt these steps to your Salesforce UI wording (menus change between releases): 1. In Salesforce Setup, open **Apps** β†’ **External Client Apps** (or equivalent) β†’ **External Client App Manager**. 2. **New External Client App** and fill basic information (name, API name, distribution state appropriate for internal use). ### Step 2: Enable OAuth and select scopes Under the app’s **API / OAuth** settings: 1. Enable **OAuth**. 2. Under **Selected OAuth Scopes**, add **Manage user data via APIs (`api`)** (move it from Available to Selectedβ€”Available alone is not enough). Do **not** use **Full access (`full`)**β€”it is too broad for Euno. Scopes such as **`web`** and **`refresh_token`** are also not supported for client credentials (Winter ’26 and later). Euno does not send a `scope` parameter or use refresh tokensβ€”Salesforce uses whatever is in **Selected OAuth Scopes**. If only unsupported scopes are selected, **Test & Save** fails with `invalid_grant: no valid scopes defined`. 3. Under **Flow enablement**, turn on **Client credentials flow**. 4. Salesforce may still require registering a **callback URL** for the Connected / External Client Appβ€”even when you only use client credentialsβ€”use an HTTPS URL permitted by your org’s policy if prompted. Euno does **not** rely on an interactive browser redirect after client credentials are configured. ### Step 3: Set the execution user (Run As) The client credentials flow has no interactive login, but Salesforce still requires a **Run As** user whose permissions apply to every API call Euno makes. 1. In **Setup β†’ App Manager** (or **External Client App Manager**), open your app and click **Manage** (not the main **Edit** screen). 2. Click **Edit Policies**. 3. Under **Client Credentials Flow**, use the **Run As** lookup (labeled **Client Credentials Flow** / **Run As**β€”not a separate β€œRun As” field elsewhere on the page, such as under a custom handler). 4. Select your integration user (see [recommended permissions](#recommended-permissions-on-the-execution-user)). 5. Ensure **Enable Client Credentials Flow** is allowed under the app’s OAuth policies, then **Save**. {% hint style="warning" %} If **Run As** is left blank, **Test & Save** in Euno fails with `invalid_grant: no client credentials user enabled`. Enabling client credentials under OAuth settings alone is not enoughβ€”you must assign **Run As** under **Edit Policies**. {% endhint %} ### Step 4: Consumer key and consumer secret 1. On the app, open **Settings** (or the credentials pane). 2. **Consumer Key / Client ID** and **Consumer Secret** β€” copy both into Euno (treat the **secret like a password**). ### Step 5: My Domain (hostname) 1. Setup β†’ **Company Settings** β†’ **My Domain** (names vary). 2. Copy **Current My Domain URL**, e.g. `https://acme.my.salesforce.com`. In Euno you can paste either: * The **full URL**, or * The **prefix** used before `.salesforce.com` (example: `acme.my`) Euno strips `https://` and `.salesforce.com` automatically so tokens are requested against the correct host. {% hint style="warning" %} Do **not** use the legacy **`login`** / **`test`** β€œenvironment” shorthand as the sole domain string for OAuth client credentials; use the My Domain host Salesforce gives you (sandbox orgs still have their own My Domain URL). {% endhint %} ### Checklist: verify Salesforce configuration Before continuing to Euno, confirm the following on your External Client App: | Salesforce setting | Required value | | ----------------------- | -------------------------------------------------------------------- | | Client credentials flow | Enabled | | Selected OAuth Scopes | **Manage user data via APIs (`api`)** β€” not **Full access (`full`)** | | Run As user | Set under **Manage β†’ Edit Policies β†’ Client Credentials Flow** | | Consumer key / secret | Copied from this app | | My Domain | Your org’s My Domain host (not `login` / `test`) | ## Stage 2: Configure the Salesforce source in Euno ### Step 1: Access the Sources page In Euno, open **Sources**, choose **Add New Source**, and select **Salesforce**. ### Step 2: General configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------------- | ------------------------------------------------------- | | **Name**\* | Friendly name for this source | | **Consumer key**\* | External Client App OAuth **Consumer Key / Client ID** | | **Consumer secret**\* | External Client App **Consumer Secret** | | **My Domain host**\* | From My Domain (full URL or `your-org.my` style prefix) | ### Step 3: Schedule Enable **Schedule** and pick a cadence that fits how often your org’s metadata changes. **Daily** or every **12–24 hours** is often enough for metadata; increase frequency if your org changes structure often. {% hint style="info" %} **Recommended**: Schedule the Salesforce integration at least daily so folder, report, and dashboard metadata stays reasonably current. {% endhint %} ### Step 4: Resource cleanup Choose a **Resource cleanup** policy the same way as for other Euno sources (for example immediate cleanup vs retaining stale resources). See [Resource sponsorship and cleanup](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). ### Step 5: Save Configuration Click **Test & Save** to complete the setup. Euno will: 1. Exchange OAuth client credentials against your My Domain host. 2. Verify REST API access to your Salesforce organization. 3. Save the configuration if validation passes. If the test fails, see [Troubleshooting](#troubleshooting) below. ## Troubleshooting When **Test & Save** fails, Euno surfaces the Salesforce OAuth error from the token endpoint. Common messages and fixes: | Error | Likely cause | Fix | | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `invalid_grant: no client credentials user enabled` | No **Run As** user under **Edit Policies β†’ Client Credentials Flow** | Open **App Manager β†’ Manage β†’ Edit Policies**, set **Run As** to your integration user, and save. See [Step 3](#step-3-set-the-execution-user-run-as). | | `invalid_grant: no valid scopes defined` | App has no supported OAuth scopes for client credentials (often **Full access (`full`)**, `web`, or `refresh_token` selected instead of **`api`**) | In **Selected OAuth Scopes**, add **Manage user data via APIs (`api`)** and remove unsupported scopes such as **`full`**. See [Step 2](#step-2-enable-oauth-and-select-scopes). | | `invalid_grant: authentication failure` / invalid client credentials | Wrong consumer key or secret, or credentials from a different app or org | Copy **Consumer Key** and **Consumer Secret** from the same External Client App you configured. Sandbox and production each need their own app and credentials. | | Authentication succeeds but crawl is incomplete | Integration user lacks object, report, or Tooling API access | Grant [recommended permissions](#recommended-permissions-on-the-execution-user). Disable **Include flows** if Tooling access is not available. | Also verify: * **My Domain host** matches the org whose consumer key and secret you entered (not `login` or `test`). See [Step 5](#step-5-my-domain-hostname). * Sandbox and production are configured separatelyβ€”credentials and app settings from one org do not work in the other. ## What Euno discovers The Salesforce source discovers: * Salesforce Organization (`salesforce_org`) * **`user`** resources (when **Observe users & groups** is enabled) * **`user_group`** resources (when **Observe users & groups** is enabled) * Salesforce Folders (reports and dashboards) * Salesforce Objects (standard and custom, subject to your patterns) * Salesforce Fields (standard, custom, formula, relationships) * Salesforce Reports * Salesforce Dashboards * Salesforce Flows (when **Include flows (Tooling API)** is enabled) Details of resource types, properties, and relationships: [Salesforce integration discovered resources](/sources/business-intelligence/salesforce-integration/salesforce-integration-discovered-resources). ## Advanced settings Open the **Advanced** section on the source form to configure optional filters and scopes. | Configuration | Description | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Object filter** | Allow/deny regexes matched against Salesforce **object API names** (e.g. `Account`, `Custom__c`) | | **Report filter** | Allow/deny regexes matched against the report’s **display Name** only (not Id or DeveloperName) | | **Dashboard filter** | Allow/deny regexes matched against the dashboard’s **Title** only (not Id or DeveloperName) | | **Only crawl custom objects** | Limits discovery to objects ending in `__c` | | **Include reports** | Discover reports (default: on); combined with report filter patterns | | **Include dashboards** | Discover dashboards (default: on); combined with dashboard filter patterns | | **Include flows (Tooling API)** | Discover Flow definitions and version metadata for lineage (default: on); requires Tooling/API permissionsβ€”disable if your connected app cannot read Flow metadata | | **Observe users & groups** | Emit users/groups for ownership linkage (default: on) | | **Ingest usage metrics** | Ingest report runs, dashboard views, object API reads/writes, and distinct-user counts from Salesforce **Event Monitoring** log files (default: off). See [Usage metrics](#usage-metrics-event-monitoring) for prerequisites. | ## Usage metrics (Event Monitoring) When **Ingest usage metrics** is enabled, Euno downloads Salesforce **Event Monitoring** log files (the `EventLogFile` object: `Report`, `Dashboard`, `API`, `RestApi`, `BulkApi`, and `BulkApi2` event types) each crawl and computes rolling usage windows: * **Reports & dashboards** β€” report runs and dashboard views populate `total_impressions_14d/30d/60d` and `distinct_impressions_users_14d/30d/60d`, shown in the resource's **Impressions** usage panel. * **Objects** β€” object API reads/writes populate `total_read_queries_14d/30d/60d` and `total_write_queries_14d/30d/60d`, shown in the resource's **Table activities** usage panel. Independently of Event Monitoring, reports also get **`last_accessed_at`** from the report's Salesforce **last run** timestamp. ### Prerequisites and limitations * **Event Monitoring availability** depends on your Salesforce edition: it is a paid add-on (part of Salesforce Shield) on Enterprise, Unlimited, and Performance editions. Without it, usage ingestion is skipped and the run report notes whyβ€”the rest of the crawl is unaffected. * **Retention-aware windows**: Salesforce retains event log files for a limited period (typically 30 days with the add-on, up to 1 year with extended retention). Euno accumulates daily usage history across crawls in its own store, so windows longer than your Salesforce retention fill up over time as the integration keeps running. Schedule the integration **at least daily** so no log days are missed. * Usage is attributed only to reports, dashboards, and objects that are discovered by the crawlβ€”assets excluded by filters do not accumulate usage. * The execution user needs **View Event Log Files** (or **View All Data**) and **API Enabled** permissions to query `EventLogFile`. ## Filtering objects, reports, and dashboards You can narrow crawl scope using: * **Object filter** plus **Only crawl custom objects** β€” control which **sObjects** and their **fields** are observed. * **Report filter** β€” include or exclude reports by **Name** regex. * **Dashboard filter** β€” include or exclude dashboards by **Title** regex. Align those patterns with your governance policy; exclusions are governed by your configuration. Reports or dashboards excluded by regex **do not** appear as resourcesβ€”downstream lineage that would have depended only on those assets will be absent for them. ## Lineage relationships Among other edges, Euno captures: * **Dashboard β†’ Report** references (when describe metadata exposes them, including at crawl time from dashboard describe `components` when available) * **Report β†’ Object / Field** dependencies derived from Analytics **report describe** columns (subject to discovered objects and fields; global processing enriches `table_dependencies` / `upstream_fields` where metadata allows) * **Flow β†’ Object / Field** lineage derived from Flow **record create/update** and related metadata (subject to Flow discovery being enabled and successful Tooling reads) * **Object β†’ Field** containment * **Folder β†’ Report / Dashboard** containment when applicable Exact property names (`table_dependencies`, `upstream_fields`, `salesforce_flow_lineage`, etc.) are summarized in [discovered resources](/sources/business-intelligence/salesforce-integration/salesforce-integration-discovered-resources). ## Personal vs. shared folders {% hint style="info" %} **Personal folders**: Reports and dashboards under personal folders (or analogous β€œUnfiled” buckets) still appear where Salesforce exposes them via API metadata, typically **directly under the organization** rather than under a folder resourceβ€”only broadly shared folders show as **`salesforce_folder`** resources depending on Salesforce’s folder model for your tenant. {% endhint %} # Salesforce Integration Discovered Resources ## Overview The Salesforce integration discovers the following resources: * Salesforce Organization * User (`user`) β€” when **Observe users & groups** is enabled * User Group (`user_group`) β€” when **Observe users & groups** is enabled * Salesforce Folder * Salesforce Object * Salesforce Field * Salesforce Report * Salesforce Dashboard * Salesforce Flow (when **Include flows** is enabled) ## Salesforce Organization A single resource of type `salesforce_org` is observed for each configured Salesforce integration, representing the top-level Salesforce organization. ### Properties | Property | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------- | | `name` | The name of the Salesforce organization | | `type` | Always `salesforce_org` | | `subtype` | Always `salesforce_org` | | `external_links` | Link to open the Salesforce organization | | `used` | After user observation runs, lists **`user`** URIs observed for this org (same crawl), using the graph **`used`** property | ## User Users are discovered when **Observe users & groups** is enabled. Euno observes active Salesforce users (with an email on the record) and links them from reports, dashboards, and flows via **`native_owners`** when Salesforce exposes **CreatedBy** and that user was observed. ### Properties | Property | Description | | --------------- | --------------------------------------------- | | `name` | The user's email address | | `type` | Always `user` | | `email_address` | The user's email address | | `label` | The user's display name | | `native_id` | The Salesforce user ID | | `created_at` | The timestamp when the user was created | | `updated_at` | The timestamp when the user was last modified | ## User Group User groups are discovered when **Observe users & groups** is enabled. Groups include public groups, role-based groups, and organization groups. ### Properties | Property | Description | | ------------- | ------------------------------------------------------------------------------------------------- | | `name` | The group name | | `type` | Always `user_group` | | `subtype` | Always `salesforce` | | `native_id` | The Salesforce group ID | | `description` | Short description derived from the Salesforce group type (for example `Regular group`) | | `created_at` | The timestamp when the group was created | | `updated_at` | The timestamp when the group was last modified | | `members` | Member **`user`** URIs (via **`has member`** / `members` toward users observed in the same crawl) | ## Salesforce Folder Folders are organizational containers for reports and dashboards in Salesforce. The integration discovers shared folders (not personal folders). ### Properties | Property | Description | | ------------------ | ------------------------------------------------------------------- | | `name` | The folder name | | `type` | Always `salesforce_folder` | | `subtype` | Always `salesforce_folder` | | `native_id` | The unique folder ID in Salesforce | | `description` | The folder type (for example "Report folder" or "Dashboard folder") | | `container_type` | Always `pure_container` β€” folders only contain other resources | | `parent_container` | Parent **`salesforce_org`** URI | ## Salesforce Object Objects represent the data structures in Salesforce, including both standard objects (Account, Contact, Opportunity, etc.) and custom objects. ### Properties | Property | Description | | --------------------------------- | ------------------------------------------------------------------------------------------------- | | `name` | The display label of the object (for example "Account", "My Custom Object") | | `type` | Always `salesforce_object` | | `subtype` | Always `salesforce_object` | | `native_id` | The API name of the object (for example `Account`, `My_Custom_Object__c`) | | `description` | The plural label of the object | | `is_calculated` | `true` if this is a custom object (API name ends in `__c`), `false` for standard objects | | `external_links` | Link to open the object in Salesforce Setup | | `parent_container` | Parent **`salesforce_org`** URI | | `total_read_queries_14d/30d/60d` | API read operations against the object per window (with **Ingest usage metrics** enabled) | | `total_write_queries_14d/30d/60d` | API write operations against the object per window (with **Ingest usage metrics** enabled) | | `distinct_users_14d/30d/60d` | Distinct users calling the object over the API per window (with **Ingest usage metrics** enabled) | Global processing may add Flow- or report-derived **`table_dependencies`** / **`upstream_fields`** when lineage is resolved against observed objects and fields. When the Airflow integration is also connected, **`table_dependencies`** additionally includes warehouse tables that an Airflow task loads into this object (for example via `SalesforceBulkOperator`); see the Airflow integration documentation. ## Salesforce Field Fields represent the individual data elements within Salesforce objects. ### Properties | Property | Description | | ------------------ | --------------------------------------------------------------------------------------------- | | `name` | The display label of the field | | `type` | Always `salesforce_field` | | `subtype` | Always `salesforce_field` | | `native_id` | The API name of the field (for example `Name`, `Custom_Field__c`) | | `native_data_type` | The Salesforce field type (for example `string`, `picklist`, `reference`, `currency`, `date`) | | `is_calculated` | `true` if this is a custom field, `false` for standard fields | | `raw_code` | For formula fields, contains the formula expression | | `parent_container` | Parent **`salesforce_object`** URI | ### Field Types The `native_data_type` property reflects the Salesforce field type, which can include: * `string` β€” Text fields * `picklist` β€” Single-select picklist * `multipicklist` β€” Multi-select picklist * `reference` β€” Lookup/Master-Detail relationship * `boolean` β€” Checkbox * `currency` β€” Currency * `date` β€” Date * `datetime` β€” Date/Time * `double` β€” Number (decimal) * `int` β€” Number (integer) * `percent` β€” Percent * `phone` β€” Phone * `email` β€” Email * `url` β€” URL * `textarea` β€” Long text area * `id` β€” Salesforce ID ## Salesforce Report Reports are analytical views of Salesforce data that can be configured to display, filter, and summarize records. ### Properties | Property | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `name` | The report name | | `type` | Always `salesforce_report` | | `subtype` | Always `salesforce_report` | | `description` | The report description, if provided | | `created_at` | The timestamp when the report was created | | `updated_at` | The timestamp when the report was last modified | | `parent_container` | Parent **`salesforce_org`** or **`salesforce_folder`** URI (shared folder vs personal/unfiled fall back to the org) | | `native_owners` | Native creator (**CreatedBy**) as a structured reference when that user was observed in the crawl | | `external_links` | Link to open the report in Salesforce | | `last_accessed_at` | The report's last run timestamp as reported by Salesforce | | `total_impressions_14d/30d/60d` | Report executions per window (with **Ingest usage metrics** enabled) | | `distinct_impressions_users_14d/30d/60d` | Distinct users running the report per window (with **Ingest usage metrics** enabled) | Euno may also compute a string **`owner`** (first native owner) for display. **Derived lineage** (global processing after the crawl): when Analytics **describe** metadata is available under `native_raw_object`, Euno derives **`table_dependencies`** (upstream **`salesforce_object`**) and **`upstream_fields`** (upstream **`salesforce_field`**) from report columns **only where** those objects and fields already exist as discovered resources. ## Salesforce Dashboard Dashboards are visual displays of key metrics and data from Salesforce reports. ### Properties | Property | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------- | | `name` | The dashboard title | | `type` | Always `salesforce_dashboard` | | `subtype` | Always `salesforce_dashboard` | | `description` | The dashboard description, if provided | | `created_at` | The timestamp when the dashboard was created | | `updated_at` | The timestamp when the dashboard was last modified | | `parent_container` | Parent **`salesforce_org`** or **`salesforce_folder`** URI | | `native_owners` | Native creator (**CreatedBy**) as a structured reference when that user was observed in the crawl | | `external_links` | Link to open the dashboard in Salesforce | | `total_impressions_14d/30d/60d` | Dashboard views per window (with **Ingest usage metrics** enabled) | | `distinct_impressions_users_14d/30d/60d` | Distinct users viewing the dashboard per window (with **Ingest usage metrics** enabled) | Euno may also compute a string **`owner`** (first native owner) for display. **`table_dependencies`** toward **`salesforce_report`** URIs is typically filled **during the crawl** when dashboard describe metadata exposes component report IDs (and may be refined by later processing). ## Salesforce Flow Flows are Salesforce automation definitions (`salesforce_flow`). They are observed when **Include flows (Tooling API)** is enabled in Euno. Euno reads **FlowDefinition** and selected **Flow** version rows via Salesforce’s **Tooling API**. The crawler stores tooling payloads under **`native_raw_object`** for auditability. Subsequent processing derives: * **`salesforce_flow_lineage`** β€” structured field-level lineage (targets and sources) parsed from Flow definition metadata **where resolvable** * **`defines`** β€” links toward **`salesforce_object`** resources the flow materially updates and can propagate lineage onto **objects** and **fields** (for example **`table_dependencies`** / **`upstream_fields`** on **`salesforce_object`** / **`salesforce_field`** from Flow-derived assignments). Lineage completeness depends on which objects and fields were discovered by the crawler and pattern filters. ### Properties | Property | Description | | ------------------ | --------------------------------------------------------------------------------------- | | `name` | Flow master label when available from the tooling version row, otherwise developer name | | `type` | Always `salesforce_flow` | | `subtype` | Always `salesforce_flow` | | `description` | From Flow definition metadata when Salesforce returns it | | `native_id` | Flow definition (tooling) id | | `parent_container` | Parent **`salesforce_org`** URI | | `native_owners` | Optional creator from tooling **CreatedById** when that user was observed in the crawl | | `external_links` | Shortcut to Salesforce Flow setup | **Derived lineage** properties (`salesforce_flow_lineage`, **`defines`**, and dependent graph edges onto objects and fields) are produced by asynchronous global processing as well as crawl-time wiring where applicable. ## Relationships ### Hierarchy and containers | Source type(s) | Relationship | Target type(s) | Notes | | ---------------------- | ------------ | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `salesforce_folder` | has parent | `salesforce_org` | `parent_container` on the folder | | `salesforce_object` | has parent | `salesforce_org` | `parent_container` on the object | | `salesforce_field` | has parent | `salesforce_object` | `parent_container` on the field | | `salesforce_report` | has parent | `salesforce_org`, `salesforce_folder` | Shared-folder reports use a folder parent; personal/unfiled reports use the org | | `salesforce_dashboard` | has parent | `salesforce_org`, `salesforce_folder` | Same pattern as reports | | `salesforce_flow` | has parent | `salesforce_org` | When Flow observation is enabled | | `salesforce_org` | `used` | `user` | After user observation, the org carries **`used`** references to observed users (see [Usage relationships](https://github.com/delphiio/delphi_backend/blob/dev/user_documentation/sources/developer-reference/technical-concepts/relationships.md)) | | `user_group` | has member | `user` | `members` on the group lists observed users | ### Lineage | Source type(s) | Relationship | Target type(s) | Notes | | ---------------------- | ------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `salesforce_dashboard` | has upstream | `salesforce_report` | **`table_dependencies`** from dashboard describe components when present | | `salesforce_report` | has upstream | `salesforce_object`, `salesforce_field` | From Analytics describe via global processing (`table_dependencies` / `upstream_fields`) where resolvable | | `salesforce_flow` | defines | `salesforce_object` | **`defines`** toward objects the flow updates | | `salesforce_flow` | has upstream | `salesforce_object`, `salesforce_field`, `salesforce_flow` | From tooling metadata and global processing (`table_dependencies`, `upstream_fields`, lineage maps) where resolvable | ### Creator metadata (users) Reports, dashboards, and flows emit **`native_owners`** from Salesforce **CreatedBy** when the creator user was observed in the same crawl. Euno may compute a display **`owner`** string from that list. For how user linkage appears in the graph alongside other products, see **Usage relationships** in the [relationships developer reference](https://github.com/delphiio/delphi_backend/blob/dev/user_documentation/sources/developer-reference/technical-concepts/relationships.md). ## Hierarchy Diagram ``` salesforce_org β”œβ”€β”€ user (active Salesforce users; linked from org via used) β”œβ”€β”€ user_group β”‚ └── has member β†’ user β”œβ”€β”€ salesforce_folder (report or dashboard folders) β”‚ β”œβ”€β”€ salesforce_report (native_owners β†’ user when observed) β”‚ └── salesforce_dashboard β†’ salesforce_report (table_dependencies) β”œβ”€β”€ salesforce_flow (optional) β”œβ”€β”€ salesforce_object β”‚ └── salesforce_field └── salesforce_object (custom __c) └── salesforce_field ``` ## Known Limitations ### Personal Folders Personal folders in Salesforce are not actual folder records that can be queried via the API. Reports and dashboards in personal folders will appear directly under the Salesforce Organization rather than within a folder container. ### Formula field lineage While formula expressions are captured in the `raw_code` property, **formula cross-field lineage** strictly from **`raw_code`** is not modeled the same way as Flow- or Report-derived lineage. Flow-derived lineage handles automations separately. ### Filtering (objects, reports, dashboards) **Object filters** hide entire objects (and usually their fields) from discovery. Reports and dashboards that depend on unseen objects/columns may show **fewer or no upstream** lineage edges until those entities are crawled. **Report** and **dashboard** regex filters remove matching reports/dashboards entirely; they leave no stub resource to attach lineage to. ### Flow tooling access Flows require **Tooling API** readable Flow metadata. Missing permissions, partial describe surfaces, or **Include flows** turned off yields **no** `salesforce_flow` assets and therefore **no** Flow-derived lineage for that crawl. # Analytics & Notebooks Connect analytics and notebook platforms to track data exploration, project usage, and semantic model definitions. ## Supported Platforms * [**Hex Technologies**](/sources/analytics-notebooks/hex-technologies) - Collaborative analytics platform * [**Hex Semantic Project**](/sources/analytics-notebooks/hex-semantic-project) - Semantic layer definitions ## What You'll Get When you connect an analytics platform, Euno discovers: * **Projects & Notebooks** - All analytics projects and their usage * **Semantic Models** - Business definitions and metrics * **Data Connections** - Which tables and sources are used * **Usage Metrics** - Execution counts, user engagement * **Lineage** - How analytics projects connect to your data stack ## Common Use Cases * **Project Discovery** - Find existing analytics work * **Usage Tracking** - Understand which projects are active * **Semantic Layer Sync** - Keep metrics in sync with dbt * **Governance** - Track data exploration and ad-hoc analysis ## Getting Started 1. Choose your platform from the list above 2. Follow the setup guide for your platform 3. Grant Euno API access 4. Run your first sync **Setup Time:** 15-20 minutes per platform *** ## Next Steps * [View All Integrations](/sources) * [Learn About Usage Data](/using-euno/usage-optimization) * [Explore Data Lineage](/using-euno/data-model-screen) # Hex Technologies Euno's Hex Technologies integration supports auto-discovery of the following Hex resources: * Hex Workspaces * Hex Projects ## Setting up Euno's Hex Technologies Integration ### Step 1: Create a Hex API Token Euno uses Hex API tokens to access the Hex API and auto-discover Hex resources. 1. Log in to your Hex workspace 2. Navigate to settings 3. Generate a new API Workspace token for Euno integration 4. Give it "Read Access" and "Read project queried tables" permission\\ 5. Copy the API token (you'll need this for the Euno configuration)
6. Copy the API token (you'll need this for the Euno configuration) ### Step 2: Configure New Hex Source in Euno #### Step 1: Access the Sources Page 1. Go to the **Sources** page. 2. Click on **Add New Source** and select **Hex Technologies** from the list of supported platforms. #### Step 2: General Configuration Asterisk (\*) means a mandatory field.
ConfigurationDescription
Name*Enter a name for your Hex source (e.g., "Hex - Analytics Projects")
API Token*Provide the API token from your Hex account.
Base URLThe Hex API base URL. Defaults to https://app.hex.tech/api/v1. Only change this if you're using a custom Hex deployment.
Workspace ID*Your Hex workspace ID. This is used to create direct links to projects in the workspace.
Workspace NameA friendly name for your Hex workspace. Defaults to "hex_workspace".
#### Step 3: Scheduling Updates 1. Enable the Schedule toggle. 2. Choose from the following: 1. **Weekly**: Specify the days and times for updates. 2. **Hourly**: Enter an interval in hours (e.g., every 6 hours). #### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. #### Step 5: Save Configuration Click the **Save** button to complete the setup. ## Features ### Project Discovery The Hex integration automatically discovers all projects within your workspace, including: * Project metadata (name, description, creation date) * Project ownership and creation information * Project status and activity indicators * Usage analytics (impressions over different time periods) * Project categorization and tags See [here](/sources/analytics-notebooks/hex-technologies/hex-integration-discovered-resources) for more details. ### Workspace Organization Projects are organized under their parent workspace, providing a clear hierarchical view of your Hex analytics environment. ### Usage Analytics The integration captures usage metrics including: * Total impressions over 14 and 30-day periods * Last viewed timestamps * Project activity status ### External Links The integration creates direct links to projects within the Hex interface for easy navigation. ## Related Integrations ### Hex Semantic Project Integration If you're using Hex's Semantic Model Sync with dbt MetricFlow, you may also want to set up the [**Hex Semantic Project**](/sources/analytics-notebooks/hex-semantic-project) integration. This integration allows you to: * Upload dbt MetricFlow YAML files to create semantic models and metrics * Establish lineage between your dbt semantic models and Hex semantic layer * Manage semantic model governance across your data stack The Hex Semantic Project integration is specifically designed for users who leverage [Hex's Semantic Model Sync feature](https://learn.hex.tech/docs/connect-to-data/semantic-models/semantic-model-sync/intro) and complements this general Hex project discovery integration. # Hex Integration Discovered Resources ## Overview The Hex Technologies integration discovers the following resources: * Hex Workspace * Hex Project ## Hex Workspace A single resource of type "hex\_workspace" is observed for each configured Hex integration. ### Properties | Property | Value | | -------- | ----------------------------- | | name | The name of the Hex workspace | | type | always `hex_workspace` | | subtype | always `hex_workspace` | ## Hex Project Projects are the main analytical resources in Hex, containing notebooks, data applications, and other analytical content. ### Properties | Property | Value | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | The title of the Hex project | | type | always `hex_project` | | subtype | always `hex_project` | | description | The description of the project, if provided | | native\_id | The unique project ID in Hex | | created\_at | The timestamp when the project was created | | updated\_at | The timestamp when the project was last edited | | created\_by | The email address of the user who created the project | | is\_active | Boolean indicating if the project is active (not trashed) | | total\_impressions\_30d | Total number of app views in the last 30 days This property is only available for published Hex Projects. | | total\_impressions\_14d | Total app views over the past 14 days. This property is only available for published Hex Projects. | | last\_viewed\_at | The timestamp when the project was last viewed | | tags | All the Hex Categories associated with the project. See [Hex Documentation](https://learn.hex.tech/docs/organize-content/statuses-categories#categories) | | collection | The Hex Collections the project belongs to, represented by a list of collection names. | | meta.status | The project status if available. See [Hex Documentation](https://learn.hex.tech/docs/organize-content/statuses-categories#statuses) | | owner | The owner of the Hex project | | is\_published | Indicates whether the Hex Project has been published as an app. | | native\_last\_data\_update | Date of the most recent data refresh. | | last\_published | Date of the most recent edit to the published project. | ### Usage Analytics The Hex integration captures detailed usage analytics for projects: * **App Views**: Track how often projects are viewed as applications * **Time-based Metrics**: Separate tracking for 14-day and 30-day periods * **Last Activity**: When the project was most recently accessed ### Project Organization Projects are organized with the following structure: * **Workspace Level**: All projects belong to a parent workspace * **Categorization**: Projects can be tagged with categories for organization * **Status Tracking**: Projects have status information (if available) * **Ownership**: Clear attribution to creating and owning users ## Lineage Hex queries are distinguishable in the data warehouse's query history. See [Hex Documentation](https://learn.hex.tech/docs/explore-data/cells/sql-cells/sql-cells-introduction#query-metadata) for how Hex tags each query with metadata about the originating project and user. Euno leverages Hex's query metadta feature to build the lineage and usage information about Hex projects. See the relevant parts in each warehouse integration documentation. ## Relationships * **Hex Workspace β†’ Hex Projects**: Projects are considered children of their parent workspace ### Container Relationships * Projects maintain a container chain showing their hierarchical position within the workspace structure * The workspace serves as the top-level container for all projects These relationships allow you to understand the organization of your Hex analytics environment and how different projects relate to each other within workspaces. ## Known Limitations Due to current Hex API constraints, the completeness of extracted metadata is affected. In particular, lineage in Euno is derived from query history, meaning only queries executed within the last 7 days are considered when building lineage for Hex projects. As a result: * **Delayed updates**: If a project is modified in a way that changes its lineage, it may take time for Euno to reflect these changes. During this period, the old and new lineage may appear overlapped. * **Missing lineage for inactive projects**: If a project has not been executed in the past 7 days, and no queries are available, Euno will be unable to detect or display its lineage. # Hex Semantic Project Euno's Hex Semantic Project integration enables you to upload dbt MetricFlow YAML files to create semantic models and metrics that can be used in Hex's semantic layer. This integration specifically targets Hex users who utilize [Hex's Semantic Model Sync with dbt MetricFlow](https://learn.hex.tech/docs/connect-to-data/semantic-models/semantic-model-sync/dbt-metricflow). {% hint style="info" %} This integration is designed for Hex users who want to manage their semantic models through dbt MetricFlow and sync them with Euno for comprehensive data lineage and governance. {% endhint %} ## How It Works The integration processes dbt MetricFlow YAML files to: 1. **Extract Semantic Models**: Identifies semantic models with Hex-specific configuration 2. **Create Hex Resources**: Generates corresponding Hex workspace, semantic project, semantic model, dimension, and measure resources 3. **Create dbt Resources**: Generates corresponding dbt semantic model, dimension, and metric resources 4. **Establish Lineage**: Creates field-level lineage between Hex and dbt resources 5. **Process Standalone Metrics**: Handles standalone metrics that reference measures from semantic models ## Setting up Euno's Hex Semantic Project Integration ### Step 1: Access the Sources Page 1. Navigate to the **Sources** page in the Euno application. 2. Click on the **Add New Source** button. ### Step 2: General Configuration 1. **Name**: Enter a name for your Hex Semantic Project source (e.g., "Hex Semantic - Customer Analytics"). 2. **Configuration Details**: * **Workspace ID**: Your Hex workspace ID (required) * **Project ID**: Your Hex semantic project ID (required) * **Workspace Name**: Your Hex workspace name * **Semantic Project Name**: The name for your semantic project in Hex * **Semantic Project Type**: Currently supports "dbt Metricflow" (this is the default and only supported type) Note: to find the semantic project id navigate to the project in the data browser and copy the id from the url. For example, if the sematic project url is `https://app.hex.tech/0197ae57-d3c0-7001-b693-280fbcb78fc4/home/data/semantic-layer/semantic-project/0198ec40-77da-7004-9731-91c06ffc5c96` The semantic project id is : `0198ec40-77da-7004-9731-91c06ffc5c96` and the workspace id is `0197ae57-d3c0-7001-b693-280fbcb78fc4` ### Step 3: Resource Cleanup Options To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Time-Based Cleanup (default)**: Remove resources that were last detected X days before the most recent successful source integration run (default is 7 days). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 4: Advanced Settings (Optional) Click on the '**Advanced**' section to display additional configurations: | Configuration | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Files Pattern | Use regular expressions to include or exclude specific YAML files. Defaults to including `.*\.yml$` and `.*\.yaml$` files. Use this to filter which files from your upload are processed. | ### Step 5: Save Configuration Click the **Save** button, and Euno will generate a trigger key. Please copy the key and save it somewhere, as the key will disappear after copying. ## **Uploading dbt MetricFlow Files** ### Preparing Your dbt MetricFlow Files Before uploading, ensure your dbt MetricFlow YAML files include the required Hex configuration. Only semantic models with Hex-specific metadata will be processed. **Required file structure:** ```yaml semantic_models: - name: customer_metrics model: ref('customer_metrics') config: meta: hex: table: analytics.prod_core.customers # Required for Hex integration entities: - name: customer_id type: primary description: "Primary customer identifier" - name: account_id type: foreign description: "Foreign key to account table" dimensions: - name: customer_name type: categorical measures: - name: total_revenue agg: sum expr: revenue create_metric: true ``` **Key requirements:** 1. **dbt\_project.yml**: Must be included in your upload 2. **Hex Configuration**: Semantic models must include `config.meta.hex.table` specification 3. **Table Reference**: The `hex.table` value should be in format `database.schema.table` or `schema.table` 4. **Entity Types**: Only entities with types `primary`, `unique`, `natural`, or `foreign` will be processed as Hex dimensions ### Upload Methods #### **Manual Upload via UI** The easiest way to upload dbt MetricFlow files is through Euno's UI: 1. Navigate to your Hex Semantic Project integration 2. Click "Upload Artifacts" 3. Upload a .zip file containing: * `dbt_project.yml` (required) * YAML files with semantic models and metrics * Any other dbt MetricFlow YAML files #### **Programmatic Upload (Recommended for Production)** For production workflows, use the trigger URL to programmatically upload files after dbt builds: To programmatically upload the files for production workflows, see reference implementations below: #### **Python Script** Use a standalone Python script to upload your files [β†’ View Python Upload Guide](/sources/zip-artifact-python-upload) ## Features ### Semantic Model Discovery The integration automatically discovers and processes: * **Semantic Models**: With Hex configuration (`config.meta.hex`) * **Dimensions**: Categorical, time, entity, boolean, and numeric dimensions * **Entities**: Primary, unique, natural, and foreign key entities (converted to dimensions in Hex) * **Measures**: Sum, count, count distinct, average, min, max, median, and sum boolean measures * **Standalone Metrics**: Simple metrics that reference measures from semantic models ### Resource Creation For each valid semantic model, the integration creates: **Hex Resources:** * Hex Workspace (container for all semantic projects) * Hex Semantic Project (container for semantic models) * Hex Semantic Model (the main semantic model resource) * Hex Dimensions (individual dimension resources) * Hex Measures (individual measure resources, including those from standalone metrics) **dbt Resources:** * dbt Semantic Model (corresponding dbt resource) * dbt Dimensions (individual dimension resources) * dbt Metrics (for measures with `create_metric: true` and standalone metrics) ### Lineage and Dependencies The integration establishes comprehensive lineage: * **Table-level**: Hex semantic models reference corresponding dbt semantic models * **Field-level**: Hex dimensions and measures reference their corresponding dbt dimensions and metrics * **Cross-platform Integration**: Integrates with Euno's existing lineage for end-to-end dependency tracking For detailed information about all discovered resources, see: [Hex Semantic Project Discovered Resources](/sources/analytics-notebooks/hex-semantic-project/hex-semantic-project-discovered-resources) ## Related Integrations * [**Hex Technologies**](/sources/analytics-notebooks/hex-technologies): For general Hex project and workspace discovery * [**dbt Core**](/sources/transformation-etl/dbt-core): For comprehensive dbt resource discovery and lineage # Hex Semantic Project Discovered Resources ## Overview The Hex Semantic Project integration discovers the following resources: * Hex Workspace * Hex Semantic Project * Hex Semantic Model * Hex Dimension * Hex Measure * dbt Semantic Model * dbt Dimension * dbt Metric ## Hex Workspace A single resource of type "hex\_workspace" is observed for each configured workspace in the integration settings. ### Properties | Property | Value | | -------- | ------------------------------------------- | | name | The workspace identifier from configuration | | type | always `hex_workspace` | | subtype | always `hex_workspace` | ## Hex Semantic Project A single resource of type "hex\_semantic\_project" is observed for each configured semantic project. ### Properties | Property | Value | | ----------------------- | -------------------------------------------------- | | name | The semantic project name from configuration | | type | always `hex_semantic_project` | | subtype | always `hex_semantic_project` | | semantic\_project\_type | Type of semantic project (e.g., "dbt\_metricflow") | | workspace\_id | The parent workspace ID | ## Hex Semantic Model Semantic models are discovered from dbt MetricFlow YAML files. Only semantic models with `config.meta.hex` configuration are processed. ### Properties | Property | Value | | ----------- | --------------------------- | | name | The semantic model name | | type | always `hex_semantic_model` | | subtype | always `hex_semantic_model` | | description | Model description from YAML | ### Relationships * **Parent**: Hex Semantic Project * **Dependencies**: References corresponding dbt semantic model as `upstream` ## Hex Dimension Dimensions are discovered from both semantic model dimensions and entities. There are two types: ### Regular Dimensions Discovered from the `dimensions` section of semantic models. These have corresponding dbt dimensions and field-level lineage. ### Entity-based Dimensions Discovered from the `entities` section of semantic models. Only entities with types `primary`, `unique`, `natural`, or `foreign` are processed. These do NOT have corresponding dbt dimensions or field-level lineage. ### Properties | Property | Value | | ----------- | ---------------------- | | name | The dimension name | | type | always `hex_dimension` | | subtype | always `hex_dimension` | | description | Dimension description | ### Relationships * **Parent**: Hex Semantic Model * **upstream\_fields**: References corresponding dbt dimension (only for regular dimensions, not entity-based dimensions) ## Hex Measure Measures are discovered from semantic models (when `create_metric: true`) and standalone simple metrics that reference existing measures. We currently do not observe non-simple dbt metrics. ### Properties | Property | Value | | ----------- | -------------------- | | name | The measure name | | type | always `hex_measure` | | subtype | always `hex_measure` | | description | Measure description | ### Relationships * **Parent**: Hex Semantic Model * **Field-level Lineage**: References corresponding dbt metric via `upstream_fields` ## dbt Semantic Model dbt semantic models correspond to each valid Hex semantic model and are created to maintain lineage. ### Properties | Property | Value | | ----------- | --------------------------- | | name | The semantic model name | | type | always `dbt_semantic_model` | | subtype | always `dbt_semantic_model` | | description | Model description | ## dbt Dimension dbt dimensions correspond to each regular dimension (from the `dimensions` section) in valid semantic models. Note that entity-based dimensions do NOT have corresponding dbt dimension resources. ### Properties | Property | Value | | -------- | ---------------------- | | name | The dimension name | | type | always `dbt_dimension` | | subtype | always `dbt_dimension` | ### Relationships * **Parent**: dbt Semantic Model ## dbt Metric dbt metrics are created for measures with `create_metric: true` and standalone simple metrics. ### Properties | Property | Value | | -------- | ------------------- | | name | The metric name | | type | always `dbt_metric` | | subtype | always `dbt_metric` | ## Relationships ### Parent/Child * **Hex Workspace** β†’ **Hex Semantic Project**: Semantic projects are children of workspaces * **Hex Semantic Project** β†’ **Hex Semantic Model**: Semantic models are children of semantic projects * **Hex Semantic Model** β†’ **Hex Dimensions** and **Hex Measures**: Dimensions and measures are children of semantic models * **dbt Semantic Model** β†’ **dbt Dimensions**: dbt dimensions are children of dbt semantic models ### Cross-Platform Lineage | Source type(s) | Relationship | Target type(s) | Notes | | -------------------- | -------------------- | -------------------- | ------------------------ | | `dbt_semantic_model` | has upstream | `hex_semantic_model` | Via `table_dependencies` | | `dbt_dimension` | has upstream\_fields | `hex_dimension` | Regular dimensions only | | `dbt_metric` | has upstream\_fields | `hex_measure` | Via `upstream_fields` | **Note**: Entity-based dimensions (from semantic model entities) do not have cross-platform lineage as they don't have corresponding dbt dimension resources. # Data Security Connect data security and classification platforms to Euno to enrich your data model with sensitivity metadata, classification tags, and governance context. ## Available Integrations * [**Cyera**](/sources/data-security/cyera-integration) - Data security and classification. * [**BigID**](/sources/data-security/bigid-integration) - Data security, discovery, and classification. * [**Microsoft Purview**](/sources/data-security/microsoft-purview-integration) - Data loss prevention policy discovery for Microsoft 365 and Power BI. ## What You'll Get When you connect a data security platform, Euno can: * **Classification Tags** - Data classes and labels applied by the platform * **Sensitivity Levels** - Sensitivity or risk levels (e.g., Unclassified, Internal, Confidential) * **DLP Policy Context** - Policies that regulate supported BI resources * **Enriched Resources** - Table and column metadata augmented with security context ## Getting Started 1. Choose an integration from the list above. 2. Follow the setup guide for your platform. 3. Configure API credentials and schedule the integration. 4. Run your first sync to enrich existing warehouse resources. *** ## Next Steps * [View All Integrations](/sources) * [Data Warehouses](/sources/data-warehouses) - Connect your data warehouses first to discover tables and columns that data security platforms can enrich # Cyera Euno's Cyera integration supports auto-discovery of tables and columns with Cyera classification tags and sensitivity metadata. * Currently, Snowflake and Bigquery are the supported data-warehouses for this integration For business value and typical use cases, see [Cyera integration: use cases & value](/sources/data-security/cyera-integration/cyera-integration-use-cases). ## Setting up Cyera integration ### Overview Cyera is a data security platform that discovers, classifies, and protects sensitive data across your data stores. The Cyera integration connects to Cyera's API to observe tables and columns that Cyera has scanned, enriching them in Euno with Cyera's classification tags and sensitivity levels. To discover Cyera-enriched resources, Euno authenticates to the Cyera API using OAuth client credentials and streams table metadata from Cyera's view of your datastores. Euno observes tables and columns that have Cyera classifications or sensitivity assignments, adding Cyera-specific properties to the corresponding table and column resources in the Euno data model. ### Step 1: Generate Cyera API credentials (partner integration) Cyera provides API tokens for partner integrations through a dedicated flow in the Cyera platform. You need credentials generated from the **Euno** integration cardβ€”not a generic API client created elsewhere in the console. #### Prerequisites * **Cyera Admin** role * Active **Cyera DSPM** license * Access to the partner integration page in the Cyera platform #### Navigate to the Euno partner integration 1. Log in to Cyera. 2. In the left navigation, go to **Integrations**. 3. Find the **Euno** integration card. 4. Click the card to open its details. #### Generate an API token 1. In the **API Token Generator** section, review the pre-filled **Token Description**. 2. Select **Token Expiration** from the dropdown: * 1 day * 7 days * 30 days * 90 days * 365 days 3. Click **Generate Token**. #### Save your credentials Cyera displays your **Client ID** and **Client Secret** after generation. {% hint style="warning" %} **Important**: These credentials are shown **only once**. Copy them immediately or use **Download CSV** to save both values. You will need the Client ID and Client Secret for Euno configuration in Step 2. {% endhint %} You can save the credentials in either of these ways: * Click the copy icon next to each credential, or * Click **Download CSV** to save both values. Store the Client Secret securely and treat it like a password. If you lose the secret, generate a new token in Cyera and update the Euno source configuration. ### Step 2: Configure New Cyera Source in Euno #### Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click on **Add New Source** and select **Cyera** from the list of supported platforms. #### General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------- | ------------------------------------------------------------------ | | Name\* | Enter a name for your Cyera source (e.g., "Cyera - Data Security") | | Client ID\* | The Cyera API Client ID from Step 1 | | Client Secret\* | The Cyera API Client Secret from Step 1 | ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). {% hint style="info" %} **Recommended**: Schedule the Cyera integration to run daily or every 12 hours to keep classification and sensitivity metadata in sync with Cyera. {% endhint %} ### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 5: Save Configuration Click the **Test & Save** button to complete the setup. Euno will: 1. Authenticate using the provided Client ID and Client Secret. 2. Verify access to the Cyera API. 3. Save the configuration if validation passes. If the test fails, review the error message and verify your credentials. {% hint style="info" %} **Initial sync**: After you connect Cyera to Euno, the first full sync of classification and sensitivity metadata may take up to **24 hours**. {% endhint %} ## Discovered Resources The Cyera integration augments warehouse table and column resources with Cyera classification and sensitivity properties. For detailed information about discovered resources and their indexed properties, see [cyera-integration-discovered-resources](/sources/data-security/cyera-integration/cyera-integration-discovered-resources). # Cyera Integration Use Cases Euno's integration with [Cyera](https://www.cyera.io/) extends data security posture management (DSPM) beyond the data warehouse and into the business layer β€” where BI tools, dashboards, reports, and AI interfaces are the primary surface through which people interact with sensitive data every day. *** ## The Challenge DSPM tools are highly effective at classifying sensitive data at rest β€” in warehouses, lakes, and storage systems. But risk doesn't stop there. The business-facing layer presents a blind spot that most security programs haven't fully addressed: * BI tools and AI interfaces frequently rely on **shared service accounts**, making it impossible to determine who actually accessed sensitive data through a dashboard or report. * **Downstream dependencies** β€” calculations, joined fields, aggregated metrics β€” often inherit sensitivity from source columns without it being tracked or flagged. * Security teams lack the **audit trails and access context** needed to respond quickly to incidents or demonstrate compliance. * Dashboards and reports that are **no longer actively used** may still be exposing sensitive data without anyone realizing it. The result: your warehouse is classified, but the business layer is a governance gap. *** ## Better Together: Cyera + Euno
Cyera classifies sensitive data in the warehouse. Euno propagates those classifications through **column-level lineage** into BI fields, dashboards, reports, and downstream assets β€” making it possible to see exactly where sensitive data travels, who uses it, and whether that access is appropriate. Together, Cyera and Euno give security and data teams a continuous, automated view of sensitive data exposure across the full data-to-decision stack. *** ## Use Cases ### Faster Incident Response When a sensitive data incident occurs, time to understanding is critical. With Euno's lineage mapped on top of Cyera's classifications, security teams can instantly trace which dashboards, reports, and pipelines are connected to the affected tables β€” and identify who accessed them and when. No manual investigation, no dead ends from shared accounts.
### Stale Asset Detection Unused dashboards and reports are easy to overlook, but they can represent a significant and silent exposure risk if they contain or reference sensitive data. Euno automatically surfaces BI assets that haven't been accessed recently but still expose Cyera-classified fields β€” giving data and security teams a clear list of assets to deprecate or restrict. ### Policy Enforcement in BI Cyera's DSPM policies can now extend their reach into BI. When sensitive data flows into a dashboard or report, Euno generates real-time alerts so security teams can enforce access policies at the point of business consumption β€” not just at the warehouse level.
### Identity Enrichment Shared service accounts mask real user activity. Euno resolves the individuals behind BI access and usage events, enriching Cyera's identity cards with business-layer context: which reports a user views, which dashboards they interact with, and which sensitive fields those assets expose. This gives security teams a fuller picture of each identity's data footprint. ### Safer AI Usage As AI agents increasingly query and summarize enterprise data, ensuring they operate with appropriate security and privacy context becomes essential. Euno enriches the data context surfaced to AI agents with Cyera's sensitivity and classification metadata, reducing the risk of sensitive data being inadvertently exposed through AI-driven queries or responses. *** ## How It Works Euno connects to Cyera via the Cyera API using OAuth client credentials. Once connected, Euno streams table and column metadata from Cyera's view of your datastores, identifying assets with active sensitivity classifications or data class assignments. Euno then maps these classifications onto the corresponding resources in its data model and traces them forward through column-level lineage β€” surfacing classification context wherever sensitive data flows downstream in your BI and analytics layer. The integration can be scheduled to sync on a recurring basis, keeping classification and lineage context continuously up to date. *** ## Get Started Ready to extend your DSPM coverage into the business layer? * **Set up the integration** β†’[ Cyera Integration setup guide](/sources/data-security/cyera-integration) # Cyera Integration Discovered Resources ## Overview The Cyera integration augments **table** and **column** resources with Cyera classification and sensitivity metadata. Cyera does not create new resource types; it observes tables and columns that Cyera has classified and adds Cyera-specific indexed properties to the corresponding resources in Euno. Only tables and columns that have Cyera classifications or a sensitivity assignment are observed. The following properties are added to those resources. ## Table – Cyera Properties When a table has Cyera classifications or a sensitivity level, the Cyera integration adds these properties to the table resource: | Property | Value | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | cyera\_tags | List of Cyera data classification tag names derived from Cyera's classifications (e.g., from `dataClassName`). Empty or absent if the table has no tags. | | cyera\_sensitivity | The sensitivity level assigned by Cyera (e.g., `"Unclassified"`, `"NotSensitive"`, `"Internal"`). | | cyera\_sensitivity\_display\_name | The human-readable display name for the Cyera sensitivity level (e.g., `"Unclassified"`, `"Public"`). | ## Column – Cyera Properties When a column has Cyera classifications, the Cyera integration adds these indexed properties to the column resource: | Property | Value | | ----------- | -------------------------------------------------------------------------------------------------------------- | | cyera\_tags | List of Cyera data classification tag names derived from Cyera's classifications (e.g., from `dataClassName`). | Columns do not have `cyera_sensitivity` or `cyera_sensitivity_display_name` in the current model; those apply at the table level in Cyera. ## Relationships The Cyera integration does not create new relationships. It enriches existing table and column resources. ## Usage in EQL You can filter and search by Cyera properties in Euno's search and EQL: **Examples**: ``` # Tables with at least one Cyera tag cyera_tags is not null # Tables with a specific sensitivity level cyera_sensitivity = "Internal" # Columns that have been classified by Cyera cyera_tags is not null and type = "column" ``` # BigID Euno's BigID integration supports auto-discovery of tables and columns with BigID attributes and classification metadata. See [BigID integration use cases](/sources/data-security/bigid-integration/bigid-integration-use-cases) for common workflows that combine BigID classifications with Euno lineage. ## Setting up BigID integration ### Overview BigID is a data security and privacy platform that discovers, classifies, and protects sensitive data across your data stores. The BigID integration connects to BigID's API and enriches warehouse tables and columns in Euno with BigID classification metadata. To discover BigID-enriched resources, Euno authenticates to the BigID API using a long-lived **API user token** and streams catalog and classification metadata from BigID's view of your datastores (BigQuery, Snowflake, and Databricks). For each structured catalog table that passes filters and maps to a supported warehouse, Euno emits a table observation with `bigid_*` properties. Column observations are emitted only when BigID reports attributes and/or column-level sensitivity tags for that column. ### Step 1: Create a BigID role for Euno Euno needs read access to the BigID catalog and scan-result findings. Create a dedicated role rather than reusing roles built for other BigID apps (for example, Jira, Databricks, or RoPA integrations). #### Prerequisites * **BigID administrator** access (to create roles and users) * Your BigID instance **base URL** (for example, `https://your-tenant.bigid.com`) #### Create the role 1. Log in to BigID as an administrator. 2. Go to **Administration β†’ Access Management β†’ Roles**. 3. Click **Add** (or equivalent) to create a new role. 4. Set: * **Name**: for example: `Euno Integration Role` * **Description**: Read-only catalog access for the Euno BigID source integration 5. Under **Scope**, select **`root`** unless you intentionally want to limit Euno to a narrower data scope. Named scopes restrict which catalog objects the role can see; Euno may then return a partial or empty catalog. 6. Under **Role Permissions**, enable at minimum: * **Catalog β†’ Read, Export, Get Attributes Value, View Sensitive Values** * **Data Sources β†’ Read** * **Scans β†’ Scan Activity β†’ Read** 7. Optionally enable: * **Catalog β†’ Manual Fields β†’ Read** - allows Euno to observe manual fields * **Catalog β†’ Business Attributes β†’ Read** - allows Euno to observe business attributes) 8. Save the role. ### Step 2: Create a BigID user and generate an API token #### Create the user 1. Go to **Administration β†’ Access Management β†’ Users**. 2. Create a new user (recommended) or select a dedicated service account. Suggested naming: * **Display name**: `Euno Integration` * **Username / email**: `euno-integration@yourdomain.com` β€” use a **dedicated** address, not your personal BigID login email * **Origin**: **Local** (recommended for API-only integration users) 3. Connect the **Euno Integration** role from Step 1 to this user (**Connect Role**). 4. Save the user. #### Generate the API token 1. Open the user profile in the right-hand detail panel. 2. Find the **Tokens** section. 3. Click **Generate**. 4. Set the token expiration (up to 999 days) and click **Generate** again. 5. **Copy the token value immediately** β€” BigID does not show it again after you close the dialog. 6. Click **Save** on the user profile so the token is active. ### Step 3: Configure New BigID Source in Euno #### Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click on **Add New Source** and select **BigID** from the list of supported platforms. #### General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | ------------------- | --------------------------------------------------------------------------------- | | Name\* | Enter a name for your BigID source (e.g., "BigID - Data Security") | | Base URL\* | The base URL of your BigID server (e.g., `https://your-bigid-instance.bigid.com`) | | System user token\* | The API token from Step 2 (paste the raw token; do not add a `Bearer` prefix) | ### Step 4: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 8 hours). {% hint style="info" %} **Recommended**: Schedule the BigID integration to run daily or every 12 hours to keep attribute and classification metadata in sync with BigID. {% endhint %} ### Step 5: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. For the freshest sensitivity metadata, use **Immediate Cleanup**. If a column was detected only because of a BigID column-level sensitivity tag and BigID later removes that tag without reporting attributes for the column, the column is no longer detected by the BigID run. **No Cleanup** preserves that last observed column metadata by design. ### Step 6: Advanced Settings (Optional) Click on the **Advanced** section to display these additional configurations. | Configuration | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Data Objects Pattern | Allow/deny regular expressions applied to BigID object fully qualified names (for example, `DataSource.schema.table`) after catalog fetch. Default allow `.*` includes all objects that pass other filters. | | Attributes Pattern | Allow/deny regular expressions applied to BigID attribute names before they are copied into `bigid_attributes`. Default allow `.*` includes all attributes that pass the confidence threshold. | | Attribute Confidence Threshold | Numeric value from **0.0** to **1.0**. Attributes with BigID confidence below this threshold are dropped. Default **0.0** observes all attributes that match the attribute pattern. | | Data Source Mapping | Optional key/value map from BigID data source name to an Euno warehouse URI prefix override. Use when BigID's source label does not match the host or account Euno uses for the same warehouse (same pattern as Fivetran database mapping). | ### Step 7: Save Configuration Click the **Test & Save** button to complete the setup. Euno validates connectivity to BigID and saves the configuration. ## Discovered Resources The BigID integration augments warehouse table and column resources with BigID attribute and sensitivity properties. For detailed information about discovered resources and their indexed properties, see [bigid-integration-discovered-resources](/sources/data-security/bigid-integration/bigid-integration-discovered-resources). # BigID Integration Use Cases BigID classifies sensitive data in warehouses and other structured sources. Euno reads that classification metadata and connects it to warehouse tables, columns, and downstream assets through lineage. This page describes common workflows when BigID and Euno are used together. Setup steps are in the [BigID integration setup guide](/sources/data-security/bigid-integration). ## What the integration provides After a scheduled sync, Euno adds `bigid_*` properties to matching **BigQuery**, **Snowflake**, and **Databricks** table resources for each structured BigID catalog row that passes filters (even when BigID metadata is empty). Column observations are added only when BigID reports attributes and/or sensitivity tags for that column. You can filter on these properties in Discover and EQL, andβ€”where the Sensitive Data sidepanel is enabledβ€”review classifications on a resource. Lineage then shows how classified columns flow into BI fields, dashboards, reports, and other downstream resources. ## Use cases ### Incident response When a table is flagged in BigID, you can use Euno to find downstream dashboards, reports, and pipelines that depend on that table or its columns. Combined with Euno usage and identity metadata where available, this reduces manual tracing across the warehouse and BI layers. ### Stale or unused BI assets Euno can surface BI resources with low or no recent usage that still reference BigID-classified columns. Teams use this to prioritize review, restriction, or retirement of assets that may no longer need access to sensitive fields. ### Governance in the BI layer Warehouse classifications from BigID do not automatically appear on Looker, Tableau, or Power BI assets. Euno propagates BigID metadata on source columns to downstream fields and reports so governance and security teams can see exposure outside the warehouse. ### Identity and access context Where Euno has usage and identity data, you can relate BI access patterns to tables and columns classified by BigIDβ€”for example, which users or service accounts have access to or have viewed assets built on sensitive source columns. ### AI and self-service analytics When AI or natural-language interfaces query data modeled in Euno, BigID sensitivity and attribute metadata on upstream tables and columns provides additional context about which assets carry classified data before results are shown or summarized. ## How it works 1. Euno authenticates to your BigID instance with a system user token and base URL. 2. On each run, the integration fetches BigID catalog and classification metadata for supported warehouse types. 3. Euno maps classified objects to existing table and column URIs and writes `bigid_*` properties (see [discovered resources](/sources/data-security/bigid-integration/bigid-integration-discovered-resources)). 4. Existing lineage and usage data in Euno connect those properties to downstream resources. Schedule the source to keep metadata aligned with BigID (daily or every 12 hours is a common choice). ## Related documentation * [BigID integration setup](/sources/data-security/bigid-integration) * [Discovered resources and EQL](/sources/data-security/bigid-integration/bigid-integration-discovered-resources) # BigID Integration Discovered Resources ## Overview The BigID integration augments **table** and **column** resources with BigID classification and sensitivity metadata. BigID does not create new resource types; it adds BigID-specific indexed properties to warehouse tables and columns that already exist in Euno. For each structured BigID catalog row that passes filters and maps to a supported warehouse (**BigQuery**, **Snowflake**, or **Databricks**), Euno emits a **table** observation with `bigid_`\* propertiesβ€”even when BigID has no attributes or sensitivity tags for that table (properties may be empty lists or null). **Column** observations are emitted only when BigID reports attributes and/or column-level sensitivity tags for that column. The following properties are written when those observations are materialized. ## Table – BigID Properties For each observed table, the integration sets these properties on the table resource (values may be empty or null when BigID provides no matching metadata): | Property | Value | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | bigid\_attributes | Distinct BigID attribute names assigned to the table and its columns (for example, `Email`, `SSN`, `businessTerm.CustomerName`). Values pass the source's attribute pattern and confidence threshold filters. An empty list when no attributes match. | | bigid\_sensitivity | The highest BigID sensitivity label on the table or any of its columns, stored as the **raw BigID value** (for example, `High`, `Restricted`, `Confidential`). | | bigid\_categories | Attribute categories from BigID metadata (for example, `PII`, `PHI`, `PCI`, `Personal Data`). Empty or absent if no categories are present. | ### Sensitivity display mapping BigID sensitivity labels are tenant-configurable. The stored `bigid_sensitivity` value is always the raw BigID label; the Euno UI maps common BigID values onto its shared seven-level display scale (badges and sidepane) as follows: | BigID label (examples) | Euno display name | | ----------------------- | ----------------- | | Restricted, Top Secret | Top Secret | | Critical, High | Very Sensitive | | Confidential | Confidential | | Medium | Sensitive | | Internal, Internal Use | Internal | | Low, Public | Public | | Missing or unrecognized | Unclassified | ## Column – BigID Properties When a column has BigID attribute or sensitivity metadata, the BigID integration adds these indexed properties to the column resource: | Property | Value | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | bigid\_attributes | List of BigID attribute names assigned to the column (for example, classification labels or data categories). | | bigid\_sensitivity | The highest BigID sensitivity label assigned directly to the column, stored as the **raw BigID value** (for example, `High` or `Restricted`). Null when BigID currently reports no sensitivity tag for the observed column. | Columns do not have `bigid_categories`; attribute categories apply at the table resource. ## Relationships The BigID integration does not create new relationships. It enriches existing table and column resources that match warehouse URIs in Euno. ## Usage in EQL You can filter and search by BigID properties in Euno's search and EQL: **Examples**: ``` # Tables with at least one BigID attribute bigid_attributes is not null # Tables whose attribute list includes a specific label 'classifier.Email' IN bigid_attributes # Case-insensitive attribute match i'email' IN bigid_attributes # Tables at a specific BigID sensitivity (raw BigID label, as configured in your BigID tenant) bigid_sensitivity = 'High' # Columns at a specific BigID sensitivity bigid_sensitivity = 'Restricted' AND type = 'column' # Tables in a BigID attribute category 'PII' IN bigid_categories # Columns with BigID attribute metadata (column observations are only emitted when metadata exists) bigid_attributes is not null AND type = 'column' ``` # Microsoft Purview Euno's Microsoft Purview integration discovers Purview Data Loss Prevention (DLP) policies, connects them to Power BI resources that Euno has already discovered, and can enrich matching Power BI resources with Microsoft Purview Data Map metadata. ## Setting up Microsoft Purview integration ### Overview Microsoft Purview is Microsoft's governance and compliance platform for Microsoft 365 and Fabric. In Euno, the Microsoft Purview integration reads sensitivity label metadata and DLP policy definitions from Microsoft Graph, then creates `purview_dlp_policy` resources in Euno. When a Purview DLP policy applies to supported Power BI items, Euno can show those Power BI resources as regulated by the Purview policy. If you also provide the Purview account name, Euno uses Microsoft Purview Data Map to add Purview metadata, such as tags, classifications, descriptions, and data policy names, to matching Power BI resources. The Power BI resources must already exist in Euno from the Power BI integration. {% hint style="info" %} The Microsoft Purview integration does not create Power BI reports, dashboards, apps, semantic models, dataflows, or columns. Run the Power BI integration first, then run Microsoft Purview to add policy and metadata context. {% endhint %} ### Step 1: Prepare Microsoft access #### Prerequisites * Microsoft Entra administrator access to create an app registration and grant admin consent. * Microsoft Purview DLP policies in the Microsoft 365 tenant. * A Power BI source in Euno if you want Purview policies linked to Power BI resources. * A Microsoft Purview account with Power BI assets in Data Map if you want Purview metadata enrichment on Power BI resources. #### Create an Entra app registration for Euno This step creates a **service application in your Microsoft tenant**. It is **not** an Azure Marketplace install of Euno, and it does not register the Euno product inside Azure. Instead, you create (or choose) an Entra **app registration** whose credentials Euno stores and uses to call Microsoft Graph when the Purview source runs. A common name is `Euno Purview Integration`. {% hint style="info" %} If you already have an Entra app for the [Power BI integration](/sources/business-intelligence/powerbi-integration), use a **separate** app registration for Microsoft Purview unless you intentionally want one shared service principal for both integrations. {% endhint %} 1. Open [Microsoft Entra](https://entra.microsoft.com/) and sign in. 2. In the sidebar, pick **App registrations**. 3. Click **New registration** (or open an existing app registration dedicated to this integration). 4. Enter a name (for example, `Euno Purview Integration`), select **Single tenant only** under **Supported account types**, and leave **Redirect URI** empty. In some Entra portal versions this option is labeled **Accounts in this organizational directory only** β€” it is the same setting.
Register an application: choose Single tenant only and leave Redirect URI empty

Register an application: choose Single tenant only and leave Redirect URI empty

5. Click **Register**. 6. On the **Overview** page, copy **Application (client) ID** and **Directory (tenant) ID**. You will enter these in Euno as **Client ID** and **Tenant ID**. 7. In the sidebar, open **Certificates & secrets**. 8. Under **Client secrets**, click **+ New client secret**, set an expiry, and click **Add**. 9. Copy the secret **Value** immediately. You will enter it in Euno as **Client Secret**. #### Grant Microsoft Graph application permissions 1. In the app registration sidebar, open **API permissions**. 2. Click **+ Add a permission**. 3. Select **Microsoft Graph**. 4. Select **Application permissions** (not Delegated permissions). 5. Search for and add the permissions needed to read sensitivity label metadata. Microsoft tenants can expose either the newer Data Security and Governance label API or the older Information Protection label API: * `SensitivityLabel.Read` β€” least-privileged permission for tenant sensitivity labels on the Microsoft Graph Data Security and Governance API. * `SensitivityLabels.Read.All` β€” broader alternative if your Microsoft tenant requires it. * `InformationProtectionPolicy.Read.All` β€” required by older Microsoft Graph Information Protection label APIs that Euno can use as a fallback. 6. Click **Add permissions** at the bottom of the panel to return to the **API permissions** page.
Request API permissions: select Application permissions and add SensitivityLabel.Read

Request API permissions: select Application permissions and add SensitivityLabel.Read

7. Back on the **API permissions** page, click **Grant admin consent for \[your organization]** above the permissions table. 8. Confirm when prompted. The **Status** column for each permission should show a green checkmark for **Granted for \[your organization]**.
API permissions after admin consent: all required Application permissions show Granted for [your organization]

API permissions after admin consent: all required Application permissions show Granted for [your organization]

#### Grant Microsoft Purview Data Map access This step is required only if you want Euno to read Purview Data Map metadata for Power BI assets. Grant the Euno app registration access to the relevant Microsoft Purview Data Map collection. Use the least-privileged role that lets the app read assets, classifications, and labels, such as **Data Reader**. Use a broader role only if your organization's Purview access model requires it. {% hint style="info" %} If you skip Data Map access or leave the Purview Account Name blank in Euno, the Microsoft Purview source still reads DLP policies from Microsoft Graph. It will skip Data Map metadata enrichment. {% endhint %} ### Step 2: Configure New Microsoft Purview Source in Euno #### Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click **Add New Source**. 3. Select **Microsoft Purview**. #### General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Name\* | Enter a name for your Microsoft Purview source, for example `Microsoft Purview - Production`. | | Tenant ID\* | Microsoft Entra directory tenant ID. | | Client ID\* | Application (client) ID for the Entra app registration. | | Client Secret\* | Client secret value for the Entra app registration. | | Purview Account Name | Optional. Enter the Microsoft Purview account name to enable Data Map metadata enrichment for matching Power BI resources. For example, use `contoso` from `https://contoso.purview.azure.com`. | Microsoft Graph DLP policy discovery does not require a Purview account name. Data Map metadata enrichment requires the Purview account name and Data Map collection access. ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours. {% hint style="info" %} Recommended: schedule Microsoft Purview after the Power BI source has refreshed. This lets Euno link Purview DLP policies to the latest Power BI resources. {% endhint %} ### Step 4: Resource Cleanup To keep your data relevant and free of outdated resources, Euno provides automatic **resource cleanup** options. For a detailed explanation on Euno's cleanup strategies, see: [Resource Sponsorship in Euno](/developer-reference/technical-concepts/resource-sponsorship-and-cleanup-in-euno). * **Immediate Cleanup**: Remove resources not detected in the most recent successful source integration run. * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected. ### Step 5: Save Configuration Click **Test & Save** to complete setup. Euno validates Microsoft Graph authentication, sensitivity label access, and DLP policy access before saving the source. If Purview Account Name is configured, Euno also performs a small Data Map search probe to validate Data Map access. ## What Euno discovers The Microsoft Purview integration creates `purview_dlp_policy` resources and can connect them to supported Power BI resources through policy regulation relationships. When Data Map is configured, it also reads Power BI Data Map asset metadata and enriches existing Power BI resources with Purview tags, classifications, descriptions, and data policy names. For detailed information about discovered resources, properties, relationships, and EQL examples, see [microsoft-purview-integration-discovered-resources](/sources/data-security/microsoft-purview-integration/microsoft-purview-integration-discovered-resources). # Microsoft Purview Integration Discovered Resources ## Overview The Microsoft Purview integration discovers Purview Data Loss Prevention (DLP) policies from Microsoft Graph `policyFiles`. Euno creates a `purview_dlp_policy` resource for each observed DLP policy. Sensitivity labels are used as metadata for DLP policy mapping. They are not created as separate Euno resources by the Microsoft Purview integration. When the Power BI integration has already discovered matching Power BI resources, Euno can connect those resources to the Purview DLP policies that regulate them. When the Microsoft Purview source is configured with a Purview account name, Euno also reads matching Power BI assets from Microsoft Purview Data Map. Euno uses those Data Map assets to enrich existing Power BI resources with Purview metadata. ## Purview DLP Policy (`purview_dlp_policy`) `purview_dlp_policy` resources represent Microsoft Purview DLP policies parsed from Microsoft Graph policy files. | Property | Value | | ------------- | ------------------------------------------------------------------- | | `name` | DLP policy name from Microsoft Purview. | | `type` | Always `purview_dlp_policy`. | | `label` | Display label for the policy, usually the policy name. | | `native_id` | Native Purview policy ID. | | `description` | Policy description when available from Microsoft Purview. | | `updated_at` | Latest policy update timestamp when available from the policy file. | ## Purview Data Map Asset (`purview_datamap_asset`) `purview_datamap_asset` resources are source facts that represent Power BI assets read from Microsoft Purview Data Map. Euno uses these source facts to enrich matching Power BI resources. These resources are not a replacement for the Power BI integration, and they do not create Power BI reports, semantic models, dashboards, apps, dataflows, or columns. | Property | Value | | ------------- | ---------------------------------------------------------------------------------------------------------- | | `name` | Asset name or display name from Purview Data Map. | | `type` | Always `purview_datamap_asset`. | | `label` | Display label for the Data Map asset. | | `native_id` | Purview Data Map asset GUID. | | `description` | Asset description from Purview Data Map, when available. | | `subtype` | Purview Data Map entity type, such as a Power BI report, dashboard, semantic model, or column entity type. | ## Power BI Metadata Enrichment When a matching Power BI resource already exists in Euno, the Microsoft Purview source can enrich it with these fields: | Property | Description | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `description` | Purview description for the matching Data Map asset. Euno fills this only when the Power BI resource does not already have a description, or when the existing description was previously filled from Purview. | | `purview_tags` | Microsoft Purview tags applied to the matching Power BI asset in Purview Data Map. | | `purview_classifications` | Microsoft Purview classifications applied to the matching Power BI asset in Purview Data Map. | | `purview_data_policy_names` | Names of Microsoft Purview DLP data policies that apply to the matching Power BI resource. | Supported Power BI resource types for this metadata are `powerbi_report`, `powerbi_dashboard`, `powerbi_semantic_model`, `powerbi_app`, `powerbi_dataflow`, and `powerbi_column`, when Purview Data Map exposes a matching asset. ## Relationships | Source type(s) | Relationship | Target type(s) | Notes | | -------------------------------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `powerbi_report`, `powerbi_dashboard`, `powerbi_semantic_model`, `powerbi_app`, `powerbi_dataflow` | has regulator | `purview_dlp_policy` | The Power BI resource is regulated by the Purview DLP policy. The Power BI resource must already exist in Euno. | | `purview_dlp_policy` | has regulated | `powerbi_report`, `powerbi_dashboard`, `powerbi_semantic_model`, `powerbi_app`, `powerbi_dataflow` | Reverse relationship: Power BI resources regulated by this policy. | ## Usage in EQL You can filter and search by Microsoft Purview resources and relationships in Euno's search and EQL. **Examples**: ``` # All Purview DLP policies type = "purview_dlp_policy" # Power BI reports regulated by any Purview DLP policy type = "powerbi_report" and has regulator(type = "purview_dlp_policy") # Power BI resources regulated by a specific Purview DLP policy has regulator(uri = "") # Purview DLP policies that regulate Power BI semantic models type = "purview_dlp_policy" and has regulated(type = "powerbi_semantic_model") # Power BI resources with a Purview tag purview_tags IN ("finance") # Power BI resources with a Purview classification purview_classifications IN ("MICROSOFT.PERSONAL.EMAIL") # Power BI resources with a Purview data policy name purview_data_policy_names IN ("Sensitive Power BI data policy") ``` ## Current Scope The Microsoft Purview integration currently supports: * DLP policy discovery from Microsoft Graph `policyFiles`. * Policy relationships to supported Power BI resources that already exist in Euno. * Data Map metadata enrichment for matching Power BI resources when Purview Data Map is configured. It does not create separate sensitivity label resources, and it does not create Power BI resources that have not already been observed by the Power BI integration. It does not expose Microsoft Purview retention policies for Power BI resources. # Data Observability Connect data observability platforms to Euno to surface data quality incidents and monitor coverage directly on the resources they impact β€” tables, dbt models, columns, and the originating monitors themselves. ### Available Integrations * [**Monte Carlo**](/sources/data-observability/monte-carlo) - Data observability and incident management. * [**Elementary**](/sources/data-observability/elementary-integration) - dbt-native data quality incident monitoring. ### What You'll Get When you connect a data observability platform, Euno can: * **Active Incidents** - Surface open data-quality incidents on impacted warehouse and dbt resources. * **Resource-level Indicators** - Boolean and count properties (e.g., `has_active_incident`, `active_incidents_count`) for fast filtering and search. ### Getting Started 1. Choose an integration from the list above. 2. Follow the setup guide for your platform. 3. Configure API credentials and schedule the integration. 4. Run your first sync to attach incidents to existing warehouse and dbt resources. *** ### Next Steps * [View All Integrations](/sources) * [Data Warehouses](/sources/data-warehouses) - Connect your data warehouses first so observability platforms can attach incidents to existing tables and columns. * [Transformation & ETL](/sources/transformation-etl) - Connect dbt so observability platforms can attach incidents to dbt models. # Monte Carlo Euno's Monte Carlo integration surfaces active data quality alerts directly on impacted warehouse tables and columns, and on the dbt resources that define them (`dbt_model`, `dbt_source`, `dbt_seed`, `dbt_snapshot`, `dbt_column`). It also introduces the originating Monte Carlo monitor as a first-class resource in the Euno data model. Monte Carlo remains the source of truth for alerts. Euno retrieves alert payloads via the Monte Carlo GraphQL API and computes a set of resource-level indicators for discovery and filtering. Only **active** alerts are ingested: * `Investigating` * `No Status` (newly opened, not yet triaged) Alerts in terminal states (`Fixed`, `No Action Needed`, `False Positive`, `Expected`, `Resolved`, `Closed`, `Acknowledged`) are not fetched, stored, or displayed. {% hint style="info" %} **Alert lookback window**: Euno fetches alerts whose `updatedTime` falls within the last **30 days**. An alert that remains in an active state but has not been updated in Monte Carlo for longer than that window may not appear in Euno until it is updated again. {% endhint %} Euno's Monte Carlo integration supports auto-discovery of: * `montecarlo_account` β€” top-level account container holding warehouse connection metadata * `montecarlo_monitor` β€” one resource per monitor returned by Monte Carlo's `getMonitors` API (including monitors with no active alerts) In addition, the integration **enriches** existing `table`, `column`, `dbt_model`, `dbt_source`, `dbt_seed`, `dbt_snapshot`, and `dbt_column` resources with active-alert properties. It does not discover or import warehouse hierarchy from Monte Carlo. ## Prerequisites * A Monte Carlo account with at least one configured warehouse connection. * Administrator access to Monte Carlo to create an Account Service Key. * The tables, columns, and dbt resources you want to enrich with alerts must already exist in Euno from another source integration (Snowflake, BigQuery, dbt, etc.). Euno resolves alerts to existing resources β€” it does not create warehouse hierarchy from Monte Carlo. ## Stage 1: Configure Monte Carlo ### Step 1: Create a Monte Carlo Account Service Key 1. Log in to your Monte Carlo account as an administrator. 2. Navigate to **Settings β†’ API**. 3. Create a new **Account Service Key**. 4. Copy the **API Key ID** and **API Key Secret** β€” you'll need these for Euno configuration. {% hint style="info" %} **Account-level token required when SSO is enabled**: If your Monte Carlo account uses SSO, you must provide an **account-level** Account Service Key (not a user-scoped key) for this integration to work. User-scoped tokens issued under SSO will not authenticate successfully against the headless GraphQL API used by the integration. {% endhint %} ## Stage 2: Configure New Monte Carlo Source in Euno ### Step 1: Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click on **Add New Source** and select **Monte Carlo** from the list of supported platforms. ### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name\*** | Enter a name for your Monte Carlo source (e.g., "Monte Carlo - Data Observability"). | | **API Key ID\*** | Account Service Key ID from Step 1. Sent on every request as the `x-mcd-id` header. | | **API Key Secret\*** | Account Service Key secret from Step 1. Sent on every request as the `x-mcd-token` header. Stored securely. | | **GraphQL API URL** | (Advanced) Base URL of the Monte Carlo GraphQL API. Leave blank to use `https://api.getmontecarlo.com/graphql`. Override only if your account uses a custom endpoint. | | **Warehouse URI prefix mapping** | (Advanced) Optional JSON mapping of Monte Carlo-reported warehouse URI prefixes or coordinates to the Euno URI prefixes or coordinates used by your warehouse/dbt sources. See [Warehouse URI prefix mapping](#warehouse-uri-prefix-mapping) below. | ### Warehouse URI prefix mapping If Monte Carlo reports warehouse coordinates differently from the URI prefixes Euno uses for tables (for example, Snowflake locator format vs. org-account format), configure **Warehouse URI prefix mapping** in the source's advanced settings. Keys are the warehouse URI prefixes or coordinate prefixes Euno derives from Monte Carlo alert payloads. Values are the Euno URI prefixes or coordinate prefixes used by the existing warehouse and dbt resources in your account. Euno applies the longest matching prefix before resolving alerts to tables, columns, and defining dbt resources (`dbt_model`, `dbt_source`, `dbt_seed`, `dbt_snapshot`, `dbt_column`). Common examples: * **Snowflake** β€” map the Monte Carlo-reported Snowflake account prefix to the Snowflake prefix used by your Snowflake/dbt source. For example, Monte Carlo may report locator format `snowflake.jgb69604.us-west-2.aws` while Euno resources use org-account format `snowflake.zwfkgsy-bdb54493`. * **BigQuery** β€” use `bigquery` or a more specific `bigquery.project` / `bigquery.project.dataset` prefix when a project or dataset needs to be remapped. * **Specific coordinates** β€” map a longer prefix such as `snowflake.old_account.analytics.public` to `snowflake.new_account.analytics.public` when only a database/schema coordinate differs. ```json { "snowflake.jgb69604.us-west-2.aws": "snowflake.zwfkgsy-bdb54493", "snowflake.old_account.analytics.public": "snowflake.new_account.analytics.public" } ``` For backward compatibility, Euno still accepts legacy keys based on Monte Carlo warehouse metadata (warehouse UUID/name or connection UUID/id), but new configurations should use reported URI prefixes or coordinate prefixes. ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 4 hours). {% hint style="info" %} **Recommended**: Schedule the Monte Carlo integration to run every 4 hours so active alert state stays close to real-time. Manual runs are also supported. {% endhint %} ### Step 4: Resource Cleanup **Immediate Cleanup** is the default and recommended setting for this integration. * **Immediate Cleanup**: `montecarlo_monitor` resources not observed in the most recent successful crawl are removed. This keeps the monitor inventory aligned with Monte Carlo's `getMonitors` results. {% hint style="warning" %} Use **Immediate Cleanup** for Monte Carlo sources. **No Cleanup** can leave stale `montecarlo_monitor` resources after monitors are deleted in Monte Carlo. `active_montecarlo_alerts` is updated when a monitor's active alerts change, but resolved alerts may leave stale data on impacted resources until those resources are updated again. Prefer **Immediate Cleanup** and re-run the integration after major alert triage in Monte Carlo. {% endhint %} ### Step 5: Save Configuration Click the **Test & Save** button to complete the setup. Euno will: 1. Authenticate to the Monte Carlo GraphQL API using the provided API Key ID and Secret. 2. Verify access via the Monte Carlo API (`getUser`). 3. Save the configuration if validation passes. If the test fails, review the error message and verify your credentials. The most common cause of failure is a user-scoped token in an SSO-enabled account β€” see the note in Step 1. ## What Euno Discovers * `montecarlo_account` β€” The top-level Monte Carlo account container. Holds warehouse connection metadata and the warehouse-to-Euno mapping context used to resolve alerts to the correct resources. * `montecarlo_monitor` β€” A Monte Carlo data quality monitor. One resource is created per monitor returned by Monte Carlo's `getMonitors` API. Stores the full monitor payload and its list of active alerts. In addition, existing **`table`**, **`column`**, **`dbt_model`**, **`dbt_source`**, **`dbt_seed`**, **`dbt_snapshot`**, and **`dbt_column`** resources are enriched with alert properties when they are the impacted asset of one or more active Monte Carlo alerts. For detailed information about discovered resources and their indexed properties, see [Monte Carlo Integration Discovered Resources](/sources/data-observability/monte-carlo/monte-carlo-integration-discovered-resources). # Monte Carlo Integration Discovered Resources ### Overview The Monte Carlo integration has two outputs: 1. It creates and maintains a first-class **`montecarlo_account`** container resource for the connected Monte Carlo account. 2. It creates and maintains a first-class **`montecarlo_monitor`** resource for every Monte Carlo monitor fetched in the crawl. 3. It enriches existing **`table`**, **`column`**, **`dbt_model`**, **`dbt_source`**, **`dbt_seed`**, **`dbt_snapshot`**, and **`dbt_column`** resources with active-alert data. The integration does not discover or import warehouse hierarchy (databases, schemas, tables, views, models, or columns) by reference β€” those resources must already exist in Euno from another source integration. Only alerts in active states (`Investigating`, `No Status`) are ingested. Alerts in terminal states (`Fixed`, `No Action Needed`, `False Positive`, `Expected`, `Resolved`, `Closed`, `Acknowledged`) are not stored or displayed. Euno fetches alerts whose `updatedTime` falls within the last **30 days**. An alert that remains active in Monte Carlo but has not been updated within that window may not appear in Euno until it is updated again. ### Ingested Resources & Properties #### Monte Carlo Account (`montecarlo_account`) The top-level container resource created once per Monte Carlo source. Holds warehouse connection metadata and the warehouse-to-Euno mapping context used to resolve alerts to the correct resources. | Property | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | The Monte Carlo account ID. | | `type` | Resource type value (`montecarlo_account`). | | `native_id` | The Monte Carlo account ID. | | `montecarlo_warehouses` | JSON list of warehouse connections returned by Monte Carlo's `getWarehouses` API. | | `database_mapping_context` | Copy of the warehouse mapping JSON from source setup (if configured). See [Warehouse mapping](/sources/data-observability/monte-carlo#warehouse-mapping) in the setup guide. | #### Monte Carlo Monitor (`montecarlo_monitor`) One resource is created per monitor returned by Monte Carlo's `getMonitors` API, including monitors with no active alerts. For `CUSTOM_SQL` monitors, custom rule details are included in the stored monitor payload; for `TABLE` monitors, table monitor details are included. | Property | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Display name of the Monte Carlo monitor. Derived from the monitor's `description` (the label shown in the Monte Carlo UI), falling back to the API `name` and then the monitor UUID when no description is set. | | `type` | Resource type value (`montecarlo_monitor`). | | `native_id` | Stable UUID of the monitor in Monte Carlo. Used for deduplication across crawls. | | `parent_container` | Parent **`montecarlo_account`** URI | | `montecarlo_monitor_is_enabled` | `true` when the monitor is enabled (neither paused nor a draft), `false` when it is paused or a draft, and null when the underlying status is unknown. Derived from the raw monitor's `isPaused` and `isDraft` flags. Indexed for EQL search. | | `active_montecarlo_alerts` | JSON list of active alerts associated with this monitor in the most recent crawl. Each entry includes the full alert detail payload plus graph-free target refs parsed from Monte Carlo table/asset MCONs. Empty when the monitor has no active alerts. | ### Resource β€” Monte Carlo Alert Enrichment When a table, column, or a dbt resource that `defines` an impacted table or column is the impacted asset of one or more active Monte Carlo alerts, the following properties are added to the resource. dbt enrichment reaches every dbt resource type that points at a physical table or column through `defines` β€” `dbt_model`, `dbt_source`, `dbt_seed`, and `dbt_snapshot` for tables, and `dbt_column` for columns: | Property | Description | Applicable Resources | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `has_active_montecarlo_alerts` | `true` when the resource has at least one attached active alert; `false` when cleared. Indexed for EQL search. | `table`, `column`, `dbt_model`, `dbt_source`, `dbt_seed`, `dbt_snapshot`, `dbt_column` | | `active_montecarlo_alerts_count` | Count of distinct active Monte Carlo alerts attached to this resource. Indexed for EQL search. | `table`, `column`, `dbt_model`, `dbt_source`, `dbt_seed`, `dbt_snapshot`, `dbt_column` | | `active_montecarlo_alerts` | JSON list of active alerts attached to this resource. Aggregated from all `montecarlo_monitor` resources whose active alerts target this resource's URI. Each entry mirrors the alert detail payload (see Alert Payload below). Visible on the resource **Observability** tab in the UI. Not indexed for EQL predicates on individual payload fields. | `table`, `column`, `dbt_model`, `dbt_source`, `dbt_seed`, `dbt_snapshot`, `dbt_column` | Use `has_active_montecarlo_alerts` and `active_montecarlo_alerts_count` to search for impacted resources in EQL. Use the Observability tab or browse `montecarlo_monitor` resources for full alert payload details. ### Alert Payload Each entry in the `active_montecarlo_alerts` JSON list captures the information Euno extracts from the Monte Carlo alert payload. #### Identity | Field | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | UUID of the alert in Monte Carlo. Used for deduplication and updates. | | `uuid` | Alternate alert UUID field (same value as `id` when both are present). | | `targets_uri` | On target-resource alert payloads, the resolved Euno resource URIs that this alert is attached to. On monitor alert payloads, graph-free table and physical-column URIs parsed directly from Monte Carlo payloads. | | `monte_carlo_target_refs` | Graph-free target refs parsed from Monte Carlo `tables`/`assets` rows before graph resolution. Each ref includes `table_uri` and, when available, `column_name`, `column_uri`, `warehouse_uuid`, `database_name`, `schema_name`, and `table_name`. These fields are also sampled in the run report for mapping troubleshooting. | #### State | Field | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `status` | One of `investigating`, `no_status`. Terminal states (`fixed`, `no_action_needed`, `false_positive`, `expected`, `resolved`, `closed`, `acknowledged`) are ignored. | | `severity` | One of `SEV-1`, `SEV-2`, `SEV-3`, `SEV-4`. | | `feedback` | Optional triage feedback recorded by the user in Monte Carlo. | #### Time | Field | Description | | ------------- | ------------------------------------------------------------------- | | `createdTime` | When the alert was first opened. | | `updatedTime` | Last change time (state change, new anomaly, or assignment change). | #### Monitoring | Field | Description | | ------------- | --------------------------------------------------------------- | | `name` | Display name of the alert. | | `type` | Alert type as returned by Monte Carlo. | | `monitorTags` | Tags attached to the originating monitor (`[{ name, value }]`). | #### Assignment / Ownership | Field | Description | | ------- | ----------------------------------------------------------------------------- | | `owner` | User identifier for the alert owner (`{ email }` as returned by Monte Carlo). | #### Asset Locator | Field | Description | | -------- | --------------------------------------------------------------------------------------------------- | | `tables` | List of Monte Carlo table references associated with the alert (`[{ mcon, tableId, isKeyAsset }]`). | | `assets` | List of Monte Carlo asset references (`[{ mcon, assetId, assetType }]`). | ### Mapping Alerts to Euno Resources Euno uses the Monte Carlo Canonical Object Name (MCON) from the alert's `tables` and `assets` fields, together with warehouse connection metadata and the optional [warehouse mapping](/sources/data-observability/monte-carlo#warehouse-mapping) from source configuration, to parse impacted table and field references. Monte Carlo processors then resolve those refs to existing Euno resources: table-scoped references resolve to table URIs, field-scoped references resolve to physical columns, and dbt resources whose `defines` relationships point at those tables or columns are also enriched. Once resolved, Euno: * attaches the alert to each impacted table, dbt model, and column resource, * updates each impacted resource's `active_montecarlo_alerts` list, * links the alert to the corresponding `montecarlo_monitor` resource. The relationship between an alert, its originating `montecarlo_monitor`, and the impacted resources is maintained on every run. ### Relationships | Source type(s) | Relationship | Target type(s) | Notes | | -------------------- | ------------ | -------------------- | --------------------------------- | | `montecarlo_monitor` | has parent | `montecarlo_account` | `parent_container` on the monitor | ### Usage in EQL You can filter Monte Carlo resource types and alert indicators in Euno's search and EQL. Individual fields inside the `active_montecarlo_alerts` JSON payload are not indexed for EQL predicates. **Examples**: ``` # All Monte Carlo monitors in the account type = "montecarlo_monitor" # All Monte Carlo account containers type = "montecarlo_account" # Warehouse tables with active Monte Carlo alerts has_active_montecarlo_alerts = true and type = "table" # dbt models with more than one active alert active_montecarlo_alerts_count > 1 and type = "dbt_model" # dbt columns with active Monte Carlo alerts has_active_montecarlo_alerts = true and type = "dbt_column" ``` # Elementary Euno's Elementary Cloud integration surfaces active data quality incidents directly on impacted resources (`dbt_model`, `dbt_source`, `dbt_column`, warehouse `table`, and `column`). Elementary remains the source of truth for incidents β€” Euno connects to Elementary Cloud and keeps resource-level indicators in sync for discovery and filtering. Only **active** incidents are shown in Euno: * `open` * `acknowledged` Incidents in terminal states are not fetched, stored, or displayed. The integration does not create new resource types. It enriches existing dbt and warehouse resources that are already present in Euno from other source integrations. ## Prerequisites * An Elementary Cloud account with API access. * A bearer token and Environment ID from Elementary Cloud. * dbt and/or warehouse resources already observed in Euno so incidents can be attached. Only incidents originating from **dbt tests on dbt model or dbt source assets** are supported. Supported test types are built-in dbt tests (`not_null`, `unique`, `relationships`, `accepted_values`) and `dbt_utils` tests. Incidents from seeds, snapshots, schema tests, custom generic tests outside these sets, or other asset types are not imported. ## Stage 1: Configure Elementary Cloud ### Step 1: Obtain Elementary Cloud credentials 1. Log in to your Elementary Cloud account. 2. Navigate to **Settings β†’ API** (or your account's token management page). 3. Generate a new **API token** (bearer token). 4. Copy the token β€” it will be needed for Euno configuration and cannot be retrieved again. 5. Note your **Environment ID** β€” this identifies the Elementary environment whose incidents Euno will sync. It is visible in the Elementary Cloud UI under your environment settings. ## Stage 2: Configure New Elementary Source in Euno ### Step 1: Access the Sources Page 1. Go to the **Sources** page in Euno. 2. Click **Add New Source** and select **Elementary Cloud** from the list of supported platforms. ### Step 2: General Configuration Asterisk (\*) means a mandatory field. | Configuration | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name**\* | Enter a name for your Elementary source (e.g., "Elementary - Data Quality"). | | **Bearer Token**\* | The API token generated in Step 1. Stored securely in Euno. | | **Environment ID**\* | The Elementary environment ID to query. Scopes all incident fetches to a single environment. | | **Elementary MCP Endpoint** | Base URL of the Elementary MCP API. Defaults to `https://prod.api.elementary-data.com/mcp/`. Change this only if your Elementary account uses a custom or self-hosted endpoint. | | **Warehouse mapping** | Optional mapping of Elementary `database.schema` identifiers to Euno `database.schema` identifiers. Use this when Elementary reports warehouse coordinates that differ from those in Euno (e.g., due to connection aliases). Keys are matched case-insensitively against the `database.schema` reported for each incident. Keys wrapped in `/…/` are treated as regular expressions. Example: `{"elem_db.prod": "euno_db.prod", "/elem_db\\.(.*)/": "euno_db.$1"}`. | ### Step 3: Schedule * Enable the Schedule option. * Choose: 1. **Weekly**: Set specific days and times. 2. **Hourly**: Define the interval in hours (e.g., every 4 hours). {% hint style="info" %} **Recommended**: Schedule the Elementary integration to run frequently (every 1–4 hours) so that active incident state stays close to real-time. Manual runs are also supported. {% endhint %} ### Step 4: Resource Cleanup The Elementary integration requires **Immediate Cleanup**. * **Immediate Cleanup**: successful runs remove incidents confirmed absent from Elementary. If an unresolved asset reference or a monitor whose `test` is null prevents Euno from reevaluating an incident, the run completes with a warning and temporarily preserves that incident's existing projections. An incident-detail batch error, missing incident, duplicate incident, or malformed response fails the run without publishing cleanup state. {% hint style="warning" %} Immediate Cleanup is the only supported cleanup mode for this integration. {% endhint %} ### Step 5: Save Configuration Click the **Test & Save** button to complete the setup. Euno will: 1. Connect to Elementary Cloud using the provided bearer token. 2. Verify access by fetching the current list of active incidents for the configured environment. 3. Save the configuration if validation passes. If the test fails, check that the bearer token is valid and that the environment ID matches an environment visible in your Elementary Cloud account. ## What Euno Discovers The Elementary integration does not create new resource types. Instead, it enriches existing **`dbt_model`**, **`dbt_source`**, **`dbt_column`**, **`table`**, and **`column`** resources with active-incident properties. For detailed information about the enriched properties, see [Elementary Integration Discovered Resources](/sources/data-observability/elementary-integration/elementary-discovered-resources). # Elementary Integration Discovered Resources ### Overview The Elementary integration does not create new resource types. It enriches existing **`dbt_model`**, **`dbt_source`**, **`dbt_column`**, **`table`**, and **`column`** resources with active-incident properties from Elementary Cloud. The integration does not discover or import warehouse hierarchy β€” those resources must already exist in Euno from another source integration (typically a dbt or warehouse integration). Only incidents in active states (`open`, `acknowledged`) are shown in Euno. Incidents in terminal states are not stored or displayed. Only incidents originating from **dbt tests on dbt model or dbt source assets** are supported. Supported test types are built-in dbt tests (`not_null`, `unique`, `relationships`, `accepted_values`) and `dbt_utils` tests (e.g. `dbt_utils.not_null_proportion`, `dbt_utils.recency`). Incidents from seeds, snapshots, schema tests, custom generic tests outside these sets, or other asset types are not imported. ### dbt Model, dbt Source, dbt Column, Table, and Column β€” Elementary Incident Properties When a dbt model, dbt source, dbt column, warehouse table, or column is affected by one or more active Elementary incidents, the following properties are added to that resource: | Property | Description | Applicable resources | | ------------------------ | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | `has_active_incident` | `true` when the resource has at least one attached active incident; `false` when cleared. | `dbt_model`, `dbt_source`, `dbt_column`, `table`, `column` | | `active_incidents_count` | Count of currently active incidents attached to this resource. | `dbt_model`, `dbt_source`, `dbt_column`, `table`, `column` | | `incidents` | Structured list of active incident details (see Incident details below). | `dbt_model`, `dbt_source`, `dbt_column`, `table`, `column` | For column-level incidents, Euno attaches the incident to the matched parent **`dbt_model`** or **`dbt_source`**, and also to the matching warehouse **`column`** and **`dbt_column`** when those resources exist in the catalog. Matching warehouse **`table`** resources are enriched as well when resolved. ### Incident details Each entry in the `incidents` list includes the information Euno displays from Elementary Cloud. #### Identity | Field | Description | | --------------- | ----------------------------------------------------------------------------------------------- | | `incident_id` | Unique identifier of the incident in Elementary Cloud. | | `asset_id` | Elementary asset ID for the affected dbt model, dbt source, table, or column. | | `incident_type` | `model` for a whole dbt model, `source` for a dbt source, `column` for a column-level incident. | #### State | Field | Description | | ---------- | --------------------------------------------------------------------------------------------------------------- | | `status` | One of `open`, `acknowledged`. Closed or fixed incidents are not shown in Euno. | | `severity` | One of `critical`, `high`, `normal`, `low`. Elementary values such as `medium` or `warning` appear as `normal`. | #### Time | Field | Description | | ------------ | ------------------------------------------------- | | `created_at` | When the incident was first opened in Elementary. | | `updated_at` | When the incident was last updated in Elementary. | #### Test information | Field | Description | | ----------- | ---------------------------------------------------------------------- | | `test_id` | Identifier of the dbt test that produced the incident, when available. | | `test_name` | Name of the failing test (e.g. `not_null`, `dbt_utils.recency`). | | `test_type` | `builtin` for core dbt tests; `dbt_utils` for `dbt_utils.*` tests. | #### Assignment and ticketing | Field | Description | | ------------- | ------------------------------------------------------------------------------- | | `assigned_to` | Elementary user the incident is assigned to, when set. | | `ticket_url` | Link to an external ticket (Jira, Linear, etc.), when configured in Elementary. | | `ticket_id` | External ticket identifier, when configured in Elementary. | #### Affected asset location | Field | Description | | ------------- | --------------------------------------------------------------- | | `name` | Display name of the incident (Elementary message or test name). | | `database` | Warehouse database for the affected asset. | | `schema` | Warehouse schema for the affected asset. | | `object_name` | Table or model name for the affected asset. | | `column_name` | Column name for column-level incidents; empty otherwise. | ### How incidents are matched to Euno resources Euno uses the warehouse coordinates from each incident (`database`, `schema`, `object_name`, and `column_name` when present), together with optional **warehouse mapping** in the source settings, to attach incidents to resources already in your catalog: 1. **dbt model or source** β€” match a `dbt_model` or `dbt_source` by database, schema, and model or source name. If more than one resource could match, Euno uses the Elementary asset path to choose the right one. 2. **Warehouse table** β€” match a `table` using the same database, schema, and table name, using the warehouse connection from the matched dbt resource when available. 3. **Warehouse column** β€” when the incident names a column and a matching `table` exists in Euno, attach the incident to the corresponding `column`. 4. **dbt column** β€” when the incident names a column and a matching `dbt_model` or `dbt_source` exists, also attach the incident to the child `dbt_column` with that name (when present in the catalog). Incidents that cannot be linked to a resource already in Euno are skipped. The integration run report lists how many incidents were skipped and why. ### Warehouse mapping If Elementary reports `database.schema` values that differ from what Euno uses for the same warehouse (for example, because of connection aliases), configure **warehouse mapping** in the source settings. Keys are matched **case-insensitively** against the `database.schema` reported for each incident. Keys wrapped in `/…/` are treated as regular expressions; you can use `$1`, `$2`, and so on in the mapped value to reuse parts of the key. ```json { "elem_db.prod_schema": "euno_db.prod_schema", "/elem_db\\.(.*)/": "euno_db.$1" } ``` In the example above, the first entry maps an exact `database.schema` pair, and the second remaps all schemas under `elem_db` to their equivalent under `euno_db`. ### Relationships The Elementary integration does not create relationships between resources. It adds incident properties to existing resources. Lineage and hierarchy from your other sources are unchanged. ### Usage in EQL You can filter and search by Elementary incident properties in Euno's search and EQL: **Examples**: ``` # dbt models with at least one active Elementary incident has_active_incident = true and type = "dbt_model" # dbt sources with at least one active Elementary incident has_active_incident = true and type = "dbt_source" # Warehouse columns currently flagged by Elementary has_active_incident = true and type = "column" # dbt columns currently flagged by Elementary has_active_incident = true and type = "dbt_column" # Resources with more than 2 active incidents active_incidents_count > 2 # Warehouse tables touched by Elementary incidents has_active_incident = true and type = "table" ``` # Custom Integrations Build custom integrations to connect Euno to any data platform or tool in your stack. ## Available Options * [**Custom Integration**](/sources/custom-integrations/custom-integration) β€” Push observations via inline JSON POST or bulk file upload ([properties reference](/sources/custom-integrations/custom-integration/custom-integration-properties-reference)) * [**OpenLineage API**](/sources/transformation-etl/openlineage-integration) β€” Use OpenLineage standard * [**Python SDK**](/developer-reference/euno-sdk) β€” Programmatic integration ## What You Can Build Custom integrations allow you to: * **Connect Any Platform** β€” Integrate tools not natively supported * **Push Metadata** β€” Add resources manually or programmatically via HTTP POST * **Track Custom Resources** β€” Define your own resource types * **Declare Lineage** β€” Wire upstream dependencies between tables, scripts, and other resources ## Getting Started 1. Choose your integration method from the options above 2. Follow the setup guide for your chosen method 3. Configure your integration 4. Test and validate **Setup Time:** 30-60 minutes depending on complexity *** ## Next Steps * [View All Integrations](/sources) * [Learn About the SDK](/developer-reference/euno-sdk) * [Explore Custom Properties](/using-euno/metadata-activation/metadata-tags) # Custom Integration Euno's Custom integration provides the simplest way to ingest data resources directly into your data model. This integration accepts raw observation objects, making it perfect for custom data sources, manual uploads, or integrating systems that don't have dedicated Euno integrations. ### How It Works The integration follows these steps: 1. **Provides a secure endpoint** Euno generates a unique trigger secret and endpoint URL for receiving custom observations 2. **Accepts Observation Objects** The integration accepts both single observations and arrays of observations via HTTP POST, using only the [supported Custom integration properties](/sources/custom-integrations/custom-integration/custom-integration-properties-reference) 3. **Validates and Processes** * Validates each observation against supported properties and value types * Stores observations directly in your data model * Tracks processing statistics and validation errors ## Setting up Euno's Custom Integration ### Step 1: Access the Sources Page 1. Navigate to the **Sources** page in the Euno application 2. Click on the **Add New Source** button 3. Select **Custom** from the available integrations ### Step 2: General Configuration | Configuration | Description | | ------------- | --------------------------------------------------------------------------- | | **Name**\* | Descriptive name for your Custom source (for example, "Manual Data Upload") | \* Required field. Custom integration requires no additional configuration fields. It is a push-based integration with no schedule β€” observations are sent whenever your pipeline or script runs. ### Step 3: Resource Cleanup Options Configure automatic **resource cleanup** options to manage outdated resources: * **Immediate Cleanup (default)**: Remove resources not detected in the most recent successful source integration run * **No Cleanup**: Keep all resources indefinitely, even if they are no longer detected in a run Time-based cleanup is not available for Custom sources. ### Step 4: Save Configuration Click **Save**. Euno opens a modal with: * Your **trigger secret** β€” copy and store it securely; it is shown only once * The **`run` endpoint URL** β€” for inline JSON observations * The **`prepare-upload` endpoint URL** β€” for bulk file uploads (see [Handling High Volume of Observations](/sources/custom-integrations/custom-integration/handling-high-volume-of-observations)) * A link to custom resource type registration guidance Use the trigger secret as a Bearer token in the `Authorization` header for the Custom integration `run` and `prepare-upload` calls. ### Step 5: API Endpoints Custom integration exposes two endpoints. Replace `YOUR_ACCOUNT_ID` and `YOUR_INTEGRATION_ID` with the values from your Euno account. | Endpoint | Use for | | ---------------------------------------------------------- | ---------------------------------------------------------------------------------- | | `POST .../integrations/YOUR_INTEGRATION_ID/run` | Inline JSON β€” single observations or small batches | | `POST .../integrations/YOUR_INTEGRATION_ID/prepare-upload` | Bulk file upload β€” request a signed URL for a `.json`, `.jsonl`, or `.ndjson` file | Both endpoints require `Authorization: Bearer YOUR_TRIGGER_SECRET`. The `/run` and `prepare-upload` URLs are shown in the post-save modal and in **Reset Trigger Key**. You can also construct them from your account ID and integration ID as in the examples below. **Rotating your trigger secret:** Open your Custom source on the **Sources** page and click **Reset Trigger Key** to generate a new secret. This invalidates the previous secret immediately. ## Register Custom Resource Types Every observation must include `properties.type`. Built-in Euno types such as `table` and `column` work without registration. Before sending a new custom type such as `campaign_view`, register it for the account: This endpoint uses normal Euno API authentication and requires permission to manage account settings. Do not use the custom integration trigger key here; the trigger key is only for `/run` and `/prepare-upload`. ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_EUNO_ACCESS_TOKEN" \ -d '{ "type": "campaign_view", "display_name_singular": "Campaign View", "display_name_plural": "Campaign Views", "description": "Custom marketing campaign view" }' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/resource_types ``` The type name must use lowercase ASCII letters, numbers, and underscores, and must start with a letter. Type registration does not currently support rename, delete, or type-level icon configuration. If a run contains an unregistered custom type, Euno skips only those observations. Other valid observations in the same run continue, and the run report lists the unknown type so you can register it and retry. If every observation in a run is skipped this way, the run fails with the rejected types listed in the report. ## Sending Custom Observations See [Custom Integration Properties Reference](/sources/custom-integrations/custom-integration/custom-integration-properties-reference) for the full list of supported properties and observation format constraints. ### Example: Table with Tags and Metadata Here's a complete example of a custom observation for a table with tags: ```json { "uri": "custom.analytics.customer_summary", "properties": { "type": "table", "name": "customer_summary", "description": "Aggregated customer metrics and analytics", "database_technology": "snowflake", "tags": ["customer_data", "analytics", "production"], "meta": { "owner": "data-team", "environment": "production", "classification": "internal" } } } ``` ### cURL Command Examples #### Single Observation Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '{ "uri": "custom.sales.monthly_revenue", "properties": { "type": "table", "name": "monthly_revenue", "description": "Monthly revenue aggregations by product line", "database_technology": "snowflake", "tags": ["revenue", "monthly", "sales"], "meta": { "department": "sales", "update_frequency": "monthly" } } }' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` #### Multiple Observations Upload ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TRIGGER_SECRET_HERE" \ -d '[ { "uri": "custom.marketing.campaigns", "properties": { "type": "table", "name": "campaigns", "tags": ["marketing", "campaigns"] } }, { "uri": "custom.marketing.campaign_analytics", "properties": { "type": "campaign_view", "name": "campaign_analytics", "description": "Derived analytics from campaign performance", "tags": ["marketing", "analytics", "derived"] } } ]' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/run ``` ## What Euno Ingests Custom integration accepts observations for built-in Euno resource types and custom resource types registered for your account. Each observation must include `type` and `name`. See the [properties reference](/sources/custom-integrations/custom-integration/custom-integration-properties-reference) for supported metadata and lineage fields. ## Handling High Volume of Observations The inline `/run` examples above are ideal for small payloads. When you need to upload **many observations in a single run** β€” especially from a pipeline or a large export β€” use Euno's **prepare-upload** flow to upload a `.json`, `.jsonl`, or `.ndjson` file directly to cloud storage. This avoids HTTP request size limits and is the recommended approach for production bulk ingestion. [β†’ Handling High Volume of Observations](/sources/custom-integrations/custom-integration/handling-high-volume-of-observations) ## Example: Homegrown ELT Pipeline For a step-by-step example of documenting a Python job that reads from one Snowflake table and writes to another β€” including which observations to emit and how to wire lineage β€” see: [β†’ Example: Homegrown ELT Pipeline](/sources/custom-integrations/custom-integration/example-homegrown-elt-pipeline) # Custom Integration Properties Reference The Custom integration accepts observations for built-in Euno resource types and custom resource types registered for your account. Only a **closed subset** of Euno built-in properties may be set in each observation. If an observation includes a property that is not listed below, the observation is **skipped** and counted in the run report as invalid. For observation structure and upload flows, see [Custom Integration](/sources/custom-integrations/custom-integration) and [Handling High Volume of Observations](/sources/custom-integrations/custom-integration/handling-high-volume-of-observations). ## Observation shape Each observation is a JSON object with: | Field | Required | Description | | ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `uri` | Yes | Unique identifier for the resource. See [URI structure](/developer-reference/technical-concepts/universal-resource-identifier-uri). | | `properties` | Yes | Object containing `type`, `name`, and any optional supported properties below. | ### Example ```json { "uri": "custom.analytics.customer_summary", "properties": { "type": "table", "name": "customer_summary", "description": "Aggregated customer metrics", "database_technology": "snowflake", "tags": ["analytics", "production"], "meta": { "owner": "data-team" }, "explicit_additional_dependencies": [ "custom.analytics.raw_customers" ] } } ``` ## Supported properties | Property | Required | Type | Description | | ---------------------------------- | -------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | **Yes** | string | Built-in Euno resource type such as `table` or `column`, or a custom type registered for your account such as `campaign_view`. | | `name` | **Yes** | string | Display name of the resource. | | `description` | No | string | Human-readable description of the resource. | | `raw_code` | No | string | Raw SQL or other source code associated with the resource. | | `qualified_code` | No | string | Code for the resource, fully qualified based on known column schemas and upstream tables. | | `tags` | No | array of strings | Tags applied to the resource. | | `meta` | No | object | Arbitrary key-value metadata (JSON object). | | `database_technology` | No | string | Warehouse or database technology where the resource lives (for example `snowflake`, `bigquery`, `redshift`). | | `table_dependencies` | No | array of strings | Upstream resource URIs for **resource-level** lineage on any type **except** `table` and `column`. Each entry must be a valid URI string. See [Lineage properties](#lineage-properties). | | `explicit_additional_dependencies` | No | array of strings | Upstream URIs for **`type: "table"`** and **`type: "column"`** only. On tables, merged into resource-level lineage; on columns, merged into field-level lineage. Do **not** use on other types. See [Lineage properties](#lineage-properties). | | `upstream_fields` | No | array of strings | Upstream field URIs for **field-level** lineage on field-like resources other than `column` (for example `dbt_metric`, `dbt_dimension`, `snowflake_metric`, `snowflake_dimension`, `looker_measure`, `looker_dimension`, `omni_measure`, `omni_dimension`). Each entry must be a valid URI string. Do **not** use on `type: "column"`; use `explicit_additional_dependencies` instead. See [Lineage properties](#lineage-properties). | | `parent_container` | No | string | URI of the parent container of the resource. Creates a `has parent` / `has child` relationship in the graph. The value must be a valid URI string. | | `parent_container_name` | No | string | Display name of the parent container. Normally derived automatically from the parent's `name` once the parent is observed; setting it directly on the child is only useful when the parent observation lags or is omitted. | | `container_chain` | No | array of objects | Ancestry of the resource from immediate parent outward. Each element is a `{uri, name, native_type, normalized_type}` object. | | `icon_url` | No | string | URL to an icon image for the resource. See [Custom type icons](#custom-type-icons) below. | ## Required properties on every observation Every observation must include both `type` and `name` in its `properties` object. Observations missing either property are skipped and reported in the run report. Observations with allowed properties but invalid values (for example a malformed `icon_url`, or `tags` provided as a string instead of an array) are also skipped and counted as invalid. Custom resource types must be registered before observations use them. Built-in types such as `table` and `column` do not need registration. If an observation uses an unregistered custom type, Euno skips that observation and records the unknown type in the run report. Register custom types with `POST /accounts/YOUR_ACCOUNT_ID/resource_types` before sending observations. ## Lineage properties Custom integration supports **three** lineage input properties. Which one to use depends on the resource `type` and whether the lineage is resource-level or field-level: | Resource `type` | Lineage kind | Property to set | Relationship created in Euno | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ---------------------------------- | ---------------------------- | | `table` | Resource-level | `explicit_additional_dependencies` | `has upstream` | | `column` | Field-level | `explicit_additional_dependencies` | `has upstream_fields` | | **Any other type** (for example `campaign_view`, `acme_transform_script`) | Resource-level | `table_dependencies` | `has upstream` | | **Any field-like type other than `column`** (for example `dbt_metric`, `dbt_dimension`, `snowflake_metric`, `snowflake_dimension`, `looker_measure`, `looker_dimension`, `omni_measure`, `omni_dimension`) | Field-level | `upstream_fields` | `has upstream_fields` | {% hint style="info" %} **Rules** * **`explicit_additional_dependencies`** β€” use **only** on `table` and `column`. On tables, do **not** use `table_dependencies`; it will not persist as lineage. * **`table_dependencies`** β€” use on **every other type** for resource-level lineage. Do **not** use `explicit_additional_dependencies` on other types; it is stored but **does not create lineage edges**. * **`upstream_fields`** β€” use for field-level lineage on field-like types **other than** `column`. For `type: "column"`, use `explicit_additional_dependencies` instead β€” Euno recomputes `upstream_fields` for column-typed resources from `explicit_additional_dependencies` (and inferred code-based column dependencies), so a value set directly on a column would be overwritten. {% endhint %} For relationship semantics in Euno, see [Relationships](/developer-reference/technical-concepts/relationships). ### `explicit_additional_dependencies` Use **only** when `type` is `table` or `column`: ```json "explicit_additional_dependencies": [ "custom.analytics.staging_orders", "custom.acme_elt.enrich_orders" ] ``` ### `table_dependencies` Use when `type` is **anything other than** `table` or `column` and you want resource-level lineage: ```json "table_dependencies": [ "custom.raw.orders", "custom.raw.customers" ] ``` ### `upstream_fields` Use on field-like types **other than** `column` to declare field-level lineage directly: ```json { "uri": "custom.analytics.revenue", "properties": { "type": "dbt_metric", "name": "revenue", "upstream_fields": [ "custom.analytics.orders.amount", "custom.analytics.orders.tax" ] } } ``` ## Parent/child relationships Use `parent_container` to attach a resource to its parent in the graph. The parent itself must also be observed (in the same run or a previous one) so the edge can resolve. `container_chain` optionally captures the ancestry from the immediate parent outward, as a list of `{uri, name, native_type, normalized_type}` objects. Example β€” a `dbt_metric` child observation, the `semantic_model` parent observation, and the ancestry: ```json [ { "uri": "custom.acme.revenue_model", "properties": { "type": "semantic_model", "name": "revenue_model" } }, { "uri": "custom.acme.revenue_model.revenue", "properties": { "type": "dbt_metric", "name": "revenue", "parent_container": "custom.acme.revenue_model", "container_chain": [ { "uri": "custom.acme.revenue_model", "name": "revenue_model", "native_type": "semantic_model", "normalized_type": "semantic_model" } ] } } ] ``` Notes: * `parent_container_name` is normally derived automatically from the parent's `name` once the parent is observed; supply it directly on the child only when the parent observation lags or is omitted. * Every entry in `container_chain` must include all four keys (`uri`, `name`, `native_type`, `normalized_type`); the `uri` must be a valid URI. Malformed entries cause the whole observation to be skipped and reported in the run report. ## Custom type icons Use `icon_url` to supply a custom icon for resources whose `type` is not a built-in Euno resource type (for example `acme_transform_script`). ```json { "uri": "custom.acme_elt.enrich_orders", "properties": { "type": "acme_transform_script", "name": "enrich_orders.py", "icon_url": "https://airflow.apache.org/images/airflow-icon.svg" } } ``` Requirements: * The URL must use `http://` or `https://` and be at most 2048 characters. * For the icon to render in the Euno UI, the URL path should end with a common image extension such as `.svg`, `.png`, `.jpg`, `.gif`, or `.webp`. * Host the image at a URL your Euno users can reach (public HTTPS is typical). * `table`, `database`, and `database_schema` resources use the warehouse icon from `database_technology` instead of `icon_url`. Set `icon_url` on each resource that should display the icon (typically every resource of the same custom type uses the same URL). ## Related documentation * [Custom Integration setup](/sources/custom-integrations/custom-integration) * [Example: Homegrown ELT Pipeline](/sources/custom-integrations/custom-integration/example-homegrown-elt-pipeline) * [Handling High Volume of Observations](/sources/custom-integrations/custom-integration/handling-high-volume-of-observations) * [Universal Resource Identifier (URI)](/developer-reference/technical-concepts/universal-resource-identifier-uri) * [Relationships](/developer-reference/technical-concepts/relationships) # Handling High Volume of Observations Use this flow when you need to ingest **many observations in a single integration run** and inline JSON POST requests are too large or impractical. For small payloads (a single observation or a modest JSON array), continue using the inline [`/run` endpoint](/sources/custom-integrations/custom-integration#sending-custom-observations) documented on the main Custom Integration page. ## When to Use File Upload | Approach | Best for | | ------------------------------ | ------------------------------------------------------------------------ | | **Inline JSON POST to `/run`** | Quick tests, single observations, small batches | | **Prepare-upload file flow** | Large exports, ETL pipelines, files that exceed HTTP request size limits | The prepare-upload flow uploads your file directly to cloud storage, then automatically triggers **one integration run** that processes every observation in the file. ## How It Works The integration follows these steps: 1. **Request a signed upload URL**\ Call `prepare-upload` with your integration key. Euno creates a pending operation and returns a signed URL for your observations file. 2. **Upload the observations file**\ PUT your `.json`, `.jsonl`, or `.ndjson` file to the signed URL. 3. **Automatic processing**\ When the upload completes, Euno starts the integration run, validates each observation, and ingests them into your data model. Track progress on the **Sources** page under the integration's run history. ## Supported File Formats Upload **exactly one** observations file per run. ### JSON (`.json`) A single observation object, or a JSON **array** of observation objects: ```json [ { "uri": "custom.marketing.campaigns", "properties": { "type": "table", "name": "campaigns", "tags": ["marketing", "campaigns"] } }, { "uri": "custom.marketing.campaign_analytics", "properties": { "type": "view", "name": "campaign_analytics", "tags": ["marketing", "analytics"] } } ] ``` ### JSONL / NDJSON (`.jsonl`, `.ndjson`) β€” recommended for large volumes One observation per line (newline-delimited JSON). Empty lines are ignored. ```jsonl {"uri": "custom.marketing.campaigns", "properties": {"type": "table", "name": "campaigns"}} {"uri": "custom.marketing.campaign_analytics", "properties": {"type": "view", "name": "campaign_analytics"}} ``` Each line must be a complete JSON object with the same structure as inline observations (`uri` plus supported `properties`). See [Custom Integration Properties Reference](/sources/custom-integrations/custom-integration/custom-integration-properties-reference). ## Prerequisites * A configured **Custom** source in Euno * Your **integration key** (Bearer token) from the Custom source's trigger key settings * Your **account ID** and **integration ID** (visible in the upload endpoint URL shown in the Euno UI) ## Upload Sequence (cURL) Replace `YOUR_ACCOUNT_ID`, `YOUR_INTEGRATION_ID`, and `YOUR_INTEGRATION_KEY_HERE` with your values. ### Step 1: Request a signed upload URL ```bash curl -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_INTEGRATION_KEY_HERE" \ -d '{"filename": "observations.jsonl", "content_type": "application/x-ndjson"}' \ https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload ``` Example response: ```json { "operation_id": 12345, "upload": { "url": "https://storage.googleapis.com/...", "method": "PUT", "headers": { "Content-Type": "application/x-ndjson" } } } ``` Supported `content_type` values: | File extension | `content_type` | | ------------------- | ---------------------- | | `.json` | `application/json` | | `.jsonl`, `.ndjson` | `application/x-ndjson` | If you omit `content_type`, Euno selects the appropriate default based on the filename. ### Step 2: Upload the file to the signed URL Use the `url`, `method`, and `headers` from the prepare-upload response: ```bash curl -X PUT \ -H "Content-Type: application/x-ndjson" \ --data-binary @observations.jsonl \ "SIGNED_URL_FROM_STEP_1" ``` When the upload succeeds, Euno automatically starts processing the file. No further API call is required. ### Step 3: Verify the run 1. Open the **Sources** page in Euno 2. Select your Custom source 3. Review the latest run in the operations history 4. Open the run report for counts of valid observations, skipped rows, and validation errors ## Python Upload Script This script follows the same prepare-upload sequence as the cURL example above. ### Prerequisites * Python 3.6+ * `requests` library (`pip install requests`) ### Script ```python import json import os from pathlib import Path import requests # Configuration β€” replace with your values endpoint_url = ( "https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/" "integrations/YOUR_INTEGRATION_ID/prepare-upload" ) integration_key = "your_key_here" observations_file_path = "observations.jsonl" headers = { "authorization": f"Bearer {integration_key}", "content-type": "application/json", } def content_type_for_filename(filename: str) -> str: lower_name = filename.lower() if lower_name.endswith(".json"): return "application/json" if lower_name.endswith((".jsonl", ".ndjson")): return "application/x-ndjson" raise ValueError("File must be .json, .jsonl, or .ndjson") def get_signed_upload(): filename = Path(observations_file_path).name body = { "filename": filename, "content_type": content_type_for_filename(filename), } response = requests.post(endpoint_url, headers=headers, json=body, timeout=30) if response.status_code != 200: print(f"Failed to obtain signed URL (status {response.status_code}): {response.text}") return None return response.json() def upload_observations_file(prepare_response: dict) -> bool: upload = prepare_response["upload"] upload_url = upload["url"] upload_headers = upload.get("headers", {}) try: with open(observations_file_path, "rb") as observations_file: response = requests.put( upload_url, data=observations_file, headers=upload_headers, timeout=300, ) if response.status_code not in (200, 201): print(f"Upload failed (status {response.status_code}): {response.text[:200]}") return False return True except Exception as exc: print("An error occurred during upload:", exc) return False def main(): if not os.path.exists(observations_file_path): print(f"Error: file '{observations_file_path}' not found.") return False print("Requesting signed upload URL...") prepare_response = get_signed_upload() if not prepare_response: return False print(f"Operation ID: {prepare_response.get('operation_id')}") print("Uploading observations file...") if upload_observations_file(prepare_response): print("Upload succeeded. Euno will process the file automatically.") print(json.dumps(prepare_response, indent=2)) return True print("Upload failed.") return False if __name__ == "__main__": main() ``` ### Usage 1. **Configure the script**: Set `endpoint_url`, `integration_key`, and `observations_file_path` 2. **Install dependencies**: `pip install requests` 3. **Prepare your file**: Export observations as `.json`, `.jsonl`, or `.ndjson` 4. **Run the script**: `python upload_observations.py` 5. **Verify in Euno**: Confirm the run completed successfully in the source's operation history ## Important Notes * **One input source per run** β€” provide either inline JSON (via `/run`) **or** one observations file, not both. * **One file per run** β€” each prepare-upload request accepts a single observations file. * **Invalid rows are skipped** β€” malformed lines or observations that fail validation are counted in the run report; the run continues processing the rest of the file. * **Reuse your integration key** β€” the same Bearer token used for inline `/run` requests works for prepare-upload. * **Generating a new trigger key** β€” go to the **Sources** page, open the three-dot menu on your Custom source, and select **Reset Trigger Key**. ## Related Documentation * [Custom Integration setup and inline uploads](/sources/custom-integrations/custom-integration) --- [Next Page](/llms-full.txt/1)