dbt Cloud

Euno's integration with dbt Cloud supports auto-discovery of dbt resources. It automates the retrieval and processing of dbt artifacts from completed runs. This allows for seamless data synchronization and analysis using the latest available dbt job runs.

How It Works

The integration follows these steps:

1. Retrieve Jobs from dbt Cloud

   • Using a dbt Cloud service token, the integration queries the dbt Cloud API to fetch all jobs associated with the projects scoped by the dbt Cloud service token provided to the integration.

2. Find the Latest Unprocessed Run for Each Job

   • For each job, the integration identifies the latest run that has completed successfully and has not yet been processed by Euno.

3. Retrieve Artifacts from the Run

   • Once an eligible job run is found, the integration retrieves its associated artifacts:

     • run_results.json

     • manifest.json

     • semantic_manifest.json

     • catalog.json

4. Process the Artifacts

   • The integration processes the retrieved artifacts to extract relevant information and adds the discovered resources to Euno's data model.

Advanced Configuration

Filtering Jobs

It's possible to define patterns to include or exclude jobs based on:

• Job names

• Environments

• Branch names

Without Advanced Filtering

If no filtering patterns are set, the integration will process all jobs retrieved from dbt Cloud. This means every available job in the configured projects will be considered.

With Advanced Filtering

By defining inclusion/exclusion patterns, users can:

• Focus on specific jobs that match certain naming conventions.

• Exclude jobs from non-production environments.

• Restrict processing to specific Git branches (e.g., main or production).

This provides better control over which jobs are processed, reducing unnecessary artifact downloads and processing time.

Prerequisites

• A dbt Cloud service token with sufficient Read-only permission. See dbt Cloud's documentation on how to create service tokens.
• The service token should grant access to the dbt projects with the job run(s) to retrieve.

Setting up Euno's dbt Cloud Integration

Step 1: Configure New dbt Cloud Source in Euno

Step 1: Access the Sources Page

1. Navigate to the Sources page in the Euno application.

2. Click on the Add New Source button.

Step 2: General Configuration

Asterik (*) means a mandatory field.

Configuration	Description
Name*	Enter a name for your dbt Cloud source (e.g., "dbt - Marketing Models")
dbt Cloud Account ID*	Enter you dbt Cloud Account ID. It can be found in the URL, e.g. https://cloud.getdbt.com/settings/accounts/{account_id}/pages/projects/{project_id}
dbt Cloud Project ID*	Enter you dbt Cloud Project ID. It can be found in the URL, e.g. https://cloud.getdbt.com/settings/accounts/{account_id}/pages/projects/{project_id}
dbt Cloud Api Token*	Enter the service token created in the previous section
dbt Cloud Job ID (optional)	Specific dbt Cloud job ID to process. If not specified, all jobs under the dbt projects associated with integration will be processed.

Step 3: Resource Cleanup Options

To keep your data relevant and free of outdated resources, Euno provides automatic resource cleanup options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: Resource Sponsorship in Euno.

• Time-Based Cleanup (default): Remove resources that were last detected X days before the most recent successful source integration run (user-defined X, default is 7 days).

• Immediate Cleanup: Remove resources not detected in the most recent successful source integration run.

• No Cleanup: Keep all resources indefinitely, even if they are no longer detected.

Step 4: Scheduling Updates

1. Enable the Schedule option.

2. Choose:

   1. Weekly: Set specific days and times.

   2. Hourly: Define the interval in hours (e.g., every 8 hours).

Step 5: Advanced Settings (Optional)

Click on the 'Advanced' section to display these additional configurations.

Configuration	Description
Job Name	Define patterns to include or exclude jobs based on their names.
Job Environment	Define patterns to include or exclude jobs based on their environment.
Job Branch	Define patterns to include or exclude jobs based on their branch name.
dbt Cloud URL	If you have a dedicated URL to access dbt Cloud enter it here. Defaults to "https://cloud.getdbt.com"