📈Data Model Sync

As your organization's shared data model is constantly evolving, Euno helps ensure that these changes are reflected in your BI tool. The data model sync automation is designed to ensure that your BI application reflects the most up-to-date shared data model of your organization.

How the Data Model Sync Works

Euno supports the automated synchronization of data model revisions, facilitating seamless updates from the dbt manifest to LookML files. Here’s how it works:

New revision: When a change is detected in the dbt manifest, Euno generates LookML files based on the updated manifest/catalog. Euno can either sync all models or be configured for a selective sync, in which models are synced based on a tag or or an input list. The sync process is triggered by changes in the dbt manifest. If no changes are detected, no sync is triggered.
Branch checkout: Euno checks out the selected LookML repository and branch configured in the data model sync configuration
PR/Commit: Euno generates a PR/commit, depending on the configurations

What Fields are Supported in the Data Model Sync?

Columns

Euno supports syncing all dbt columns in the model to dimensions in LookML. The dimension data type will be dependent on the dialect used in your DW (see Dialect Conversion Charts). For time dimensions, Euno automatically syncs these columns as a dimension_group of type; time, in LookML.

dimension_group: dimension_group_name {
  type: time
  sql: SQL expression ;;  
  timeframes: [raw, time, hour_of_day, date, day_of_the_week, week,  
              month, quarter, year]
  datatype: # field type conversion - dependent on dialect
  convert_tz: yes
}

Metrics

Euno supports automatically syncing dbt metrics to Looker, if the metric is built from a single semantic model. Meaning; the measures or metrics used to create the metric is from the same semantic model, and are of type:

Simple
Ratio
Derived

If the metric includes a filter, Euno will sync an additional dimension specifically for the filter that will hold the filter's logic within the 'sql' parameter of that dimension. The dimension will be of type 'yesno', with the parameter 'hidden' set to 'yes', tagged with 'euno-generated-filter-dimension', and named '<measure_name>_euno_generated_filter_dim' , which can be overridden by placing the macro key; under the meta section of the metric. Inside the measure, under the 'filter' parameter, it will reference this field with the dimension as the key and 'yes' as the value. Essentially, this means that if the data matches the logic in the filter, the metric will be applied to the data.

Note: These filters will only be generated for simple metrics and will specifically apply to dbt filter templates utilizing entities, dimensions, or time dimensions within the identical semantic model. Metrics filtered by other metrics are not currently supported.

Below is an example of how the Data Model Sync syncs a metric with a filter (simple_metric) in LookML.

dbt Metric with Filter

dbt Metric Synced to LookML with Filter

semantic_models:
  - name: users
    model: ref('users')
    entities:
      - type: primary
        name: pe
        description: This is a delphi auto-generated primary entity (Change according to desired primary entity)
    defaults:
      agg_time_dimension: time_dim
    dimensions:
      - name: time_dim
        type: time
        type_params:
          time_granularity: day
  measures:
      - name: simple_measure
        label: simple_measure
        agg: sum
        expr: 1
metrics:
  - name: simple_metric
    label: simple_metric
    type: simple
    type_params:
      measure: simple_measure
    filter: |
      {{ TimeDimension('pe__time_dim', 'day') }} IS NULL

  dimension: simple_metric_euno_generated_filter_dim {
    type: yesno
    hidden: yes
    sql: DATE_TRUNC('day', ${TABLE}.time_dim) IS NULL ;;
    tags: ["euno-generated-filter-dimension"]
  }

  measure: simple_metric {
    type: sum
    sql: 1 ;;
    filters: [simple_metric_euno_generated_filter_dim: "TRUE"]
  }

    type: yesno
    sql: metric_time__day IS NOT NULL ;;
  }

Euno also supports syncing filters using our built-in macro 'getdelphi.io/lookml_filters'. As described below, the above macro key should be placed in the metric's config, with the value being a list containing the 'filter_field' as the key and the 'filter_value' as the value inside the list. If the metric includes both a filter, and the above macro, the macro will override the filter within the metric.

getdelphi.io/lookml_filters: '[["filter_field", "filter_value"]]'

Limitations of the Data Model Sync

Euno does not support syncing the following dbt metrics:

Cross model metric, a metric that is built from metrics across multiple semantic models
Cumulative metrics
Conversion metrics
Metrics which include an offset window

Formatting Fields in the Data Model Sync

Users can utilize Euno’s built-in macros using field meta to sync specific formatting like labels and value_format to LookML. See Customizing Generated LookML Files.

Configuring the Data Model Sync

The Data Model Sync section explains how to set up, configure, and manage the synchronization of data models between dbt and Looker using Euno's automation framework. This guide covers running the sync locally, automating the sync process, and pushing the sync to Looker repositories.

Overview of the Data Model Sync Interface

The Data Model Sync interface is organized into two main tabs:

Data Sync Jobs Tab
- This tab is where you create, configure, edit, and manage Data Sync Jobs.
- Jobs define how, when, and where data model synchronization occurs. Each job is displayed as a tile with key details, including target repositories, sync triggers, and configuration options.
- Use this tab to manage all your jobs and initiate manual syncs.
Data Sync Runs Tab
- This tab tracks the execution history of Data Sync Jobs.
- Each run corresponds to a job execution and displays detailed information about its lifecycle, including status (Pending, Running, Completed, Failed, or Canceled) and artifacts (e.g., LookML packages).
- Use this tab to monitor the status of your syncs and download generated artifacts.

Both tabs work together to provide a complete view of your Data Model Sync operations: the Jobs tab focuses on setup and configuration, while the Runs tab provides visibility into execution and outcomes.

Setting Up the Data Model Sync

1. Creating a Data Sync Job

A Data Sync Job defines the configuration for synchronizing your data models to your BI tool.

Steps:

Navigate to Automations > Data Model Sync and open the Data Sync Jobs Tab.
- If no jobs exist, click the Create New Job button.
Fill in the Configuration Details:
- General Settings:
  - Job Name: Assign a name for easy identification.
  - SQL Dialect: Select the appropriate SQL dialect for your DW that Looker is connected to. For details on how we convert certain data types to Looker data types, click here:
    Snowflake
    Databricks
    Trino
    Presto
    BigQuery (Standard SQL or Legacy SQL)
- Sync Trigger: Choose whether the sync should be triggered automatically.
  - By default, the automatic trigger is not configured. To automate the sync, toggle on the automatic sync option.
  - If the automatic sync is enabled, select the Euno project that should trigger the sync. The connected job or dbt artifacts defined in the selected Euno project will trigger the sync whenever a change is detected in the dbt manifest.
- Sync Target: The sync target is where Euno will write the LookML artifacts from the sync.
  - Select the target repository
    If a sync target has been enabled, a target repository must be selected, or the configuration cannot be saved.
  - Select the target branch
    If a sync target has been enabled, a target branch must be selected, or the configuration cannot be saved.
  - Select the target directory
    The default behavior is for Euno to push to a separate directory named /dbt_sync. This can be configured to any directory you prefer.
    This directory should be a separate directory used exclusively for Euno sync purposes. The data model sync works by replacing all files in the directory. Therefore, this directory should only be managed by Euno, as any manual changes will be overridden during the next sync.
  - Choose a merge strategy:
    Create Pull Request (default)
    Commit Directly to Branch
- What To Sync: Select the sync contents.
  - Select dbt project: Select the dbt project which is being synced.
  - dbt Models:
    Sync All Models: Includes all models from the selected dbt project.
    Selective Sync: Specify the desired models by typing the model name and pressing Enter to add it.
    Models with Specific Tags: Only models with specified tags will sync.
  - Metrics:
    Include All Metrics: Sync all supported metrics associated with the selected models.
    No Metrics: Exclude metrics from the sync.
Save or Cancel the Job:
- Click Save to apply your configuration. Confirm your action in the dialog box.
- Use Cancel to discard changes.

Saved jobs appear as tiles under the Data Sync Jobs Tab. Each tile includes job details and options to edit, delete, or manually trigger the job.

2. Running the Data Model Sync Locally

To validate data model sync outputs prior to automating, you can run the sync manually without pushing changes to the Looker repository. To run the job locally, either create a new job and leave the Target Information option disabled, or edit an existing job (see image below), toggle off Target Information, and save the job. If no target information is configured, the job will run locally, and the artifacts can be downloaded to your machine.

3. Automating the Data Sync

Automated syncs ensure data models remain up to date without manual intervention.

Steps:

Enable Automatic Sync During Job Configuration.
Specify the Euno project that triggers the sync. The Euno project is connected to either a dbt Cloud job, a webhook, or dbt artifacts, and will trigger the sync to run automatically whenever a change is detected in the dbt manifest.
- Please note: The sync is triggered once the latest dbt artifacts are published ().
Save the job. The sync will automatically execute based on updates to the project.

4. Configuring Sync to Push to Looker

For environments integrated with Looker:

Set the Target Repository, Branch, and Directory during job configuration.
Choose a merge strategy:
- Create Pull Request (default): Push changes via a PR for review.
- Commit Directly: Automatically merge changes into the branch.
Save the job. Changes sync directly to the specified Looker repository.

Managing Data Sync Jobs and Runs

Data Sync Jobs Tab

This tab displays all configured sync jobs as tiles, with the following details:

Header: Job name, status of last run, and an ellipsis menu for options:
- Edit job: Opens the job configuration form.
- View jobs runs: Opens the Data Sync Runs tab with the table filtered by the specific job.
- Delete job: Confirm deletion via dialog box.
Body: Includes target project, automatic sync status, and target branch.
Footer: Buttons for manual sync or canceling a running sync, and the timestamp of the last time the job ran.

Data Sync Runs Tab

This tab displays the execution history of syncs in a table, with each row representing a different run. The table is sorted by timestamp and, by default, includes all runs from every job. You can filter the table by a specific job using the filter located in the top-right corner of the page.

Columns in the Table:

Timestamp: The timestamp of when the job was triggered.
Job: The name of the job.
Trigger: How the job was triggered—either automatic or manual.
Triggered By: What triggered the job (e.g. user or euno project)
Run Status:
- Pending: Sync triggered but not yet started.
- Running: Sync currently in progress.
- Completed: Sync finished successfully.
- Failed: Sync encountered an error.
- Canceled: Sync was manually canceled.
Action Buttons
- Rerun Job Button:
  - In the event of a failed job, users can click the "Rerun" button for a specific run to trigger a new sync job.
- Cancel Job Button:
  - While a job is running, users can cancel it by clicking the "Cancel Job" button .
- Download Run Artifacts:
  - For successful runs, users can download the contents of the job (e.g., LookML files) as a .zip file by clicking on the download button .
Open Report:
- Once a job has completed, whether successfully or with a failure, users can view the run report, which summarizes the events of the sync job by clicking the "Open repot" button .

Use the Runs Tab to monitor ongoing syncs, investigate failures, or retrieve generated artifacts.

Sync Data Model Now

To manually trigger a sync:

Navigate to the Data Sync Jobs Tab and select a job tile.
Click Run now to start the job.
- Progress and results will appear in the Data Sync Runs Tab.

Customizing LookML Structure for Auto-code Generation

Euno uses Jinja2 templates to generate the LookML files from a given dbt manifest and catalog.

Out-of-the-box, every model that is selected by the tags (or all models, if no tags are specified), will automatically generate a single view file in the target directory. Users can customize the behavior by either specifying meta fields in the model or the column, or by overriding the default Euno Jinja2 macros.

Customizing Generated LookML Files

Model level keys and tags

Add the below meta keys to a dbt model's .yml file in order to customize the corresponding synced LookML view.

Meta key

Description

Default

getdelphi.io/lookml_filename

Full path of the resulting view file Note: This will override lookml_directory, lookml_filename_prefix, and lookml_filename_suffix

./<model name>.view.lkml (The filepath needs to include the suffix for the file format)

getdelphi.io/lookml_directory or euno.ai/lookml_directory

Adds the file to a sub-directory within the directory configured in the data application sync setup (see )

No sub-directory by default

getdelphi.io/lookml_filename_prefix or euno.ai/lookml_filename_prefix

Adds a prefix to the view's file name (e.g. prefix_model-name.view.lkml) Note: This can be used in tandem with euno.ai/lookml_filename_suffix and euno.ai/lookml_directory, but not euno.ai/lookml_filename, which will override this value.

No filename prefix by default

getdelphi.io/lookml_filename_suffix or euno.ai/lookml_filename_suffix

Adds a suffix to the view's file name (e.g. model-name_suffix.view.lkml) Note: This can be used in tandem with euno.ai/lookml_filename_prefix and euno.ai/lookml_directory, but not euno.ai/lookml_filename, which will override this value.

No filename suffix by default

getdelphi.io/lookml_sql_table_schema

Schema to use for the table

Schema specified in the manifest file

getdelphi.io/lookml_sql_table_name

Name of the table

Table name specified in the manifest file

euno.ai/lookml_view_name or getdelphi.io/lookml_view_name

Name of the view

Model name

euno.ai/lookml_view_name_prefix

Adds a prefix to the default view name (e.g. prefix_model-name) Note: This can be used in tandem with euno.ai/lookml_view_name_suffix, but not euno.ai/lookml_view_name, which will override this value.

No view name prefix by default

euno.ai/lookml_view_name_suffix

Adds a suffix to the default view name (e.g. model-name_suffix) Note: This can be used in tandem with euno.ai/lookml_view_name_prefix, but not euno.ai/lookml_view_name, which will override this value.

No view name suffix by default

getdelphi.io/lookml_sync_timeframes or euno.ai/lookml_sync_timeframes

Only relevant for time dimensions. Overrides ALL the default timeframes within the view for a timestamp or date column. This is a comma separated list of time grains. Note: If this key is used at both the model and column level, the more specific option, the column, will override.

raw, time, hour_of_day, date, day_of_week, week, month, quarter, year

getdelphi.io/lookml_sync_convert_tz or euno.ai/lookml_sync_convert_tz

Only relevant for time dimensions. Overrides the default for converting by timezone in ALL the time and date dimensions in the view. Note: If this key is used at both the model and column level, the more specific option, the column, will override.

The default is set to yes.

dbt column / dbt metric level keys and tags

By default, Euno will generate a single dimension clause for each column and a single measure for each dbt metric (MetricFlow).

Please note that only simple metrics and ratios are currently supported for dbt Metrics. In addition, we do not support metrics across multiple dbt models.

Meta field/Config key/Tag

Description

Default

getdelphi.io/lookml_label

Overrides lookml label for the field

No label

getdelphi.io/lookml_view_label

Overrides the lookml view the field is placed

getdelphi.io/lookml_group_label

Overrides the field in lookml group_label

No group label

getdelphi.io/lookml_hidden

Overrides the field in lookml hidden

Not hidden

getdelphi.io/lookml_value_format

Overrides the field in lookml value_format

No value format

getdelphi.io/lookml_value_format_name

Overrides the field in lookml value_format_name

No value format name

getdelphi.io/lookml_extra

For this specific key we expect users to insert a string with a lookml compliment fragment to be placed under the dimension or measure. For example "can_filter: yes"

No extra fields

getdelphi.io/lookml_tags

Overrides fields tags in the lookml file

Use the tags from the metric or column

getdelphi.io/lookml_sync_sql

Overrides the sql used for this dimension, or measure

For dimension - ${TABLE}.<column name> For measure:

Simple - "expr" attribute of underlying measure in dbt
Ratio - ${measure}/${measure}

getdelphi.io/lookml_sync_dimension_name

Only relevant for dimensions. Overrides the name of the dimension

Column name used by default

getdelphi.io/lookml_sync_measure_name

Only relevant for measures. Overrides the name of the measure

Metric named used by default

getdelphi.io/lookml_primary_key

Only relevant for dimension. Overrides the primary key of the dimension

The primary key of the column, if it exists

getdelphi.io/skip_lookml_sync

Set this tag to skip this column/metric

The column/metric is included in the sync

getdelphi.io/lookml_sync_timeframes or euno.ai/lookml_sync_timeframes

Only relevant for time dimensions. Overrides the default timeframes for a timestamp or date column. This is a comma separated list of time grains.

raw, time, hour_of_day, date, day_of_week, week, month, quarter, year

getdelphi.io/lookml_sync_convert_tz or euno.ai/lookml_sync_convert_tz

Only relevant for time dimensions. Overrides the default for converting by timezone.

The default is set to yes.

getdelphi.io/lookml_filters

Only relevant for measures. Overrides the filter of the measure.

No filter

euno.ai/analytical_semantic_role

Only relevant for dbt columns. Adding this key with the value "measure" overrides the sync's default semantic role and syncs a column to a LookML measure.

Unless stated otherwise, the measure’s type defaults to sum. Adding the following key: euno.ai/default_aggregation with the desired aggregation as value (e.g., "sum", "average", "count_distinct", etc.) will override the aggregation type.

By default, a dbt column syncs to a LookML dimension

To configure any of the above options:

For columns - Insert the meta key & and overriding value underneath the column in the YML file the model is found.

For metrics - Insert the config key and overriding value in metric's YML file underneath the config key.

Customizing Generated LookML by Overriding Built-in Macros[

Euno will process any file in the dbt’s repository, under the directory .delphi/lookml_sync/macros/ as a Jinja2 template, and override any build-in macros found in those files.

Note: The macro overrides are taken from the head of a branch, while the sync is triggered by a dbt build. This means that if a macro is changed or removed, and a build or revision occurs in between these changes, the macro override changes might not take effect until the next build or revision occurs.

Deploying LookML Changes Made in GitHub to Looker

Euno syncs to LookML any changes introduced to dbt, by utilizing the GitHub integration to commit changes directly to the LookML repository in GitHub. However, Looker does not automatically pull these changes from GitHub and deploy them to the production environment.

You have the option to deploy changes to the project's production environment in Looker either manually for every commit or by setting up automatic pushes from the master branch.

Note: This step is crucial to ensure a successful sync with Looker.

Manually

To manually deploy the changes from a branch to the Looker’s production environment, follow the following steps from within Looker:

Enter “development mode”
Go to “Develop”, and pick the Looker project
Go to “Git Actions”
Click “Pull from…” and pick “Pull from remote”
Once the operation is done, click “Deploy to Production”

Automatically

Users can take advantage of Looker’s built-in Advanced deploy mode capability so that changes would automatically be deployed using either a webhook, API endpoint or manually using the deployment manager.

For Looker configurations involving GitHub, we can recommend configuring a webhook so that commits into the master branch would be automatically applied in Looker.

For more detailed instructions see Google’s instructions.

PreviousAutomations NextDialect Conversion Charts

Last updated 2 days ago