📈Data Application Sync

As your organization's shared data model is constantly evolving, Euno helps ensure that these changes are reflected in your BI tool. The data application sync automation is designed to ensure that your BI application reflects the most up-to-date shared data model of your organization.

How the Data Application Sync Works

Euno supports the automated synchronization of data model revisions, facilitating seamless updates from the dbt manifest to LookML files. Here’s how it works:

  1. New revision: When a change is detected in the dbt manifest, Euno generates LookML files based on the updated manifest/catalog. Euno can either sync all models or be configured for a selective sync, in which models are synced based on a tag. The sync process is triggered by changes in the dbt manifest. If no changes are detected, no sync is triggered.

  2. Branch checkout: Euno checks out the selected LookML repository and branch configured in the data application sync configuration

  3. PR/Commit: Euno generates a PR/commit, depending on the configurations

What Fields are Supported in the Data Application Sync?

  • Columns - Euno supports syncing all dbt columns in the model to dimensions in LookML. The dimension data type will be dependent on the dialect used in your DW (see Dialect Conversion Charts). For time dimensions, Euno automatically syncs these columns as a dimension_group of type; time, in LookML.

dimension_group: dimension_group_name {
  type: time
  sql: SQL expression ;;  
  timeframes: [raw, time, hour_of_day, date, day_of_the_week, week,  
              month, quarter, year]
  datatype: # field type conversion - dependent on dialect
  convert_tz: yes
}
  • Metrics - Euno supports automatically syncing dbt metrics to Looker, if the metric is built from a single semantic model. Meaning; the measures or metrics used to create the metric is from the same semantic model, and are of type:

    • Simple

    • Ratio

    • Derived

    The metric will be synced as a single field with all the metric logic under a single measure. Currently, Euno does not support syncing the filters included in the metric. However, if you add the the below metadata, which utilizes Euno's built in macros, the filter will be added to the synced measure.

getdelphi.io/lookml_filters: '[["filter_field", "filter_value"]]'

Limitations of the Data Application Sync

Although Euno supports syncing dbt metrics, certain metric types are not currently supported:

  • Cross model metric, a metric that is built from metrics across multiple semantic models

  • Cumulative metrics

  • Conversion metrics

In addition to the above metrics limitations, Euno does not support syncing other dbt semantic elements, such as:

  • Entities

  • Dimensions

  • Measures

Formatting Fields in the Data Application Sync

Users can utilize Euno’s built-in macros using field meta to sync specific formatting like labels and value_format to LookML. See Customizing Generated LookML by Specifying Meta Fields.

Configuring the Data Application Sync

To leverage one of Euno's distinctive features, syncing your dbt changes back to your BI application, it is necessary to configure the specific data application for syncing with your dbt project.

Configuring LookML

From the project Configuration screen, users can link a specific branch in the project to a particular LookML file by clicking the "Sync to Data Application" button next to the branch. Clicking this button opens a dialogue box prompting you to input:

  • LookML Repository

  • Target branch

  • Directory

  • Selective sync

Users can also specify whether they want the sync to be automatically committed to a branch (selected by default) or have Euno create a pull request.

Setting up this data sync ensures that with each new build from the selected job, Euno will automatically trigger a data application sync.

Customizing LookML Structure for Auto-code Generation

Euno uses Jinja2 templates to generate the LookML files from a given dbt manifest and catalog.

Out-of-the-box, every model that is selected by the tags (or all models, if no tags are specified), will automatically generate a single view file in the target directory. Users can customize the behavior by either specifying meta fields in the model or the column, or by overriding the default Euno Jinja2 macros.

Customizing Generated LookML by Specifying Meta Fields

Node Meta Fields

Column Meta Fields

By default, Delphi will generate a single dimension clause for each column and a single measure for each dbt metric (MetricFlow).

Please note that only simple metrics and ratios are currently supported for dbt Metrics. In addition, we do not support metrics across multiple dbt models.

To configure any of the above options:

  • For columns - Insert the meta key & and overriding value underneath the column in the YML file the model is found.

  • For metrics - Insert the config key and overriding value in metric's YML file underneath the config key.

Customizing Generated LookML by Overriding Built-in Macros[

Euno will process any file in the dbt’s repository, under the directory .delphi/lookml_sync/macros/ as a Jinja2 template, and override any build-in macros found in those files.

Deploying LookML Changes Made in GitHub to Looker

Euno syncs to LookML any changes introduced to dbt, by utilizing the GitHub integration to commit changes directly to the LookML repository in GitHub. However, Looker does not automatically pull these changes from GitHub and deploy them to the production environment.

You have the option to deploy changes to the project's production environment in Looker either manually for every commit or by setting up automatic pushes from the master branch.

Note: This step is crucial to ensure a successful sync with Looker.

Manually

To manually deploy the changes from a branch to the Looker’s production environment, follow the following steps from within Looker:

  • Enter “development mode”

  • Go to “Develop”, and pick the Looker project

  • Go to “Git Actions”

  • Click “Pull from…” and pick “Pull from remote”

  • Once the operation is done, click “Deploy to Production”

Automatically

Users can take advantage of Looker’s built-in Advanced deploy mode capability so that changes would automatically be deployed using either a webhook, API endpoint or manually using the deployment manager.

For Looker configurations involving GitHub, we can recommend configuring a webhook so that commits into the master branch would be automatically applied in Looker.

For more detailed instructions see Google’s instructions.

Last updated