πData Application Sync
As your organization's shared data model is constantly evolving, Euno helps ensure that these changes are reflected in your BI tool. The data application sync automation is designed to ensure that your BI application reflects the most up-to-date shared data model of your organization.
How the Data Application Sync Works
Euno supports the automated synchronization of data model revisions, facilitating seamless updates from the dbt manifest to LookML files. Hereβs how it works:
New revision: When a change is detected in the dbt manifest, Euno generates LookML files based on the updated manifest/catalog. Euno can either sync all models or be configured for a selective sync, in which models are synced based on a tag. The sync process is triggered by changes in the dbt manifest. If no changes are detected, no sync is triggered.
Branch checkout: Euno checks out the selected LookML repository and branch configured in the data application sync configuration
PR/Commit: Euno generates a PR/commit, depending on the configurations
What Fields are Supported in the Data Application Sync?
Columns
Euno supports syncing all dbt columns in the model to dimensions in LookML. The dimension data type will be dependent on the dialect used in your DW (see Dialect Conversion Charts). For time dimensions, Euno automatically syncs these columns as a dimension_group of type; time, in LookML.
Metrics
Euno supports automatically syncing dbt metrics to Looker, if the metric is built from a single semantic model. Meaning; the measures or metrics used to create the metric is from the same semantic model, and are of type:
Simple
Ratio
Derived
If the metric includes a filter, Euno will sync an additional dimension specifically for the filter that will hold the filter's logic within the 'sql'
parameter of that dimension. The dimension will be of type 'yesno'
, with the parameter 'hidden'
set to 'yes'
, tagged with 'euno-generated-filter-dimension'
, and named '<measure_name>_euno_generated_filter_dim'
, which can be overridden by placing the macro key; under the meta section of the metric. Inside the measure, under the 'filter'
parameter, it will reference this field with the dimension as the key and 'yes'
as the value. Essentially, this means that if the data matches the logic in the filter, the metric will be applied to the data.
Note: These filters will only be generated for simple metrics and will specifically apply to dbt filter templates utilizing entities, dimensions, or time dimensions within the identical semantic model. Metrics filtered by other metrics are not currently supported.
Below is an example of how the Data Application Sync syncs a metric with a filter (simple_metric
) in LookML.
dbt Metric with Filter | dbt Metric Synced to LookML with Filter |
---|---|
Euno also supports syncing filters using our built-in macro 'getdelphi.io/lookml_filters'
. As described below, the above macro key should be placed in the metric's meta, with the value being a list containing the 'filter_field'
as the key and the 'filter_value'
as the value inside the list. If the metric includes both a filter, and the above macro, the macro will override the filter within the metric.
Limitations of the Data Application Sync
Euno does not support syncing the following dbt metrics:
Cross model metric, a metric that is built from metrics across multiple semantic models
Cumulative metrics
Conversion metrics
Metrics which include an offset window
Formatting Fields in the Data Application Sync
Users can utilize Eunoβs built-in macros using field meta to sync specific formatting like labels and value_format to LookML. See Customizing Generated LookML by Specifying Meta Fields.
Configuring the Data Application Sync
To leverage one of Euno's distinctive features, syncing your dbt changes back to your BI application, it is necessary to configure the specific data application for syncing with your dbt project.
Configuring Target Repository
From the project Configuration screen, users can link a specific branch in the project to a particular LookML file by clicking the "Sync to Data Application" button next to the branch. Clicking this button opens a dialogue box prompting you to input the target repository information. First, select the Git hosting option, GitHub or GitLab, and follow the input requirements detailed below.
Setting up this data sync ensures that with each new build from the selected job, Euno will automatically trigger a data application sync.
Configuring GitHub LookML Repository
Repository: The repository where the LookML files are found for the data sync.
Branch: The branch that will be used to push changes.
Directory: The target directory for the LookML files generated by Euno. By default, Euno generates a new directory:
/dbt_sync
.SQL dialect: Users must select the SQL dialect to be written in LookML (e.g., which DW your Looker instance is connected to). By default, Snowflake is the selected dialect. Currently, the additional supported dialects that can be selected are Spark, Databricks, and Presto.
Selective sync: Adding a tag in the selective sync input will instruct the system to only sync models with the specified tag. Multiple tags can be input.
Users need to specify whether they want the sync to be automatically committed to a branch (selected by default) or have Euno create a pull request.
Configuring GitLab LookML Repository
Project ID: The project ID where the LookML files are found for the data sync.
Project token: The project token for the project where the LookML files are found. If a group-level token has been defined in the GitLab integration, this field is optional.
Branch: The branch that will be used to push changes.
Directory: The target directory for the LookML files generated by Euno. By default, Euno generates a new directory:
/dbt_sync
.SQL dialect: Users must select the SQL dialect to be written in LookML (e.g., which DW your Looker instance is connected to). By default, Snowflake is the selected dialect. Currently, the additional supported dialects that can be selected are Spark, Databricks, and Presto.
Selective sync: Adding a tag in the selective sync input will instruct the system to only sync models with the specified tag. Multiple tags can be input.
Users need to specify whether they want the sync to be automatically committed to a branch (selected by default) or have Euno create a pull request.
Configuring LookML Repository Outside Euno
If there is an error pushing to the dbt repository due to permissions, or if you prefer to set up the configuration for the data application sync outside Euno, simply copy the configuration file from the "Preview changes" dialogue box and add a new file called project_configuration.yaml
under a new directory called .delphi
in your git repo, which houses the dbt project you are configuring the sync with.
This file can be easily updated at any time in your preferred code editor. However, it is important to note that the target_repo
and target_branch
need to exist; otherwise, the sync will fail as Euno will not automatically create anything except for the file directory where the files from the sync are added.
Customizing LookML Structure for Auto-code Generation
Euno uses Jinja2 templates to generate the LookML files from a given dbt manifest and catalog.
Out-of-the-box, every model that is selected by the tags (or all models, if no tags are specified), will automatically generate a single view file in the target directory. Users can customize the behavior by either specifying meta fields in the model or the column, or by overriding the default Euno Jinja2 macros.
Customizing Generated LookML by Specifying Meta Fields
Node Meta Fields
Meta key | Description | Default |
---|---|---|
getdelphi.io/lookml_filename | Full path of the resulting view file Note: This will override lookml_directory, lookml_filename_prefix, and lookml_filename_suffix | ./<model name>.view.lkml (The filepath needs to include the suffix for the file format) |
getdelphi.io/lookml_directory or euno.ai/lookml_directory | Adds the file to a sub-directory within the directory configured in the data application sync setup (see configuring target repository) | No sub-diorectory by default |
getdelphi.io/lookml_filename_prefix or euno.ai/lookml_filename_prefix | Adds a prefix to the view's file name (e.g. prefix_model-name.view.lkml) Note: This can be used in tandem with euno.ai/lookml_filename_suffix and euno.ai/lookml_directory, but not euno.ai/lookml_filename, which will override this value. | No filename prefix by default |
getdelphi.io/lookml_filename_suffix or euno.ai/lookml_filename_suffix | Adds a suffix to the view's file name (e.g. model-name_suffix.view.lkml) Note: This can be used in tandem with euno.ai/lookml_filename_prefix and euno.ai/lookml_directory, but not euno.ai/lookml_filename, which will override this value. | No filename suffix by default |
getdelphi.io/lookml_sql_table_schema | Schema to use for the table | Schema specified in the manifest file |
getdelphi.io/lookml_sql_table_name | Name of the table | Table name specified in the manifest file |
euno.ai/lookml_view_name or getdelphi.io/lookml_view_name | Name of the view | Model name |
euno.ai/lookml_view_name_prefix | Adds a prefix to the default view name (e.g. prefix_model-name) Note: This can be used in tandem with euno.ai/lookml_view_name_suffix, but not euno.ai/lookml_view_name, which will override this value. | No view name prefix by default |
euno.ai/lookml_view_name_suffix | Adds a suffix to the default view name (e.g. model-name_suffix) Note: This can be used in tandem with euno.ai/lookml_view_name_prefix, but not euno.ai/lookml_view_name, which will override this value. | No view name suffix by default |
getdelphi.io/lookml_sync_timeframes or euno.ai/lookml_sync_timeframes | Only relevant for time dimensions. Overrides ALL the default timeframes within the view for a timestamp or date column. This is a comma separated list of time grains. Note: If this key is used at both the model and column level, the more specific option, the column, will override. | raw, time, hour_of_day, date, day_of_week, week, month, quarter, year |
getdelphi.io/lookml_sync_convert_tz or euno.ai/lookml_sync_convert_tz | Only relevant for time dimensions. Overrides the default for converting by timezone in ALL the time and date dimensions in the view. Note: If this key is used at both the model and column level, the more specific option, the column, will override. | The default is set to yes. |
Column Meta Fields
By default, Delphi will generate a single dimension clause for each column and a single measure for each dbt metric (MetricFlow).
Please note that only simple metrics and ratios are currently supported for dbt Metrics. In addition, we do not support metrics across multiple dbt models.
Meta field/Config key/Tag | Description | Default |
---|---|---|
getdelphi.io/lookml_label | Overrides lookml label for the field | No label |
getdelphi.io/lookml_view_label | Overrides the lookml view the field is placed | |
getdelphi.io/lookml_group_label | Overrides the field in lookml group_label | No group label |
getdelphi.io/lookml_hidden | Overrides the field in lookml hidden | Not hidden |
getdelphi.io/lookml_value_format | Overrides the field in lookml value_format | No value format |
getdelphi.io/lookml_value_format_name | Overrides the field in lookml value_format_name | No value format name |
getdelphi.io/lookml_extra | For this specific key we expect users to insert a string with a lookml compliment fragment to be placed under the dimension or measure. For example "can_filter: yes" | No extra fields |
getdelphi.io/lookml_tags | Overrides fields tags in the lookml file | Use the tags from the metric or column |
getdelphi.io/lookml_sync_sql | Overrides the sql used for this dimension, or measure | For dimension - ${TABLE}.<column name> For measure:
|
getdelphi.io/lookml_sync_dimension_name | Only relevant for dimensions. Overrides the name of the dimension | Column name used |
getdelphi.io/lookml_sync_measure_name | Only relevant for measures. Overrides the name of the measure | Measure name used |
getdelphi.io/lookml_primary_key | Only relevant for dimension. Overrides the primary key of the dimension | The primary key of the column, if it exists |
getdelphi.io/skip_lookml_sync | Set this tag to skip this column/metric | The column/metric is included in the sync |
getdelphi.io/lookml_sync_timeframes or euno.ai/lookml_sync_timeframes | Only relevant for time dimensions. Overrides the default timeframes for a timestamp or date column. This is a comma separated list of time grains. | raw, time, hour_of_day, date, day_of_week, week, month, quarter, year |
getdelphi.io/lookml_sync_convert_tz or euno.ai/lookml_sync_convert_tz | Only relevant for time dimensions. Overrides the default for converting by timezone. | The default is set to yes. |
getdelphi.io/lookml_filters | Only relevant for measures. Overrides the filter of the measure. | No filter |
To configure any of the above options:
For columns - Insert the meta key & and overriding value underneath the column in the YML file the model is found.
For metrics - Insert the config key and overriding value in metric's YML file underneath the config key.
Customizing Generated LookML by Overriding Built-in Macros[
Euno will process any file in the dbtβs repository, under the directory .delphi/lookml_sync/macros/
as a Jinja2 template, and override any build-in macros found in those files.
Note: The macro overrides are taken from the head of a branch, while the sync is triggered by a dbt build. This means that if a macro is changed or removed, and a build or revision occurs in between these changes, the macro override changes might not take effect until the next build or revision occurs.
Deploying LookML Changes Made in GitHub to Looker
Euno syncs to LookML any changes introduced to dbt, by utilizing the GitHub integration to commit changes directly to the LookML repository in GitHub. However, Looker does not automatically pull these changes from GitHub and deploy them to the production environment.
You have the option to deploy changes to the project's production environment in Looker either manually for every commit or by setting up automatic pushes from the master branch.
Note: This step is crucial to ensure a successful sync with Looker.
Manually
To manually deploy the changes from a branch to the Lookerβs production environment, follow the following steps from within Looker:
Enter βdevelopment modeβ
Go to βDevelopβ, and pick the Looker project
Go to βGit Actionsβ
Click βPull fromβ¦β and pick βPull from remoteβ
Once the operation is done, click βDeploy to Productionβ
Automatically
Users can take advantage of Lookerβs built-in Advanced deploy mode capability so that changes would automatically be deployed using either a webhook, API endpoint or manually using the deployment manager.
For Looker configurations involving GitHub, we can recommend configuring a webhook so that commits into the master branch would be automatically applied in Looker.
For more detailed instructions see Googleβs instructions.
Last updated