⬅️Shift-left Proposals
Last updated
Last updated
If Looker integration is enabled, Euno facilitates the integration of logical elements from Looker into your dbt model. It identifies dimensions, measures, and custom fields in Looker that can be shifted left, and generates the code for inclusion in dbt in the form of a proposal.
If Data Application Sync is enabled, the changes that have been added to dbt will also be synced back to Looker.
Add a column to an existing dbt model - Euno generates proposals to “shift-left” (up stack) a dimension in a LookML View or a custom dimension in a Tile or Look to a column in an existing dbt model. For a dimension to generate a proposal, it must be built from fields within a single view, not over a join (explore) and must include the definition in the "sql" field.
Limitations:
Dimension groups
Dimension of type bin
Dimensions where logic is in a LookML-specific field parameters like "case" (not to be confused with the use of a case statement in the sql field)
No proposals can be generated for dbt models that only utilize Jinja, and do not include from or select statements
Add a measure to a dbt semantic model - Euno generates proposals to “shift-left” (up stack) a measure in a LookML View or a custom measure in a Tile or Look to a metric in the dbt semantic layer. If the underlying semantic model in dbt does not exist, Euno will generate the semantic model in the proposal as well.
Euno supports proposals for the following measure types:
Average
Count
Max
Median
Min
Percentile
Sum
Measures with filters will generate proposals with the equivalent dbt filter.
If a measure includes specific formatting (value_format, value_format_name, group_label, view_label), Euno will include these fields in the fields metadata in dbt to ensure that when synced back to the data application the formatting will be included.
Limitations:
Measures using fields from multiple views, joined over an explore
Measures in Looker using average_distinct, sum_distinct, percentile_distinct, and median_distinct work on denormalized views and calculate distinct values before aggregation. These are not supported by the dbt semantic layer.
Measures that reference measures using distinct aggregations.
Measures utilizing unsupported filters ('date’, 'date_time', 'location', or Individual time and date types)
Add a model to dbt project - Euno generates proposals to “shit-left” (up stack) a LookML derived view to dbt model. In order to qualify, the LookML derived view must be a SQL based derived view, and not a native derived view. In addition, the base fields must be known to Euno. Meaning; the fields must be based on dbt fields known to Euno, and from within the same dbt project.
In all of the above proposal types, if a measure includes specific formatting (value_format, value_format_name, group_label, view_label), Euno will include these fields in the fields metadata in dbt to ensure that when synced back to the data application the formatting will be included.
Note: Elements using liquid parameters, or constants are supported, as long as the parameters are known to Euno. Contact customer support team via email at support@euno.ai.
Once identified by Euno, these change proposals can be found in two locations:
The Data Model Screen
The Proposals Screen under the Model Changes section (In a specific project)
When generating a shift-left proposal, Euno automatically generates a code diff that includes all the logic necessary to shift the items to dbt. The following explains how the shift-left process works for each change type:
Add a column to an existing dbt model: Euno generates a code diff to add the Looker column to the associated dbt model.
Add a measure to a dbt semantic model: Euno generates a code diff to add the measures, entities, and metrics necessary to shift the Looker measure to a dbt metric. In the case of compound measures, Euno generates a diff not only for the underlying measures but also for the measures needed to create the compound metric in dbt. If the underlying semantic model does not exist, Euno will generate it as well and include all the necessary entities. The shift-left logic for measures is the same here as in derived views, except that in this case, Euno collects and shifts left all underlying measures for compound measures.
Add a model to a dbt project: Euno generates a code diff with all the logic within the derived view to create a model in dbt. If there are measures in the associated derived view, Euno generates a diff with both a dbt model and a semantic model, including the necessary measures, entities, and metrics needed to shift the LookML measure to a dbt metric. However, for derived views, compound measures are shifted-left slightly differently than when they are shifted by themselves. Instead of creating a diff for both the underlying measures and the measures needed to create the compound metric in dbt, Euno only generates a single shallow measure with all the logic. If this newly generated model is configured to sync back to Looker via the data model sync, it will be synced as a single shallow measure as it was created.
Note: LookML dimensions and measures that are not supported in the shift-left automation (see "Add a measure to a dbt semantic model" above) will not be included in the shifted derived view. Please be aware that if a dimension or measure is not included in the PR due to these limitations, Euno does not alert the user to this.
A user can then click the node or row in the table to expand the node's sidebar. Once expanded, the icon will be located next to the components associated with the proposals. Users can then directly select a specific component with the proposal, opening the sidebar for that specific component. In the component, users will see a banner titled 'Shift to dbt Review change draft', which, if clicked, will trigger the Review Change Draft dialogue box detailed below.
The proposal tab, found under the Model Changes section of a project is where you can view all the logic identified by the Euno crawlers from your integrated BI applications that are associated with a specific project. It serves as a platform for users to review and evaluate proposed suggestions, determining their viability as potential change proposals. Users can analyze the details of each proposal, considering factors such as relevance, feasibility, and alignment with the organization's goals. This screen acts as a preliminary filter, ensuring that only well-considered and relevant ideas progress to the next stages of the change management process.
The proposals table is divided into six columns, each serving a specific purpose:
Proposal Number: An identifier assigned by Euno for the proposal, determined by the chronological order in which the change was detected.
Type: The nature of the change proposed by Euno; for instance, "Add Column," indicating a suggestion to incorporate a new column into an existing dbt data model.
Title: An automatically generated title by Euno, offering additional context regarding the suggested actions in the proposal.
Source: The origin or location where the logical element was identified.
Owner: The individual from the BI application who crafted the logic based on the suggestion; represented by their username in the BI application.
Date: The timestamp indicating when Euno generated the proposal.
In order to facilitate identifying the specific suggestions, Euno allows searching for proposals based on owner, name of the field, source (i.e. Look’s name) as well as filtering the list by time of suggestion (please note the table is sorted in chronological order from when the logical element was identified by Euno).
Clicking on a specific suggestion will trigger the Review Change Draft dialogue box, detailing all the change proposal details, allowing also editing the proposal.
In the “Review Change Draft” dialogue box, the following information will be displayed:
Proposal Title: By default Euno auto-generates a proposal title, but this can be edited.
Proposal Description: By default Euno auto-generates a proposal description, but this can be edited.
Change Type: “Add Column”/”Add Measure”/"Add Model", can’t be edited.
Description: This is the description of the new column, which will be added to the dbt model. By default Euno auto-generates a description, but this can be edited.
Name: The name of the column being added to the dbt model. By default Euno will use the name of the field pulled from the BI application. This can be edited.
Target Branch: The branch in the repository the change will be pushed to. By default Euno will select the Main branch from the repository the Euno project is configured to. This can be edited.
Tags: By default added by Euno, but can be edited by the user. This can be edited.
Meta: By default Euno will add the key owner, and the name or user_name of the source owner. Additional metadata can be entered by inputting a key in the text box, and the value next to the key.
Tests (Not supported for measures or models) : Currently Euno supports the four generic tests from dbt
Unique: By default not filled out by Euno, but can be edited by the user
Not Null: By default not filled out by Euno, but can be edited by the user
Accepted Values: A list of strings, which by default is empty, but can be edited by the user.
Relationships: By default not filled out by Euno, but can be edited by the user. The user will select from a dropdown list the table, and then from a second dropdown list the column.
Once satisfied with the proposal, clicking the “Create change draft” button will transform the proposal into a change draft.
The Change Draft screen is a fundamental component in the change management process. It provides insight into the change, and its associated properties used to manage the workflow associated with the lifecycle of the change.
A change has five different stages:
Open - This is the first stage of change, after this is chosen to become a change draft. This is the point in the lifecycle where the change can be altered within the Euno web application.
Pending merge - Once a pull request has been created, the change will move to the “Pending merge” stage, in which a member of your organization will need to merge change from outside of Euno.
Merged - Once the pull request has been merged, and the code committed to your dbt repository, the change stage will be automatically moved to the “Merged” stage.
Deployed - Once your integrated orchestration tool has conducted a job that has built the transformation successfully, which includes the newest change on the database, the change stage will be automatically moved into the “Deployed” stage.
Published - Once the change has been synced with the configured data application. If no data application has been configured, the change will skip the deployed state and skip directly to “Published”, once the transformation has been completed.
This screen is split into four sections:
Change Details
Change Overview
Code
The details of the change draft should include the following information (outlined in red in the image below):
Change number - Unique identifier
Change title
Description of the change
Created by - Euno user who created the change draft
Timestamp of change draft creation
Last edited by - Euno user who last edited the change draft
Timestamp of the last edit
GitHub link - Once the pull request has been created
Build link - Once the dbt build has been deployed
In addition to the above, alongside the change details is a progress bar indicating the current stage of the change's lifecycle, specifically highlighting the critical path (Open->Pending merge->Merged->Deployed->Published). If the change is outside the critical path (i.e. “Rejected”), the general progress should not be displayed, but a message about the state.
If the change is in “Open” stage, a button labeled "Create pull request" can be found beneath the lifecycle progress bar, along with an option to reject the change. Clicking the and selecting the "Reject proposal" option will trigger a confirmation dialogue box, allowing you to confirm your decision to reject the change.
Once satisfied with the content of the change, you can click the "Create pull request" button found underneath the lifecycle progress bar. Upon clicking, a popup message will request confirmation to proceed, with options to cancel or continue. If "cancel" is chosen, the dialogue box should close without making any changes to the proposal. If "continue" is selected, the dialogue box should display the progress of creating the pull request, including three steps: Commit branch, Commit change, and Submit pull request. Once finished, the dialogue box should provide a link to the pull request in GitHub. Users can exit the dialogue box at any point by clicking "Done".
Once the change moves from the “Open” to “Pending Merge”, the “Create Pull Request” button will be replaced with a button called “View Pull Request”, which will link to the pull request on Github.
After the change has been “Merged”, and the orchestration tool has triggered the build, and the transformation has been completed successfully in the data warehouse, the “View Pull Request” button will be replaced with a button called “View Build”, which will link to the build section of the project screen. A link for viewing the pull request can be accessed at any point by clicking the Github icon on the change details upper section.
The change draft details holds critical written contextual information regarding the change attributes itself. Information such as the type of change, the metadata associated with the change and tests related to the change would be found in this section.
This section also includes a graphical representation of the change. From this visualization, users can access the side panel for either the source or target resource. Within the side panel, users can isolate a specific resource in the DAG and expand it to display its upstream or downstream connections. Additionally, users can click on options such as "Downstream Dependencies" or "Upstream Dependencies" to navigate directly to a detailed view of the downstream dependencies for the target resource or the upstream dependencies for the source resource. These features provide users with a broader context, helping them understand the impact and placement of the change within the overall workflow.
The code section includes separate blocks of code for each of the different files being altered, both the YAML file with the contextual information, and the SQL file with the transformations. The code block will include a diff between the code prior and after the change, with whatever is being removed highlighted in red, and the additions highlighted in green.
At the moment, this screen is not editable, and once an actual pull request has been created, a link to the GitHub pull request can be found above the code block. Once the pull request has been merged, the user will no longer be able to see the code block through Euno. Instead, the user will see the following message with a link to the pull request:
While in the “Open” stage, and prior to creating a pull request, the above information can be edited by clicking on the “Edit” button found in the top right corner of the screen found above the “Change details” section. If clicked, a dialogue box will be displayed where the user can edit the details of the change, similar to in the proposal screen.
The “In Progress” tab, found under the Model Changes section of a project is designed to provide insight into the current lifecycle stage each change is in. The screen includes changes which are currently “In progress”, meaning in the following stages of the change lifecycle:
Open
Pending merge
Merged
Deployed
This screen includes a table of all the changes in the above stages, and includes the following columns:
Change title
Created by - The Euno user who created the change
Last time the change was edited
Source of change
Timestamp when the change was created
Stage of lifecycle
This table can be searched, and filtered to assist the user in finding the change they were looking for. The filters include:
Change lifecycle stage
Timeframe of last edit
Timeframe of creation
Opened by
The “Archive” screen, found under the Model Changes section of a project is identical to the “In Progress” screen, including functionality with the one exception, only showing changes which are in the two following states:
Published
Draft Rejected
If Euno has identified a proposal, the user will see the Proposals Icon in the Data Model screen placed over the node in the Graph view and in the table view under the column 'Proposals', indicating the existence of a proposal underneath the node.
