Bigquery
Euno's Bigquery integration supports auto-discovery of:
Bigquery views
Query statistics for usage information by Tableau and Thoughtspot
Setting up Bigquery integration
Overview
Bigquery is Google Cloud Platform's flagship analytical data platform. In Bigquery, data is stored in tables and views. Each table or view belongs to one Bigquery dataset, and each dataset belong to a Google Cloud Platform (GCP) Project. A GCP Project can have no datasets, one dataset, or more.
To discover Bigquery resources, Euno executes queries on the system-defined, read-only dynamic INFORMATION_SCHEMA. In Bigquery, a query runs under one Project, and can access tables and views of other Projects. In other words, there is no constraint that a query must run in some project P in order to query for tables or views in project P.
Euno utilizes this fact to allow users to configure the project under which to run queries, called execution project, and separately configure which GCP projects Euno should discover, called discovered projects.
To sum up:
execution project - the GCP Project under which Euno's integration will execute queries
discovered projects - the list of GCP Projects for which Euno's integration will discover resources
To set up Euno's Bigquery integration, complete the following stages:
Step 1: Execution Project
Create a GCP Service Account on the Execution Project
On GCP choose the execution project.
Browse to GCP's Service Accounts page.
Click "Create service account"
Enter a service account name (e.g. Euno BigQuery Integration)
Copy service account Unique Id (e.g. [email protected])
Click "Create and continue"
Under "Permissions" choose "BigQuery User" role (the technical identifier of this role is
roles/bigquery.user).Click "continue" and the "done"
Create a Service Account Key
Go to the service account page that you created in stage 1.
On the Keys tab click "Add key" --> "Create new key"
Choose JSON file type and click "Create".
Save the file.
Step 2: Discovered Projects
For each discovered project, complete these steps
Choose the discovered project.
Browse to the project's IAM configuration. page Click "GRANT ACCESS"
Under "New principals", enter the email address of the service account from stage 1
Under "Assign Roles", click the "select a role" and pick the role "BigQuery Metadata Viewer" (the technical identifier of this role is
roles/bigquery.metadataViewer)Click "Add another role" and pick "BigQuery Resource Viewer" (the technical identifier of this role is
roles/bigquery.resourceViewer)Click "Save"
Step 3: Configure Euno's BigQuery integration
Asterik (*) means a mandatory field.
Service Account Key*
Copy the entire content of the service account key file you downloaded in step 1 and paste here.
Query History Regions*
A comma separated list of BQ regions to scan for query history (e.g. us-central1)
Execution Project ID
The id of the GCP project used as execution project. If not specified, Euno will use the project under which the Service Account is defined.
Query Location
The BigQuery Location to run the queries under.
Skip SSL Verification
Enable this option to skip SSL checks. Defaults to "true".
Step 5: Schedule
Enable the Schedule option.
Choose:
Weekly: Set specific days and times.
Hourly: Define the interval in hours (e.g., every 8 hours).
Step 6: Resource Cleanup
Immediate Cleanup: Remove resources not detected in the most recent successful source integration run.
No Cleanup: Keep all resources indefinitely, even if they are no longer detected.
To keep your data relevant and free of outdated resources, Euno provides automatic resource cleanup options. These settings determine when a resource should be removed if it is no longer detected by a source integration. For a detailed explanation on Euno's cleanup strategies, see: Resource Sponsorship in Euno.
Step 7: Advanced Settings
Click on the 'Advanced' section to display these additional configurations.
Discover Views
Check this box to have Euno discover BigQuery views. Defaults to "true".
Discover dataset labels as meta
Check this box to have Euno discover dataset labels as meta for BigQuery datasets. Defaults to "true".
Discover table labels as meta
Check this box to have Euno discover table labels as meta for BigQuery datasets. Defaults to "true".
Pattern
Use a regular expression to allow or exclude specific projects, for example, ".*" will include/exclude all projects.
Step 8: Save Configuration
Click the Test & Save button to complete the setup.
Discovered Resources
The Euno-BigQuery integration discovers various resources including GCP projects, datasets, and views. For detailed information about discovered resources and their properties, see bigquery-integration-discovered-resources.
Last updated