BigQuery Integration Discovered Resources
The Euno-BigQuery integration discovers the following resources:
BigQuery Project
name
The GCP project's display name
native_id
The GCP project's id
type
always gcp_project
Database (BigQuery Project)
name
The GCP project's display name
description
The GCP project's id
type
always database
Only GCP projects with views or tables are discovered
Schema (BigQuery Dataset)
name
The GCP dataset's name
description
The dataset's description, if available
database_technology
always bigquery
created_at
date resource was created
updated_at
date resource was last updated
parent container
The GCP project containing the dataset
type
always schema
meta
Any dataset label will be observed as a meta entry
Table
name
The GCP view's name
subtype
base table, view, or materialized view
description
The table description, if available
created_at
updated_at
parent container
The GCP dataset containing the view
type
always table
materialized
for tables and materialized views: true
for views: false
meta
Any view label will be observed as a meta entry
table properties
The SQL definition of the view
table dependencies
The list of views or tables this view depends on based on its SQL definition
database technology
always bigquery
schema
The name of the BigQuery schema containing the table
database
The name of the BigQuery dataset containing the table
BigQuery Usage & Optimization Properties
Usage and cost optimization properties are derived from BigQuery’s query logs, specifically from the INFORMATION_SCHEMA.JOBS and INFORMATION_SCHEMA.JOBS_BY_PROJECT views.
Read Properties
Read properties capture activity from SQL statements such as SELECT. These properties apply to all BigQuery table subtypes, and some also apply at the column level.
total_read_queries_14d total_read_queries_30d total_read_queries_60d
Total number of queries referencing (SELECT queries) the resource over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views, columns
total_read_slots_14d total_read_slots_30d total_read_slots_60d
Total compute credits consumed by read operations (SELECT queries) referencing the resource over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views
average_read_slots_14d average_read_slots_30d average_read_slots_60d
Average compute credits per read operation (SELECT query) referencing the resource over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views
total_read_bytes_processed_14d total_read_bytes_processed_30d total_read_bytes_processed_60d
Total number of bytes processed by read queries (SELECT queries) referencing the resource over the last 14/30/60 days,.
tables, base_tables, temporary tables, materialized_views
average_read_bytes_processed_14d average_read_bytes_processed_30d average_read_bytes_processed_60d
Average number of bytes processed by read queries (SELECT queries) referencing the resource over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views
total_read_runtime_14d total_read_runtime_30d total_read_runtime_60d
Total runtime of read queries (SELECT queries) referencing the resource over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views
distinct_users_14d distinct_users_30d distinct_users_60d
Number of distinct users who queried (SELECT queries) the resource over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views, columns
Write Properties
Write-related properties capture activity from SQL statements such as INSERT, UPDATE, DELETE, and MERGE. These metrics apply to both tables and materialized views.
total_write_queries_14d total_write_queries_30d total_write_queries_60d
Total number of write operations (INSERT, UPDATE, DELETE, MERGE) involving the table in the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_view
total_write_slots_14d total_write_slots_30d total_write_slots_60d
Total slots consumed by write operations (INSERT, UPDATE, DELETE, MERGE) involving the table in the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_view
average_write_slots_14d average_write_slots_30d average_write_slots_60d
Average slots consumed by write operations (INSERT, UPDATE, DELETE, MERGE) involving the table in the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_view
total_write_bytes_processed_14d total_write_bytes_processed_30d total_write_bytes_processed_60d
Total number of bytes processed by write queries (INSERT, UPDATE, DELETE, MERGE) involving the table over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_view
average_write_bytes_processed_14d average_write_bytes_processed_30d average_write_bytes_processed_60d
Average number of bytes processed by write queries (INSERT, UPDATE, DELETE, MERGE) involving the table over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_view
total_write_runtime_14d total_write_runtime_30d total_write_runtime_60d
Total runtime of write-related operations (INSERT, UPDATE, DELETE, MERGE) involving the table over the last 14/30/60 days.
tables, base_tables, temporary tables, materialized_views
Storage Properties
Storage properties reflect the current size of the table as stored in BigQuery. They are used to represent storage utilization and are applicable to all table types.
volume
The number of rows in the table.
tables, base_tables, temporary tables, materialized_views
row_count
Current table volume as stored in the database (in bytes).
tables, base_tables, temporary tables, materialized_views
Relationships
database
child
gcp_project
schema
child
database
table
child
schema
column
child
table
table
upstream
table
For views, this is based on analyzing the SQL defining the view. For tables, the lineage is based on analyzing the queries in the log. Euno considers all write queries in the last 7 days to infer the lineage.
column
upstream_fields
column
For views, this is based on analyzing the SQL defining the view. For tables, the lineage is based on analyzing the queries in the log. Euno considers all write queries in the last 7 days to infer the lineage.
Known Limitations
For columns of type STRUCT, column-level lineage (upstream_fields relationships) is tracked only at the column level, from the source BigQuery column to the target BigQuery column. Lineage is not captured for individual nested keys within the STRUCT.
Hex Projects
Properties
name
The name of the Hex project
type
always hex_project
subtype
always hex_project
native_id
The unique project ID from Hex
Lineage Information
The BigQuery integration tracks which BigQuery tables and columns are accessed by each Hex project through analysis of query history. This enables:
Data Dependencies: Understand how Hex projects depend on specific BigQuery resources.
Access Patterns: Identify which tables and datasets are most frequently accessed by Hex projects.
Impact Analysis: Determine which Hex projects might be affected by changes to BigQuery resources.
Last updated