> For the complete documentation index, see [llms.txt](https://docs.euno.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.euno.ai/sources/transformation-etl/dbt-core/dagster-upload.md).

# Dagster Upload

This guide shows how to run your dbt Core project with [Dagster](https://dagster.io/) (using the `dagster-dbt` integration) and automatically upload the resulting artifacts to Euno after every successful build.

Euno's dbt Core integration is **orchestrator-agnostic**: it only needs your dbt artifacts (`manifest.json`, `catalog.json`, `run_results.json`, and optionally `semantic_manifest.json`) delivered through the two-step `prepare-upload` flow. Dagster is therefore fully compatible — it is responsible only for running dbt and ordering the upload after the build. The same upload logic used in the Python, GitHub Actions, and Prefect guides applies here.

### Prerequisites <a href="#toc_1" id="toc_1"></a>

* Python 3.9+
* A dbt Core project you already run with Dagster (`dagster` + `dagster-dbt`)
* Euno **trigger secret** and **endpoint URL** for your dbt source

```bash
pip install "dagster" "dagster-dbt" "dbt-core" "dbt-<your-adapter>" "requests"
```

Replace `<your-adapter>` with your warehouse adapter, e.g. `dbt-snowflake`, `dbt-bigquery`, `dbt-redshift`, or `dbt-databricks`.

### Configuration <a href="#toc_2" id="toc_2"></a>

Provide these as environment variables (recommended) so secrets stay out of code:

```bash
export EUNO_ENDPOINT_URL="https://api.app.euno.ai/accounts/YOUR_ACCOUNT_ID/integrations/YOUR_INTEGRATION_ID/prepare-upload"
export EUNO_TOKEN="your_trigger_secret_here"   # Bearer token from your dbt source settings
export DBT_PROJECT_DIR="/path/to/your/dbt/project"
export DBT_PROFILES_DIR="/path/to/your/dbt/profiles"   # optional
```

### How it works <a href="#toc_3" id="toc_3"></a>

The example defines three things:

1. **`euno_dbt_assets`** — a `@dbt_assets` definition that runs `dbt build` (producing `manifest.json` and `run_results.json`) and then `dbt docs generate` (producing `catalog.json`, which Euno requires).
2. **`euno_artifacts_upload`** — an asset that `deps` on the dbt assets, so it runs **only after a successful build**. It zips the artifacts from `target/` and uploads them to Euno.
3. **A job + daily schedule** wiring the two together.

The upload is the same two steps used everywhere in Euno's dbt Core integration:

* **Step 1 – prepare-upload:** `POST` the zip filename to your endpoint with the Bearer token and receive a short-lived **signed URL**.
* **Step 2 – upload:** `PUT` the zip to that signed URL.

> The helper handles both `prepare-upload` response shapes Euno may return (`{"body": {"upload": {"url": ...}}}` and `{"upload": {"url": ...}}`).

### Example <a href="#toc_4" id="toc_4"></a>

A complete, ready-to-adapt module is provided here: `dagster_euno_upload.py`.

The key parts:

```python
from dagster import AssetExecutionContext, AssetKey, Definitions, ScheduleDefinition, asset, define_asset_job
from dagster_dbt import DbtCliResource, DbtProject, dbt_assets

dbt_project = DbtProject(project_dir=DBT_PROJECT_DIR)
dbt_project.prepare_if_dev()

@dbt_assets(manifest=dbt_project.manifest_path)
def euno_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()      # manifest.json + run_results.json
    dbt.cli(["docs", "generate"], context=context).wait()        # catalog.json

@asset(deps=[euno_dbt_assets])
def euno_artifacts_upload(context: AssetExecutionContext) -> None:
    target_dir = Path(dbt_project.project_dir) / "target"
    zip_path = _zip_artifacts(target_dir)                        # zips the 3-4 artifact files
    upload_url, _ = _get_signed_url(endpoint_url, token, Path(zip_path).name)  # step 1
    _put_file(zip_path, upload_url)                              # step 2
```

### Integrating with an existing dagster-dbt project <a href="#toc_5" id="toc_5"></a>

If you already have a `@dbt_assets` definition, you don't need to rewrite it. Just:

1. Add a `dbt docs generate` call after your `dbt build`/`dbt run` so `catalog.json` is produced (Euno needs it for schema information).
2. Copy the upload helpers and the `euno_artifacts_upload` asset from the example, pointing `deps` at **your** dbt assets definition.
3. Add `euno_artifacts_upload` to your job/`Definitions` and select it alongside your dbt assets.

### Notes <a href="#toc_6" id="toc_6"></a>

* `manifest.json` and `catalog.json` are required; `run_results.json` and `semantic_manifest.json` are included when present.
* `dbt build` can be replaced with `dbt run` (or `dbt compile`) depending on your pipeline — just make sure `dbt docs generate` still runs so `catalog.json` exists.
* To run on a schedule, keep the `ScheduleDefinition` (defaults to daily at 06:00). To upload only on demand, remove the schedule and trigger the job manually or from a Dagster sensor.
* If you split dbt models across multiple Dagster assets/jobs, run `dbt docs generate` and upload once per full build so Euno receives a complete, consistent set of artifacts.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.euno.ai/sources/transformation-etl/dbt-core/dagster-upload.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.