Running the Tests#

The suite runs inside the Dagster image, so the pinned runtime packages are exactly the ones deployed. The code under test is your checkout, mounted read-only. CI and local runs use the same entry point, tests/run.sh.

Quick start#

tests/run.sh                         # whole suite (needs Docker)
tests/run.sh tests/definitions -q    # one directory
tests/run.sh -k customer_summary -x  # any pytest arguments

tests/run.sh uses the newest local opendatastack/dagster image, and the newest opendatastack/spark image for the Spark tests (skipped if there is none). To pick them:

ODS_TEST_IMAGE=opendatastack/dagster:<tag> ODS_SPARK_IMAGE=opendatastack/spark:<tag> tests/run.sh

If no image exists yet, build one first with onprem/core/docker/build.sh. You also need a rebuilt image whenever requirements.txt changes, because the image provides the dependencies.

What the suite covers#

Directory

What it proves

Needs Docker

Typical time

tests/definitions/

Every asset follows the lake convention (D12) and carries the required metadata (D14); table names are unique; Delta-managed inputs are type-annotated (D5 addendum)

No

~2 s

tests/ingest/

The landing sensor’s cursor finds each file exactly once, including same-second and out-of-key-order arrivals

No

<1 s

tests/lake/

The I/O manager’s partition filter reads a whole time window (regression test for the dagster-deltalake first-partition-only bug), with DATE, timestamp and static partitions

No

<1 s

tests/storage/

SeaweedFS honours S3 conditional writes; 16 concurrent delta-rs writers lose nothing (D3)

Yes

~10 s

tests/pipeline/

The demo graph end to end on SeaweedFS (Phase 2): dlt merge into raw/, 7 daily partitions with one Delta version each, an idempotent re-run, Polars staging, the pandas mart, D14 runtime metadata, and the schema-drift check. Also landing/ → raw/: the upstream job drops CSV files, the sensor requests one run per batch of new files, nothing is requested twice

Yes

~40 s

tests/transform/

dbt on SeaweedFS (Phase 3): models land as Delta in their layers; the mart matches the source data; dbt tests and the schema check run as asset checks; a later run of one model reads its upstream model from the lake

Yes

~90 s

tests/spark/

Spark through Pipes on SeaweedFS (Phase 4): the job reads Delta tables from Python and dbt, writes a mart that matches the source data, and reports its metadata; the schema check passes. Skipped when there is no Spark image

Yes, plus the Spark image

~75 s

tests/parity/

The same graph, unchanged (dlt, Python and dbt assets), gives identical tables (and Delta versions) on SeaweedFS and on S3, with moto standing in for AWS until Phase 1. The off-site replica copies the lake to S3 as readable Delta tables, incrementally, logs after data

Yes

~60 s

Tests that need storage start a throwaway SeaweedFS container of the exact image pinned in onprem/core/docker/docker-compose.yml (via testcontainers and the Docker socket). They never touch the live stack. The AWS side is an in-process moto S3 server (moto_s3 fixture); real AWS parity follows Phase 1.

How CI runs it#

.github/workflows/onprem-core.yml builds opendatastack/dagster:<sha12>, then runs:

ODS_TEST_IMAGE="opendatastack/dagster:$TAG" tests/run.sh tests

A failing test stops the deploy.

Lint#

CI also runs ruff, pinned in requirements-dev.txt:

docker run --rm -v "$PWD:/repo:ro" -w /repo python:3.13.18-slim-trixie \
  sh -c "pip install -q ruff==$(grep -oP '^ruff==\K\S+' requirements-dev.txt) && ruff check --no-cache ."

Without Docker-in-Docker#

To run under a local Python 3.13 instead:

python3.13 -m venv .venv && . .venv/bin/activate
pip install -r onprem/core/docker/requirements.txt -r requirements-dev.txt
pip install --no-deps -e .
pytest tests/definitions tests/lake    # no Docker needed
pytest tests                           # storage and pipeline tests still need a Docker daemon

mmh3 has no wheel for Python 3.13 and needs a C compiler (plan3 finding 9).

Writing tests#

  • Storage: take the session fixture seaweedfs (from tests/conftest.py) and call seaweedfs.new_bucket("<prefix>"), so each module gets its own bucket.

  • Pipelines: use tests/helpers.py: lake_on(seaweedfs) or lake_on(moto_s3, target="aws") gives a lake in a fresh bucket, materializer(lake) returns (run, defs), and run("raw/demo__orders", partition_key=...) materializes a selection. run_demo_graph(run) builds the whole demo graph. See tests/pipeline/ and tests/parity/.

  • dlt state: point DLT_DATA_DIR at a temporary directory, and build dlt sources fresh for each run (plan3 finding 5).

  • Order: tests inside tests/pipeline/test_demo_pipeline.py run in file order and build on each other (raw → staging → mart). Keep that order when adding to it.

  • New directories need an __init__.py, because tests import shared helpers as tests.conftest.