opendatastack#

A data stack you own. Stack it yourself. Swap any layer.

An open data platform that runs on your own hardware by default, and uses a curated set of AWS services where they are clearly cheaper, better or less work. Not on-prem or cloud: both, with every swappable layer proven to run either way.

Open by Default#

Managed warehouses and lakehouses sell convenience, and charge for it per credit, per DBU or per seat, with a proprietary layer that makes leaving expensive. Running everything yourself in the cloud swaps license fees for always-on infrastructure. opendatastack takes a third path:

Open source, open formats

Open source components and open formats (Delta tables on S3-compatible storage), run on hardware you own. No platform subscription.

AWS by delegation

AWS where it earns its place: off-site backup and archive, identity, burst compute, notifications. Bulk data flows in; only small results flow back.

Proven swappable

Every layer is a configuration choice, not a rewrite. The same asset code runs against either side, and parity tests check that the results match.

The mechanism is simple. In Dagster, every external system is a resource. Asset code calls the resource, and configuration decides whether it is on-prem or AWS.

ELT, in layers. Data lands in the lake unchanged, then every transformation runs inside it as versioned, tested code: raw → staging → intermediate → marts. Roughly bronze, silver and gold, with one caveat. Remember, one team’s gold is another team’s bronze: the medallion model is not a universal absolute, it’s a relative measure in your platform.

The lake is the warehouse. Snowflake and Databricks both sell you the illusion that a warehouse is a thing: a product, a service, a SKU. Strip it away and a warehouse is:

  • files, laid out by a convention: here, Delta tables (Parquet plus a transaction log, which is what makes files into tables: atomic commits, schema enforcement, time travel);

  • a query engine that can read those files: DuckDB, Trino, Spark, Athena, and no single engine owns the data;

  • a schema registry;

  • access control;

  • and the rest the product bundles: orchestration, lineage, quality checks, table maintenance and compute scheduling.

Here each of those is its own layer, so each one can be swapped. That makes the lake schema a first-class artifact: written down in the repository, reviewed like code, checked on every run, and published to whichever catalogs and engines you use. Engines and catalogs are built from the schema or checked against it, never the other way round, so swapping one doesn’t change what your data means.

Components#

Each layer of the stack is an open-source component you run yourself, with a curated AWS counterpart it can swap to. The on-prem column is the default; the AWS column exists to prove the layer swaps, or to take on what AWS does better (off-site backup, identity, burst compute).

✅ built · 🟡 planned · ⚪ placeholder. Last updated 2026-11-03.

Layer

On-prem (open source)

AWS (curated)

Storage

Object storage

✅ SeaweedFS (S3 API)

🟡 S3

Table format

✅ Delta Lake

🟡 Delta Lake on S3

Backup and archive

✅ incremental replica to any S3 bucket

🟡 S3 → Glacier Deep Archive

Pipelines

Orchestration

✅ Dagster + Postgres

(the same Dagster drives both sides)

Ingestion

✅ dlt, plus a sensor on landed files

🟡 the same dlt code, writing to S3

Transformation

✅ pandas, Polars, PyArrow, DuckDB

🟡 the same code on AWS compute

SQL modeling

✅ dbt (DuckDB), writing Delta

🟡 the same dbt project on AWS compute

Large-scale compute

✅ Spark, via Dagster Pipes

🟡 Glue, running the same job scripts

Streaming and CDC

⚪ Kafka-compatible broker (e.g. Apache Kafka), Debezium

⚪ Amazon MSK

Data quality

✅ asset checks, dbt tests, required metadata

(the same checks)

Analysis

Interactive SQL

🟡 DuckDB, Trino

🟡 Athena

Notebooks

🟡 JupyterHub, with a SQL toolset: JupySQL, duckdb-kernel, jupyterlab-sql-editor, sqlfluff, sqlvis

🟡 JupyterHub on AWS, the same toolset

Database explorer

⚪ CloudBeaver, through Trino

⚪ CloudBeaver, through Athena

BI and dashboards

🟡 Superset

🟡 Superset on Athena

Catalog and data apps

✅ portal (tool tiles, pipeline status) · 🟡 data catalog, example data app

(the same apps)

Governance

Catalog and lineage

🟡 Unity Catalog OSS or Hive Metastore; OpenLineage

🟡 Glue Data Catalog, DataZone

Access control

🟡 Trino with Ranger or OPA

🟡 Lake Formation

PII detection

🟡 Presidio

🟡 Macie

Audit

🟡 storage audit log → Loki

🟡 CloudTrail

Operations

Edge and TLS

✅ Traefik, self-signed TLS · 🟡 Let’s Encrypt or your own certificates

🟡 ACM

Identity and SSO

🟡 Keycloak

🟡 Cognito

Secrets

🟡 OpenBao

🟡 Secrets Manager

Monitoring and alerts

🟡 Prometheus, Grafana, Loki; email

🟡 CloudWatch, SNS

Deployment

✅ Docker Compose, CI-built releases

🟡 OpenTofu

AI

Models, vector search, chat

⚪ Ollama, Open WebUI

⚪ Bedrock

Status#

The project is built in phases. This section separates what runs today from what is designed but not yet built. Last updated 2026-11-03.

Built and running

  • Core platform: Dagster on Docker Compose, a SeaweedFS S3-compatible lake, Postgres, TLS and authentication, and CI that tests every change and deploys it as a release.

  • Portal: the install’s front door, with a tile per tool (built and planned kept apart) and live pipeline status from Dagster. It grows into the data catalog.

  • Lake tables: Delta tables in a layered lake (raw → staging → intermediate → marts), written and read from pandas, Polars or PyArrow through one I/O manager. Naming, ownership and sensitivity metadata are required and enforced by tests.

  • Ingestion: dlt loads into raw/ with merges and daily partitions; a sensor turns files arriving in landing/ into loads.

  • SQL models: dbt (DuckDB) models written back to the lake as Delta tables; dbt tests run as Dagster asset checks.

  • Spark: PySpark jobs written once with Dagster Pipes, run on-prem in their own container, reading and writing Delta.

  • Off-site copy: an incremental replica of the lake to any S3 bucket.

  • Parity: the whole demo graph (Python, dbt and Spark) runs unchanged against SeaweedFS and against an S3 API, with identical tables on both. The S3 side is an emulator today; the same suite against real AWS comes with the AWS foundation phase.

Designed, not yet built

  • AWS foundation: identity, budgets, the backup bucket and its archive tier.

  • AWS Glue for burst Spark, running the same job scripts.

  • Self-service: notebooks over the lake, interactive SQL, BI.

  • Governance on both sides: catalog, lineage, fine-grained access, PII classification, audit.

  • The data catalog (in the portal app) and an example data app.

  • Hardening; AI tooling as a later phase.

Explore#

Feature comparison

Coverage against Databricks, Snowflake and dbt Labs: built, planned and missing.

Feature Comparison: Databricks, Snowflake, dbt Labs
Cost comparison

Where the money goes in each approach, on a relative scale.

Cost Comparison
Tutorial

From an empty lake to a dashboard-ready mart. Coming soon.

Tutorial
Documentation

Conventions, decisions and how the pieces work.

Documentation