opendatastack#

A data stack you own. Stack it yourself. Swap any layer.

An open data platform that runs on your own hardware by default, and uses a curated set of AWS services where they are clearly cheaper, better or less work. Not on-prem or cloud: both, with every swappable layer proven to run either way.

Open by Default#

Managed warehouses and lakehouses sell convenience, and charge for it per credit, per DBU or per seat, with a proprietary layer that makes leaving expensive. Running everything yourself in the cloud swaps license fees for always-on infrastructure. opendatastack takes a third path:

Open by default

Open source components and open formats (Delta tables on S3-compatible storage), run on hardware you own. No platform subscription.

AWS by delegation

AWS where it earns its place: off-site backup and archive, identity, burst compute, notifications. Bulk data flows in; only small results flow back.

Proven swappable

Every layer is a configuration choice, not a rewrite. The same asset code runs against either side, and parity tests check that the results match.

The mechanism is simple. In Dagster, every external system is a resource. Asset code calls the resource, and configuration decides whether it is on-prem or AWS.

Components#

Each layer of the stack is an open-source component you run yourself, with a curated AWS counterpart it can swap to. The on-prem column is the default; the AWS column exists to prove the layer swaps, or to take on what AWS does better (off-site backup, identity, burst compute).

✅ built · 🟡 planned · ⚪ placeholder. Last updated 2026-11-02.

Layer

On-prem (open source)

AWS (curated)

Storage

Object storage

✅ SeaweedFS (S3 API)

🟡 S3

Table format

✅ Delta Lake

🟡 Delta Lake on S3

Backup and archive

✅ incremental replica to any S3 bucket

🟡 S3 → Glacier Deep Archive

Pipelines

Orchestration

✅ Dagster + Postgres

(the same Dagster drives both sides)

Ingestion

✅ dlt, plus a sensor on landed files

🟡 the same dlt code, writing to S3

Transformation

✅ pandas, Polars, PyArrow, DuckDB

🟡 the same code on AWS compute

SQL modeling

✅ dbt (DuckDB), writing Delta

🟡 the same dbt project on AWS compute

Large-scale compute

✅ Spark, via Dagster Pipes

🟡 Glue, running the same job scripts

Streaming and CDC

⚪ Kafka-compatible broker (e.g. Apache Kafka), Debezium

⚪ Amazon MSK

Data quality

✅ asset checks, dbt tests, required metadata

(the same checks)

Analysis

Interactive SQL

🟡 DuckDB, Trino

🟡 Athena

Notebooks

🟡 JupyterHub

🟡 JupyterHub on AWS

BI and dashboards

🟡 Superset

🟡 Superset on Athena

Catalog and data apps

🟡 portal, data catalog, example data app

(the same apps)

Governance

Catalog and lineage

🟡 Unity Catalog OSS or Hive Metastore; OpenLineage

🟡 Glue Data Catalog, DataZone

Access control

🟡 Trino with Ranger or OPA

🟡 Lake Formation

PII detection

🟡 Presidio

🟡 Macie

Audit

🟡 storage audit log → Loki

🟡 CloudTrail

Operations

Edge and TLS

✅ Traefik, self-signed TLS · 🟡 Let’s Encrypt or your own certificates

🟡 ACM

Identity and SSO

🟡 Keycloak

🟡 Cognito

Secrets

🟡 OpenBao

🟡 Secrets Manager

Monitoring and alerts

🟡 Prometheus, Grafana, Loki; email

🟡 CloudWatch, SNS

Deployment

✅ Docker Compose, CI-built releases

🟡 OpenTofu

AI

Models, vector search, chat

⚪ Ollama, Open WebUI

⚪ Bedrock

Status#

The project is built in phases. This section separates what runs today from what is designed but not yet built. Last updated 2026-10-30.

Built and running

  • Core platform: Dagster on Docker Compose, a SeaweedFS S3-compatible lake, Postgres, TLS and authentication, and CI that tests every change and deploys it as a release.

  • Lake tables: Delta tables in a layered lake (raw → staging → intermediate → marts), written and read from pandas, Polars or PyArrow through one I/O manager. Naming, ownership and sensitivity metadata are required and enforced by tests.

  • Ingestion: dlt loads into raw/ with merges and daily partitions; a sensor turns files arriving in landing/ into loads.

  • SQL models: dbt (DuckDB) models written back to the lake as Delta tables; dbt tests run as Dagster asset checks.

  • Spark: PySpark jobs written once with Dagster Pipes, run on-prem in their own container, reading and writing Delta.

  • Off-site copy: an incremental replica of the lake to any S3 bucket.

  • Parity: the whole demo graph (Python, dbt and Spark) runs unchanged against SeaweedFS and against an S3 API, with identical tables on both. The S3 side is an emulator today; the same suite against real AWS comes with the AWS foundation phase.

Designed, not yet built

  • AWS foundation: identity, budgets, the backup bucket and its archive tier.

  • AWS Glue for burst Spark, running the same job scripts.

  • Self-service: notebooks over the lake, interactive SQL, BI.

  • Governance on both sides: catalog, lineage, fine-grained access, PII classification, audit.

  • Hardening and an app layer; AI tooling as a later phase.

Explore#

Feature comparison

Coverage against Databricks, Snowflake and dbt Labs: built, planned and missing.

Feature Comparison: Databricks, Snowflake, dbt Labs
Cost comparison

Where the money goes in each approach, on a relative scale.

Cost Comparison
Tutorial

From an empty lake to a dashboard-ready mart. Coming soon.

Tutorial
Documentation

Conventions, decisions and how the pieces work.

Documentation