Getting started#

This page installs the on-prem core of opendatastack on one Linux host: Dagster for orchestration, a SeaweedFS S3-compatible lake, Postgres for Dagster’s metadata, Traefik for HTTPS and login, and the portal, the install’s front page. It’s everything in the Built and running list on the home page. The AWS side and the self-service tools (notebooks, interactive SQL, BI) aren’t built yet, so they aren’t part of the install.

Last updated 2026-11-03.

What you need#

A Linux host (x86-64) with Docker Engine and the Docker Compose v2 plugin. Tested with Docker Engine 29.7 and Compose 5.4.
git, bash, openssl and curl.
Disk: about 7 GB for the container images, plus room for your lake data.
Memory: the core stack uses about 1 GB when idle. Every pipeline run starts its own container, and Spark jobs need more on top of that, so leave headroom for your workloads.
Ports: HTTPS and HTTP ports reachable on your network (set in .env), plus a few ports bound to 127.0.0.1 only, for local troubleshooting. If a port is already taken on your host, change it in .env.

1. Clone the repository#

git clone https://github.com/middleforkgis/opendatastack.git
cd opendatastack

Everything for the on-prem core lives in onprem/core/. Its Docker files are in onprem/core/docker/.

2. Configure#

Create your settings file from the example:

cd onprem/core/docker
cp .env.example .env

.env holds your secrets and ports. It is never committed. Replace the two CHANGE_ME secrets with random values, for example:

sed -i \
  -e "s/^DAGSTER_PG_PASSWORD=CHANGE_ME/DAGSTER_PG_PASSWORD=$(openssl rand -hex 24)/" \
  -e "s/^LAKE_SECRET_KEY=CHANGE_ME/LAKE_SECRET_KEY=$(openssl rand -hex 24)/" \
  .env

Leave ODS_ADMIN_PASSWORD=CHANGE_ME as it is: the next step generates it.

Then review the ports in .env:

Setting	What it is	Reachable from
`ODS_HTTPS_PORT`	The web interface (HTTPS, login required)	Your network
`ODS_HTTP_PORT`	Redirects to HTTPS	Your network
`ODS_DAGSTER_PORT`	Dagster directly, no login	This host only
`ODS_LAKE_API_PORT`, `ODS_LAKE_CONSOLE_PORT`	The lake’s S3 API and file browser	This host only

3. Create the certificate and login#

../scripts/init_traefik.sh

This creates:

a self-signed TLS certificate for this host’s name, its network addresses and localhost;
the login for the web interface: user admin (or ODS_ADMIN_USER), with a generated password written to ODS_ADMIN_PASSWORD in .env. The password is never printed; read it from .env.

Browsers warn about a self-signed certificate. Accept the warning, or trust onprem/core/traefik/certs/ods.crt on the machines you browse from. Certificates from Let’s Encrypt or your own certificate authority are planned.

4. Build the images#

./build.sh
../../apps/docker/build.sh

This builds three images from the pinned requirements: opendatastack/dagster, which runs Dagster and your pipelines, opendatastack/spark, for Spark jobs, and opendatastack/apps, for the portal. The first build takes several minutes; later builds reuse the cache.

5. Start the stack#

docker compose up -d

Compose starts Postgres, the lake, Dagster (webserver, daemon and code server) and Traefik, and creates the lake’s bucket. Check that everything is up:

docker compose ps

All services should be running (Postgres also healthy). The bucket-creation service, lake-init, runs once and exits.

Then start the portal. It’s a second Compose project that shares the core’s .env and network:

(cd ../../apps/docker && docker compose --env-file ../../core/docker/.env up -d --wait)

6. Open the portal and Dagster#

Browse to https://<your-host>:<ODS_HTTPS_PORT>/ and log in as admin with the password from .env. You land on the portal: a tile per tool, built and planned kept apart, and a status strip with the pipeline’s health. Open Dagster (at /dagster/) to see the demo pipeline’s assets in the asset graph.

To check from the host itself without a browser:

curl -s -H 'Content-Type: application/json' \
  -d '{"query":"{ version }"}' \
  "http://127.0.0.1:$(grep ^ODS_DAGSTER_PORT= .env | cut -d= -f2)/dagster/graphql"

It answers with Dagster’s version.

7. Run the demo pipeline#

The install comes with a small demo: a fictional shop’s customers, orders and returns (see Sample data). Running it touches every built layer: dlt ingestion, the landing-file sensor, Python, dbt and Spark, all writing Delta tables to your lake. In the Dagster UI, under Assets:

Customers. Select raw/demo__customers and click Materialize.
Orders, a few days. Orders are split into daily partitions starting 2026-10-01. Select raw/demo__orders and staging/stg_demo__orders, click Materialize, and choose a few days (for example 2026-10-01 to 2026-10-03). Dagster launches one run per day.
Customer summary. Select staging/stg_demo__customers and marts/customer_summary, and materialize them.
Returns, through the landing zone. Returns arrive as CSV files from an “upstream system”. Under Jobs, open demo_upstream_drop_returns, and in the Launchpad set the day:
```
ops:
  drop_returns_file:
    config:
      day: "2026-10-01"
```
Launch it. Within about a minute the landing_demo__returns sensor sees the new file and loads it into raw/demo__returns. You’ll see the run appear under Runs.
dbt models. Select staging/stg_demo__returns, intermediate/int_demo__order_returns and marts/daily_revenue, and materialize them. The dbt tests run as asset checks.
Spark. Materialize marts/customer_ltv. It runs as a PySpark job in its own container.

Each run gets its own container; follow them under Runs. On a small host the whole demo takes a few minutes. The Tutorial will explain each step in depth.

Stop, restart, remove#

Run these from onprem/core/docker:

Task	Command
Stop the stack (data is kept)	`docker compose stop`
Start it again	`docker compose up -d`
Remove the containers (data is kept)	`docker compose down`
Remove everything, including all lake and Dagster data	`docker compose down -v`

The portal is its own Compose project: run the same commands from onprem/apps/docker with --env-file ../../core/docker/.env. Remove it before the core, because it uses the core’s network.

Where to go next#

Features: what’s built and planned, compared with Databricks, Snowflake and dbt Labs.
Docs: the lake’s naming convention, required table metadata, dbt, Spark and testing.