Feature Comparison: Databricks, Snowflake, dbt Labs

Version	1.2
Date	2026-10-30
Baseline	Live release 0f8504f plus Phase 4 on-prem; decisions D1–D15 (plan2)

A high-level comparison of opendatastack’s coverage, built and scoped, against the three commercial platforms it is most often measured against. Vendor product names and packaging change often, so read them as orientation, not a spec. Revise this page when a phase lands or a gap is scoped, bump the version, and add a row to the revision history.

Key: ✅ built · 🟢 decided (D#) · 🟡 planned (phase) · ⚪ placeholder (open question) · ❌ not scoped

In short: built is ✅; planned is 🟢 or 🟡; missing is ⚪ or ❌. Rows with more than one mark are partly built.

Comparison

Capability	Databricks	Snowflake	dbt Labs	opendatastack	Status
Storage + ACID table format, time travel	Delta, Unity Catalog managed tables	Native tables, Iceberg tables	(relies on the warehouse)	Delta on SeaweedFS / S3	✅ D3, D13
Orchestration	Lakeflow Jobs	Tasks	dbt Cloud jobs	Dagster asset graph	✅
Ingestion (batch connectors)	Lakeflow Connect	Openflow, Snowpipe	(none)	dlt, plus a sensor on landed files	✅ D6
Streaming / CDC	Structured Streaming, declarative pipelines	Snowpipe Streaming, Streams	(none)	(none yet)	⚪ open question 20
Incremental / declarative transforms	Declarative pipelines	Dynamic Tables	Incremental models	dbt incremental + Dagster partitions and automation	✅ partitions · 🟡 dbt incremental
SQL modeling	SQL, notebooks	SQL, Snowpark	dbt core	dbt-duckdb on both sides, writing Delta	✅ on-prem (D13) · 🟡 AWS, Phase 3
Semantic layer / metrics	Metric views	Semantic views	Semantic Layer (MetricFlow)	(none yet)	⚪ open question 19
Interactive SQL engine	SQL warehouses (Photon)	Virtual warehouses	(none)	DuckDB / Trino / Athena	🟡 Phase 5
Large-scale compute	Spark	Snowpark	(none)	Spark via Pipes, Glue for burst	✅ on-prem Spark · 🟡 Glue (Phase 4, after Phase 1)
Notebooks	Notebooks	Notebooks	(none)	JupyterHub + DuckDB	🟢 D11 · 🟡 Phase 5
BI / dashboards	AI/BI dashboards	Snowsight dashboards	(none)	Superset, Dash	🟡 Phase 5, 7
Hosted apps	Databricks Apps	Streamlit in Snowflake	(none)	Dash + Flask stack	🟢 D7 · 🟡 Phase 7
Data quality	Lakehouse monitoring, expectations	Data metric functions	dbt tests	Asset checks, dbt tests, pandera, schema-drift check	✅ checks, dbt tests, schema drift (D14) · 🟡 pandera, Phase 7
Catalog / discovery	Unity Catalog	Horizon	Catalog (Explorer)	UC OSS / HMS, OpenMetadata ↔ Glue, DataZone	🟢 D10 · 🟡 Phase 6
Lineage	Unity Catalog lineage	Horizon lineage	Explorer lineage	Dagster (built in) + our own OpenLineage emitter	✅ Dagster · 🟡 Phase 6
Fine-grained access (column, row, tag)	Unity Catalog ABAC	Masking and row access policies, tags	(none)	Trino + OPA ↔ Lake Formation, tag-driven	🟢 D10, D11, D14 · 🟡 Phase 6
PII classification	Data classification	Sensitive-data classification	(none)	Presidio ↔ Macie, plus D14 tags	🟡 Phase 6
Audit	System tables	Access history	(none)	SeaweedFS audit log → Loki ↔ CloudTrail	🟡 Phase 6
Identity / SSO	Workspace SSO	SSO	SSO	Cognito; interim basic auth	🟡 Phase 1
CI/CD for data code	Asset Bundles	(none)	dbt Cloud CI	GitHub Actions, release deploys	✅ D9
Dev/prod environments, data cloning, branching	Workspaces, shallow clone	Zero-copy cloning	Environments, defer	Code releases only; no separate data environments yet	⚪ open question 18
Data sharing / marketplace	Delta Sharing, Marketplace	Secure sharing, Marketplace	(none)	(none yet)	⚪ open question 21
Backup / disaster recovery	Cloud storage + replication	Fail-safe, replication	(none)	SeaweedFS → S3 → Glacier, plus a DR drill	✅ replica to S3 · 🟡 Glacier, DR drill (Phase 1, 7)
Observability / cost	System tables, budgets	Resource monitors	(none)	Prometheus/Grafana/Loki, AWS Budgets	🟡 Phase 1, 7
AI (LLMs in SQL, vector search, text-to-SQL, assistants)	Mosaic AI, Genie, Assistant	Cortex, Cortex Analyst	dbt Copilot	Ollama/Bedrock through a gateway, Open WebUI	⚪ open question 14
ML lifecycle / feature store	MLflow, feature store	Snowflake ML	(none)	(none yet)	⚪ open question 22
Cross-project modularity	Multiple workspaces	Multiple accounts	dbt Mesh	(none yet)	⚪ open question 23
Documentation	Catalog docs	(none)	dbt docs	Sphinx (this site)	✅ D15 · ⚪ fold in the plans (open question 16)
OLTP alongside analytics	Lakebase (Postgres)	Hybrid tables, Postgres	(none)	Postgres exists, for Dagster only	❌ likely out of scope
Runs on-prem and on a cloud, swappable	No (managed cloud only)	No	Partly (core is open source)	Yes, proven by parity tests	✅ against an S3 emulator · 🟡 real AWS, Phase 1

Reading the table

• Built: the on-prem lakehouse loop runs end to end: storage and table format, orchestration, batch and file ingestion, SQL modeling, data quality checks, lineage in the asset graph, and an off-site copy. Each piece is also run against an S3 API in the parity tests.

• Scoped, not built: catalog, fine-grained access, PII classification and audit (governance phase); notebooks, interactive SQL and BI (self-service phase); Spark and burst compute.

• Placeholders (plan0 open questions 18–23, added in 1.0) (plan0 open questions 18–23), roughly in order of importance:

  • dev/prod data environments

  • semantic layer

  • streaming / CDC

  • data sharing

  • ML lifecycle / feature store

  • cross-project modularity

• Where opendatastack is structurally different:

  • no per-credit billing;

  • every layer can run on-prem or on AWS, with tests proving it;

  • Dagster’s asset graph and lineage are arguably stronger than the vendors’ built-in schedulers.

• The cost: we operate it ourselves. There’s no serverless autoscaling, and capacity depends on the deployment’s hardware (plan0 open question 15).

Revision history

Version	Date	Change
1.0	2026-10-30	First version. Gaps 1–6 recorded as plan0 open questions 18–23.
1.1	2026-10-30	Statuses updated for Phases 2–3 (on-prem): dlt and landing ingestion, dbt writing Delta, checks, replica, parity against an S3 emulator. Published on opendatastack.io (D15).
1.2	2026-10-30	Large-scale compute: Spark via Pipes built on-prem.