Lake Layout and Naming Convention#

Status: adopted as plan2 D12 (2026-10-30), resolving plan0 open question 8. The changes to D6 and D11 13a (§12) were approved the same day. Related:

  • plan2 D3 (SeaweedFS), D5 (Delta I/O manager), D6 (dlt), D10 (governance), D11 (notebook access)

  • docs/notebook_lake_access.md

1. Principle: the path is the table name#

The source doc’s thesis is “the warehouse is a directory”. This convention makes that literal. The lake has the same three-level namespace as every catalog and engine we plan to use:

Lake

Catalog / engine name

Example

bucket

catalog

lake

layer (first path segment)

schema / database

marts

table (second path segment)

table

customer_revenue

So s3://lake/marts/customer_revenue/ is lake.marts.customer_revenue in Trino and Unity Catalog, marts.customer_revenue in DuckDB views, and <env>_marts.customer_revenue in Glue/Athena (§9). The Dagster asset key is ["marts", "customer_revenue"]. One name, everywhere.

This matches what the I/O manager already does (verified in dagster-deltalake 0.29.27). A table’s URI is <root_uri>/<schema>/<table>. The schema comes from, in order: output metadata schema, then the I/O manager’s schema config, then the second-to-last asset-key component, and finally public. Two-part asset keys therefore map straight onto <layer>/<table>.

2. Buckets#

  • One bucket per environment, named by configuration (LAKE_BUCKET):

    • on-prem: lake

    • AWS: <prefix>-lake-<env> (bucket names are global; plan0 open question 4)

  • Not split by access tier. SeaweedFS 4.48 scopes identities to prefixes correctly (§8 test), and S3 IAM does the same. Access is granted per table, not per bucket. [changes D11 13a], whose leaning was separate buckets until prefix policies were verified.

3. Top-level prefixes#

Prefix

Contents

Format

Written by

Mutability

landing/<source>/<yyyy>/<mm>/<dd>/<file>

Files exactly as received (CSV, JSON, Excel, …), where originals matter

Original

Uploads, external pushes, dlt filesystem staging

Immutable; lifecycle → archive/

raw/<source>__<entity>/

Source data, normalized by dlt (typed, flattened, plus _dlt_load_id/_dlt_id)

Delta

dlt via Dagster (D6)

Append / merge

staging/stg_<source>__<entity>/

Cleaned, renamed, typed; one table per source table

Delta

dbt / Python assets

Rebuilt

intermediate/int_<purpose>/

Optional reusable joins and logic, not meant for analysts

Delta

dbt / Python assets

Rebuilt

marts/<name>/

Business-facing tables: fct_…, dim_…, or plain business names

Delta

dbt / Python assets

Rebuilt / merged

reports/<report>/<yyyy-mm-dd>/<file>

Rendered outputs: HTML, PNG, CSV extracts, static Dash data

Files

Dagster report assets

Immutable per date

sandbox/<user>/…

Personal scratch from notebooks (D11 13c)

Any

Notebooks

User-owned; lifecycle-expired

archive/…

Aged-out landing files and exports; Glacier Deep Archive on AWS (plan0 Phase 1)

Any

Lifecycle rules

Cold

_system/…

Platform internals: smoke tests, health checks, test scratch

Any

Platform

Disposable

  • landing/ replaces raw/incoming/. [changes D6] In D6, original files lived under raw/incoming/. That mixes non-Delta files into the raw schema, so a raw table called incoming would collide with them, and catalog crawlers would trip over them. Keeping one schema to one kind of thing removes both problems. The Phase 2 sensor watches landing/.

  • No _metadata/ prefix. The source doc floated it for schema snapshots and lineage exports. Schema history is already in each table’s _delta_log, lineage goes to OpenLineage backends (D10), and the catalogs are the registry. A second copy would only drift out of date.

  • Prefixes starting with _ are reserved for the platform. The Phase 0 smoke asset moves from raw/smoke/hello_lake.parquet to _system/smoke/.

  • dlt bookkeeping lives in raw/ (found in Phase 2). dlt keeps its own state next to the tables of its dataset: raw/_dlt_loads, raw/_dlt_version, raw/_dlt_pipeline_state and raw/init. They start with _ or aren’t Delta tables, so catalog registration (open question 10) and access grants skip them. Each pipeline’s state is keyed by pipeline name, so sources can share raw/.

  • dlt naming keeps __ (found in Phase 2). dlt’s default snake_case collapses __ in table names (demo__customers → demo_customers), because it reserves __ for nested tables. dagster_platform.ingest.naming normalizes a table name as a path instead, so raw/demo__customers is kept and nested data lands in raw/demo__customers__<child>. Columns are still plain snake_case.

4. Names#

  • Characters: lowercase snake_case, ^[a-z][a-z0-9_]*$, at most 64 characters, so it’s valid unquoted in Trino, Athena/Glue, DuckDB and Unity Catalog.

  • Table names are unique across the whole lake, not just within a layer. dbt requires this anyway, since model names are global. The layer prefix therefore organizes tables; it doesn’t disambiguate them.

  • Source tables: <source>__<entity>, with a double underscore. Two sources can both have a customers table: raw/crm__customers and raw/shop__customers. [changes D6] D6’s translator keys ["raw", <resource>]; it becomes ["raw", "<source>__<resource>"] (dlt table_name hint, or a naming function in the translator).

  • Layer prefixes on names: stg_ in staging, int_ in intermediate (dbt’s standard); fct_/dim_ are optional in marts. These follow dbt practice and keep model names unique.

  • Columns:

    • snake_case;

    • avoid SQL reserved words (date, timestamp, order, user, year, value, …): use order_date, event_ts;

    • suffixes: _id, _date (DATE), _ts (timestamp, UTC; not _at), is_/has_ for booleans;

    • dlt’s _dlt_* columns stay in raw and are dropped from staging onward.

5. Dagster asset keys#

  • Every asset that writes to the lake has a two-part key, [layer, table]. A test over the definitions enforces:

    • exactly two parts;

    • the layer is one of the Delta layers;

    • the name matches the regex and the layer’s prefix rule;

    • names are unique across layers.

  • reports/ and _system/ assets write files through their own resource, not the Delta I/O manager. They still use [layer, name] keys.

  • Group = domain (e.g. finance, customers), not layer. The layer is already in the key, and domain is the organizing axis D10 needs (DataZone domains, OpenMetadata). The domain also goes in asset metadata (open question 11).

  • dlt translator: ["raw", f"{source}__{resource}"]. dbt translator: [<model folder = layer>, <model name>].

6. Partitioning#

  • A Dagster partition is a Delta partition column, one to one. dagster-deltalake always writes partitioned assets with partition_by=<partition_expr> and overwrites just that partition with a predicate (verified in the handler source). There is no “Dagster-partitioned but physically unpartitioned” mode.

  • Hence: partition only tables that are processed incrementally by time or key (backfills, late data). Everything else stays unpartitioned and is rebuilt or merged.

  • Time partitions:

    • Dagster key format YYYY-MM-DD;

    • the partition column is named for its meaning (order_date, event_date), type DATE. To verify in the Phase 2 test: which predicate types the handler emits for DATE vs TIMESTAMP columns.

  • Granularity: coarse enough that a partition holds tens of MB or more. Use monthly for small tables. Small files are the main Delta performance trap; compaction (OPTIMIZE) runs as a scheduled maintenance asset (Phase 7).

  • Categorical partitions (static or dynamic, e.g. region) use a column of the same name. Avoid high-cardinality keys.

  • Hive-style folders (order_date=2026-09-07/) come from Delta itself. Never write or read them by hand; always go through the Delta log.

7. Table properties and retention (defaults, per layer)#

Layer

delta.deletedFileRetentionDuration

delta.logRetentionDuration

Why

raw

30 days

30 days

Room to reprocess and compare loads

staging, intermediate

7 days

30 days

Rebuildable from raw

marts

7 days

30 days

Limits how long time travel exposes dropped columns (D11)

VACUUM and OPTIMIZE run as scheduled Dagster maintenance assets (Phase 7). These values are starting points, set per table with configuration= on first write.

8. Access mapping (D11)#

Grants are per table prefix, default deny, generated from the policy file (plan0 open question 12):

Principal

On-prem (SeaweedFS identity actions)

AWS (IAM)

Pipelines (Dagster runs)

Read, Write, List on <bucket>

s3:*Object, s3:ListBucket on the bucket

data_engineering

Read:<bucket>, List:<bucket>

s3:GetObject on arn:…:<bucket>/*, s3:ListBucket

analyst (direct read)

Read:<bucket>/marts/<table>/* + List:<bucket>/marts/<table>/* for each table that is published (table-level rules only)

The same prefixes as Resource ARNs, plus s3:prefix conditions on ListBucket

Notebook writes

Write:<bucket>/sandbox/<user>/*

The same prefix

Verified on SeaweedFS 4.48 (2026-10-30):

  • An identity with Read:lake/marts/* + List:lake/marts/* can read and list under marts/.

  • It is denied raw/, denied the look-alike martsx/, and denied writes.

  • A bucket-scoped identity is likewise confined to its bucket.

Delta readers need List on the table prefix (for _delta_log/), which the per-table grant includes.

9. Catalog and engine mapping (feeds plan0 open question 10)#

System

Name for marts/customer_revenue

Dagster

["marts", "customer_revenue"]

Trino (Delta connector) / Unity Catalog OSS

lake.marts.customer_revenue

Glue / Athena

database <env>_marts (Glue database names are per account and region, so the environment is part of the name), table customer_revenue

DuckDB / JupyterHub views

CREATE SCHEMA marts; CREATE VIEW marts.customer_revenue AS FROM delta_scan('s3://lake/marts/customer_revenue')

10. Engine findings that affect the layout (for Phase 3 and open question 9)#

  • dbt-duckdb cannot write Delta out of the box. Its delta plugin is read-only (load only, checked in 1.11.1). DuckDB 2.0’s Delta write support is limited: COPY … (FORMAT delta) doesn’t exist, and ATTACH … (TYPE delta) rejected both CTAS and a named CREATE TABLE. Phase 3 needs one of: a small dbt-duckdb plugin whose store() calls write_deltalake, Python models, or DuckDB writing Parquet that a Dagster asset then commits as Delta.

  • Athena does not write Delta. To verify: as far as I know, Athena reads Delta natively but writes only Iceberg or Hive/Parquet. That would make dbt-athena’s outputs on AWS Iceberg or plain Parquet, not Delta, which conflicts with the plan0 matrix row “Table format: Delta on both”. This is open question 9’s core issue: either AWS-side dbt writes Iceberg, or dbt on AWS runs as dbt-duckdb in Fargate/Glue writing Delta, or we accept mixed formats behind the catalog.

11. Changes needed in code (Phase 2)#

  • dagster_platform/assets/hello.py: move the smoke output to _system/smoke/.

  • dlt translator: ["raw", "<source>__<resource>"] and table name hints.

  • Add a definitions test enforcing §4–5.

  • Lake resource: bucket from LAKE_BUCKET, plus layer constants.

  • create_bucket.py: no prefix creation needed (S3 has no directories); leave as is.

12. Changes to earlier decisions (approved 2026-10-30)#

  1. D6:

    • original files move from raw/incoming/ to landing/<source>/<yyyy>/<mm>/<dd>/;

    • raw table names become <source>__<entity>.

  2. D11 13a: one bucket with per-table prefix grants, replacing the separate-bucket leaning. Verified on SeaweedFS 4.48; standard on S3.

  3. plan0 matrix “Table format: Delta on both” is at risk on the AWS dbt path (§10). This is not changed here; it is raised for open question 9.