Required Asset Metadata#

Status: adopted as plan2 D14 (2026-10-30), resolving plan0 open question 11. Related:

plan2 D10 (governance), D11 (notebook access), D12 (lake convention), D13 (Delta)
plan0 open questions 10 (catalog registration) and 12 (policy source)

1. Purpose#

Dagster is where every asset is defined, so it is the one place to capture what the catalogs, lineage backends and access policies need. Metadata declared on the asset flows outward:

to OpenLineage events (our own emitter, D10), and from there to Marquez/OpenMetadata on-prem and DataZone on AWS;
to catalog registration (open question 10): Unity Catalog/HMS comments and properties, Glue table parameters;
to access policy (open question 12): sensitivity tags drive column and row rules, including Lake Formation tag-based access control (LF-Tags) and the matching OPA/Ranger rules;
to D11: whether a table can be read directly or only through the engine follows from its column sensitivity.

If metadata isn’t declared on the asset, it doesn’t exist anywhere downstream. So the required set is small, checked in CI, and mostly filled in automatically.

2. What Dagster 1.13 supports (verified 2026-10-30, image `3ef6d4e22d8f`)#

@asset(description=, owners=, group_name=, tags=, kinds=, metadata=, code_version=).
owners must be an email address or team:<name>; a bare name is rejected.
Custom tag keys with a namespace (ods/domain, ods/sensitivity) are accepted.
TableColumn(name, type, description, constraints, tags): column-level tags work.
Standard table metadata (TableMetadataSet): column_schema, column_lineage, row_count, partition_row_count, table_name, storage_kind.

3. The required set (every asset that writes to the lake)#

Field	Where	Rule	Feeds
Key	`key=[layer, table]`	D12 shape and names	Everything
Description	`description=`	Non-empty; the first line is a one-sentence summary	Catalog comments; OpenLineage documentation facet
Owner	`owners=`	At least one `team:<name>`; emails optional	OpenLineage ownership facet; DataZone/OpenMetadata ownership
Domain	`group_name=` and tag `ods/domain`	The same value (snake_case); the group is for Dagster’s UI, the tag travels with lineage	DataZone/OpenMetadata domains
Table sensitivity	tag `ods/sensitivity`	One of `public`, `internal`, `confidential`, `restricted`; no default; must be at least the highest column sensitivity	Policy (open question 12), D11
Kinds	`kinds=`	Storage + compute, e.g. `{"delta", "polars"}`	Dagster UI
Source (raw only)	tag `ods/source`	The `<source>` part of the table name	Lineage, catalog

Columns (`metadata={"dagster/column_schema": TableSchema(...)}`)#

Layer	Requirement
All layers	Every sensitive column is declared at definition time with tag `ods/sensitivity` (`confidential` or `restricted`), plus `ods/pii` (`email`, `ssn`, `phone`, `name`, `address`, `dob`, …) where it applies. PII lands in `raw`, so it must be declared there too.
`marts`	Every column is declared, with type and description
`staging`, `intermediate`	Columns are recommended; sensitive ones are required (as above)
`raw`	Full schemas come from dlt at run time; only sensitive columns are declared

Optional, recommended for `marts`#

A freshness policy (Dagster freshness checks).
code_version= on Python assets, so changes in code mark downstream assets stale.
Tag ods/tier: gold (contracted, monitored) or bronze (best effort).

4. Filled in automatically at materialization#

The lake I/O manager and resources add these, so authors never write them:

Key	Value
`dagster/table_name`	`lake.<layer>.<table>` (the D12 catalog name)
`dagster/storage_kind`	`delta`
`dagster/row_count`, `dagster/partition_row_count`	From the write
`dagster/column_schema` (runtime)	The actual Delta schema after the write
`ods/uri`	`s3://<bucket>/<layer>/<table>`
`ods/delta_version`	The table version this run committed, for reproducibility and time travel

5. Sources of the metadata, by asset type#

Python assets: declared directly on @asset. A small helper (ods_asset(...) or a DomainSpec) fills in the defaults (domain → group + tag, kinds) so authors only supply what is specific to the asset.

dbt models: declared in schema.yml and mapped by a DagsterDbtTranslator:

models:
  - name: customer_revenue
    description: Revenue per customer per month.
    config:
      meta:
        ods: {owner: "team:finance", domain: finance, sensitivity: internal}
    columns:
      - name: email
        description: Customer contact email.
        config:
          meta:
            ods: {sensitivity: confidential, pii: email}

dbt’s own column descriptions satisfy the marts rule, and dbt build tests become asset checks.

dlt sources: the dlt translator sets owner, domain, ods/source and table sensitivity per source. Sensitive columns are declared in a per-source mapping next to the source definition.

6. Enforcement#

Definitions test (CI, tests/definitions/) loads Definitions and fails on:
- missing or invalid description, owner, domain or sensitivity;
- group ≠ ods/domain;
- table sensitivity lower than its highest column sensitivity;
- a marts table with undeclared columns;
- D12 key and naming violations.
- an input of a Delta-managed asset with no type annotation (D5 addendum).
Schema-drift asset check (runtime, marts): the declared columns must match the materialized Delta schema. It fails on a missing or undeclared column, so a column can’t be added without a description and sensitivity.
PII backstop (D10): the Presidio check flags data that looks like PII in a column not tagged as sensitive. That catches declarations that are wrong, not just missing.

7. How it maps outward#

Our field	OpenLineage	DataZone / Glue	OpenMetadata / Unity Catalog	Policy
description	documentation facet	asset/table description	description / comment	n/a
owners	ownership facet	owner	owners	approver for access requests
`ods/domain`	tags / custom facet	domain	domain	rule scope
`ods/sensitivity` (table, column)	tags / schema facet	LF-Tag	classification tag	LF-TBAC / OPA rule input
`ods/pii`	schema facet	Macie comparison	PII tag	masking rules
column schema	schema facet	Glue columns	columns	column grants

Driving access from tags rather than column lists (Lake Formation LF-Tags; OPA rules keyed on tags) means a new confidential column is protected the moment it is declared. That is the main input to open question 12.

8. Changes needed in code (Phase 2)#

Add the ods_asset helper (or a spec builder) in dagster_platform/.
Lake I/O manager: add the runtime metadata in §4. It subclasses the multi-type DeltaLakeIOManager (D5).
dlt and dbt translators: map the metadata above.
tests/definitions/test_conventions.py: the D12 + §6 checks.
The smoke asset gets owner team:platform, domain platform, sensitivity public.