AI in the Platform#

Status: placeholder / stub: nothing is decided. This document frames the problem and seeds the options. Decisions will be recorded in plan2 when made. Origin: plan0 §9, open question 14. Starting point: KDnuggets, “5 Docker Containers You Need for a Home AI Lab” (Nahla Davies, 2026-10-29): Ollama, Open WebUI, Qdrant, n8n, Portainer. Related:

  • plan2 D10 (governance), D11 (notebook lake access)

  • docs/notebook_lake_access.md

1. What the commercial platforms sell#

Capability

Databricks

Snowflake

AWS (curated counterpart)

Model serving / LLM API

Mosaic AI Model Serving, Foundation Model APIs

Cortex LLM functions

Bedrock

AI inside SQL

ai_query() and AI functions

COMPLETE, SUMMARIZE, SENTIMENT, EMBED

Bedrock called from Athena/Lambda UDFs, or batch jobs

Vector / hybrid search

Mosaic AI Vector Search

Cortex Search

S3 Vectors, OpenSearch Serverless, Bedrock Knowledge Bases

Natural-language questions over data (text-to-SQL)

Genie (AI/BI)

Cortex Analyst (semantic model)

Amazon Q in QuickSight

Assistant in notebooks and the SQL editor

Databricks Assistant

Copilot

Amazon Q Developer

ML lifecycle

MLflow (managed), feature store

Snowflake ML

SageMaker (including managed MLflow)

Agents / RAG apps

Agent Framework

Cortex Agents

Bedrock Agents, Knowledge Bases

To verify: product names and scope change quickly. Treat this table as orientation, not a spec.

The commercial point is the same as in the warehouse thesis (plan1): these products are a model endpoint, a vector index and some glue, billed per credit. The analog for us is a set of protocols with an on-prem and an AWS implementation behind each one.

2. The KDnuggets home-lab set, mapped onto opendatastack#

Container

Role in the article

Fit here

Proposed treatment

Ollama (ollama/ollama, port 11434)

Local model runtime

Yes, as the on-prem model server, on CPU or GPU (NVIDIA through the Container Toolkit, AMD through the rocm tag)

On-prem model runtime. Which models it can serve depends on the deployment’s hardware tier (§5). It always handles sensitive data that must not leave the premises.

Open WebUI (ghcr.io/open-webui/open-webui, port 8080 in the container)

Browser chat UI

Yes, as the self-service chat front end, comparable to JupyterHub for notebooks

Behind Traefik and the Phase 1 OIDC. It points at the model gateway (§3), not at Ollama directly, so it can reach both local models and Bedrock.

Qdrant (ports 6333/6334)

Vector database

Candidate. It competes with pgvector (we already run Postgres) and with lake-native vectors (embeddings as Delta columns; LanceDB or DuckDB vss).

Open question 14b. The lake-native option fits the “warehouse is a directory” thesis.

n8n (port 5678)

Workflow automation

Overlaps with Dagster. Scheduled ingestion, embedding and summarizing are Dagster assets (dlt, D6) with lineage, retries and backfills.

Not adopted as an orchestrator (raised here rather than swapped in; see the no-silent-substitutions rule). It might be reconsidered later for human or webhook workflows only.

Portainer CE (port 9443)

Docker management UI

Overlaps with Phase 7 observability (Grafana, Loki) and the D9 deploy model. It mounts the Docker socket, which is root-equivalent.

Deferred / optional. If it is ever added, only on localhost or behind Traefik auth.

The article itself recommends pinned versions, but its examples use moving tags (:main). We pin as usual.

3. Proposed swap points (protocols over products)#

Layer

Protocol / interface

On-prem

AWS

Notes

Model access

OpenAI-compatible HTTP API, reached through a model gateway (candidate: LiteLLM proxy)

Ollama

Bedrock (including Claude models)

One Dagster LLMResource with a base_url and a model alias. The gateway routes aliases such as fast-local, embed and smart to a backend, logs usage and enforces budgets.

Embeddings

Same gateway, /embeddings

Ollama embedding model

Bedrock embedding model

Embeddings must be reproducible: the model name and version are recorded as asset metadata.

Vector index

Store embeddings in the lake as Delta (the contract); serve from an index

Qdrant or pgvector or LanceDB/DuckDB vss (14b)

S3 Vectors or OpenSearch Serverless (14b)

The index can be rebuilt from the Delta table.

Batch AI over data (the Cortex COMPLETE / ai_query analog)

Dagster assets calling LLMResource

Ollama via the gateway

Bedrock via the gateway (or Bedrock batch inference)

Partitioned, cached, with lineage and asset checks. Prefer assets over per-query AI functions in SQL: they cost less and can be reproduced.

Chat / RAG UI

OpenAI-compatible API + vector index

Open WebUI

(same UI; Bedrock through the gateway)

Like notebooks (D11 13e), we don’t build an AWS-hosted UI for the MVP.

Text-to-SQL (the Genie / Cortex Analyst analog)

LLM + catalog/semantic layer + query engine under the user’s identity

Candidates to evaluate: Vanna, WrenAI, or a thin custom agent; Trino as the engine

Athena as the engine; Amazon Q as an optional demo

Must go through Trino/Athena as the user (D11). An agent never gets raw lake keys. dbt schema.yml and the catalog (D10) supply the schema context.

Notebook assistant

OpenAI-compatible API

Jupyter AI extension → gateway

(same)

To verify: the extension’s compatibility with JupyterLab 4.6.

ML lifecycle

MLflow tracking API

MLflow server (Postgres + lake artifacts)

SageMaker managed MLflow

Only if we train or track models. Not needed for LLM-only use.

Agent tool access

MCP

MCP servers for Dagster, Trino/DuckDB and the catalog

(same)

Exploratory. Every tool call must respect D10/D11 identities.

4. Governance applied to AI (D10 / D11)#

  • Data residency by sensitivity: columns tagged sensitive (open question 11) may go only to on-prem models. The gateway or LLMResource refuses to send them to Bedrock. The policy file (open question 12) can carry this as a rule.

  • PII: the Presidio checks (D10) run before any external call, and on the output of AI-derived assets.

  • Audit: gateway request logs (who, which model, how many tokens; prompts optional) go to Loki on-prem and CloudWatch on AWS.

  • Cost: Bedrock spend is included in AWS Budgets ($25/$50 alerts, plan0 Phase 1). The gateway enforces a per-key budget.

  • Lineage: AI-derived assets (summaries, classifications, embeddings) are ordinary Dagster assets that emit OpenLineage, with the model and prompt version recorded as metadata.

  • Evaluation: asset checks on AI outputs (schema, non-empty, sampled quality) are the AI version of data-quality checks.

5. Hardware is a deployment choice, not a design input#

opendatastack is an open platform. The design must not assume any particular host. The model gateway (§3) is what keeps it independent of hardware: each deployment maps model aliases to what it can run.

Tier (placeholder)

Local models

Typical routing

CPU only

Embedding models, small quantized instruct models

Heavy generation → Bedrock; sensitive data → local small model

Single GPU

Mid-size instruct models, faster embeddings

Most work local; Bedrock for the largest models

Multi-GPU / dedicated inference host

Large models

Bedrock optional (burst, comparison)

Minimum and recommended hardware for the whole platform, AI included, is a separate document still to be written (plan0 open question 15). AI services start on demand (a Compose profile), so a deployment that doesn’t use them pays nothing for them.

6. Open questions (14a–14g)#

#

Question

Seed options

14a

Model gateway

LiteLLM proxy / call Ollama and Bedrock directly from LLMResource / none

14b

Vector store

Qdrant / pgvector in the existing Postgres / lake-native (LanceDB, DuckDB vss) on-prem; S3 Vectors / OpenSearch Serverless on AWS

14c

Which local models

A default embedding model plus an instruct model per hardware tier (§5)

14d

Text-to-SQL

Vanna / WrenAI / a custom agent over Trino and Athena; evaluate against the dbt marts

14e

Stack placement

A new onprem/ai/ Compose stack (Ollama, Open WebUI, gateway, vector store) with a Compose profile for on-demand start

14f

Where sensitive data may go

A residency rule in the policy file (open question 12): local only vs Bedrock allowed

14g

Phase

Post-MVP Phase 8, or fold the gateway and batch-AI assets into an earlier phase as a demo

7. A first slice (when we get there)#

  1. onprem/ai/ stack: Ollama + gateway + Open WebUI behind Traefik, on demand.

  2. LLMResource in dagster_platform/resources/, bound to onprem or aws by configuration like every other resource.

  3. One batch asset: embed a mart’s text column into a Delta table, partitioned, with the model recorded as metadata.

  4. Parity: the same asset runs with Ollama and with Bedrock. Compare the shapes and pass the asset checks; outputs won’t be identical, so compare outcomes, not bytes.