duocthu/docs/14-data-stores.md

14 — Data stores

Detail on schema and indexing is in 07-indexing-and-storage.md. This page covers operational shape: what is deployed, who touches it, and what is missing.

Deployed stores

Store	Image	Deployed in	Volume	Host port
Qdrant	qdrant/qdrant:latest	docker-compose.prod.yml	qdrant-data	none in prod; 6333/6334 in local dev
PostgreSQL 16	postgres:16-alpine	docker-compose.prod.yml	postgres-data	none in prod; 5432 in local dev
Prometheus TSDB	prom/prometheus:v3.3.0	observability overlay	prometheus-data	127.0.0.1:9090
Tempo	grafana/tempo:2.7.2	observability overlay	tempo-data	none
Grafana	grafana/grafana:11.5.2	observability overlay	grafana-data	127.0.0.1:3002
Caddy	caddy:2-alpine	prod	caddy-data, caddy-config	80, 443

qdrant/qdrant:latest is an unpinned tag — a rebuild can silently move the Qdrant version underneath a loaded collection. Every other image is pinned.

Redis — declared, never used

Redis appears in three places and is used by none of them:

• infra/docker/docker-compose.yml (local dev) starts redis:7-alpine.
• infra/k8s/base/redis/ is an empty directory.
• The pre-existing docs/architecture.md reserves it for session cache, rate-limit counters and a future job queue.

No source file in the repository imports a Redis client, and it is absent from docker-compose.prod.yml and from the Helm chart. middleware.ts names Redis as where its in-memory rate limiter should move when web scales past one replica.

Who touches what

Component	Qdrant	PostgreSQL	Local disk
ai-service startup	read (manifest, collection list)	—	reads ENTITIES_PATH JSON
ai-service query path	read (scroll + query_points)	write trace, read/write conversation turns	—
ai-service /v1/rag/feedback	—	upsert feedback	—
ingestion load	create collection, create indexes, upsert, count	—	reads chunks.jsonl, reads/writes embedding cache
web	—	—	reads the source PDF for /api/pdf

Consistency and idempotency

• Qdrant writes are idempotent. Point ids are uuid5(namespace, chunk_id), so re-loading the same corpus converges.
• Migrations are idempotent. All four are CREATE TABLE IF NOT EXISTS / ADD COLUMN IF NOT EXISTS / CREATE INDEX IF NOT EXISTS. There is no migration-version table and no down-migration; migrate.py simply replays all four every deploy.
• No transactions span stores. A trace row and a Qdrant read are unrelated; a failed trace write leaves the answer already returned.
• No cache layer. The only cache in the system is the offline embedding cache on disk. Query embeddings, retrieval results and generations are not cached — every identical question re-pays for every model call.

Connection handling

adapters/postgres.py opens a new connection per call with connect_timeout=5 and no pool. Both classes document this as a known simplification (F-09: "a real pool, with startup-time lifecycle, is a further improvement not made here"). The timeout is load-bearing: an unreachable but non-refusing host otherwise hangs on the OS TCP timeout, which defeats the caller's fail-open try/except just as completely as no try/except at all.

The Qdrant client is a single long-lived QdrantClient(timeout=30) built in bootstrap.py.

Backup, restore, retention

Concern	State
PostgreSQL backup	Not found — no dump job, no cron, no snapshot automation
Qdrant backup	Not found in code; ingestion/README.md recommends snapshot + restore for moving a corpus, done manually
EBS snapshots	Unverifiable from the repository
rag_conversation_turn retention	None — append-only, grows without bound
rag_retrieval_trace retention	None
Prometheus retention	7d in Helm values; the Compose overlay sets no --storage.tsdb.retention flag, so the Prometheus default applies
Tempo retention	24h in Helm values; Compose uses whatever infra/docker/tempo/tempo.yml specifies

Data classification

rag_retrieval_trace.query_text and rag_conversation_turn.line store the raw user turn. Because the product asks clinicians to supply patient context — age, weight, comorbidities, allergies, current medications, eGFR/CrCl, Child-Pugh, pregnancy status, lab values (rag/clinical.py::PatientContext) — those columns can contain clinical detail about a third party. There is no redaction, no encryption at rest beyond whatever the host volume provides, no access control on the database, and no retention limit. See 16-security.md.