Wire the guarded conversational RAG answer layer end-to-end

coordination/CLAUDE_TASK.md

@@ -0,0 +1,160 @@
+# Task for Claude: AWS Bedrock embedding setup
+
+## Objective
+
+Prepare and verify the smallest safe AWS Bedrock integration needed to benchmark
+embedding models. Do not modify parsing, segmentation, table, or formula code.
+
+## Verified current state
+
+- Repository: `D:\VSF-DUOCTHU`
+- AWS CLI is installed and resolves credentials for IAM user `ai-lab-user`.
+- Configured region: `us-east-1`.
+- `aws sts get-caller-identity` succeeded on 2026-08-03.
+- `aws bedrock list-foundation-models --region us-east-1` failed on 2026-08-03
+  with `AccessDeniedException` for `bedrock:ListFoundationModels`.
+- No Bedrock embedding invocation has succeeded yet.
+
+## Models to benchmark
+
+1. `cohere.embed-v4:0`, 1024-dimensional float embeddings.
+2. `amazon.titan-embed-text-v2:0`, 1024 dimensions with normalization enabled.
+3. `BAAI/bge-m3` local as the zero-API-cost control.
+
+For Cohere, corpus records must use `input_type=search_document`; queries must
+use `input_type=search_query`. Never mix vectors from different models in one
+Qdrant collection.
+
+## Requested work
+
+1. Diagnose the current IAM restriction without exposing credentials.
+2. Provide or add a least-privilege IAM policy for listing and invoking only the
+   two embedding models above. Cohere may additionally need AWS Marketplace
+   subscription permissions for first use.
+3. Add provider adapters behind an interface under the existing embedding
+   boundary; do not couple retrieval/domain code directly to Boto3.
+4. Add a no-cost smoke test with mocked Bedrock responses.
+5. Only after permissions work, make one minimal live call per cloud model and
+   report request shape, vector dimension, latency, and actual error/success.
+6. Do not run full-corpus embedding yet. Leave that for the shared benchmark:
+   10 hard cases, then 100, then full corpus only after acceptance gates pass.
+
+## Required handoff
+
+Update this file with:
+
+- files changed;
+- exact commands and scope run;
+- observed results;
+- remaining permissions or account actions required;
+- anything not tested.
+
+## Handoff — Claude, 2026-08-03
+
+**Status: items 1-4 done. Item 5 (live calls) blocked on an IAM change that has
+not been applied. No AWS spend has occurred.**
+
+### Files changed
+
+Added:
+
+- `ingestion/ingestion/embed/ports.py` — `EmbeddingProvider` ABC,
+  `EmbeddingVector`, `EmbeddingBatch`, `text_digest`
+- `ingestion/ingestion/embed/bedrock_runtime.py` — `BedrockInvoker` protocol +
+  `Boto3BedrockInvoker`; the only module that imports boto3, lazily
+- `ingestion/ingestion/embed/bedrock_titan.py` — `amazon.titan-embed-text-v2:0`
+- `ingestion/ingestion/embed/bedrock_cohere.py` — `cohere.embed-v4:0`
+- `ingestion/ingestion/embed/local_bge_m3.py` — `BAAI/bge-m3` local control
+- `ingestion/ingestion/embed/registry.py` — name → provider
+- `ingestion/ingestion/embed/probe.py` — one live call, one short string
+- `ingestion/tests/test_embed_providers.py` — 22 tests, all mocked
+- `infra/aws/iam/bedrock-embedding-invoke.json`
+- `infra/aws/iam/bedrock-model-access-bootstrap.json`
+- `infra/aws/iam/README.md`
+
+Modified:
+
+- `ingestion/ingestion/embed/__init__.py` — was empty, now the package's
+  public surface
+- `ingestion/pyproject.toml` — added optional extras `bedrock` (boto3) and
+  `local-embed` (sentence-transformers)
+
+**No parser, segmentation, table, formula, chunking or `cli.py` file was
+touched.** `cli.py` carries a pre-existing lint finding from the other
+worktree owner (`F401 evaluate_clinical imported but unused`) which was left
+alone deliberately.
+
+### Commands run and their observed results
+
+Diagnosis (all read-only, all free):
+
+| Command | Result |
+|---|---|
+| `aws sts get-caller-identity` | `arn:aws:iam::<account>:user/ai-lab-user` |
+| `aws iam list-attached-user-policies --user-name ai-lab-user` | `[]` |
+| `aws iam list-user-policies --user-name ai-lab-user` | `[]` |
+| `aws iam list-groups-for-user --user-name ai-lab-user` | `AI-Lab-Group` |
+| `aws iam list-attached-group-policies --group-name AI-Lab-Group` | `AmazonEC2FullAccess`, `IAMFullAccess`, `ElasticLoadBalancingFullAccess`, `AmazonVPCFullAccess` |
+| `aws iam list-group-policies --group-name AI-Lab-Group` | `[]` |
+| `aws bedrock list-foundation-models --region us-east-1` | `AccessDeniedException` — `bedrock:ListFoundationModels` |
+| `aws bedrock-runtime invoke-model --model-id amazon.titan-embed-text-v2:0 …` | `AccessDeniedException` — `bedrock:InvokeModel` |
+
+**Diagnosis:** `ai-lab-user` has no inline and no attached user policy. Its one
+group grants EC2, IAM, ELB and VPC full access and nothing else. There is no
+`bedrock:*` permission anywhere on this identity — the denial is a plain
+absence of grant, not an explicit `Deny` and not a model-access problem. No
+credential value was read or printed at any point.
+
+Tests and lint:
+
+| Command | Scope | Result |
+|---|---|---|
+| `python -m pytest tests/test_embed_providers.py -q` | the new suite only | **22 passed** |
+| `python -m pytest -q` | whole `ingestion/` suite | **203 passed** (181 before this task, +22) |
+| `python -m ruff check --select F,E9,B,ARG .` | whole `ingestion/` tree | 1 error, and it is the pre-existing `cli.py` one above; **0 in any file added here** |
+| `python -m ingestion.embed.probe --help` | CLI wiring | parses, lists all three providers |
+
+Request/response shapes were taken from the AWS Bedrock user guide pages
+"Amazon Titan Embeddings G1 - Text" (V2 tabs) and "Cohere Embed v4", both read
+2026-08-03 — not from memory.
+
+### Remaining permissions / account actions required
+
+1. Create and attach `infra/aws/iam/bedrock-embedding-invoke.json` to
+   `AI-Lab-Group` (or directly to `ai-lab-user`). Commands are in
+   `infra/aws/iam/README.md`. `ai-lab-user` holds `IAMFullAccess`, so it can
+   do this itself — **not done here because it changes permissions on a shared
+   company account.**
+2. Enable model access for both models in the Bedrock console (or via the
+   bootstrap policy). `cohere.embed-v4:0` is third-party and may additionally
+   need an AWS Marketplace subscription on first use.
+3. Then run, one call each:
+   `python -m ingestion.embed.probe --provider titan-v2`
+   `python -m ingestion.embed.probe --provider cohere-v4`
+
+### Not tested / not measured / uncertain
+
+- **No live Bedrock call has ever succeeded.** Every request-body claim in
+  `bedrock_titan.py` and `bedrock_cohere.py` is documentation-derived and
+  unproven against the service. The probe is what settles it.
+- Whether the drafted IAM policies are *sufficient* is unproven in both
+  directions — nothing was attached, so nothing was retried.
+- Whether an SCP or a permissions boundary would still block Bedrock after
+  attachment cannot be determined from inside this identity.
+- `bge-m3` has **never been run** on this machine; no weights were downloaded.
+  Its 1024 dimensions and its no-instruction-prefix property come from the
+  published model card. The dimension is asserted at runtime, so a wrong
+  assumption fails on the first call rather than silently.
+- Cohere's float vectors are recorded as `normalized=None` because AWS's
+  documentation does not state it. The probe prints a *measured* L2 norm,
+  which is how that gets settled.
+- No embedding cost has been incurred. Nothing has been written to Qdrant.
+  No corpus run was started.
+
+## Message the user can send Claude
+
+> Read `D:\VSF-DUOCTHU\CLAUDE.md` and everything in
+> `D:\VSF-DUOCTHU\coordination`. Claim the Claude task in
+> `coordination\README.md`, then perform the AWS Bedrock embedding setup exactly
+> within that scope. Do not touch parser/chunking files and do not expose AWS
+> credentials. Record all results back into the coordination folder.