Add Langfuse as a self-hosted eval and trace viewer
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
# Ragas scoring — setup and pins
|
||||
|
||||
`scripts/run_all_evals.py` checks invariants (decision, citations, drug
|
||||
provenance). `scripts/score_evals_ragas.py` is the other half: it scores the
|
||||
*quality* of answers already recorded by that run, so a change to retrieval or
|
||||
prompting cannot quietly degrade answers while every invariant still passes.
|
||||
|
||||
## Why a separate virtualenv
|
||||
|
||||
Ragas pulls the LangChain stack, which conflicts with this service's pinned
|
||||
dependencies. Installing it into the service environment on 2026-08-18 broke
|
||||
`botocore` and `langchain-core` system-wide. Keep it isolated.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
python -m venv /path/to/ragas_env
|
||||
/path/to/ragas_env/bin/python -m pip install --upgrade pip
|
||||
/path/to/ragas_env/bin/python -m pip install ragas langchain-aws boto3
|
||||
```
|
||||
|
||||
That alone does **not** work. `pip` resolves `langchain-openai` /
|
||||
`langchain-community` to versions that need a newer `langchain-core` than
|
||||
ragas accepts, and the import dies in `ragas/embeddings/base.py` on
|
||||
`from langchain_openai.embeddings import OpenAIEmbeddings`. Pin all three:
|
||||
|
||||
```bash
|
||||
/path/to/ragas_env/bin/python -m pip install \
|
||||
"langchain-core<0.4,>=0.3.85" \
|
||||
"langchain-community<0.4,>=0.3" \
|
||||
"langchain-openai<0.4,>=0.3" \
|
||||
"langchain-aws<1.0"
|
||||
```
|
||||
|
||||
Verify before running anything real:
|
||||
|
||||
```bash
|
||||
/path/to/ragas_env/bin/python -c \
|
||||
"import ragas, langchain_aws; from ragas.llms import llm_factory; print(ragas.__version__)"
|
||||
```
|
||||
|
||||
Known-good: ragas **0.4.3**, boto3 1.43.x.
|
||||
|
||||
## Models
|
||||
|
||||
Judged by Bedrock in `us-east-1`, using the same credentials as the rest of
|
||||
this repo (instance role or `AWS_*` env). Pay per call: roughly three LLM
|
||||
calls per case per metric, so a 90-case run is a few hundred calls.
|
||||
|
||||
| role | model | why |
|
||||
|---|---|---|
|
||||
| judge | `qwen.qwen3-next-80b-a3b` | same model production answers with |
|
||||
| embeddings | `cohere.embed-multilingual-v3` | **not** cohere-v4 |
|
||||
|
||||
cohere-v4 is what production embeds the corpus with, but `langchain-aws`
|
||||
cannot parse v4's response envelope — `BedrockEmbeddings.embed_query` raises a
|
||||
bare `KeyError(0)`. It does not matter here: this embedder only measures how
|
||||
close an answer sits to its question and never touches the index, so it has no
|
||||
need to match the retrieval model. Multilingual does matter — the corpus and
|
||||
the questions are Vietnamese.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# 1. record responses (hits the deployment)
|
||||
python3 scripts/run_all_evals.py \
|
||||
--base-url https://realvuxbaro.me --output-dir /tmp/evals
|
||||
|
||||
# 2. score them (offline, re-runnable, no production traffic)
|
||||
/path/to/ragas_env/bin/python scripts/score_evals_ragas.py \
|
||||
--input /tmp/evals/production60.jsonl \
|
||||
--output /tmp/evals/production60.ragas.jsonl
|
||||
```
|
||||
|
||||
## Reading the numbers
|
||||
|
||||
Only `answerable` turns are scored. An abstain or a clarify has no claims to be
|
||||
faithful to, and averaging them in would move the mean for no reason.
|
||||
|
||||
- **faithfulness** — every claim traceable to the cited evidence. This is the
|
||||
hallucination check and the one that must stay at 1.0. Anything below means
|
||||
the answer asserted something its own citations do not support.
|
||||
- **context_precision** — how much of what was retrieved was actually useful.
|
||||
Low values are a *retrieval* signal, not a safety one: the answer can be
|
||||
perfectly faithful while most of the retrieved chunks were noise.
|
||||
- **answer_relevancy** — whether the answer addresses the question. Catches a
|
||||
well-grounded answer to a different question. Expect below 1.0 on clinical
|
||||
answers that legitimately add safety context the question did not ask for.
|
||||
|
||||
## The trap that already caught us once
|
||||
|
||||
Contexts must carry the drug name. A citation's `evidence_text` is raw section
|
||||
prose that often never names its own drug ("Tăng huyết áp (dùng đơn trị
|
||||
liệu...)"). The service knows the drug from a separate field; a judge handed
|
||||
the bare text does not.
|
||||
|
||||
Scored that way on 2026-08-18, a multi-drug answer came out at **faithfulness
|
||||
0.251** — every claim marked unsupported because no context could be
|
||||
attributed to any drug. The identical run scored **1.000** once `[drug_name]`
|
||||
was prefixed. That was a defect in the measurement, not in the service, and it
|
||||
would have been reported as a model regression. `_contexts_for` in the scoring
|
||||
script exists solely to prevent it; do not simplify it away.
|
||||
|
||||
## The judge confuses lookalike drug names — verify before believing a low score
|
||||
|
||||
Run of 2026-08-19, case **G14** ("Viêm phổi mắc phải ở cộng đồng dùng thuốc
|
||||
gì?"), scored **faithfulness 0.43**. It is not a hallucination. The answer
|
||||
reproduces the GEMIFLOXACIN indication line from printed page 720 almost
|
||||
verbatim, pathogen for pathogen.
|
||||
|
||||
Dumping the per-claim verdicts shows the judge contradicting itself:
|
||||
|
||||
> "Mycoplasma pneumoniae chỉ được liệt kê trong chỉ định của GEMIFLOXACIN cho
|
||||
> viêm phổi cộng đồng, nhưng không phải với GEMIFLOXACIN — mà là với
|
||||
> GEMIFLOXACIN trong phần đầu context"
|
||||
|
||||
> "...chỉ được liệt kê trong chỉ định của GEMIFLOXACIN, không phải
|
||||
> GEMIFLOXACIN"
|
||||
|
||||
The same name sits on both sides of the contradiction. The judge is mixing up
|
||||
**GEMIFLOXACIN** and **GATIFLOXACIN**, which differ by two letters, and marks
|
||||
correct claims unsupported on that basis.
|
||||
|
||||
This is a formulary full of near-identical stems — -floxacin, -azolam,
|
||||
-tidine, -pril, -sartan — so expect it wherever one answer cites two drugs
|
||||
from the same class.
|
||||
|
||||
**Treat faithfulness below 1.0 as a question, not a verdict.** Before reporting
|
||||
one as a regression, dump the claim-level verdicts and read them against the
|
||||
printed page:
|
||||
|
||||
```python
|
||||
stmts = (await metric._create_statements(sample.to_dict(), None)).statements
|
||||
verdicts = await metric._create_verdicts(sample.to_dict(), stmts, None)
|
||||
for v in verdicts.statements:
|
||||
print(v.verdict, v.statement, v.reason)
|
||||
```
|
||||
|
||||
Both low scores this project has investigated turned out to be measurement
|
||||
faults, not service faults: the missing drug names on 2026-08-18, and this on
|
||||
2026-08-19. That record is the reason for the rule above.
|
||||
Reference in New Issue
Block a user