RAG RESEARCH · EVALUATION · HUMAN VALIDATION
Validated RAG Eval Harness
Does LLM-as-judge agree with human reviewers on RAG answer correctness and citation support? Compare naive, rubric-based, and stepwise judge prompts against independent human labels.
Measure agreement—not just model scores
- Three-class answer labels: correct, partially correct, or wrong.
- Separate binary judgments for whether cited passages support the answer.
- Nominal Cohen's κ, raw agreement, confusion counts, and per-type disagreements.
- Judge cost from provider token usage and optional human labor cost.
Human labels are blank until reviewers complete them. The included 50-case fixture is illustrative and cannot establish agreement or represent multi-hop, numeric, or negation question performance.
Prepare independent human labels first
cd rag-judge-eval
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
rag-judge-eval prepare
# Complete results/human-review.csv before seeing judge outputs.
export OPENAI_API_KEY="..."
rag-judge-eval run --model gpt-4o-mini