RAG RESEARCH · RETRIEVAL · CONTROLLED ABLATION
Chunking Strategy Ablation
Is fixed-size chunking really the best default? Compare fixed, recursive, sentence-window, and semantic strategies with one corpus, one TF-IDF embedder, 50 hand-written questions, and fixed retrieval depth k=5.
Measure retrieval quality against context cost
- Recall@5 against explicit gold evidence sentences.
- Answer exact match, optional LLM-judge score, and editable human review.
- cl100k context tokens, retrieval and end-to-end latency, and configurable token cost.
- Identical corpus, embedder, question set, answerer, and retrieval depth across chunkers.
The bundled engineering corpus is original illustrative fixture text, not private papers or domain guidance. Local extractive mode makes no provider calls; LLM mode is optional and requires an approved endpoint.
Run the benchmark
cd chunking-ablation
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
chunking-ablation
pytest