OPEN-SOURCE TOOLKIT · PYTHON · DOCKER

BenchmarkKit

Make LLM serving and RAG trade-offs measurable. BenchmarkKit runs repeatable workloads against OpenAI-compatible endpoints and evaluates retrieval and generation results with transparent, documented metrics.

Inference, under load

Compare vLLM, TGI, SGLang, and TensorRT-LLM using throughput, P50/P95/P99 latency, streaming TTFT, token throughput, and estimated cost per 1,000 tokens. Export run metrics to Prometheus and inspect JSON results in the Streamlit dashboard.

RAG, with evidence

Measure recall@k against labeled relevant documents, end-to-end latency, and a deterministic answer-term support proxy. The proxy is clearly distinguished from semantic faithfulness or an LLM-judge evaluation.

Bring JSONL records transformed from public datasets such as HotpotQA or Natural Questions, with dataset versions and licenses documented for each experiment.

Start a reproducible run

pip install -e ".[dashboard]"
benchmarkkit inference \
  --dataset samples/prompts.jsonl \
  --endpoint http://localhost:8000/v1/chat/completions \
  --model your-served-model --framework vllm

The toolkit includes Docker Compose for the results dashboard and Prometheus, sample JSONL fixtures, metric assumptions, and contributor guidance.

Back to profile