RAG Evaluation Infrastructure
Project Summary
Client: Enterprise Platform Industry: Enterprise Search / Knowledge Management
Impact Metrics:
- Replaced generic quality metrics with calibrated measurements, such as answer accuracy, retrieval accuracy, and stability
- CI/CD regression testing for search quality on every deployment
- Open-source models in the judge role, cutting the cost per evaluation run
- A documented framework the team runs for ongoing quality monitoring
Challenge
The client had a production RAG-based search assistant and no reliable way to measure whether it was working. The same evaluation run twice gave different numbers, so teams couldn't tell if a change improved or degraded search quality. Without trustworthy evaluation, every deployment was still guesswork.
Approach
I built a systematic measurement layer designed for production reliability:
- LLM-as-a-judge framework: evaluation against defined rubrics, with the judge calibrated on a labelled reference set
- Open-source judges: OSS models in the judge role, which cut the recurring cost of every evaluation run
- CI/CD integration: automated regression testing, so search quality is verified on every deployment
- Metric calibration: metrics tied to the reference set, so a movement in the number reflects a movement in the system
What the client has now
- A rubric-based judge, calibrated against a labelled reference set
- A regression suite that runs in CI on every deployment
- Evaluation runs cheap enough to repeat often, using open-source judges
- Documented metric definitions, so the same number means the same thing next quarter
Tech Stack
- Python
- LLM-as-a-judge evaluation framework
- Open-source language models for cost-efficient judging
- CI/CD pipeline integration
- Statistical calibration and metric design
My Role
Designed and built the entire evaluation infrastructure, from metric definition and judge calibration to CI/CD integration and production deployment.
The reasoning behind the layered judge is in The judge does not score. The judge diagnoses..
-
Need to measure your RAG system's quality?
Tell me what your system does and where you suspect it drifts, and I'll say what measuring it would take.