Jev as a RAG judge — live numbers on rag-configurator

Shipping a RAG product is one thing. Trusting that your evals mean the same thing on Monday as they did on Friday is another. LLM-as-judge is the default now — paste a rubric, ask for a score, ship a dashboard. The bill is manageable. The variance is not. Re-run the same frozen trace ten times and the quality score wanders. Pass/fail can flip. CI noise goes up. You start writing “soft” thresholds so the pipeline does not fail on weather.

That is the problem I care about on rag-configurator: not “can a model grade an answer,” but “is the grade stable enough that I can use it as a gate.” LangChain’s Jev-as-a-Judge for Agent Evals piece pushed the same question with typed Choice / Score / Noul instead of free-text JSON. I wanted live numbers on this product’s Naive RAG traces — not a weather-agent demo, not a mock, not ASIL theatre.

So we wired TypeSafe Jev into the existing eval path and ran a live COMPARE against an Ollama Cloud LLM judge. Same seven fixtures. Ten repeats each. USD cap $2. Observed proxy spend $0.0258. Binary agreement tied at 1.0. Quality variance and latency did not.

What we built (extend, do not fork)

rag-configurator already had an evaluation/ surface. The job was to extend it, not grow a second scoring universe next to it.

Merged in PR #1:

• Frozen Naive RAG traces — question, retrieved chunks, answer, human oracle pass/quality where labeled.
• Two judges on identical state: TypeSafe Jev (jev-1.13.0) with Score + Noul, and an LLM JSON rubric via Ollama Cloud (deepseek-v4-flash:0731).
• A compare harness that reports agreement, repeatability, signal value, quality variance, p50/p95 latency, and a USD cost proxy.
• Offline mock for CI (make jev-eval-compare) and a live path behind JEV_EVAL_LIVE=1 with a hard spend cap.

Jev never writes prose judgments here. It returns typed Score / Noul over compact JSON. The LLM path asks for a structured rubric JSON the way most teams already do. Control flow and report generation stay in code.

This is the same “System One gate” idea I used in the coding harness post — closed schema, cheap judgment, escalate or accept — pointed at RAG eval instead of model routing.

Protocol — frozen traces, K=10, live keys

Protocol matters more than slogans:

1. Frozen traces. Seven cases across grounded answers, hallucinations, partial answers, wrong retrieval, and empty-context fabrications (PTO, remote stipend, HMO premium, parental leave). No live retrieval during judging. The retrieval already happened; both judges see the same blob.
2. Same rubric shape. Binary does_pass plus a 1–5 quality score. Jev uses Score + Noul. LLM returns JSON with the same fields.
3. K=10 repeats per case per judge (70 calls each). Repeatability and quality variance are measured across those repeats, not guessed from a single shot.
4. Live mode. Real TypeSafe + Ollama Cloud calls. Cap $2. Observed proxy $0.0258 for the whole COMPARE (both judges).
5. Honest labels. Agreement is against human oracle_pass when present. Numbers below are observational for this fixture set — not a general ranking of judges, not a safety certification, not an ASIL claim.

If your compare harness re-samples retrieval or rewrites the prompt between judges, you are not comparing judges — you are comparing weather.

Live metrics (2026-09-21)

Mode: live. Cases: 7. Repeats: 10. Cap: $2. Observed proxy: $0.0258.

Highlights from the same run:

• LLM quality variance was 169.9× Jev (0.03809524 vs 0.00022429).
• Mean latency was 15.3× Jev (~4472 ms vs ~293 ms; Jev p50 226.5 ms, LLM p50 4180.0 ms).
• Cost proxy: LLM was 0.6× Jev ($0.009706 vs $0.016086).
• Binary agreement: both 1.000. Signal value: both 1.000.

So on this product slice, pass/fail accuracy tied. Stability and wall-clock did not. Dollar proxy favored the LLM judge — which is an honest result, not a footnote to bury.

Honest takeaways

When Jev wins on this fixture set

• You re-score the same traces in CI or overnight sweeps and you cannot afford quality scores that jitter by tenths.
• You care about p50 under a few hundred milliseconds so a judge call can sit inside a developer loop or a cheap pre-merge gate.
• You want a closed schema (Score / Noul / Choice) instead of parsing free-text “mostly good, maybe 3.5?” from a chat model.
• Binary agreement is already saturated and the remaining pain is variance — exactly what showed up here (both at 1.0 agreement; Jev ~170× tighter on quality var).

When the $ proxy favors the LLM

• On this run the LLM path was cheaper by proxy (~0.6× Jev total). If your bottleneck is dollars per thousand traces and you only need a coarse pass/fail with majority vote, a fast flash model on Ollama Cloud can win the spreadsheet.
• If you only ever call the judge once per trace and never look at continuous quality, you will not feel the variance tax. You will feel it the day you chart score distributions or tighten a gate.
• Latency still hurts. ~4 s p50 is fine for a batch nightly job. It is not fine if you want “judge this answer before I merge.”

What this is not

• Not a claim that Jev beats every LLM judge on every domain.
• Not a general leaderboard.
• Not ASIL / SIL / PL / functional-safety evidence of any kind.
• Not a replacement for human review on high-stakes answers.

Jev closes the schema and calms the score. It can still be wrong. The harness’s job is cheap, repeatable judging you can put next to retrieval metrics — not magic correctness.

How to reproduce

Repo: github.com/ozkannceylan/rag-configurator (merged PR #1).

# Offline mock/demo — no API keys, CI-safe
make jev-eval-compare

# Live COMPARE (TypeSafe Jev + Ollama Cloud LLM judge)
export TYPESAFE_API_KEY=...    # your key — never commit it
export OLLAMA_API_KEY=...      # Ollama Cloud
JEV_EVAL_LIVE=1 make jev-eval-compare-live

Optional: JEV_EVAL_USD_CAP=2.0 (default in the live target we used). Artifacts land as COMPARE report + CSV + latest.json. Read docs/jev-eval.md in the repo for field definitions.

No secrets in the blog, no secrets in the PR. Keys stay in the environment.

Links

• Repo: https://github.com/ozkannceylan/rag-configurator
• Prior harness write-up: Jev as a coding harness — Cursor + Claude
• Inspiration: LangChain — Jev-as-a-Judge for Agent Evals

Closing

For this Naive RAG slice on rag-configurator, live COMPARE says: same binary accuracy, far calmer quality scores, and roughly fifteen times faster p50 for Jev — while the LLM judge still won the crude dollar proxy. That is enough to keep Jev in the eval path as the stable gate, and enough to keep the LLM path around when batch cost dominates.

Typed System One judging will not replace careful retrieval work. It will stop your scoreboard from lying about noise. That is the practical win.