JEV cascade keeps 99% of GPT-6 accuracy at about 57% of the fee

A paper introducing Jev-as-a-Judge finds that most evaluations should use a cheap judge, with only uncertain calls sent to a frontier model. It tests that approach with JEV, TypeSafe AI’s decision-only judge. On 510 held-out preference pairs, a cascade that accepted JEV’s confident verdicts and escalated the rest to GPT-6 Astra kept 99% of GPT-6’s accuracy at about 57% of its fee.

JEV returns a verdict and label probabilities and does not produce reasoning text. It costs $0.044 per 1,000 judgments, with a median latency of 0.152 seconds, compared with $12.182 and 1.885 seconds for GPT-6, about 277 times cheaper.

On ordinary preference and evidence-grounded factuality, JEV stays within 3 points of GPT-6: 92.2% versus 93.5% on RewardBench, and 87.5% versus 86.7% on HaluEval. The gap grows to 9 to 20 points on tasks that require checking a derivation or rejecting an elaborately written wrong answer, including 78.6% versus 93.1% on JudgeBench. On several benchmarks, JEV’s gap to GPT-6 is concentrated in its low-confidence decisions. The escalation threshold did not transfer to every fallback model, so the authors recommend setting it on one’s own data. The paper is Jev-as-a-Judge: accept when confident, escalate when unsure.