# Paper uses one Jev yes/no score to flag alignment failures

A paper reports that TypeSafe AI’s Jev, with no extra training, separates alignment failures from good responses at a median AUROC of 0.886. Across 19 benchmarks, one Jev pass cost $0.30, compared with $18.96 for the LLM judges those benchmarks use.

Language: en
Time zone: America/Los_Angeles
Current time (UTC, cached as of): 2026-11-17T15:01:00.000Z
Query date (America/Los_Angeles): 2026-11-17

HTML: https://didcodexreset.com/news/a8f1cfebf3db4b759dcd1271.html

Content language: en
Localization state: sameLanguage

Source: elvis · Published 9/27/2026, 13:03:40

Researchers built RLCDAlignBench from 44 existing benchmarks covering ten failure types, including sycophancy, jailbreaks, deception, prompt injection, and reward hacking. The method asks Jev one generic yes/no question about a model’s response and uses that probability as the score. Jev, TypeSafe AI’s calibrated decision model, can answer many typed questions about one input in a single call, each with its own probability.

On StrongREJECT, the paper says Jev matches the GPT-4o-mini scorer’s agreement with human labels and ranks responses better, with an AUROC of 0.971 versus 0.929. Where Jev confidently disagreed with benchmark labels, it found label errors in three benchmarks. Question wording mattered little, but thresholds did not transfer across benchmarks; fitting one on 10 labelled items raised median F1 from 0.706 to 0.793.

The results are reported in the [arXiv paper](https://arxiv.org/abs/2609.29429), with a [paper chat](https://academy.dair.ai/papers/just-ask-jev-reinforcement-learning-for-calibrated-decisions-as-a-zero-shot-dete-2609.29429) also available.

Tags: Jev, TypeSafe AI, Alignment, RLCDAlignBench, Benchmarks

[View original post](https://x.com/omarsar0/status/2104295014117589053)
