Only 35.8% of the cleanest public terminal-agent RL collection passed a Salesforce AI Research audit

Salesforce AI Research audited public reinforcement-learning collections for terminal agents. Only 35.8% of the environments in the cleanest collection passed. Two other public collections were 10.1% and 3.3% clean. The audit found reward errors in both directions: some environments give reward 1 for copying a leaked answer or passing a weak verifier without doing the task, while others give reward 0 to a correct solution because the reference answer or oracle is wrong.

The authors' recipe, RIVER, filters defective environments and penalizes turns that repeat an earlier command with nearly the same output. They argue that RL mainly shapes behaviors already learned in pre-training and SFT, including inspecting before acting, verifying before finishing, and dropping an approach that keeps failing.

With the budget held at 3.5K environments, River-8B averaged 19.4 across four terminal benchmarks, against 17.7 for RL on 3.5K environments sampled at random from the same collection. Of the open RL-trained 8B models the authors evaluated, River-8B was the best on all four benchmarks. Across models from 2B to 27B, and using fewer than 30% of TMax's environments, they report that RIVER increases RL gains by 106% on Terminal-Bench-Lite and 30% on Terminal-Bench v2.1. The paper is Learning generalizable behaviors for terminal agents.