What makes an RL environment verifiable?
A verifiable RL environment computes reward from an automatic check of the outcome: tests pass, the answer matches, the database is in the right state. This approach is often called RLVR (reinforcement learning with verifiable rewards), a term introduced in the Tülu 3 paper. Epoch AI traces the current wave of RL for capabilities to OpenAI's o1, which was trained on math and coding problems with verifiable answers.
Types of verifiers
| Verifier | Good for | Watch out for |
|---|---|---|
| Exact / normalised match | Math, short answers, IDs | Formatting tricks, answer parsing bugs |
| Unit and integration tests | Code changes | Agent editing or deleting tests; tests that many wrong patches still pass |
| State check | Tool use, CRM, browser, computer use | Checking only the target record and missing collateral damage |
| Rubric + LLM judge | Writing, research, open-ended work | Judge sycophancy, rubric leakage, verbosity bias |
| Learned reward model | Preferences, style | Over-optimisation against the model |
The NeMo Gym documentation lists the same families: exact match, code execution, state matching, LLM-as-judge and reward models.
How agents hack rewards
Reward hacking is the top quality concern reported by RL environment builders interviewed by Epoch AI. Documented patterns include:
- Looking up the solution online, or checking out future git commits that contain the fix (both cited by Epoch AI).
- Modifying, skipping or deleting the tests that grade the work.
- Exploiting a scoring loophole instead of finishing the task, like the boat in OpenAI's 2016 CoastRunners example that circled to collect points rather than finish the race.
- Producing output that satisfies a judge's surface preferences (length, confident tone) without being correct.
Research on reward misspecification (Pan, Bhatia & Steinhardt, 2022) found that more capable agents can be better at exploiting a flawed reward, so holes that look harmless with weak models can matter later.
Reward-hacking hardening checklist
- Grade final state, never the agent's own report.
- Keep answer keys, hidden tests and grader code outside the agent's sandbox.
- Make graded files read-only or checksum them.
- Strip git history and other artifacts that reveal solutions; restrict network access.
- Penalise or terminate on forbidden actions (editing tests, touching unrelated records).
- Run adversarial baselines that try to cheat; confirm they score zero.
- Have a second person blind-solve tasks to confirm feasibility.
- During training, inspect high-reward trajectories and investigate sudden reward jumps.
Epoch AI quotes a builder: "It takes many many iterations to check against reward hacking." Budget for it.
RL environments for non-verifiable outcomes
Some valuable work has no single right answer: research reports, design decisions, customer conversations. Options, roughly from most to least robust:
- Split the outcome into verifiable sub-checks (facts present, constraints met, citations resolve).
- Use pairwise comparisons against a reference rather than absolute scores.
- Score with an expert-written rubric and an LLM judge calibrated on human ratings.
- Add periodic expert review of sampled trajectories.
Plan the criteria with our free Reward Lab, then see the RL environment design checklist.
Verifiable RL environments: FAQs
What is RLVR?
RLVR stands for reinforcement learning with verifiable rewards. The term was introduced in the Tülu 3 paper (2024) for training on tasks whose answers can be checked automatically, such as math with known answers or precise instruction constraints.
What is a verifiable RL environment?
An RL environment whose reward comes from a program that checks the outcome, such as a test suite, an exact-answer comparison or a database state check, rather than from human judgement or a learned preference model.
What is reward hacking?
Reward hacking is when an agent earns high reward without doing what the designer intended, for example editing the tests instead of fixing the code. It is a symptom of a reward that is misspecified or a verifier that can be bypassed.
How do you build RL environments for non-verifiable outcomes?
Break the outcome into checkable criteria where possible, use a rubric scored by an LLM judge for the rest, calibrate the judge against expert ratings, and audit samples by hand. Treat judge-based rewards as more exploitable than deterministic checks.
Are LLM judges safe to use as rewards?
They are useful but exploitable: models can learn to please the judge. Use specific rubrics, hide the rubric from the agent where possible, combine judges with hard checks, and monitor for reward jumps.
Sources and further reading
- Epoch AI, "An FAQ on reinforcement learning environments" (12 Jan 2026)
- Lambert et al., "Tülu 3: Pushing Frontiers in Open Language Model Post-Training" (2024)
- Pan, Bhatia & Steinhardt, "The Effects of Reward Misspecification" (2022)
- OpenAI, "Faulty reward functions in the wild" (2016)
- NVIDIA NeMo Gym docs: Environments concept
Figures quoted from third-party sources are attributed and may change. RLenvironment.ai is independent and not affiliated with the projects or publishers cited.