How to build an RL environment

What you are building

Building an RL environment means writing a program that an agent can repeatedly reset() and step() through, and that returns a trustworthy reward. For LLM agents the program usually wraps a sandbox (a container, VM or simulated app), a set of tools, a task dataset, and a grader. If you are new to the concepts, start with what RL environments are.

Pick one capability and a checkable outcome

Write one sentence describing what the agent should be able to do, and one describing how a program can tell whether it did. "Resolve refund requests according to policy" is buildable; "be a great support agent" is not. If you cannot describe the check, you are not ready to build the environment yet.

Define observations and actions

Decide what the agent sees (task text, tool output, screenshots, DOM or accessibility tree) and what it can do (tool calls, shell commands, clicks). Keep the action space close to what the agent will have in production. Epoch AI notes enterprise environments come in several forms: MCP-style tool integrations, Playwright-style browser control, and screenshot-based computer use.

Model state and make reset reproducible

Everything the agent can change must be restorable: files, databases, app data, browser sessions. Seed random generators, pin dependency versions, and start each episode from a snapshot. Epoch AI reports that environments are often delivered as Docker containers, which makes this easier.

Write the verifier before the tasks scale

The verifier (grader) is the most important code you will write. Check the resulting state, not the agent's description of what it did. Common patterns, as listed in the NeMo Gym docs, are exact match, running tests, state matching, LLM-as-judge, and reward models. Prefer deterministic checks and use judges only where necessary.

A minimal RL environment in Python

This toy tool-use environment follows the Gymnasium-style reset() / step() convention. The reward comes from the final order state, so an agent cannot earn reward just by saying "refund issued."

import random

class RefundEnv:
    """Toy tool-use RL environment: refund only orders inside policy."""

    POLICY_DAYS = 30

    def __init__(self, seed=None):
        self.rng = random.Random(seed)

    def reset(self):
        self.order = {"id": self.rng.randint(1000, 9999),
                      "days_since_delivery": self.rng.randint(1, 60),
                      "refunded": False}
        self.steps = 0
        obs = (f"Customer requests a refund for order {self.order['id']}. "
               "Tools: lookup, refund, deny.")
        return obs, {}

    def step(self, action):
        self.steps += 1
        if action == "lookup":
            obs = (f"Order {self.order['id']} was delivered "
                   f"{self.order['days_since_delivery']} days ago. "
                   f"Policy: refunds within {self.POLICY_DAYS} days.")
            truncated = self.steps >= 5
            return obs, 0.0, False, truncated, {}
        if action == "refund":
            self.order["refunded"] = True
        # Verifier: score the final STATE, not the agent's wording.
        eligible = self.order["days_since_delivery"] <= self.POLICY_DAYS
        reward = 1.0 if self.order["refunded"] == eligible else 0.0
        return "Episode finished.", reward, True, False, {"eligible": eligible}

env = RefundEnv(seed=7)
obs, info = env.reset()
obs, reward, terminated, truncated, info = env.step("lookup")
obs, reward, terminated, truncated, info = env.step("refund")
print(reward, info)

Real environments replace the dictionary with a sandboxed application and the string actions with structured tool calls, but the contract is the same.

Write tasks with a spread of difficulty

Create varied tasks that exercise related skills. Practitioners interviewed by Epoch AI described aiming for a minimum pass rate of around 2–3% (about one success in 64–128 attempts), mixing tasks at 0%, 5% and 30%, and retiring tasks once they reach roughly 70%. Have a second person blind-solve a sample to confirm tasks are feasible.

Attack your own reward

Before training, try to get reward without doing the task. Can the agent read the answer key, edit the tests, look up the solution online, or check out a future git commit? (Epoch AI cites that last one as a real failure.) Run weak and adversarial baselines and confirm they score low. The reward-hacking checklist lists common holes.

Package it for training and evaluation

Expose the environment through a framework your training stack supports. Gymnasium is the classic standard; Verifiers, NeMo Gym and OpenEnv target LLM agents. See choosing an RL environment framework. Log full trajectories, tool calls and rewards so failures can be inspected.

Monitor once training starts

Watch for sudden reward jumps, shorter episodes, or repeated unusual actions. These often signal a new exploit rather than real progress. Read a sample of high-reward trajectories by hand every run.

How to build an RL environment: FAQs

Sources and further reading

1. Epoch AI, "An FAQ on reinforcement learning environments" (12 Jan 2026)
2. NVIDIA NeMo Gym docs: Environments concept
3. Gymnasium documentation: Env API
4. OpenEnv on GitHub

Figures quoted from third-party sources are attributed and may change. RLenvironment.ai is independent and not affiliated with the projects or publishers cited.