HUD
A platform for building RL environments and evals for AI agents — define an environment once, then evaluate and train any model on it.
Positive: CI uses minimal contents:read permissions and seccomp hardening for the coding template; example tests explicitly defend against path traversal (../, /etc/passwd) and clear staging directories (including symlinks) between tasks; external effects (remote deploy, Docker, traces recorded on hud.ai) are clearly disclosed. Deducted: no user-confirmation flows visible (hud deploy, hud set execute directly); sensitive-data handling shows only env-var access with no storage/transport detail; the httpx2 dependency is suspicious (not a mainstream PyPI package — possible typo or supply-chain risk) and allow-direct-references is enabled; rollback covers only package migration (hud-python→hud), not runtime operations.
Positive: README, pyproject, CI and tests are mutually consistent (templates are actually built and task lifecycles exercised in CI); tests show graceful degradation (empty criteria yields 0.0 reward instead of crashing) and a typed error (DataFileError). Deducted: exact pins such as required-version==0.12.2 for uv and fastmcp==3.2.0 make environments brittle; availability of the httpx2 dependency is questionable.
Positive: covers coding, browser, computer-use, and robotics scenarios; template mechanism (blank/coding/cua) and bring-your-own-harness design give wide adaptability; the protocol manifest declares capability boundaries explicitly and robot is flagged beta. Deducted: Python support limited to 3.11–3.12 with a hard uv 0.12.2 requirement, narrowing environment fit.
Positive: well-organized docs (quickstart/reference/advanced); thorough install notes including the hud-python→hud migration guidance; complete MIT LICENSE with a named copyright holder; CONTRIBUTING, contributor attribution, and a release workflow exist. Deducted: no CHANGELOG file (version read dynamically from version.py); known limitations appear only as scattered beta tags rather than a consolidated statement.
Positive: protocol-first design (manifest / tasks.start / tasks.grade) lets one environment definition serve eval and RL across models; Run/trace/reward outputs are structured and usable. Deducted: cost-benefit is weakened by dependence on the hud.ai cloud platform and a required HUD_API_KEY — platform lock-in with an unclear fully-offline path.
Positive: major claims cross-corroborate across files (pyproject script entry matches README CLI; CI actually validates templates; tests verify manifest args and x-hud-hint declarations); beta/experimental labels separate maturity claims. Deducted: scale claims ('at any scale') have no in-repo evidence; the citation's author list cannot be statically verified.
- The httpx2 dependency is not a mainstream PyPI package and may be a typo or supply-chain risk; static review cannot confirm its legitimacy — verify manually before installing.
- Core workflows (deploy, eval --remote, trace recording) depend on the hud.ai cloud platform and a required HUD_API_KEY: platform lock-in plus outbound data; the Scarf tracking pixel in the README is also external telemetry.
- Evaluation sandboxes expose high-power capabilities (shell, browser, VNC); run untrusted environments only in isolated networks/containers. Note the CI uses systempaths=unconfined for the coding template.
- Exact pins (uv required-version==0.12.2, fastmcp==3.2.0) may block upgrades or offline installs.
What does this agent do, and when should you use it?
HUD is a platform for building reinforcement learning environments for AI agents across coding, browser, computer-use, and robotics domains. It is protocol-first: an agent and an environment exchange only a manifest, a tasks.start call returning the prompt, and a tasks.grade call returning the reward, so any model or harness can plug into any environment. Developers define tasks with async templates registered via @env.template(), build and register environments with hud deploy, and run evaluations with hud eval. Every rollout produces a Run carrying a trace_id and reward, which can be fed directly into TrainingClient.step() for custom GRPO/PPO training loops. The SDK natively supports Claude, OpenAI, OpenAI-compatible endpoints, and Gemini models, and exposes sandboxed shells, tools, browsers, full computer control, and robots through the ssh, mcp, cdp, rfb, and robot capability protocols.
You install the hud CLI via pip or uv, set HUD_API_KEY, scaffold an environment with hud init, and define tasks in template functions that yield a prompt and a reward; then run hud eval tasks.py claude --group 3 against any supported model. During evaluation the agent interacts with the environment through capability protocols: ssh (sandboxed shell and files), mcp (Model Context Protocol tools), cdp (Chrome DevTools Protocol browser control), rfb (full computer-use over VNC), and robot (beta, WebSocket observation/action loop). Each graded evaluation is recorded as a trace on hud.ai when HUD_API_KEY is set, and the live SDK handle is a Run. hud deploy builds and registers the environment image in one step; hud sync tasks then hud eval --remote runs it remotely. The hud.integrations.harbor adapter converts Harbor task directories into runnable HUD tasksets and exports HUD tasks back to Harbor folders. TrainingClient.step() consumes grouped, graded runs to supply environment and reward signals to your own GRPO/PPO loop.
- An RL research team that needs to reuse the same task environment across models for GRPO/PPO training, feeding graded rollouts from TrainingClient into their own training loop.
- An evaluation engineer comparing models on an identical taskset, running grouped batches with hud eval and inspecting traces on hud.ai.
- A computer-use agent developer testing against real desktops via the rfb capability (VNC screen, keyboard, mouse) and cdp browser control.
- An organization with existing Harbor-format benchmarks using hud.integrations.harbor to adapt task directories into HUD tasksets without rewriting tasks.
- A platform team packaging environments as images with hud deploy and registering them for remote batch runs across the team.
- A robotics researcher driving a schema-based observation/action loop over WebSocket through the beta robot capability to evaluate VLA policies.
What are this agent's strengths and limitations?
- Protocol-first design (manifest / tasks.start / tasks.grade) keeps environments, benchmarks, and tasks usable across new harnesses and models without rework.
- One environment serves one-shot Q&A through full computer-use and robotics rollouts via the ssh, mcp, cdp, rfb, and robot capabilities, with variants and grading unchanged.
- Evals are already training data: every rollout returns a Run with trace_id and reward, directly consumable by TrainingClient.step() for GRPO/PPO without changing the environment.
- Native support for Claude, OpenAI, OpenAI-compatible endpoints, and Gemini, plus documented paths to wrap custom harnesses (e.g., browser-use on cdp, a VLA policy on robot).
- Platform dependency: full value (remote runs, batch comparisons, trace inspection) requires the hud.ai platform and HUD_API_KEY.
- The package rename from hud-python to hud forces a migration (uninstall both, reinstall, update requirements and CI) or the two packages clobber each other's files.
- The robot capability is labeled beta, so production reliability is unproven.
- Actual cost and resource quotas for remote evaluation and training are not documented in the source; verify before committing.
How do you install or deploy this agent?
Install the CLI (recommended, Python 3.12):
uv tool install hud --python 3.12Or as a library:
pip install hudNote: the package was renamed from hud-python to hud; if the old package is installed, run pip uninstall -y hud-python hud && pip install hud first (uninstalling both prevents file clobbering) and update requirements files and CI configs. Then get an API key at https://hud.ai/project/api-keys and set it:
hud set HUD_API_KEY=your-key-here
or: export HUD_API_KEY=your-key-hereHow do you use this agent?
- Scaffold an environment: hud init my-env (coding), hud init my-desktop-env --template cua, or hud init my-custom-env --template blank.
- Define tasks with a template: decorate an async function with @env.template(), yield a prompt, receive the answer, yield a reward (see the letter-count example), and call the template to mint task variants.
- Run evals: hud eval tasks.py claude --group 3, or build and register with hud deploy, then hud sync tasks my-taskset and hud eval my-taskset --remote for remote runs.
- Train: run a group of rollouts per task and pass the graded Run objects to TrainingClient.step() to drive your GRPO/PPO loop.
All evals are recorded on hud.ai when HUD_API_KEY is set, where you can compare models and inspect traces.