Coder Eval

What does this agent do, and when should you use it?

Coder Eval is UiPath's open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills, published on PyPI as coder-eval and installed with pip install coder-eval or uv tool install coder-eval on Python 3.13+. An evaluation is a YAML task file: an initial_prompt, an agent config, a sandbox driver, and a list of success_criteria. At run time the framework starts a real agent — Claude Code, OpenAI Codex, Google Antigravity (Gemini), OpenCode, or Pi — inside an isolated sandbox, then scores the files and commands it actually produced. Scoring is continuous and weighted from 0.0 to 1.0, with criterion types ranging from file_exists and run_command to code similarity and LLM-graded rubrics, plus a skill_triggered activation check. It also ships an A/B experiment layer, per-tool-call token and cost telemetry, run.json/variant.json/task.json reports, a GitHub Actions composite action, and a Claude Code plugin marketplace front end. It never supplies model access: you bring the agent runtime and the model credentials.

You author a YAML task (task_id, description, initial_prompt, agent, sandbox, success_criteria), then use the coder-eval CLI: coder-eval plan tasks/hello_date.yaml validates it without spending tokens, coder-eval run tasks/hello_date.yaml starts the configured agent in a tempdir or Docker sandbox to complete the task, and coder-eval report runs/latest shows the scored result. agent.type is the only harness-specific line, so switching to codex, antigravity, or opencode (or overriding per run with -D agent.type=opencode) leaves tasks, criteria, scoring, telemetry, and reports unchanged. After the run, each success_criterion is evaluated against the artifacts and commands the sandbox produced, yielding a weighted score against pass/fail thresholds. The experiment layer fans the same tasks across different models, tools, or prompts side by side, while telemetry records every tool call, token count, and cost with real-time streaming. For CI, the UiPath/coder_eval composite action installs a pinned CLI version, runs the tasks, writes JUnit XML and run.md, and fails the step on any task failure. Inside Claude Code, the repo doubles as a plugin marketplace exposing six slash commands — /coder-eval:init, /coder-eval:check-skill, /coder-eval:task, /coder-eval:lint-tasks, /coder-eval:analyze, and /coder-eval:ci — which drive the same CLI.

1. Benchmark authors who want to compare Claude Code, Codex, Gemini (Antigravity), OpenCode, and Pi on task suites from their own domain instead of a shared public dataset.
2. Teams shipping Claude Code skills or plugins who need the skill_triggered criterion to confirm the skill actually fires in the agent their users run.
3. Skill maintainers who wire a scheduled GitHub Actions job so skills are continuously re-validated and quietly stop triggering before users notice.
4. Platform or infrastructure teams that want coding-agent quality to be a CI gate, failing the build when a regression appears.
5. Researchers and product teams running A/B experiments — model vs. model, tool-on vs. tool-off, prompt vs. prompt — who also want per-tool token and cost data.
6. Teams with existing task data that use Bring Your Own Dataset to fan one task across many rows and scale the suite.

How do you install or deploy this agent?

Prerequisites: Python 3.13+, uv 0.8+, and the runtime of at least one coding agent plus that agent's own model credentials. From a clone:

uv sync                      # install the framework (add --extra codex / --extra antigravity for those runtimes)
cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip it: an existing `claude login` is picked up automatically

If you only want the CLI without cloning:

uv tool install coder-eval
uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
coder-eval --version                              # verify the install

To add it as a project dependency, use uv add coder-eval or pip install coder-eval. Two of the agents need their runtime installed separately:

brew install claude                     # Claude Code
npm install -g opencode-ai              # OpenCode

How do you use this agent?

Your first end-to-end run, using the default claude-code agent:

uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
uv run coder-eval report runs/latest           # view the result

A minimal task file, tasks/hello_world.yaml:

task_id: "hello_world"
description: "Create a Python script that prints Hello, World!"
initial_prompt: "Create hello.py that prints 'Hello, World!'"

agent:
  type: "claude-code"
  permission_mode: "acceptEdits"
  allowed_tools: ["Read", "Write", "Bash"]

sandbox:
  driver: "tempdir"
  python: {}

success_criteria:
  - type: "file_exists"
    path: "hello.py"
    description: "hello.py must be created"
  - type: "run_command"
    command: "python hello.py"
    timeout: 10
    description: "Script must execute successfully"

Install the Claude Code plugin front end for the six slash commands:

/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval

Use it as a GitHub Actions gate (the default claude-code agent needs the claude CLI first):

- uses: actions/setup-node@v4
  with: { node-version: '20' }
- run: npm install -g @anthropic-ai/claude-code

- uses: UiPath/coder_eval@v0
  id: eval
  with:
    args: |
      tests/tasks/**/*.yaml
      --model
      claude-sonnet-5
    env: |
      ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

- if: always()
  run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
- uses: mikepenz/action-junit-report@v5
  if: always()
  with:
    report_paths: ${{ steps.eval.outputs.junit-path }}

Do not run the action under pull_request_target with secrets exposed to untrusted fork PRs. To turn off usage telemetry, set TELEMETRY_ENABLED=false in your .env or environment.

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

FAQ