Coder Eval

Playwright for coding agents: run real agents against declarative YAML tasks in a sandbox, score them with weighted criteria, and gate CI on the result.

Stars
★ 160
Last updated
2d ago
License
Apache-2.0
Primary language
Python

At a glance

How it runs
CLIAgent plugin / skill
Works with
Universal · cross-platformCodex · Claude Code · OpenAI API · Claude API
Cost
Free software; you pay for model usage
Setup effort
Medium · a few setup steps
You'll need
Python 3.13+uv 0.8+Claude Code CLINode.js 20Docker (only for container isolation driver)Shell / CLINetwork accessLocal filesystemMCP Server
Typical use
Benchmark authors who want to compare Claude Code, Codex, Gemini (Antigravity), OpenCode, and Pi on task suites from their own domain instead of a shared public dataset.
Not a fit if
  • Teams that want a ready-made fixed leaderboard instead of authoring their own tasks and scoring criteria
  • Users unwilling to install and configure a separate agent runtime such as Claude Code or OpenCode
Website
coder-eval.com ↗ set on GitHub
Maintenance
Commits in the last 30 dayslast commit 2026-11-19latest release v0.12.18 (2026-11-09)
Source review
79/100 · Good

What does this agent do, and when should you use it?

Coder Eval is UiPath's open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills, published on PyPI as coder-eval and installed with pip install coder-eval or uv tool install coder-eval on Python 3.13+. An evaluation is a YAML task file: an initial_prompt, an agent config, a sandbox driver, and a list of success_criteria. At run time the framework starts a real agent — Claude Code, OpenAI Codex, Google Antigravity (Gemini), OpenCode, or Pi — inside an isolated sandbox, then scores the files and commands it actually produced. Scoring is continuous and weighted from 0.0 to 1.0, with criterion types ranging from file_exists and run_command to code similarity and LLM-graded rubrics, plus a skill_triggered activation check. It also ships an A/B experiment layer, per-tool-call token and cost telemetry, run.json/variant.json/task.json reports, a GitHub Actions composite action, and a Claude Code plugin marketplace front end. It never supplies model access: you bring the agent runtime and the model credentials.

You author a YAML task (task_id, description, initial_prompt, agent, sandbox, success_criteria), then use the coder-eval CLI: coder-eval plan tasks/hello_date.yaml validates it without spending tokens, coder-eval run tasks/hello_date.yaml starts the configured agent in a tempdir or Docker sandbox to complete the task, and coder-eval report runs/latest shows the scored result. agent.type is the only harness-specific line, so switching to codex, antigravity, or opencode (or overriding per run with -D agent.type=opencode) leaves tasks, criteria, scoring, telemetry, and reports unchanged. After the run, each success_criterion is evaluated against the artifacts and commands the sandbox produced, yielding a weighted score against pass/fail thresholds. The experiment layer fans the same tasks across different models, tools, or prompts side by side, while telemetry records every tool call, token count, and cost with real-time streaming. For CI, the UiPath/coder_eval composite action installs a pinned CLI version, runs the tasks, writes JUnit XML and run.md, and fails the step on any task failure. Inside Claude Code, the repo doubles as a plugin marketplace exposing six slash commands — /coder-eval:init, /coder-eval:check-skill, /coder-eval:task, /coder-eval:lint-tasks, /coder-eval:analyze, and /coder-eval:ci — which drive the same CLI.

  1. Benchmark authors who want to compare Claude Code, Codex, Gemini (Antigravity), OpenCode, and Pi on task suites from their own domain instead of a shared public dataset.
  2. Teams shipping Claude Code skills or plugins who need the skill_triggered criterion to confirm the skill actually fires in the agent their users run.
  3. Skill maintainers who wire a scheduled GitHub Actions job so skills are continuously re-validated and quietly stop triggering before users notice.
  4. Platform or infrastructure teams that want coding-agent quality to be a CI gate, failing the build when a regression appears.
  5. Researchers and product teams running A/B experiments — model vs. model, tool-on vs. tool-off, prompt vs. prompt — who also want per-tool token and cost data.
  6. Teams with existing task data that use Bring Your Own Dataset to fan one task across many rows and scale the suite.

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Install / run
  • uv tool install coder-eval
  • uv tool install "coder-eval[codex,antigravity]" # same, with agent extras
  • brew install claude # Claude Code
  • npm install -g opencode-ai # OpenCode
Environment variables
ANTHROPIC_API_KEY
Config files
.env.example .env tasks/hello_date.yaml tasks/hello_world.yaml
Runtime
Python 3.13+
Install notes cover
macOS

How do you install or deploy this agent?

Prerequisites: Python 3.13+, uv 0.8+, and the runtime of at least one coding agent plus that agent's own model credentials. From a clone:

uv sync                      # install the framework (add --extra codex / --extra antigravity for those runtimes)
cp .env.example .env         # then set ANTHROPIC_API_KEY — or skip it: an existing `claude login` is picked up automatically

If you only want the CLI without cloning:

uv tool install coder-eval
uv tool install "coder-eval[codex,antigravity]"   # same, with agent extras
coder-eval --version                              # verify the install

To add it as a project dependency, use uv add coder-eval or pip install coder-eval. Two of the agents need their runtime installed separately:

brew install claude                     # Claude Code
npm install -g opencode-ai              # OpenCode

How do you use this agent?

Your first end-to-end run, using the default claude-code agent:

uv run coder-eval plan tasks/hello_date.yaml   # validate (no tokens spent)
uv run coder-eval run  tasks/hello_date.yaml   # run your first evaluation
uv run coder-eval report runs/latest           # view the result

A minimal task file, tasks/hello_world.yaml:

task_id: "hello_world"
description: "Create a Python script that prints Hello, World!"
initial_prompt: "Create hello.py that prints 'Hello, World!'"

agent:
  type: "claude-code"
  permission_mode: "acceptEdits"
  allowed_tools: ["Read", "Write", "Bash"]

sandbox:
  driver: "tempdir"
  python: {}

success_criteria:
  - type: "file_exists"
    path: "hello.py"
    description: "hello.py must be created"
  - type: "run_command"
    command: "python hello.py"
    timeout: 10
    description: "Script must execute successfully"

Install the Claude Code plugin front end for the six slash commands:

/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval

Use it as a GitHub Actions gate (the default claude-code agent needs the claude CLI first):

- uses: actions/setup-node@v4
  with: { node-version: '20' }
- run: npm install -g @anthropic-ai/claude-code

- uses: UiPath/coder_eval@v0
  id: eval
  with:
    args: |
      tests/tasks/**/*.yaml
      --model
      claude-sonnet-5
    env: |
      ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

- if: always()
  run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
- uses: mikepenz/action-junit-report@v5
  if: always()
  with:
    report_paths: ${{ steps.eval.outputs.junit-path }}

Do not run the action under pull_request_target with secrets exposed to untrusted fork PRs. To turn off usage telemetry, set TELEMETRY_ENABLED=false in your .env or environment.

What are this agent's strengths and limitations?

Pros
  • Tasks, criteria, scoring, telemetry, and reports stay identical across harnesses: agent.type is the only agent-specific line, and swapping to Codex, Antigravity, OpenCode, or Pi requires no task edits.
  • Scoring is continuous and weighted (0.0–1.0) with thresholds, and the skill_triggered criterion exists specifically to verify that a target skill actually fires.
  • Three documented delivery paths — the coder-eval CLI, the UiPath/coder_eval GitHub Action with JUnit XML and run.md outputs, and a Claude Code plugin marketplace — make it usable as a CI gate out of the box.
  • Full per-tool-call telemetry covering token counts and cost, plus an A/B experiment layer for comparing models, tool toggles, and prompts on the same tasks.
  • Sandboxing supports both a tempdir driver and a Docker container driver, and task dependencies can be pinned for reproducibility.
Limitations
  • Python 3.13+ only, which is a hard blocker for teams still on older interpreters.
  • No model access is provided: you must supply agent runtimes (Claude Code and OpenCode are separate CLIs) and your own model credentials, so real API spend is unavoidable.
  • Anonymous usage telemetry is on by default and must be explicitly disabled with TELEMETRY_ENABLED=false.
  • Tasks execute agent-generated code, and the docs state the tempdir driver is not a security boundary — untrusted tasks must use the container driver.
  • The GitHub Action exposes eight inputs but none of them is a coder-eval run flag, so flags and task globs must all go through args; GitHub silently ignores an input the referenced tag does not define, which can produce a run that measured something else and still exits 0.

How does this agent compare with similar options?

The docs compare Coder Eval against several named alternatives: versus fixed benchmarks like SWE-bench and SkillsBench, which score a canonical dataset, Coder Eval scores your tasks with continuous 0.0–1.0 weighted criteria (and can still wrap a fixed dataset via Bring Your Own Dataset); versus large-scale/RL harnesses such as Harbor, which target scale and RL rollouts, Coder Eval targets weighted, skill-aware suites gated in CI; versus model-output eval tools like OpenAI Evals, which grade model text, Coder Eval runs a full agent in a sandbox and scores the files and commands it produced; versus hand-rolled scripts, it ships reproducible sandboxes, weighted criteria, cost/token telemetry, A/B experiments, and CI-ready pass/fail gates.

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
Coder Eval This agent 79 · Good CLIFree + model costs ★ 160 2d ago Python Codex · Claude Code · OpenAI API · Claude API
Yardlet 80 · Good Desktop appFree ★ 46 today Swift Codex · Claude Code
wsp 59 · Major gaps Desktop appFree + model costs ★ 15 8d ago TypeScript Codex · Claude Code
Loop Engineering 65 · Some gaps CLIFree + model costs ★ 12k 1d ago TypeScript Codex · Claude Code

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Good
79/ 100 5-point scale 4.0 / 5
This review covers the source as of Nov 08, 2026. The project has shipped 1 stable release(s) since, which have not been re-reviewed. The score and verdict describe the reviewed version only. Recent releases ↓
Trust 19/29
Reliability 11/14
Adaptability 15/18
Convention 16/18
Effectiveness 12/13
Verifiability 6/8
Why each dimension lost points
Trust19 / 29 · 3.3/5

The docs describe container isolation, resource limits, the risks of executing real task code, and that tempdir is not a security boundary. The CI example uses read-only permissions and warns against exposing secrets to untrusted forks. Telemetry is disclosed as enabled by default with listed fields and an opt-out; the security policy also says agent_judge uses evaluator credentials. Deductions: permissions and confirmation are largely left to user configuration, and the packet does not establish product-wide sensitive-data safeguards or result rollback. Publisher identity remains unknown as instructed.

Reliability11 / 14 · 3.9/5

README, pyproject metadata, and source broadly align on Python 3.13+, the CLI, supported harnesses, and optional dependencies. The docs say missing CLI or harness prerequisites produce guidance, and the error source defines explicit configuration, timeout, and crash errors. Deduction: some setup depends on separate CLIs, model credentials, or optional extras, and the supplied files do not support checking every documentation-to-implementation detail.

Adaptability15 / 18 · 4.2/5

The materials identify benchmark authors, CLI and skill builders, and CI users; they describe several evaluation scenarios, harnesses, an extension SPI, and the isolation distinction between docker and tempdir. Deductions: the sampled skills focus on the Claude Code plugin, Python 3.13+ is required, and several agents need separately installed runtimes and credentials.

Convention16 / 18 · 4.4/5

The README provides a structured documentation index, installation and quick start, task YAML, CI examples, skill guidance, limitations, and security reporting channels. The license file and package metadata both identify Apache-2.0. Deductions: metadata shows version 0.12.17 while the action example uses the moving @v0 tag; the changelog contents were not supplied for review. Maintenance contacts and update paths are stated, but do not verify publisher identity.

Effectiveness12 / 13 · 4.6/5

The product combines task definitions, sandboxed runs, scoring, A/B comparisons, telemetry, and CI gates into usable team outputs, with report and JUnit paths documented. It also states cost, code-execution, and runtime prerequisites. Deductions: value depends on users authoring suitable tasks and providing agent runtimes and credentials, while live model runs incur cost.

Verifiability6 / 8 · 3.8/5

Major README claims are corroborated across package entry points and extras, the Agent abstraction, error types, skill files, workflows, and security policy; the test sample also shows concrete assertion design. Deductions: some claims are promotional and the packet contains excerpts rather than the complete repository; no execution occurred, so the sample cannot establish all implementation or release behavior.

Risks and how to mitigate them
  • Evaluation tasks execute agent-generated code; the docs state that tempdir is not a security boundary. Use container isolation for untrusted tasks and avoid exposing credentials to untrusted PRs.
  • Telemetry is enabled by default; set TELEMETRY_ENABLED=false if you do not want the documented anonymous usage data sent.
  • This is a static review; tests and runtime behavior were not verified. Publisher identity is unknown.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision 68cd17f3ca19 New commits since this review; the score may not cover them
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/coder-eval.svg)](https://followagents.com/en/agents/coder-eval)
<a href="https://followagents.com/en/agents/coder-eval"><img src="https://followagents.com/badge/coder-eval.svg" alt="FARS review"></a>

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

  1. v0.12.18
    Release notes excerpt
    Bug Fixes - deps: Cap typer below 0.29 ([#255](https://github.com/UiPath/coder_eval/pull/255), [bf4e638](https://github.com/UiPath/coder_eval/commit/bf4e6382eb1eed65243ff26253f50caf6b8bb89b)) - evalboard: Keep a null arm pass rate null and cap Series run.json loads ([#255](https://github.com/UiPath/coder_eval/pull/255), [bf4e638](https://github.com/UiPath/coder_eval/commit/bf4e6382eb1eed65243ff26253f50caf6b8bb89b)) Features - evalboard: Add a Scheduled tab with per-pipeline trends and fold Scribe into it …
    View on GitHub ↗
  2. v0.12.17
    Release notes excerpt
    Bug Fixes - Address grading recovery before-merge feedback ([#248](https://github.com/UiPath/coder_eval/pull/248), [bc45225](https://github.com/UiPath/coder_eval/commit/bc45225fe4b836ae9376a66fc5864f5b49e6851f)) - Align Delegate smoke prompt with automation use ([#248](https://github.com/UiPath/coder_eval/pull/248), [bc45225](https://github.com/UiPath/coder_eval/commit/bc45225fe4b836ae9376a66fc5864f5b49e6851f)) - Make grading failure and budget returns explicit ([#248](https://github.com/UiPath/coder_eval/pull/248), …
    View on GitHub ↗
  3. v0.12.16
    Release notes excerpt
    Bug Fixes - codex: Crash a turn codex ended as failed instead of grading it ([#230](https://github.com/UiPath/coder_eval/pull/230), [30c62ce](https://github.com/UiPath/coder_eval/commit/30c62ce212efd1bf517332497422327cf34bf7a7))
    View on GitHub ↗

FAQ

Does it ship models or a benchmark dataset?
No. Coder Eval never supplies model access, and it only bundles an agent runtime where an extra says so (codex, antigravity) — Claude Code and OpenCode are separate CLIs you install. It is also not a fixed leaderboard: you bring the tasks and the scoring, and the repo only ships example tasks.
What does a run cost?
The software is free under Apache-2.0; the cost is whatever your agent spends calling models. The framework records every tool call, token count, and cost via telemetry so you can track it, and coder-eval plan <task>.yaml validates a task without spending any tokens.
Is it safe to run tasks that execute generated code?
Tasks execute real agent-generated code. The documentation states that the tempdir driver is not a security boundary, so untrusted tasks must run under the Docker container driver, and in GitHub Actions you should avoid pull_request_target with secrets exposed to untrusted fork PRs.
How do I switch agents or models without rewriting tasks?
Change the agent.type field (claude-code, codex, antigravity, opencode, pi), or override it for a single run with -D agent.type=opencode. Tasks, criteria, scoring, telemetry, and reports stay the same.
How does it relate to Claude Code?
Claude Code is both one of the agents you can evaluate (the default agent.type) and the authoring front end: the repo is a Claude Code plugin marketplace whose six slash commands — init, check-skill, task, lint-tasks, analyze, ci — drive the same coder-eval CLI, which you must install separately for those commands to work.
Is Coder Eval open source?
Yes. The source code is on GitHub (UiPath/coder_eval) under the Apache-2.0 licence.
Is Coder Eval still maintained?
Yes. The latest commit on GitHub is from 2026-11-19, within the last 30 days. The latest release is v0.12.18 (2026-11-09).
Who makes Coder Eval?
Coder Eval is published on GitHub by the account UiPath. FollowAgents has not verified which person or company is behind that account.
What is Coder Eval's official website?
The project's GitHub repository lists https://coder-eval.com as its website. That link is set by the repository owner; the source code itself is on GitHub (UiPath/coder_eval).
Was this page helpful?
View on GitHub ↗ Install ↓

Host ecosystems

Compare agents like this one

The same FARS review applied across the shortlist this agent qualifies for.

Related concepts