PinchBench Agent Benchmark

53 real-world tasks that measure how well an LLM actually performs as an OpenClaw coding agent.

Source repo
pinchbench/skill
Stars
★ 1.4k
Last updated
6mo ago
License
MIT
Primary language
Python

At a glance

How it runs
CLI
Works with
Portable with changesOpenAI API · Claude API
Cost
Free software; you pay for model usage
Setup effort
Medium · a few setup steps
You'll need
Python 3.10+uvRunning OpenClaw instanceOpenRouter API key (optional, default provider)KILO_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY (judge dependent)Shell / CLINetwork accessLocal filesystem
Typical use
Researchers and engineering teams comparing how different models handle tool calling and multi-step reasoning on genuine agent work
Not a fit if
  • Users who cannot run an OpenClaw instance
  • Readers who only want leaderboard results, not their own runs
Website
pinchbench.com ↗ set on GitHub
Maintenance
No commits for 6+ monthslast commit 2026-05-22latest release v2.0.0 (2026-05-06)

What does this agent do, and when should you use it?

PinchBench is a benchmarking system for evaluating LLM models as the brain of an OpenClaw coding agent. This repository holds the benchmark skill and task set; the public leaderboard lives separately at pinchbench.com, and official results are curated in pinchbench/scripts/default-models.yml. The suite contains 53 tasks spread across eight categories — Productivity, Research, Writing, Coding, Analysis, Email, Memory, and Skills — covering things like calendar scheduling, stock lookups, blog writing, weather scripts, spreadsheet and PDF analysis, inbox triage, context recall, and ClawHub skill discovery. Everything is driven by ./scripts/run.sh, which requires Python 3.10+, the uv package manager, and a running OpenClaw instance. Grading is automatic, LLM-judged, or both; results land in a local results/ directory and can optionally be uploaded to the leaderboard, with a separate official key for official runs. Session transcripts are archived as JSONL files under results/{run_id}_transcripts/ for post-run analysis.

PinchBench reads task definitions under tasks/ and organizes them into real-world categories: event creation and time parsing for calendars, web search and data extraction for research, tone and formatting for writing, code generation and file operations for coding, data processing for spreadsheets and PDFs, inbox management for email, long-term recall for memory, and OpenClaw ecosystem integration for skills. The entry point is ./scripts/run.sh, which accepts --model, --judge, --suite, --runs, --timeout-multiplier, --thinking, --output-dir, --no-upload, --register, --upload, and --official-key. Model IDs must carry a provider prefix such as openrouter/ or anthropic/, with OpenRouter as the default router. Each run drives the model as an OpenClaw agent session that must call tools and chain actions to complete the task. Grading is done automatically, by an LLM judge, or both; without --judge the judge runs as an OpenClaw session, while passing --judge calls the model API directly (OpenRouter, Kilo Gateway, Anthropic, OpenAI, or a headless claude CLI) and bypasses OpenClaw personality injection. Output goes to a results/ JSON file, optionally uploaded to pinchbench.com, with per-task agent conversations saved as JSONL transcripts.

  1. Researchers and engineering teams comparing how different models handle tool calling and multi-step reasoning on genuine agent work
  2. Model vendors who want to submit results to the public leaderboard and distinguish ordinary submissions from official runs
  3. Developers choosing a primary model for an OpenClaw agent who need real-task evidence rather than synthetic capability scores
  4. Teams running targeted regressions by narrowing scope with --suite, for example only task_calendar and task_stock
  5. Contributors who want to extend the benchmark by submitting tasks following tasks/TASK_TEMPLATE.md
  6. Debuggers who replay results/{run_id}_transcripts/ JSONL files to inspect exactly what the agent did on each task

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Environment variables
OPENROUTER_API_KEY KILO_API_KEY ANTHROPIC_API_KEY OPENAI_API_KEY PINCHBENCH_OFFICIAL_KEY
Runtime
Python 3.10+

How do you install or deploy this agent?

Prerequisites: Python 3.10+, the uv package manager, and a running OpenClaw instance. Clone the repository and the scripts are ready to use; no separate package installation step is documented.

git clone https://github.com/pinchbench/skill.git
cd skill

Set an API key for the provider you use. OpenRouter is the default routing provider; when you pass --judge, prepare the matching environment variable for the judge's prefix (OPENROUTER_API_KEY, KILO_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY).

How do you use this agent?

Run the full suite against a model. Model IDs must include a provider prefix:

./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4

Run only specific tasks, or average over multiple runs and scale timeouts for slower models:

./scripts/run.sh --model openrouter/openai/gpt-4o --suite task_calendar,task_stock
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4 --runs 3 --timeout-multiplier 2

Register for an upload token so results reach the leaderboard, or keep everything local with --no-upload:

./scripts/run.sh --register
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4 --no-upload

Submit an official run, or switch the judge to a direct API call:

export PINCHBENCH_OFFICIAL_KEY=your_official_key
./scripts/run.sh --model anthropic/claude-sonnet-4
./scripts/run.sh --model openai/gpt-4o --judge openrouter/anthropic/claude-sonnet-4-5
./scripts/run.sh --model openai/gpt-4o --judge claude

What are this agent's strengths and limitations?

Pros
  • Tests real work instead of synthetic prompts: 53 tasks spanning calendar scheduling, email triage, file management, spreadsheet and PDF analysis across eight categories
  • Flexible grading — automatic, LLM judge, or both; the judge defaults to an OpenClaw session and can instead call OpenRouter, Kilo Gateway, Anthropic, OpenAI, or a headless claude CLI directly
  • Model-agnostic switch: any model ID with a provider prefix works, with OpenRouter as the default router
  • Results stay local or upload to the public pinchbench.com leaderboard, with a separate official-run marking path
  • Every run archives JSONL session transcripts so you can inspect the agent's behavior task by task
  • A documented TASK_TEMPLATE.md and explicit task quality criteria make community contributions straightforward
Limitations
  • Hard dependency on a running OpenClaw instance; without it the benchmark cannot execute
  • Requires third-party model API keys and incurs per-use API costs rather than running free locally
  • Not the source of official leaderboard results — adding models officially means editing pinchbench/scripts/default-models.yml
  • Judge comparability depends on which model and provider you choose, and the documentation offers no normalization guidance
  • No packaged installer or container image is documented; setup relies on cloning and supplying Python, uv, and OpenClaw yourself
  • The repository lists no topics and ships no additional guidance file, so metadata around the project is thin

How does this agent compare with similar options?

The README positions itself against benchmarks that 'test isolated capabilities' but names no specific competing product, so no named comparison is offered.

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
PinchBench Agent Benchmark This agent 50 · Major gaps CLIFree + model costs ★ 1.4k 6mo ago Python OpenAI API · Claude API
Up.computer Coding Workbench 65 · Some gaps Desktop appFree + model costs ★ 252 1d ago TypeScript Codex · Claude Code
Nightshift Coding Shifts 62 · Some gaps Agent plugin / skillFree + model costs ★ 551 1d ago Shell Codex · Claude Code
Multica — Multi-Agent Task Collaboration Platform 61 · Some gaps Web appFreemium ★ 57k today Go ChatGPT · Codex · Claude Code

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Major gaps
50/ 100 5-point scale 2.5 / 5
Trust 12/29
Reliability 5/14
Adaptability 12/18
Convention 10/18
Effectiveness 7/13
Verifiability 4/8
Why each dimension lost points
Trust12 / 29 · 2.1/5

The README and SKILL say the benchmark runs tasks, saves full session transcripts, and can upload results to a public leaderboard. They document --no-upload and official-key options, so data flows are partly described, but default upload behavior, handling of sensitive transcript content, and concrete execution permissions are not fully explained. Least privilege, confirmation, sensitive-data handling, external effects, and rollback therefore receive partial credit. pyproject lists dependencies but provides no security review evidence. Repository and author references are traceable, while publisher identity remains unknown.

Reliability5 / 14 · 1.8/5

The README and SKILL give different task counts, 53 and 23, and point to different command entry points, scripts/run.sh and benchmark.py, indicating documentation inconsistency in scope or version. Python, uv, and OpenClaw prerequisites are clear, and the sample includes a lint workflow, but user-facing troubleshooting or error guidance is not shown.

Adaptability12 / 18 · 3.3/5

The README describes a benchmark for OpenClaw coding agents and lists varied real-world tasks. The SKILL offers task selection, run counts, timeouts, and reasoning-depth options. Its trigger description is reasonably specific, but the required OpenClaw, Python, and uv environment is narrow, and unsupported scenarios are only lightly addressed.

Convention10 / 18 · 2.8/5

The README, SKILL, command tables, and task template provide a clear entry structure, prerequisites, and usable examples; the MIT license is complete. The README and SKILL disagree on task lists and entry points, the version is marked 2.0.0-rc1, and the sample shows no changelog or clear maintenance responsibility. Naming stability, limitations, version history, and ownership therefore receive only partial credit.

Effectiveness7 / 13 · 2.7/5

The project packages multi-step tool use and realistic tasks into a runnable benchmark, with JSON results, per-task scores, and leaderboard submission, making outputs actionable and offering value beyond isolated capability tests. Runtime costs, API charges, and privacy implications before upload are not quantified or fully discussed, so cost-benefit earns partial credit.

Verifiability4 / 8 · 2.5/5

The README makes traceable claims about task categories, grading, and upload options, and the SKILL shows how to inspect result files. However, the two product documents disagree on task counts and entry points, and the sample does not include benchmark implementation, graders, or result provenance for cross-checking. This static review reports only what the files state and deducts for what they leave unspecified.

Risks and how to mitigate them
  • The README and SKILL disagree on task counts and run entry points; verify against the actual scripts and task files before use.
  • Runs save full session transcripts and may upload results; transcripts may contain sensitive information, so review configuration and data before submission.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision 819384ae8304
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/skill.svg)](https://followagents.com/en/agents/skill)
<a href="https://followagents.com/en/agents/skill"><img src="https://followagents.com/badge/skill.svg" alt="FARS review"></a>

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

  1. v2.0.0
    Release notes excerpt
    PinchBench 2.0.0 🦀 A major release with significant expansion of the benchmark suite and infrastructure improvements. Highlights - 148 tasks (up from ~25 in v1.x) — nearly 6x more comprehensive benchmark coverage - Parallel judge execution — overlaps grading with task execution for faster benchmarks - Haiku judge by default — faster grading without sacrificing accuracy - Thinking-level support — benchmark models at different reasoning intensities - Axiom observability — detailed logging for benchmark runs …
    View on GitHub ↗
  2. v2.0.0-rc11
    Release notes excerpt
    Fix The model was responding correctly all along! gpt-oss-120b said 'Hello, I'm ready!' — we just weren't looking in the right place. The grader was checking for type='message' entries but OpenClaw trajectory format uses type='model.completed' with data.assistantTexts. Changes in this release: - Grader now checks type='model.completed' and data.assistantTexts - rc4-rc10 fixed the bootstrap preamble issue (BOOTSTRAP.md now correctly missing) - rc11 fixes the grader to actually find the response Full fix chain: …
    View on GitHub ↗
  3. v2.0.0-rc10
    Release notes excerpt
    Fix rc9 tried to delete workspace-state.json, but that doesn't work because shutil.rmtree already clears the workspace. By the time our delete code runs, workspace-state.json doesn't exist yet — it gets created when openclaw agent is invoked. The actual fix: CREATE a workspace-state.json with bootstrapSeededAt set BEFORE OpenClaw runs. This tells OpenClaw that bootstrap has already been seeded, so it won't create BOOTSTRAP.md. Verified: when workspace-state.json with bootstrapSeededAt exists, openclaw agents add does NOT create BOOTSTRAP.md.
    View on GitHub ↗

FAQ

What does running PinchBench cost?
The repository is MIT-licensed and free, but benchmarking calls third-party models, so you pay per-use API rates for the tested model and, if you pick one, for a separately invoked judge model.
Do I have to run it myself to see results?
No — the public leaderboard at pinchbench.com is browsable. Running locally is for validating your own model or a specific subset of tasks; the README notes this repository is not the source of official leaderboard results.
What is the difference between an ordinary and an official submission?
After ./scripts/run.sh --register gives you a token, results upload automatically as ordinary submissions. Adding --official-key or setting PINCHBENCH_OFFICIAL_KEY marks the run as official.
Can I keep results off the leaderboard?
Yes, pass --no-upload. The run still writes its results JSON and the JSONL session transcripts under results/{run_id}_transcripts/ to disk.
How do I add a new task?
Follow the format in tasks/TASK_TEMPLATE.md; good tasks are real-world, measurable, reproducible, and challenging, and are contributed through the repository's issue tracker or pull requests.
Is PinchBench Agent Benchmark open source?
Yes. The source code is on GitHub (pinchbench/skill) under the MIT licence.
Is PinchBench Agent Benchmark safe to use?
FollowAgents' FARS review (Nov 08, 2026) scored PinchBench Agent Benchmark 12/29 on trust and safety. None of the eight safety controls the review checks for was missing outright.
Is PinchBench Agent Benchmark still maintained?
The GitHub repository has had no commits since 2026-05-22, more than six months ago. That can mean the project is finished or abandoned; check its issues before relying on it. The latest release is v2.0.0 (2026-05-06).
Who makes PinchBench Agent Benchmark?
PinchBench Agent Benchmark is published on GitHub by the account pinchbench. FollowAgents has not verified which person or company is behind that account.
What is PinchBench Agent Benchmark's official website?
The project's GitHub repository lists https://pinchbench.com as its website. That link is set by the repository owner; the source code itself is on GitHub (pinchbench/skill).
Was this page helpful?
View on GitHub ↗ Install ↓

Host ecosystems

Compare agents like this one

The same FARS review applied across the shortlist this agent qualifies for.

Related concepts