PinchBench Agent Benchmark
53 real-world tasks that measure how well an LLM actually performs as an OpenClaw coding agent.
- Source repo
- pinchbench/skill
- Stars
- ★ 1.4k
- Last updated
- 6mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 50/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Researchers and engineering teams comparing how different models handle tool calling and multi-step reasoning on genuine agent work
- Not a fit if
- Users who cannot run an OpenClaw instance
- Readers who only want leaderboard results, not their own runs
- Website
- pinchbench.com ↗ set on GitHub
- Maintenance
- No commits for 6+ monthslast commit 2026-05-22latest release v2.0.0 (2026-05-06)
- Source review
- 50/100 · Major gaps
What does this agent do, and when should you use it?
PinchBench is a benchmarking system for evaluating LLM models as the brain of an OpenClaw coding agent. This repository holds the benchmark skill and task set; the public leaderboard lives separately at pinchbench.com, and official results are curated in pinchbench/scripts/default-models.yml. The suite contains 53 tasks spread across eight categories — Productivity, Research, Writing, Coding, Analysis, Email, Memory, and Skills — covering things like calendar scheduling, stock lookups, blog writing, weather scripts, spreadsheet and PDF analysis, inbox triage, context recall, and ClawHub skill discovery. Everything is driven by ./scripts/run.sh, which requires Python 3.10+, the uv package manager, and a running OpenClaw instance. Grading is automatic, LLM-judged, or both; results land in a local results/ directory and can optionally be uploaded to the leaderboard, with a separate official key for official runs. Session transcripts are archived as JSONL files under results/{run_id}_transcripts/ for post-run analysis.
PinchBench reads task definitions under tasks/ and organizes them into real-world categories: event creation and time parsing for calendars, web search and data extraction for research, tone and formatting for writing, code generation and file operations for coding, data processing for spreadsheets and PDFs, inbox management for email, long-term recall for memory, and OpenClaw ecosystem integration for skills. The entry point is ./scripts/run.sh, which accepts --model, --judge, --suite, --runs, --timeout-multiplier, --thinking, --output-dir, --no-upload, --register, --upload, and --official-key. Model IDs must carry a provider prefix such as openrouter/ or anthropic/, with OpenRouter as the default router. Each run drives the model as an OpenClaw agent session that must call tools and chain actions to complete the task. Grading is done automatically, by an LLM judge, or both; without --judge the judge runs as an OpenClaw session, while passing --judge calls the model API directly (OpenRouter, Kilo Gateway, Anthropic, OpenAI, or a headless claude CLI) and bypasses OpenClaw personality injection. Output goes to a results/ JSON file, optionally uploaded to pinchbench.com, with per-task agent conversations saved as JSONL transcripts.
- Researchers and engineering teams comparing how different models handle tool calling and multi-step reasoning on genuine agent work
- Model vendors who want to submit results to the public leaderboard and distinguish ordinary submissions from official runs
- Developers choosing a primary model for an OpenClaw agent who need real-task evidence rather than synthetic capability scores
- Teams running targeted regressions by narrowing scope with --suite, for example only task_calendar and task_stock
- Contributors who want to extend the benchmark by submitting tasks following tasks/TASK_TEMPLATE.md
- Debuggers who replay results/{run_id}_transcripts/ JSONL files to inspect exactly what the agent did on each task
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Environment variables
OPENROUTER_API_KEYKILO_API_KEYANTHROPIC_API_KEYOPENAI_API_KEYPINCHBENCH_OFFICIAL_KEY- Runtime
- Python 3.10+
How do you install or deploy this agent?
Prerequisites: Python 3.10+, the uv package manager, and a running OpenClaw instance. Clone the repository and the scripts are ready to use; no separate package installation step is documented.
git clone https://github.com/pinchbench/skill.git
cd skillSet an API key for the provider you use. OpenRouter is the default routing provider; when you pass --judge, prepare the matching environment variable for the judge's prefix (OPENROUTER_API_KEY, KILO_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY).
How do you use this agent?
Run the full suite against a model. Model IDs must include a provider prefix:
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4Run only specific tasks, or average over multiple runs and scale timeouts for slower models:
./scripts/run.sh --model openrouter/openai/gpt-4o --suite task_calendar,task_stock
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4 --runs 3 --timeout-multiplier 2Register for an upload token so results reach the leaderboard, or keep everything local with --no-upload:
./scripts/run.sh --register
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4
./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4 --no-uploadSubmit an official run, or switch the judge to a direct API call:
export PINCHBENCH_OFFICIAL_KEY=your_official_key
./scripts/run.sh --model anthropic/claude-sonnet-4
./scripts/run.sh --model openai/gpt-4o --judge openrouter/anthropic/claude-sonnet-4-5
./scripts/run.sh --model openai/gpt-4o --judge claudeWhat are this agent's strengths and limitations?
- Tests real work instead of synthetic prompts: 53 tasks spanning calendar scheduling, email triage, file management, spreadsheet and PDF analysis across eight categories
- Flexible grading — automatic, LLM judge, or both; the judge defaults to an OpenClaw session and can instead call OpenRouter, Kilo Gateway, Anthropic, OpenAI, or a headless claude CLI directly
- Model-agnostic switch: any model ID with a provider prefix works, with OpenRouter as the default router
- Results stay local or upload to the public pinchbench.com leaderboard, with a separate official-run marking path
- Every run archives JSONL session transcripts so you can inspect the agent's behavior task by task
- A documented TASK_TEMPLATE.md and explicit task quality criteria make community contributions straightforward
- Hard dependency on a running OpenClaw instance; without it the benchmark cannot execute
- Requires third-party model API keys and incurs per-use API costs rather than running free locally
- Not the source of official leaderboard results — adding models officially means editing pinchbench/scripts/default-models.yml
- Judge comparability depends on which model and provider you choose, and the documentation offers no normalization guidance
- No packaged installer or container image is documented; setup relies on cloning and supplying Python, uv, and OpenClaw yourself
- The repository lists no topics and ships no additional guidance file, so metadata around the project is thin
How does this agent compare with similar options?
The README positions itself against benchmarks that 'test isolated capabilities' but names no specific competing product, so no named comparison is offered.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| PinchBench Agent Benchmark This agent | 50 · Major gaps | CLIFree + model costs | ★ 1.4k | 6mo ago | Python | OpenAI API · Claude API |
| Up.computer Coding Workbench | 65 · Some gaps | Desktop appFree + model costs | ★ 252 | 1d ago | TypeScript | Codex · Claude Code |
| Nightshift Coding Shifts | 62 · Some gaps | Agent plugin / skillFree + model costs | ★ 551 | 1d ago | Shell | Codex · Claude Code |
| Multica — Multi-Agent Task Collaboration Platform | 61 · Some gaps | Web appFreemium | ★ 57k | today | Go | ChatGPT · Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The README and SKILL say the benchmark runs tasks, saves full session transcripts, and can upload results to a public leaderboard. They document --no-upload and official-key options, so data flows are partly described, but default upload behavior, handling of sensitive transcript content, and concrete execution permissions are not fully explained. Least privilege, confirmation, sensitive-data handling, external effects, and rollback therefore receive partial credit. pyproject lists dependencies but provides no security review evidence. Repository and author references are traceable, while publisher identity remains unknown.
The README and SKILL give different task counts, 53 and 23, and point to different command entry points, scripts/run.sh and benchmark.py, indicating documentation inconsistency in scope or version. Python, uv, and OpenClaw prerequisites are clear, and the sample includes a lint workflow, but user-facing troubleshooting or error guidance is not shown.
The README describes a benchmark for OpenClaw coding agents and lists varied real-world tasks. The SKILL offers task selection, run counts, timeouts, and reasoning-depth options. Its trigger description is reasonably specific, but the required OpenClaw, Python, and uv environment is narrow, and unsupported scenarios are only lightly addressed.
The README, SKILL, command tables, and task template provide a clear entry structure, prerequisites, and usable examples; the MIT license is complete. The README and SKILL disagree on task lists and entry points, the version is marked 2.0.0-rc1, and the sample shows no changelog or clear maintenance responsibility. Naming stability, limitations, version history, and ownership therefore receive only partial credit.
The project packages multi-step tool use and realistic tasks into a runnable benchmark, with JSON results, per-task scores, and leaderboard submission, making outputs actionable and offering value beyond isolated capability tests. Runtime costs, API charges, and privacy implications before upload are not quantified or fully discussed, so cost-benefit earns partial credit.
The README makes traceable claims about task categories, grading, and upload options, and the SKILL shows how to inspect result files. However, the two product documents disagree on task counts and entry points, and the sample does not include benchmark implementation, graders, or result provenance for cross-checking. This static review reports only what the files state and deducts for what they leave unspecified.
- The README and SKILL disagree on task counts and run entry points; verify against the actual scripts and task files before use.
- Runs save full session transcripts and may upload results; transcripts may contain sensitive information, so review configuration and data before submission.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
PinchBench 2.0.0 🦀 A major release with significant expansion of the benchmark suite and infrastructure improvements. Highlights - 148 tasks (up from ~25 in v1.x) — nearly 6x more comprehensive benchmark coverage - Parallel judge execution — overlaps grading with task execution for faster benchmarks - Haiku judge by default — faster grading without sacrificing accuracy - Thinking-level support — benchmark models at different reasoning intensities - Axiom observability — detailed logging for benchmark runs …View on GitHub ↗ -
Release notes excerpt
Fix The model was responding correctly all along! gpt-oss-120b said 'Hello, I'm ready!' — we just weren't looking in the right place. The grader was checking for type='message' entries but OpenClaw trajectory format uses type='model.completed' with data.assistantTexts. Changes in this release: - Grader now checks type='model.completed' and data.assistantTexts - rc4-rc10 fixed the bootstrap preamble issue (BOOTSTRAP.md now correctly missing) - rc11 fixes the grader to actually find the response Full fix chain: …View on GitHub ↗ -
Release notes excerpt
Fix rc9 tried to delete workspace-state.json, but that doesn't work because shutil.rmtree already clears the workspace. By the time our delete code runs, workspace-state.json doesn't exist yet — it gets created when openclaw agent is invoked. The actual fix: CREATE a workspace-state.json with bootstrapSeededAt set BEFORE OpenClaw runs. This tells OpenClaw that bootstrap has already been seeded, so it won't create BOOTSTRAP.md. Verified: when workspace-state.json with bootstrapSeededAt exists, openclaw agents add does NOT create BOOTSTRAP.md.View on GitHub ↗
FAQ
What does running PinchBench cost?
Do I have to run it myself to see results?
What is the difference between an ordinary and an official submission?
Can I keep results off the leaderboard?
How do I add a new task?
Is PinchBench Agent Benchmark open source?
Is PinchBench Agent Benchmark safe to use?
Is PinchBench Agent Benchmark still maintained?
Who makes PinchBench Agent Benchmark?
What is PinchBench Agent Benchmark's official website?
Thanks — we use this to improve the page.