Arbor Research Optimizer
A hypothesis-tree agent for running, validating, and retaining metric-driven improvements.
- Source repo
- RUC-NLPIR/Arbor
- Stars
- ★ 1.1k
- Last updated
- 5d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 63/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformCodex · Claude Code · OpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- An ML research engineer with a runnable training benchmark wants to test optimizer or architecture hypotheses under a fixed cycle budget while protecting a held-out test split.
- Not a fit if
- Projects without a runnable eval script and a clean Git baseline
- Users on Python 3.10/3.11 who need the built-in alphaXiv literature search
- Source review
- 63/100 · Some gaps
What does this agent do, and when should you use it?
Arbor is an autonomous research runtime for tasks that have a measurable target and an executable evaluation path. Its Coordinator manages an Idea Tree and research decisions, while independent Executors implement one hypothesis at a time in isolated Git worktrees. The Python CLI covers intake, configuration, dashboards, checkpoints, reports, replay, export, and read-only session monitoring, with run artifacts stored under .arbor/sessions/. It can use OpenAI, Anthropic, or OpenAI-compatible providers in its native runtime, or operate inside Codex and Claude Code through skills and optional deterministic MCP tools. It is best suited to benchmark or repository optimization work with a clean Git baseline and dev/held-out evaluation discipline, rather than tasks lacking a defined metric.
The arbor command begins with an intake conversation that turns the objective, target directory, metric, baseline, budget, dev/test rules, and artifact paths into an Arbor Research Contract. The Coordinator executes the Observe, Ideate, Select, Dispatch, Backpropagate, and Decide cycle over the Idea Tree, then dispatches chosen hypotheses to Executors. An Executor changes code and runs the task in its own branch and Git worktree, iterating on the dev signal and reporting evidence; a change is retained only after held-out test validation and the configured threshold are satisfied. arbor replay replays a session timeline, arbor report renders REPORT.md, arbor export produces self-contained HTML or JSONL, and arbor web serves a read-only monitor. arbor idea-check can query the built-in alphaXiv backend for a concise related-work and novelty assessment.
- An ML research engineer with a runnable training benchmark wants to test optimizer or architecture hypotheses under a fixed cycle budget while protecting a held-out test split.
- A developer maintaining an executable coding benchmark needs candidate changes evaluated in separate Git worktrees before any run result is promoted toward main.
- A team improving a retrieval, browsing, or terminal-evaluation harness wants a durable record of failed directions, scores, and distilled lessons rather than an unstructured series of attempts.
- A Codex or Claude Code subscriber without a separate model API key wants the host coding model to run an Arbor-style research loop backed by skills and optional MCP tools.
- A researcher considering a new technical branch wants to run arbor idea-check before allocating compute, using the documented alphaXiv-based related-work check.
How do you install or deploy this agent?
Requirements are Python 3.10+ and Git. Install the native CLI with:
pip install arbor-agent
arbor doctorThen run arbor quickstart to set up a free key or local Ollama, or use arbor setup to configure provider, model, base_url, and API key. Configuration is written to ~/.arbor/config.yaml. For a keyless demo, run arbor replay --demo. To install the skill suite for Codex or Claude Code, run pip install arbor-agent followed by arbor install. Optional MCP support requires pip install "arbor-agent[mcp]" and registration of arbor mcp with an MCP-capable host.
How do you use this agent?
From a clean Git repository containing a runnable evaluation script and evaluation data, preferably with dev and held-out test splits, run:
arbor "improve validation score without touching the test split" --cwd ./benchmarkConfirm the resolved research contract before experiments launch. For repeatable settings, place task, coordinator.max_cycles, coordinator.max_depth, merge_threshold, and executor.max_turns in research_config.yaml, then run:
arbor --cwd ./benchmark --config research_config.yamlResume an interrupted run with arbor --resume --run-name <run_name>. When satisfied, promote its separate trunk with git merge research/run_xxx/trunk.
What are this agent's strengths and limitations?
- The Coordinator/Executor split is paired with an Idea Tree that preserves results, failure modes, and propagated insights across research cycles.
- Each experiment runs in an isolated Git worktree, develops against the dev signal, and is retained only after held-out validation clears a configurable merge threshold.
- The repository documents three delivery modes: a native CLI, a standalone skill suite, and optional MCP tools, with native support for OpenAI, Anthropic, and OpenAI-compatible backends.
- Sessions leave inspectable, resumable artifacts including REPORT.md, events.jsonl, run_stats.json, the Idea Tree, and per-experiment outputs; HTML and JSONL export are available.
- Native research runs require provider/model configuration and credentials; the keyless route depends on a host coding agent such as Codex or Claude Code for reasoning.
- Adoption presumes a clean Git repository and a runnable evaluation setup, ideally with dev and held-out test data; projects without them must establish that infrastructure first.
- Built-in alphaXiv search requires Python 3.12+; on Python 3.10 or 3.11 the documented backend degrades, and in-run network search is disabled by default.
- Validated changes are merged into a per-run trunk rather than main, so final promotion remains an explicit user-managed Git decision.
How does this agent compare with similar options?
The repository reports higher held-out scores for Arbor than its listed Claude Code and Codex baselines on six tasks. For example, it reports 67.67 on BrowseComp versus 53.33 for Claude Code and 50.00 for Codex, and 77.36 on Terminal-Bench 2.0 versus 71.70 and 73.59. These are repository-reported benchmark results, not evidence of a general performance guarantee for every project.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Arbor Research Optimizer This agent | 63 · Some gaps | CLIFree + model costs | ★ 1.1k | 5d ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| wmux Agent Workspace | 75 · Good | Desktop app | ★ 447 | today | TypeScript | Codex · Claude Code |
| Agent Manager | 75 · Good | CLIFree | ★ 646 | today | Go | Codex · Claude Code |
| no_human | 75 · Some gaps | CLIFree + model costs | ★ 320 | 1mo ago | Python | Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The docs describe review modes, permission and run-mode clarification, budgets, environment-based keys, redacted config snapshots, worktrees, protected paths, and a read-only monitor, supporting meaningful confirmation, data-flow visibility, and recovery. Deductions: the Executor can modify code and run experiments, and auto mode permits autonomous activity; entry-point code also edits .gitignore, with the Coordinator attempting to commit that change. Least privilege and external effects are therefore only partly addressed. Dependencies have lower-bound versions, but the supplied material shows no security audit or vulnerability mitigation. Author and repository attribution are stated, but publisher identity is unverified and provenance evidence is limited.
The materials specify workflow stages, tree states, split discipline, retries, and some failure handling; CI configuration and tests cover selected behavior, including missing search packages and access failures. Deduction: README performance and win claims lack matching independent result evidence in this packet. Some behavior is documented but not visible in implementation, and no execution was performed, so end-to-end reliability is not established.
The docs cover native CLI, skill suite, MCP, multiple model providers, benchmark work, and CI, with interaction modes, plugins, and configuration paths. Deductions: boundaries are partly specified by skill responsibilities and evaluation rules, but general tasks still depend on a runnable evaluator and clear metric from the user; platform support has limits, including Windows being omitted from CI.
The repository provides a structured README, installation, quickstart, and configuration docs, English and Chinese installation and quickstart guides, substantial examples, an Apache-2.0 LICENSE, and a tag-derived versioning convention. Deductions: news and version changes are recorded, but the supplied material does not show a complete changelog; author and repository links do not establish a support commitment or maintenance process.
The hypothesis tree, isolated experiments, dev/held-out evaluation discipline, reports, and optional human review provide useful structure for iterative research; cycle and time budgets are configurable. Deductions: claimed benefits rely largely on README benchmark and example statements that this packet cannot verify; model, compute, and benchmark preparation costs make value task-dependent.
Requirements, commands, configuration fields, evaluation metadata, and several tool behaviors are described concretely; test files corroborate selected search behavior. Deductions: the packet does not provide enough corresponding tests or result artifacts for the main performance claims or the overall research capability, and it does not always clearly distinguish illustrative figures and claims from verified facts.
- Autonomous runs may modify the target repository and start lengthy or billable experiments. Review the run mode, budget, evaluation commands, and protected paths first. The entry-point code may update .gitignore, and the Coordinator may attempt to commit that change.
- Performance improvements in the README are repository claims and were not independently verified in this static review.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/arbor)<a href="https://followagents.com/en/agents/arbor"><img src="https://followagents.com/badge/arbor.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
<!-- Release notes generated using configuration in .github/release.yml at main --> What's Changed Other Changes * docs: refresh README first screen + add roadmap Direction 4 (adoption/DX/community) by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/38 * docs(roadmap): rewrite around open problems and preliminary ideas by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/39 * fix(llm): make Responses API cache hits visible; default example to auto by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/40 …View on GitHub ↗ -
Release notes excerpt
<!-- Release notes generated using configuration in .github/release.yml at main --> What's Changed Other Changes * Harden MCP tool path boundaries + web monitor headers by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/21 * feat(search): zero-config alphaXiv backend + idea-check novelty review by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/22 * feat(search): grounded ideation + pluggable backends + keyless visit by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/23 …View on GitHub ↗ -
Release notes excerpt
<!-- Release notes generated using configuration in .github/release.yml at main --> What's Changed Other Changes * feat(cli): export session artifacts (HTML / JSONL) by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/9 * feat(examples): add AlgoTune-style k-NN speedup benchmark by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/10 * docs(readme): introduce the runnable algotune_knn example task by @ignorejjj in https://github.com/RUC-NLPIR/Arbor/pull/11 …View on GitHub ↗
FAQ
Does every Arbor run need an API key?
Will it modify my main branch directly?
How does it guard against tuning only to development data?
Can I try it before configuring a model?
Does literature search contact the network by default?
Thanks — we use this to improve the page.