Arbor Research Optimizer
A hypothesis-tree agent for running, validating, and retaining metric-driven improvements.
- Source repo
- RUC-NLPIR/Arbor
- Stars
- ★ 1.1k
- Last updated
- 16d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 55/100 · Major gaps
At a glance
- Works with
- Universal · cross-platformCodex · Claude Code · OpenAI API · Claude API
- You'll need
- Typical use
- An ML research engineer with a runnable training benchmark wants to test optimizer or architecture hypotheses under a fixed cycle budget while protecting a held-out test split.
- Main limitation
- Native research runs require provider/model configuration and credentials; the keyless route depends on a host coding agent such as Codex or Claude Code for reasoning.
- Source review
- 55/100 · Major gaps
What does this agent do, and when should you use it?
Arbor is an autonomous research runtime for tasks that have a measurable target and an executable evaluation path. Its Coordinator manages an Idea Tree and research decisions, while independent Executors implement one hypothesis at a time in isolated Git worktrees. The Python CLI covers intake, configuration, dashboards, checkpoints, reports, replay, export, and read-only session monitoring, with run artifacts stored under .arbor/sessions/. It can use OpenAI, Anthropic, or OpenAI-compatible providers in its native runtime, or operate inside Codex and Claude Code through skills and optional deterministic MCP tools. It is best suited to benchmark or repository optimization work with a clean Git baseline and dev/held-out evaluation discipline, rather than tasks lacking a defined metric.
The arbor command begins with an intake conversation that turns the objective, target directory, metric, baseline, budget, dev/test rules, and artifact paths into an Arbor Research Contract. The Coordinator executes the Observe, Ideate, Select, Dispatch, Backpropagate, and Decide cycle over the Idea Tree, then dispatches chosen hypotheses to Executors. An Executor changes code and runs the task in its own branch and Git worktree, iterating on the dev signal and reporting evidence; a change is retained only after held-out test validation and the configured threshold are satisfied. arbor replay replays a session timeline, arbor report renders REPORT.md, arbor export produces self-contained HTML or JSONL, and arbor web serves a read-only monitor. arbor idea-check can query the built-in alphaXiv backend for a concise related-work and novelty assessment.
- An ML research engineer with a runnable training benchmark wants to test optimizer or architecture hypotheses under a fixed cycle budget while protecting a held-out test split.
- A developer maintaining an executable coding benchmark needs candidate changes evaluated in separate Git worktrees before any run result is promoted toward main.
- A team improving a retrieval, browsing, or terminal-evaluation harness wants a durable record of failed directions, scores, and distilled lessons rather than an unstructured series of attempts.
- A Codex or Claude Code subscriber without a separate model API key wants the host coding model to run an Arbor-style research loop backed by skills and optional MCP tools.
- A researcher considering a new technical branch wants to run arbor idea-check before allocating compute, using the documented alphaXiv-based related-work check.
How do you install or deploy this agent?
Requirements are Python 3.10+ and Git. Install the native CLI with:
pip install arbor-agent
arbor doctorThen run arbor quickstart to set up a free key or local Ollama, or use arbor setup to configure provider, model, base_url, and API key. Configuration is written to ~/.arbor/config.yaml. For a keyless demo, run arbor replay --demo. To install the skill suite for Codex or Claude Code, run pip install arbor-agent followed by arbor install. Optional MCP support requires pip install "arbor-agent[mcp]" and registration of arbor mcp with an MCP-capable host.
How do you use this agent?
From a clean Git repository containing a runnable evaluation script and evaluation data, preferably with dev and held-out test splits, run:
arbor "improve validation score without touching the test split" --cwd ./benchmarkConfirm the resolved research contract before experiments launch. For repeatable settings, place task, coordinator.max_cycles, coordinator.max_depth, merge_threshold, and executor.max_turns in research_config.yaml, then run:
arbor --cwd ./benchmark --config research_config.yamlResume an interrupted run with arbor --resume --run-name <run_name>. When satisfied, promote its separate trunk with git merge research/run_xxx/trunk.
What are this agent's strengths and limitations?
- The Coordinator/Executor split is paired with an Idea Tree that preserves results, failure modes, and propagated insights across research cycles.
- Each experiment runs in an isolated Git worktree, develops against the dev signal, and is retained only after held-out validation clears a configurable merge threshold.
- The repository documents three delivery modes: a native CLI, a standalone skill suite, and optional MCP tools, with native support for OpenAI, Anthropic, and OpenAI-compatible backends.
- Sessions leave inspectable, resumable artifacts including REPORT.md, events.jsonl, run_stats.json, the Idea Tree, and per-experiment outputs; HTML and JSONL export are available.
- Native research runs require provider/model configuration and credentials; the keyless route depends on a host coding agent such as Codex or Claude Code for reasoning.
- Adoption presumes a clean Git repository and a runnable evaluation setup, ideally with dev and held-out test data; projects without them must establish that infrastructure first.
- Built-in alphaXiv search requires Python 3.12+; on Python 3.10 or 3.11 the documented backend degrades, and in-run network search is disabled by default.
- Validated changes are merged into a per-run trunk rather than main, so final promotion remains an explicit user-managed Git decision.
How does this agent compare with similar options?
The repository reports higher held-out scores for Arbor than its listed Claude Code and Codex baselines on six tasks. For example, it reports 67.67 on BrowseComp versus 53.33 for Claude Code and 50.00 for Codex, and 77.36 on Terminal-Bench 2.0 versus 71.70 and 73.59. These are repository-reported benchmark results, not evidence of a general performance guarantee for every project.
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| Arbor Research Optimizer This agent | 55 · Major gaps | ★ 1.1k | 16d ago | Python | Codex · Claude Code · OpenAI API · Claude API |
| Happier | 82 · Good | ★ 1.7k | today | TypeScript | Codex · Claude Code |
| Bernstein | 78 · Good | ★ 1.3k | today | Python | Codex · Claude Code · OpenAI API |
| Loop Engineering | 69 · Some gaps | ★ 11k | today | TypeScript | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: README describes multiple interaction modes (auto/direction/review/collaborative), with review mode pausing before nodes and executors, indicating user confirmation; experiments run in isolated git worktrees, main branch untouched, providing rollback; but evidence for least privilege, data flow transparency, sensitive data handling, dependency security, external effects, and source attribution is insufficient. Deductions: no explicit tool permission scopes, data flow paths, sensitive data protection measures, dependency vulnerability scanning, clear control over external network requests (except literature search), or third-party code source verification.
Evidence shows: README and code structure are consistent, describing a clear six-step cycle and Idea Tree mechanism; dependencies are listed, and alphaxiv-py degrades gracefully on Python 3.10/3.11; tests cover failure scenarios (e.g., search failure, missing package). Deductions: no specific runtime error message examples, and dependency availability not actually verified.
Evidence shows: README describes multiple usage scenarios (CLI, skill suite, MCP), provides configuration options and plugin mechanism; capability boundaries are indicated through interaction modes and permission settings (e.g., review mode); trigger precision via explicit commands and config (e.g., --max-cycles); environment fit via support for multiple model backends and platforms (e.g., Claude Code, Codex). Deductions: capability boundaries not detailed, no explicit list of limitations.
Evidence shows: README is well-structured, including install, config, CLI reference, examples, and docs links; install notes and example config provided; naming stable (arbor command and package name consistent); example task and FAQ links provided; license is Apache-2.0; version auto-generated via setuptools-scm. Deductions: known limitations not explicitly listed, no changelog, maintenance responsibility not clearly stated.
Evidence shows: README provides output examples (e.g., reports, dashboard), claims superiority on multiple benchmarks; marginal value via automated research process and hypothesis tree; cost-benefit via configurable budget and interaction modes. Deductions: output usability lacks specific format examples, marginal value and cost-benefit claims lack independent verification.
Evidence shows: README cites arXiv paper and project page, provides benchmark result tables; tests verify some functionality (e.g., alphaXiv backend). Deductions: benchmark results lack reproduction details, facts and inferences not clearly separated, cross-source verification insufficient.
- This agent has autonomous capabilities to run experiments and modify code; use in controlled environments and ensure user confirmation mechanisms (e.g., review mode) are enabled.
- Dependencies are not vulnerability-scanned; check dependency security before deployment.
- Benchmark performance claims (e.g., 2.5× improvement) lack independent reproduction; treat with caution.