SciPIvot Research Judgment Benchmark
Evaluate how research agents choose the next scientific step using real project records.
- Source repo
- AutoResearch-Factory/SciPIvot
- Stars
- ★ 14
- Last updated
- 20d ago
- License
- CC-BY-4.0
- Primary language
- Python
- FA score
- 51/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesCodex · Claude Code (Partial support)
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A research-agent team wants to see whether a model catches flawed next-step plans using records from real projects.
- Not a fit if
- Teams seeking general literature search or autonomous experiment execution
- Evaluation teams unable to access the Hugging Face dataset
- Source review
- 51/100 · Major gaps
What does this agent do, and when should you use it?
SciPIvot is a 248-case benchmark for judging scientific decisions at pivotal moments in research projects. Each case contrasts a research agent’s original plan with an alternative reconstructed from a senior researcher’s correction; evaluators use the project record from before that correction to choose the plan that better advances the scientific question. The repository includes the question set, an index of 230 project records, evaluation prompts, results, and Python scripts for prompt generation and scoring. The project condition requires records downloaded from Hugging Face, and the README provides a runner for Claude Code or Codex inside a bubblewrap sandbox. SciPIvot is suited to evaluating research agents and reproducing benchmark runs, rather than carrying out an end-to-end research program.
Use scripts/make_prompt.py to render a case as a project or options-only prompt and select the option order; the project condition also needs that case’s project record. Give the prompt to a model and collect JSON containing argument and response, then score it with scripts/score.py against the reference answer. Alternatively, scripts/run_project.py invokes Claude Code or Codex in a sandbox, runs both option orders, and saves outputs and transcripts. scripts/fetch_records.py downloads, verifies, and unpacks records from Hugging Face.
- A research-agent team wants to see whether a model catches flawed next-step plans using records from real projects.
- A benchmark researcher needs to compare judgment with project context against a plans-only evaluation.
- An evaluation team wants to reproduce the documented Claude Code or Codex runs at medium reasoning effort and score the responses.
- A researcher wants to inspect model preference for human-corrected plans or review the published results across evaluation conditions.
How do you install or deploy this agent?
The repository provides Python scripts rather than a packaged application. For the project condition, the documented setup requires Python 3.12+, huggingface_hub, and zstandard, plus acceptance of the Hugging Face dataset access terms. Running run_project.py additionally requires Linux, bubblewrap, and either the Claude Code or Codex CLI on PATH.
How do you use this agent?
Create an environment and install the record-download dependencies:
python3 -m venv .venv && . .venv/bin/activate
pip install huggingface_hub zstandard
hf auth loginFetch one case’s record and render a project-condition prompt:
python scripts/fetch_records.py --id case_000 --out records
python scripts/make_prompt.py --id case_000 --condition project --order abAfter asking a model to answer, save its response in predictions.json and score it:
python scripts/score.py predictions.jsonTo run Claude Code through the documented runner, set its API key and invoke:
export ANTHROPIC_API_KEY=...
python scripts/run_project.py --id case_000 --condition project --engine claude --model claude-opus-5 --effort medium --records records --out runs/opus.jsonl
python scripts/score.py runs/opus.jsonlFor Codex, use OPENAI_API_KEY and --engine codex.
What are this agent's strengths and limitations?
- The 248 cases derive from human corrections made during six months of frontier research across 26 PhD-level projects and six scientific fields.
- The benchmark can include the pre-correction project record, and the repository provides record checks, exact prompts, scoring scripts, and aggregate results.
- Evaluation covers project and options-only conditions, five effort settings, and two reconsideration stages; the README documents the protocol.
- The project archives total about 21 GB and unpack to 91 GB; access to the Hugging Face dataset requires accepting its terms.
- Reproducing scripted runs depends on Linux, bubblewrap, a Claude Code or Codex CLI, and the corresponding API key; the model session needs network access to its API.
- The task measures choosing between two candidate plans against a reference; it does not establish that an agent can conduct research autonomously end to end.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| SciPIvot Research Judgment Benchmark This agent | 51 · Major gaps | CLI | ★ 14 | 20d ago | Python | — |
| XScientist | 80 · Good | CLIFree + model costs | ★ 127 | 7d ago | Python | OpenAI API · Claude API |
| Sotopia | 46 · Major gaps | Library / SDKFree + model costs | ★ 342 | 10mo ago | Python | OpenAI API |
| Autonomous Agents Research Papers | 24 · Major gaps | Library / SDKFree | ★ 1.4k | 4mo ago | — | — |
How does FollowAgents rate this agent?
Why each dimension lost points
README documents a concrete sandbox: bubblewrap isolation, read-only /project mount, empty scratch HOME, Claude Code pre-approving Read/Grep/Glob and denying Bash/Write/Edit/WebFetch, and Codex running danger-full-access only inside bubblewrap with network kept up for API access. Least privilege has real evidence but Codex full-access plus open network weaken it, so 2. User confirmation: run_project.py spends API keys and runs models with no interactive confirmation or cost prompt, so 1. Data-flow transparency: README explains HF sourcing, cache location, 91GB unpack size and --remove-after-verify behavior, so 2. Sensitive data: question set and prompts were scrubbed with identifier substitutions in 19 cases and digests of originals preserved, so 2. Dependency security: only huggingface_hub, zstandard, bubblewrap named, no lockfile or hash pinning, so 1. External effects: scripts only download, unpack and score, no destructive defaults, so 2. Rollback: --remove-after-verify keeps verified trees, failed sessions are dropped and rerun, runs resume, so 2. Source attribution: arXiv, HF dataset, BibTeX and author contacts provided, so 2.
Self-consistency: 248 cases, 230 records, 12 conditions and scoring rules (order averaging, unparsed counted incorrect) are described consistently, so 2. Dependency availability: no requirements/lock file, only inline pip commands; Python 3.12+ and bubblewrap stated but unpinned, so 1. Failure messages: only a harmless Codex 'failed to install system skills' line and score.py refusing mixed files are mentioned; no systematic error-handling documentation, so 1.
Audience and scenarios: aimed at benchmark researchers with project and options-only conditions and multiple engines, so 2. Capability boundaries: clearly states records exclude .git, external links are not mounted, run_project needs Linux+bubblewrap and only claude/codex, so 2. Trigger precision: this is a benchmark rather than a trigger-based agent; script flags are documented but no input-validation evidence, so 1. Environment fit: hard dependence on Linux, bubblewrap, specific CLIs and API keys limits portability, so 1.
Information architecture: README has path table, task description, quickstart, protocol, results table and data schema, so 2. Install notes: venv, pip, hf auth and bubblewrap steps given, so 2. Naming stability: anonymized case ids in uniform format, consistent A/B and ab/ba ordering, so 2. Examples and FAQ: predictions.json and command examples exist but no FAQ or troubleshooting, so 1. Known limitations: mentions no history, missing external links and below-chance models, but no systematic benchmark limitations (e.g., construction bias), so 1. License: full CC-BY-4.0 text matching README, so 2. Versioning/changelog: no CHANGELOG or version number, so 0. Maintenance responsibility: corresponding-author emails listed but no governance, issue process or update commitment, and publisher identity is unverified, so 1.
Output usability: make_prompt.py, score.py, run_project.py and a structured JSON output format make it directly usable for evaluation, so 2. Marginal value: a 248-case human-correction benchmark of PI judgment fills a gap, so 2. Cost benefit: 91GB unpack, 21GB cache, paid APIs and many conditions are expensive with no lightweight path offered, so 1.
Claim traceability: README cites arXiv, HF dataset, results/ and prompts/sources.json digests, so 2. Cross-source corroboration: only README and LICENSE are in the packet; results and prompts files are not present to corroborate numbers, so 1. Fact/inference separation: README separates paper results from repository contents, but some claims (e.g., four of five models below chance) lack in-packet data, so 1.
- Publisher identity is unverified; maintenance responsibility and update path are unclear, so verify independently before use.
- The Codex engine runs danger-full-access inside the sandbox with network kept up; isolation strength depends on correct bubblewrap configuration.
- No dependency lockfile; huggingface_hub and zstandard versions are unpinned, creating reproducibility and supply-chain risk.
- Full evaluation needs ~91GB disk and paid APIs; validate the pipeline on a few cases first.
- Only README and LICENSE are in the packet; results tables and prompt files are absent, so headline numbers cannot be cross-checked in-packet.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/scipivot)<a href="https://followagents.com/en/agents/scipivot"><img src="https://followagents.com/badge/scipivot.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v1.0.0 SciPIvot v1.0.0 — 248 cases, prompts, results, and project-record access
Release notes excerpt
First complete release of SciPIvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents. - Question set — 248 cases (data/questions.json, plain text twin): both plans in exact wording, reference answers, and a snapshot pointer to the frozen project record each case is asked against - Project records — 230 frozen records (26 projects, 21 GB compressed) on [Hugging Face](https://huggingface.co/datasets/sun1245/SciPIvot), byte-for-byte as mounted for the paper's models; scripts/fetch_records.py downloads and verifies them …View on GitHub ↗
FAQ
Do I need to download the project records?
Can the scripts run the model for me?
scripts/run_project.py supports Claude Code or Codex, with the relevant CLI, API key, and documented sandbox setup.How does scoring account for option order?
ab and ba order. When both are present, scores are averaged within each case before cases are weighted equally.Is this a research agent that conducts studies autonomously?
Thanks — we use this to improve the page.