AutoResearch
Let coding agents autonomously improve nanochat training on a single NVIDIA GPU.
The README confines agent edits to train.py, emphasizes reviewable diffs, and recommends disabling all permissions, providing meaningful but unenforced least-privilege guidance. The overnight autonomous loop has no per-experiment confirmation or approval gate, so user_confirmation is 0. Downloads, code mutation, and GPU execution are disclosed at a high level, but endpoints, telemetry, network boundaries, and the complete set of external effects are not documented. No credential or sensitive-data handling policy is supplied. Dependencies are visible and PyTorch is pinned, but most packages use lower bounds; no lockfile, integrity verification, audit process, or vulnerability policy is shown, and installation includes a curl-to-shell command. Keeping or discarding experiments and reviewable diffs suggests limited reversibility, but no concrete rollback mechanism is documented. The README attributes nanochat, the author handle, and related forks, although publisher identity remains unverified by the stated registry.
README and pyproject are broadly consistent about purpose, Python requirements, uv, and the PyTorch/CUDA environment. The claim of only a few small external packages is somewhat simplified relative to the nine declared dependencies, and the core implementation files are absent. Named dependencies, constraints, and a dedicated PyTorch index support ordinary dependency resolution, but no locked resolution or fallback source is evidenced. The supplied material contains no failure messages, exception handling, or recovery behavior, so failure_messages is 0.
The documentation addresses researchers, newcomers, H100 users, and people adapting the project to smaller machines, with concrete model-scaling suggestions, though it lacks complete role-specific workflows. Capability boundaries are unusually explicit: one NVIDIA GPU, one agent-edited file, a fixed five-minute budget, one val_bpb metric, and no direct comparability across different hardware. Only one sample activation prompt is shown, while the behavior-defining program.md is not supplied, leaving trigger precision thin. Environment requirements and tested hardware are clear and alternative-platform forks are listed, but the base project lacks CPU, MPS, AMD, and automatic device support.
The README has strong information architecture across quick start, operation, structure, design choices, platform support, and forks. Installation steps are concrete, but the curl-to-shell step lacks integrity guidance and common setup failures are not covered. Core filenames are used consistently and pyproject declares version 0.1.0, although the early version and absence of compatibility commitments limit naming stability. A prompt example, manual run path, and extensive small-device tuning guide are useful, but there is no formal FAQ. Hardware, portability, and cross-machine comparability limitations are documented thoroughly. MIT is asserted only in the README; no LICENSE file or pyproject license metadata is supplied. A package version exists, but no tags, release notes, or changelog are evidenced. First-person statements and an invitation to discuss forks suggest an update contact, but there is no formal maintainer list, support policy, security contact, or maintenance commitment.
Expected outputs are an experiment log, reviewable train.py changes, and a val_bpb measurement, all useful for a research loop. Full marks are withheld because no log schema, sample run, result summary interface, or program.md is provided. Automating modification, training, comparison, and keep-or-discard decisions offers clear marginal value over manual iteration, but no supplied results establish the size or consistency of improvements. A fixed five-minute experiment budget, an estimate of roughly twelve experiments per hour, and the single-GPU requirement make time costs understandable; energy, cloud expense, failure rate, and benefit distribution are not quantified.
Major claims can be associated with README file-role descriptions and pyproject dependency configuration, but train.py, prepare.py, program.md, a lockfile, tests, and experiment logs are missing, preventing static cross-checking of the core behavior. README and pyproject provide only limited corroboration for the project name, Python requirement, and dependency environment; there are no independent results or test sources. The prose generally separates design facts, limitations, and speculation—for example, using 'hopefully,' describing broader platform support as possible in principle, and warning that results across hardware are not comparable—but throughput and autonomous-improvement claims remain unsupported by supplied artifacts.
- The agent is intended to modify and execute training code unattended; run it in an isolated workspace with restricted networking and review every diff before adoption.
- The README recommends disabling permissions, but the supplied evidence does not show technical enforcement of the train.py boundary, confirmation gates, or network restrictions.
- prepare.py downloads data, yet the supplied material does not identify exact sources, licensing, integrity checks, cache locations, or data that may be transmitted.
- Most dependencies lack exact pins or upper bounds and no lockfile is shown; lock and audit dependencies before execution, and avoid an unverified curl-to-shell installation path.
- The base project explicitly supports one NVIDIA GPU and was tested on H100; val_bpb and throughput from different hardware are not directly comparable.
- The core program.md instructions, training implementation, tests, and experiment logs are absent, so this assessment cannot verify actual safety boundaries, failure behavior, or research outcomes.
What does this agent do, and when should you use it?
AutoResearch is an autonomous experimentation framework built around a small but real LLM training workload derived from nanochat. Its three central components are prepare.py for fixed data and evaluation utilities, train.py for the agent-editable model and training loop, and program.md for human-authored agent instructions. Every experiment receives a fixed five-minute wall-clock training budget and is judged using val_bpb, where lower is better. An agent can repeatedly change the architecture, hyperparameters, optimizer, and batch size, then keep or discard each result and leave a log of the experiments. It is a self-hosted research workflow requiring local file and shell access, networked data preparation, and one NVIDIA GPU—not a hosted service or a general-purpose training platform.
The workflow begins with uv run prepare.py, which downloads training data, trains a BPE tokenizer, and provides the dataloader and evaluation utilities. Claude Code, Codex, or another coding agent then reads program.md and edits only train.py, which contains the complete GPT model, Muon + AdamW optimizer, and training loop. Each candidate is trained with uv run train.py for a fixed five-minute budget and evaluated through val_bpb. The agent keeps an improvement, discards a regression, and repeats while recording an experiment log. Humans change program.md to refine the research strategy; prepare.py is intentionally fixed.
- An ML researcher with one NVIDIA GPU who wants a coding agent to run roughly a hundred short experiments overnight.
- An agent-workflow developer comparing different program.md instructions to see which research strategy produces faster progress.
- A nanochat user searching architectures, optimizers, batch sizes, or training hyperparameters under a fixed compute-time budget.
- An individual researcher who wants reviewable experiment diffs by restricting automated edits to one train.py file.
- An engineer building a Mac, Windows, or AMD-oriented fork from a deliberately compact single-GPU training baseline.
What are this agent's strengths and limitations?
- Automated changes are concentrated in train.py, keeping model, optimizer, and training-loop diffs relatively easy to review.
- A fixed five-minute budget makes architectures, model sizes, and batch choices comparable on the same hardware.
- The vocabulary-size-independent val_bpb metric supports fairer comparisons when experiments alter the vocabulary or architecture.
- The system is intentionally compact, centered on prepare.py, train.py, and program.md without distributed training or complex configuration.
- The current implementation explicitly requires one NVIDIA GPU and has only been tested on H100; CPU, MPS, AMD, and other environments need a fork or code changes.
- Results are not directly comparable across different hardware because the fixed-time design optimizes for each machine's throughput.
- Dataset preparation and repeated training require network access, filesystem writes, and sustained GPU compute.
- The default program.md is a bare-bones baseline; stronger research policies and multi-agent organizations must be designed by the adopter.
- The source provides no estimate of operating cost, documented recovery mechanism, or guarantee that autonomous experiments will improve the model.
How do you install or deploy this agent?
The documented runtime requires Python 3.10+, uv, and one NVIDIA GPU; testing is reported on an H100. Install and prepare the project with:
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
uv run prepare.pyprepare.py downloads the dataset and trains the tokenizer; the one-time preparation is described as taking about two minutes. No API key or other credential is documented.
How do you use this agent?
First verify a manual experiment with:
uv run train.pyA run takes approximately five minutes. For autonomous mode, launch Claude Code, Codex, or another coding agent inside the repository and direct it to program.md. The provided starter prompt is:
Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.
The agent should repeatedly edit train.py, run training, inspect val_bpb, and keep or discard each change. Humans refine the research behavior by editing program.md rather than directly conducting the usual Python-editing workflow.
How does this agent compare with similar options?
Compared with the full nanochat project, AutoResearch uses a simplified single-GPU implementation and frames research as agent edits to train.py followed by fixed-time val_bpb comparisons. nanochat offers broader platform techniques such as generic device support, autodetection, and a Flash Attention 3 fallback, while AutoResearch intentionally avoids that added complexity. The README also names community forks for macOS, MLX, Windows RTX, and AMD hardware.