Data & Analysis deep-learningexperiment-automationpytorchgpu-monitoringhyperparameter-tuningslurmresearch-automation

Deep Researcher Agent

An autonomous operations loop for planning, running, monitoring, and iterating deep-learning experiments on GPUs.

FollowAgents review · FARS-2.0
Not yet reviewed
See the full review method →

What does this agent do, and when should you use it?

Deep Researcher Agent is a 24/7 automation framework for deep-learning experiments, not a general-purpose coding assistant. Its core combines the THINK→EXECUTE→REFLECT loop in core.loop, Leader-Worker dispatch, zero-LLM monitoring in core.monitor, and a two-tier memory design. A project is driven by PROJECT_BRIEF.md and records operational state in MEMORY_LOG.md, state.json, experiments.jsonl, DEAD_ENDS.md, and INSIGHTS.md. Experiments can run locally, on one SSH-connected host, or through Slurm; the controller remains local in the remote modes. It ships as Claude Code slash commands and Codex local skills, with Anthropic-compatible, OpenAI-compatible, and limited CLI subscription provider modes.

On each cycle, the framework reads PROJECT_BRIEF.md, HUMAN_DIRECTIVE.md, MEMORY_LOG.md, and the experiment ledger, then has the Leader plan a next experiment and optionally dispatch an Idea, Code, or Writing worker. The Code worker can read and modify workspace code and configuration, perform a dry-run, and call launch_experiment to start training. During training, core.monitor polls process state, logs, and nvidia-smi without LLM calls. After a run ends, REFLECT parses logs and metrics, appends the hypothesis, metrics, and outcome to workspace/experiments.jsonl, and updates state, memory, and journals; the Slurm backend obtains terminal states such as FAILED, TIMEOUT, and CANCELLED from sacct. Supporting commands include /experiment-status, /gpu-monitor, /progress-report, /obsidian-sync, /daily-papers, /paper-analyze, and /conf-search.

  1. A researcher with an existing PyTorch project who wants bounded, repeated ablations under explicit GPU, epoch, and search-space constraints.
  2. An ML engineer tuning optimizers, learning rates, or augmentation from the outcomes and logs of prior training runs.
  3. A user with one remote GPU host who wants the controller on a laptop while code operations, training, logging, and GPU checks run over SSH.
  4. A researcher on a Slurm GPU cluster who needs sbatch submission and sacct-based reporting of actual job completion states.
  5. A project lead who needs persistent records of hypotheses, metrics, failed directions, and durable experiment observations.
  6. A Claude Code or Codex user who wants local commands or skills for launching loops, checking status, and reporting progress.

What are this agent's strengths and limitations?

Pros
  • Monitoring is explicitly designed to avoid LLM calls while training, using process, log, and GPU polling instead.
  • Local, SSH, and Slurm backends cover workstation GPUs, a remote host, and cluster submission; Slurm records terminal outcomes through sacct.
  • The experiments.jsonl ledger plus DEAD_ENDS.md and INSIGHTS.md preserve structured results and failed directions across cycles.
  • Bounded two-tier memory and one-worker dispatch are concrete controls on context growth and concurrent LLM cost.
Limitations
  • Running real experiments requires Python 3.10+, an NVIDIA GPU, a project workspace, and runnable training commands; it does not supply datasets or research code.
  • API operation depends on an Anthropic-compatible or OpenAI-compatible endpoint and credentials; subscription operation requires a locally installed and logged-in Claude or Codex CLI.
  • The repository documents codex_cli only for the Leader/THINK path: its internal agent loop prevents the framework from recovering PIDs for experiments it launches.
  • The default is one worker and experiment.max_parallel is 1, so it is not positioned as a large-scale parallel experiment scheduler.

How do you install or deploy this agent?

Provide Python 3.10+, at least one NVIDIA GPU, and either Anthropic/OpenAI-compatible API credentials or a logged-in Claude/Codex CLI. Run:
git clone https://github.com/Xiangyue-Zhang/auto-deep-researcher-24x7.git
cd auto-deep-researcher-24x7
pip install -r requirements.txt
python install.py
python -m core.loop --check

Create a project directory containing at least PROJECT_BRIEF.md. For API mode, set agent.provider to "anthropic" or "openai" in the project config.yaml, choose model, and set the corresponding credential environment variable.

How do you use this agent?

A minimal direct invocation is:
python -m core.loop --project /path/to/project --gpu 0 --max-cycles 5

Remove --max-cycles for continuous operation. In Claude Code, run /auto-experiment --project /path/to/project --gpu 0; the installer also provides matching Codex local skills. For remote execution, set execution.mode: "ssh" with ssh_host and remote_workspace. For Slurm, set execution.mode: "slurm" and provide ssh_host, remote_workspace, slurm_partition, and slurm_time. Check active work with /experiment-status and steer the next cycle with workspace/HUMAN_DIRECTIVE.md.

How does this agent compare with similar options?

The project compares itself with Claude Scholar, AI Scientist, OpenHands, and SWE-Agent, emphasizing continuous experiment runs, zero-cost monitoring during training, GPU management, and constant-size memory. Its own table also presents a narrower paper-writing and general-coding scope.

FAQ

Does it keep making paid model calls while a job trains?
No. Training-time monitoring is designed around process checks, log reads, and nvidia-smi; LLM usage is concentrated in THINK and REFLECT. The README gives an illustrative estimate of about $0.08 for a 24-hour cycle with eight hours of training.
Can it modify an existing training codebase?
Yes. The Code Agent can read, write, and modify project-workspace files, perform a dry-run, and launch training. The README identifies PROJECT_BRIEF.md and MEMORY_LOG.md as protected files.
What happens when training crashes or a Slurm job times out?
The monitor captures the finished run's logs for REFLECT. In Slurm mode, sacct supplies terminal states including FAILED, TIMEOUT, and CANCELLED, which are propagated to state, the ledger, and reflection context.
Can the controller stay local while training runs elsewhere?
Yes. SSH mode sends workspace operations, training, logs, PID checks, and GPU queries to one remote host. Slurm mode submits jobs through SSH while controller state remains local.

Related agents