AgentFlow
Train an in-system planner with Flow-GRPO for more reliable multi-tool, long-horizon reasoning.
Per-dimension scores and reasoning
Evidence shows: The repository provides no documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust-related criteria are unsupported, hence scored 0.
Evidence shows: README and test scripts provide partial consistency in tool testing and LLM engine testing, but no comprehensive error handling or failure messages. Dependencies are listed in pyproject.toml but availability is not verified. Thus, self-consistency, dependency availability, and failure messages are scored 1.
Evidence shows: README identifies target audience (researchers and developers) and scenarios (search, math, science), but capability boundaries are not clearly stated. Trigger precision (tool selection) is exemplified but not detailed. Environment fit (Python version, API keys) is documented but not fully configured. Hence, audience and scenarios scored 2, others 1.
Evidence shows: README provides clear installation instructions, examples, and benchmark information, but lacks known limitations, versioning/changelog, and explicit maintenance responsibility. License is MIT, but no versioning or changelog. Thus, information architecture, install notes, examples, and license scored 2, naming stability and maintenance responsibility 1, known limitations and versioning/changelog 0.
Evidence shows: Output formats (JSON, logs) are described in benchmark scripts, but detailed output specifications are not provided. Marginal value is supported by benchmark results in README, but cost-benefit is not detailed. Hence, output usability and marginal value scored 2, cost-benefit 1.
Evidence shows: Claims in README (e.g., benchmark results) lack traceable detailed data, but links to paper and project page are provided. Cross-source corroboration is limited, and fact-inference separation is unclear. Thus, all criteria scored 1.
- The repository lacks documentation on permission management, data flow transparency, or sensitive data handling; assess security risks before use.
- Dependencies are not security-audited, and no version pinning or vulnerability mitigation is provided.
- Performance claims in README lack reproducible detailed data; verify via paper and project page.
What does this agent do, and when should you use it?
AgentFlow is a trainable, tool-integrated modular reasoning system centered on online planner optimization. Its Planner, Executor, Verifier, and Generator coordinate over multiple turns through evolving memory and a toolkit. At inference time, it can use search, encyclopedia, and Python-coding tools, producing answers, process summaries, and execution logs. Its training workflow applies Flow-based Group Refined Policy Optimization (Flow-GRPO) directly to the Planner, with example data drawn from Natural Questions and DeepMath-103K. The repository exposes local Python commands, tmux-based training services, vLLM serving, and benchmark scripts rather than a hosted chat product.
With python quick_start.py, AgentFlow initializes its tools; the Planner analyzes a query and predicts an action, the Executor runs calls such as Google_Search_Tool, the Verifier cross-checks results, and the Generator returns the answer and a process summary. For training, data/get_train_data.py and data/aime24_data.py prepare data, train/serve_with_logs.sh starts the service, and train/train_with_logs.sh runs Flow-GRPO training. For evaluation, scripts/serve_vllm.sh serves a trained planner, then task scripts such as bash bamboogle/run.sh run from test; task directories receive logs/, results/output_i.json, and finalscore_*.log.
- A research group training a tool-planning model for Natural Questions-style search and mathematical reasoning workloads.
- An evaluation engineer who needs per-problem logs, JSON answers, and final scores from benchmark task scripts such as Bamboogle.
- A developer with Google Search API credentials who wants a multi-module workflow to search, verify, and answer factual questions.
- A team serving a trained 7B planner through vLLM and changing planner-model settings in benchmark scripts.
- A researcher comparing modular Planner, Executor, Verifier, and Generator coordination with a single model that interleaves reasoning and tool calls.
What are this agent's strengths and limitations?
- Separates planning, execution, verification, and generation into four explicit modules coordinated by evolving memory and tools.
- Uses Flow-GRPO to optimize the Planner while it operates in the system, targeting long-horizon planning and tool use with sparse rewards.
- Provides runnable paths for tool and engine checks, logged training, vLLM serving, and benchmark artifacts.
- Documents several engine paths, including OpenAI, DashScope, Gemini, DeepSeek, Together, and local vLLM serving.
- Search and judging require user-supplied API keys, and the repository does not document the cost of those services.
- Executor, Verifier, and Generator default to Qwen-2.5-7B-Instruct through DashScope; using another model requires code configuration changes.
- Full training requires tmux, service scripts, data preparation, and suitable compute resources, making it heavier than one-off inference.
- The documented delivery modes are local scripts, environment variables, and vLLM; no hosted API, web UI, or MCP integration is described.
How do you install or deploy this agent?
Python 3.11 is recommended. Run:
bash setup.sh
source .venv/bin/activateCopy agentflow/.env.template to agentflow/.env, then configure OPENAI_API_KEY for judging and GOOGLE_API_KEY for Google Search. DASHSCOPE_API_KEY or TOGETHER_API_KEY are optional paths for Qwen-2.5-7B-Instruct. For optional parallel benchmark runs, install GNU parallel with sudo apt-get update and sudo apt-get install parallel.
How do you use this agent?
First run bash ./tools/test_all_tools.sh from agentflow/agentflow, then use python agentflow/scripts/test_llm_engine.py to check available engines. After configuring credentials, run python quick_start.py for a first inference. For training, run bash train/serve_with_logs.sh in one tmux window and bash train/train_with_logs.sh in another; hyperparameters are in train/config.yaml. For benchmarks, run bash scripts/serve_vllm.sh, enter test, and execute the chosen task script.
How does this agent compare with similar options?
The repository contrasts AgentFlow with Search-R1: it describes Search-R1 as training a single LLM to interleave reasoning steps and tool calls, while AgentFlow uses four modules and online Flow-GRPO optimization of the Planner.
FAQ
Which credentials are needed for an initial run?
GOOGLE_API_KEY is needed for the Google Search tool, and the README identifies OPENAI_API_KEY as used for judging. DashScope or Together credentials are optional for their respective engine paths.Can I use my own model?
llm_engine_name can be changed in the relevant run.sh; the other modules can be changed through configuration in planner.py and solver.py.How can I catch environment or credential failures before training?
bash ./tools/test_all_tools.sh to test integrated tools and python agentflow/scripts/test_llm_engine.py to test LLM-engine initialization and responses.What does training and evaluation produce?
output_i.json answers, and finalscore_*.log scores.