AgentFlow
Train an in-system planner with Flow-GRPO for more reliable multi-tool, long-horizon reasoning.
- Source repo
- lupantech/AgentFlow
- Stars
- ★ 2.1k
- Last updated
- 9mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 31/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A research group training a tool-planning model for Natural Questions-style search and mathematical reasoning workloads.
- Not a fit if
- Users wanting a one-command experience without tmux, service scripts, and dataset preparation
- Anyone without GPUs or suitable compute for full Flow-GRPO training
- Source review
- 31/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
AgentFlow is a trainable, tool-integrated modular reasoning system centered on online planner optimization. Its Planner, Executor, Verifier, and Generator coordinate over multiple turns through evolving memory and a toolkit. At inference time, it can use search, encyclopedia, and Python-coding tools, producing answers, process summaries, and execution logs. Its training workflow applies Flow-based Group Refined Policy Optimization (Flow-GRPO) directly to the Planner, with example data drawn from Natural Questions and DeepMath-103K. The repository exposes local Python commands, tmux-based training services, vLLM serving, and benchmark scripts rather than a hosted chat product.
With python quick_start.py, AgentFlow initializes its tools; the Planner analyzes a query and predicts an action, the Executor runs calls such as Google_Search_Tool, the Verifier cross-checks results, and the Generator returns the answer and a process summary. For training, data/get_train_data.py and data/aime24_data.py prepare data, train/serve_with_logs.sh starts the service, and train/train_with_logs.sh runs Flow-GRPO training. For evaluation, scripts/serve_vllm.sh serves a trained planner, then task scripts such as bash bamboogle/run.sh run from test; task directories receive logs/, results/output_i.json, and finalscore_*.log.
- A research group training a tool-planning model for Natural Questions-style search and mathematical reasoning workloads.
- An evaluation engineer who needs per-problem logs, JSON answers, and final scores from benchmark task scripts such as Bamboogle.
- A developer with Google Search API credentials who wants a multi-module workflow to search, verify, and answer factual questions.
- A team serving a trained 7B planner through vLLM and changing planner-model settings in benchmark scripts.
- A researcher comparing modular Planner, Executor, Verifier, and Generator coordination with a single model that interleaves reasoning and tool calls.
How do you install or deploy this agent?
Python 3.11 is recommended. Run:
bash setup.sh
source .venv/bin/activateCopy agentflow/.env.template to agentflow/.env, then configure OPENAI_API_KEY for judging and GOOGLE_API_KEY for Google Search. DASHSCOPE_API_KEY or TOGETHER_API_KEY are optional paths for Qwen-2.5-7B-Instruct. For optional parallel benchmark runs, install GNU parallel with sudo apt-get update and sudo apt-get install parallel.
How do you use this agent?
First run bash ./tools/test_all_tools.sh from agentflow/agentflow, then use python agentflow/scripts/test_llm_engine.py to check available engines. After configuring credentials, run python quick_start.py for a first inference. For training, run bash train/serve_with_logs.sh in one tmux window and bash train/train_with_logs.sh in another; hyperparameters are in train/config.yaml. For benchmarks, run bash scripts/serve_vllm.sh, enter test, and execute the chosen task script.
What are this agent's strengths and limitations?
- Separates planning, execution, verification, and generation into four explicit modules coordinated by evolving memory and tools.
- Uses Flow-GRPO to optimize the Planner while it operates in the system, targeting long-horizon planning and tool use with sparse rewards.
- Provides runnable paths for tool and engine checks, logged training, vLLM serving, and benchmark artifacts.
- Documents several engine paths, including OpenAI, DashScope, Gemini, DeepSeek, Together, and local vLLM serving.
- Search and judging require user-supplied API keys, and the repository does not document the cost of those services.
- Executor, Verifier, and Generator default to Qwen-2.5-7B-Instruct through DashScope; using another model requires code configuration changes.
- Full training requires tmux, service scripts, data preparation, and suitable compute resources, making it heavier than one-off inference.
- The documented delivery modes are local scripts, environment variables, and vLLM; no hosted API, web UI, or MCP integration is described.
How does this agent compare with similar options?
The repository contrasts AgentFlow with Search-R1: it describes Search-R1 as training a single LLM to interleave reasoning steps and tool calls, while AgentFlow uses four modules and online Flow-GRPO optimization of the Planner.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AgentFlow This agent | 31 · Major gaps | CLIFree + model costs | ★ 2.1k | 9mo ago | Python | OpenAI API |
| AReaL Async RL Platform | Insufficient evidence | CLIFree | ★ 5.9k | 1d ago | Python | — |
| LatentMAS | 42 · Major gaps | CLIFree | ★ 1.2k | 4mo ago | Python | — |
| Agent Lightning | 40 · Major gaps | CLIFree + model costs | ★ 19k | 1d ago | Python | OpenAI API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: The repository provides no documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All trust-related criteria are unsupported, hence scored 0.
Evidence shows: README and test scripts provide partial consistency in tool testing and LLM engine testing, but no comprehensive error handling or failure messages. Dependencies are listed in pyproject.toml but availability is not verified. Thus, self-consistency, dependency availability, and failure messages are scored 1.
Evidence shows: README identifies target audience (researchers and developers) and scenarios (search, math, science), but capability boundaries are not clearly stated. Trigger precision (tool selection) is exemplified but not detailed. Environment fit (Python version, API keys) is documented but not fully configured. Hence, audience and scenarios scored 2, others 1.
Evidence shows: README provides clear installation instructions, examples, and benchmark information, but lacks known limitations, versioning/changelog, and explicit maintenance responsibility. License is MIT, but no versioning or changelog. Thus, information architecture, install notes, examples, and license scored 2, naming stability and maintenance responsibility 1, known limitations and versioning/changelog 0.
Evidence shows: Output formats (JSON, logs) are described in benchmark scripts, but detailed output specifications are not provided. Marginal value is supported by benchmark results in README, but cost-benefit is not detailed. Hence, output usability and marginal value scored 2, cost-benefit 1.
Evidence shows: Claims in README (e.g., benchmark results) lack traceable detailed data, but links to paper and project page are provided. Cross-source corroboration is limited, and fact-inference separation is unclear. Thus, all criteria scored 1.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository lacks documentation on permission management, data flow transparency, or sensitive data handling; assess security risks before use.
- Dependencies are not security-audited, and no version pinning or vulnerability mitigation is provided.
- Performance claims in README lack reproducible detailed data; verify via paper and project page.
FAQ
Which credentials are needed for an initial run?
GOOGLE_API_KEY is needed for the Google Search tool, and the README identifies OPENAI_API_KEY as used for judging. DashScope or Together credentials are optional for their respective engine paths.Can I use my own model?
llm_engine_name can be changed in the relevant run.sh; the other modules can be changed through configuration in planner.py and solver.py.How can I catch environment or credential failures before training?
bash ./tools/test_all_tools.sh to test integrated tools and python agentflow/scripts/test_llm_engine.py to test LLM-engine initialization and responses.What does training and evaluation produce?
output_i.json answers, and finalscore_*.log scores.