AgentSquare
Automatically search combinations of planning, reasoning, tool-use, and memory modules for LLM agents.
- Source repo
- tsinghua-fib-lab/AgentSquare
- Stars
- ★ 235
- Last updated
- 1y ago
- Primary language
- HTML
- FA score
- 41/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specificOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- An LLM-agent researcher who wants to search combinations across planning, reasoning, tool-use, and memory modules.
- Not a fit if
- Users unwilling to install external WebShop or Sciworld task environments
- Those wanting turnkey use without preparing ALFWorld task data themselves
- Source review
- 41/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
AgentSquare is the official implementation of “AgentSquare: Automatic LLM Agent Search in Modular Design Space,” including code, prompts, and results. It organizes agent designs into planning, reasoning, tool-use, and memory modules, with documented module I/O interfaces. The main search experiment runs through search/agent_search.py. The repository supplies task entry points for ALFWorld, WebShop, M3Tooleval, and Sciworld, and points developers to workflow.py for integrating encapsulated custom tasks. Its operating boundary is a local Python and shell environment, task data or external task environments, and OpenAI API credentials.
The workflow starts by setting OPENAI_API_KEY and installing requirements.txt in a Python 3.9.12 environment. The search flow runs python agent_search.py from the search directory. For ALFWorld, sh run.sh or alfworld_run.py combines modules selected with --planning, --reasoning, --tooluse, --memory, and --model to solve the task; ALFWORLD_DATA must point to the task data. Separate commands are provided for WebShop, M3Tooleval, and Sciworld; the Sciworld example invokes eval_main_sci.py with a configuration file, baseline-results directory, model, and module arguments for evaluation.
- An LLM-agent researcher who wants to search combinations across planning, reasoning, tool-use, and memory modules.
- A developer with ALFWorld data who needs to run a task-solving example with explicit module selections.
- An evaluator who has installed and launched the WebShop environment and needs the repository’s task runner.
- A researcher using the AgentBoard Sciworld environment who needs to evaluate a model and module combination with the supplied configuration and baseline-results path.
- A developer with an encapsulated task who wants to integrate it by following the workflow.py and tasks/alfworld structure.
How do you install or deploy this agent?
Set credentials and create the documented environment:
export OPENAI_API_KEY=<YOUR KEY HERE>
git clone https://github.com/tsinghua-fib-lab/AgentSquare.git
conda create -n agentsquare python=3.9.12
conda activate agentsquare
cd AgentSquare
pip install -r requirements.txtBefore running other tasks, the README also specifies:
cd tasks
pip install -r requirements.txtHow do you use this agent?
Run the main search experiment:
cd search
python agent_search.pyFor the ALFWorld example, set the data path and run:
export ALFWORLD_DATA=<Your path>/AgentSquare/tasks/alfworld
cd tasks/alfworld
sh run.shThe documented direct invocation is:
python3 alfworld_run.py --planning deps --reasoning cot --tooluse none --memory dilu --model gpt-3.5-turbo-0125What are this agent's strengths and limitations?
- Defines a modular design space with four named areas—planning, reasoning, tool use, and memory—and documents module interfaces.
- Provides both a search entry point and task runners for ALFWorld, WebShop, M3Tooleval, and Sciworld.
- Its ALFWorld and Sciworld examples expose module and model choices as command-line parameters, supporting reproducible experiment setup.
- The documented setup requires OPENAI_API_KEY, and examples use gpt-3.5-turbo-0125 and gpt-4o-2024-08-06; no alternative-provider path is documented.
- WebShop and Sciworld require separately installed external environments following instructions from other projects.
- ALFWorld depends on local task data through ALFWORLD_DATA; the supplied material does not fully document obtaining or deploying that data.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AgentSquare This agent | 41 · Major gaps | CLIFree + model costs | ★ 235 | 1y ago | HTML | OpenAI API |
| AI Agents Projects & Tutorials | Insufficient evidence | Library / SDKFree + model costs | ★ 2.9k | 1d ago | Jupyter Notebook | OpenAI API · Claude API |
| MS-Agent | 67 · Some gaps | Web appFree + model costs | ★ 4.4k | 5d ago | Python | — |
| AnimaWorks | 59 · Major gaps | — | ★ 268 | today | Python | Codex · Claude Code · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The README asks users to put an OpenAI API key in an environment variable, but the provided files do not describe key storage, log redaction, or data retention, so sensitive-data handling and data-flow transparency receive limited credit. The code sample calls tools, executes task actions, and sends requests, with no shown user confirmation, permission isolation, or rollback mechanism; those criteria score low. The README links the paper and provides citation details, and the module contribution process mentions review and acknowledgment, supporting some source-attribution credit.
requirements.txt pins several direct dependency versions, and the M3ToolEval code uses backoff, a turn limit, and an imported traceback facility, providing limited evidence of availability and failure handling. The supplied runtime excerpt is incomplete, and the packet does not show consistent error handling or cross-file consistency checks, so self-consistency and failure-message scores remain limited. The lack of executed tests was not treated as a deduction.
The README covers ALFWorld, WebShop, M3ToolEval, Sciworld, and module search, and explains how to add tasks and modules, giving reasonable evidence about intended scenarios and setup. Module interfaces and tool-execution boundaries are only partly represented; precise rules for when capabilities should trigger and explicit out-of-scope cases are not shown, so boundary and trigger-precision scores are low.
The README is organized around setup, quick start, other tasks, search experiments, and module contributions, with command examples and findable entry points. The provided files do not include an FAQ, known limitations, or a clear maintenance owner or update commitment. A README badge states Apache-2.0, but license metadata is unknown and the packet does not establish the full license terms, so license and maintenance scores are conservative. The dated news list is a brief update record, not a sustained versioning policy.
Modular search and the demonstration of combining planning, reasoning, tool use, and memory show added value for research and experimentation. The README offers reusable commands and multiple benchmark entry points, but it does not explain API or search costs or resource requirements, limiting confidence in the cost-benefit score.
The README connects the paper, code release, demos, and module process, making some claims traceable to specific documentation. The supplied material lacks the paper contents, execution records, or independent corroboration for performance claims. The README and code excerpts provide limited cross-source support, but not enough to confirm results. The packet does not systematically separate verified facts from promotional or inferred claims, so that score is limited.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Running the project requires an OpenAI API key; the provided files do not explain how requests or task data are handled or logged.
- Tool actions execute in task environments. Check the target environment, dependencies, and API costs before use; this static review did not run the code or verify results.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/agentsquare)<a href="https://followagents.com/en/agents/agentsquare"><img src="https://followagents.com/badge/agentsquare.svg" alt="FARS review"></a>FAQ
What credentials are required?
Can I run the search directly?
Which listed tasks have runners?
Can I add my own task?
Thanks — we use this to improve the page.