AgentSquare
Automatically search combinations of planning, reasoning, tool-use, and memory modules for LLM agents.
- Source repo
- tsinghua-fib-lab/AgentSquare
- Stars
- ★ 232
- Last updated
- 10mo ago
- Primary language
- HTML
- FA score
- 28/100 · Major gaps
At a glance
- Works with
- Platform-specificOpenAI API
- You'll need
- Typical use
- An LLM-agent researcher who wants to search combinations across planning, reasoning, tool-use, and memory modules.
- Main limitation
- The documented setup requires OPENAI_API_KEY, and examples use gpt-3.5-turbo-0125 and gpt-4o-2024-08-06; no alternative-provider path is documented.
- Source review
- 28/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
AgentSquare is the official implementation of “AgentSquare: Automatic LLM Agent Search in Modular Design Space,” including code, prompts, and results. It organizes agent designs into planning, reasoning, tool-use, and memory modules, with documented module I/O interfaces. The main search experiment runs through search/agent_search.py. The repository supplies task entry points for ALFWorld, WebShop, M3Tooleval, and Sciworld, and points developers to workflow.py for integrating encapsulated custom tasks. Its operating boundary is a local Python and shell environment, task data or external task environments, and OpenAI API credentials.
The workflow starts by setting OPENAI_API_KEY and installing requirements.txt in a Python 3.9.12 environment. The search flow runs python agent_search.py from the search directory. For ALFWorld, sh run.sh or alfworld_run.py combines modules selected with --planning, --reasoning, --tooluse, --memory, and --model to solve the task; ALFWORLD_DATA must point to the task data. Separate commands are provided for WebShop, M3Tooleval, and Sciworld; the Sciworld example invokes eval_main_sci.py with a configuration file, baseline-results directory, model, and module arguments for evaluation.
- An LLM-agent researcher who wants to search combinations across planning, reasoning, tool-use, and memory modules.
- A developer with ALFWorld data who needs to run a task-solving example with explicit module selections.
- An evaluator who has installed and launched the WebShop environment and needs the repository’s task runner.
- A researcher using the AgentBoard Sciworld environment who needs to evaluate a model and module combination with the supplied configuration and baseline-results path.
- A developer with an encapsulated task who wants to integrate it by following the workflow.py and tasks/alfworld structure.
How do you install or deploy this agent?
Set credentials and create the documented environment:
export OPENAI_API_KEY=<YOUR KEY HERE>
git clone https://github.com/tsinghua-fib-lab/AgentSquare.git
conda create -n agentsquare python=3.9.12
conda activate agentsquare
cd AgentSquare
pip install -r requirements.txtBefore running other tasks, the README also specifies:
cd tasks
pip install -r requirements.txtHow do you use this agent?
Run the main search experiment:
cd search
python agent_search.pyFor the ALFWorld example, set the data path and run:
export ALFWORLD_DATA=<Your path>/AgentSquare/tasks/alfworld
cd tasks/alfworld
sh run.shThe documented direct invocation is:
python3 alfworld_run.py --planning deps --reasoning cot --tooluse none --memory dilu --model gpt-3.5-turbo-0125What are this agent's strengths and limitations?
- Defines a modular design space with four named areas—planning, reasoning, tool use, and memory—and documents module interfaces.
- Provides both a search entry point and task runners for ALFWorld, WebShop, M3Tooleval, and Sciworld.
- Its ALFWorld and Sciworld examples expose module and model choices as command-line parameters, supporting reproducible experiment setup.
- The documented setup requires OPENAI_API_KEY, and examples use gpt-3.5-turbo-0125 and gpt-4o-2024-08-06; no alternative-provider path is documented.
- WebShop and Sciworld require separately installed external environments following instructions from other projects.
- ALFWorld depends on local task data through ALFWORLD_DATA; the supplied material does not fully document obtaining or deploying that data.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| AgentSquare This agent | 28 · Major gaps | ★ 232 | 10mo ago | HTML | OpenAI API |
| AI Agents Projects & Tutorials | 9 · Major gaps | ★ 2.9k | 4d ago | Jupyter Notebook | OpenAI API · Claude API |
| MS-Agent | 48 · Major gaps | ★ 4.4k | 2d ago | Python | — |
| Anda | 44 · Major gaps | ★ 440 | 7d ago | Rust | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows no permission model, user confirmation mechanism, data flow transparency, sensitive data handling, dependency security audit, external effects description, rollback mechanism, or source attribution. All trust criteria are unmet, hence score 0.
Self-consistency: README installation steps align with requirements.txt, but no complete configuration example is provided, and some tasks depend on external environments, so self-consistency is insufficient. Dependency availability: requirements.txt lists specific versions but no completeness or compatibility verification. Failure messages: No documentation of error handling or failure messages.
Audience and scenarios: README clearly identifies target audience (research community) and multiple task scenarios (ALFWorld, WebShop, etc.), but lacks detailed scenario configuration. Capability boundaries: No clear statement of AgentSquare's capability boundaries, such as supported models, task types. Trigger precision: Provides command-line parameter examples but not detailed parameter meanings. Environment fit: Provides Python version and dependency installation instructions but not OS compatibility.
Information architecture: README is well-structured with sections for installation, quick start, task running. Install notes: Detailed installation steps provided. Naming stability: Module naming not clearly specified. Examples and FAQ: Multiple task examples provided, but no FAQ. Known limitations: Not mentioned. License: README shows Apache-2.0 but no LICENSE file. Versioning and changelog: Not provided. Maintenance responsibility: Contact email provided but maintenance responsibility not explicit.
Output usability: Provides run examples but not output format. Marginal value: AgentSquare proposes modular design space search, which has research value. Cost-benefit: No performance or cost analysis provided.
Claim traceability: README cites paper and website but no specific result data. Cross-source corroboration: Only relies on README, no other sources. Fact-inference separation: No clear distinction between facts and inferences.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- No LICENSE file provided despite README claiming Apache-2.0.
- Dependencies are pinned but no security audit or vulnerability information is provided.
- No security or privacy documentation is provided.
- No rollback or recovery mechanism is provided.