AgentBench
A multi-environment benchmark for measuring how language models execute multi-turn tasks.
- Source repo
- THUDM/AgentBench
- Stars
- ★ 3.8k
- Last updated
- 9mo ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- Insufficient evidence
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
- Not a fit if
- Individuals wanting a ready-to-run tool without Docker and data-service setup
- Laptops or small machines with under 16GB of RAM
- Teams needing long unattended runs despite the alfworld memory leak
- Source review
- Insufficient evidence 8 safety controls not found
What does this agent do, and when should you use it?
AgentBench is a benchmark for evaluating large language models operating as autonomous task executors. The current repository contains AgentBench FC, a function-calling version integrated with AgentRL. Its Docker Compose deployment runs an AgentRL Controller, task workers, a Freebase service, and Redis for five task families: alfworld, dbbench, knowledgegraph, os_interaction, and webshop. The README also documents the original v0.2 benchmark, which has eight environments and Dev/Test splits. That older workflow uses task workers, a controller, and an assigner, with an OpenAI Chat configuration as the documented example. It fits research or model-evaluation teams that need self-hosted, repeatable environment-based testing rather than a general-purpose end-user assistant.
For FC, docker compose -f extra/docker-compose.yml up starts the AgentRL Controller, five task-worker types, the Freebase service required by knowledgegraph, and Redis for container allocation. dbbench uses mysql:8, while os_interaction requires locally built local-os/default, local-os/packages, and local-os/ubuntu images. knowledgegraph requires Freebase data at ./virtuoso_db/virtuoso.db, unless the mount point in extra/docker-compose.yml is changed. In the older workflow, python -m src.client.agent_test checks an agent configuration, python -m src.start_task -a starts workers, and python -m src.assigner launches evaluation. The process produces model results on benchmark test tasks, with full results presented through the project leaderboard.
- A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
- A researcher reproducing LLM-as-Agent experiments can use the original v0.2 Dev/Test splits, task services, and assigner workflow.
- An engineer with an OpenAI API key who wants to validate a
gpt-3.5-turbo-0613setup can runsrc.client.agent_test. - An individual researcher with constrained capacity can use the v0.2 lite start and assignment presets, which run one worker per task.
- A team needing a local knowledge-graph evaluation service can set up Freebase and point
configs/tasks/kg.yamlto its own SPARQL endpoint.
How do you install or deploy this agent?
The current FC setup requires Docker and Docker Compose. Prepare images with docker pull mysql:8, then run docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles, docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles, and docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles. Download and extract the Freebase-Setup data, then place the database at ./virtuoso_db/virtuoso.db. The README does not document FC model credentials or a command for submitting an evaluation job. For v0.2, it recommends a conda Python 3.9 environment followed by pip install -r requirements.txt, then entering an OpenAI API key in configs/agents/openai-chat.yaml.
How do you use this agent?
Start the current FC stack with docker compose -f extra/docker-compose.yml up. It starts the Controller, five task-worker types, Freebase, and Redis; if Redis 7+ already runs on the machine, its Compose service may be omitted. For a first complete v0.2 evaluation, run python -m src.start_task -a, wait for task services to become ready, then run python -m src.assigner. The lite equivalents are python -m src.start_task -a --config configs/start_task_lite.yaml and python -m src.assigner --config configs/assignments/lite.yaml.
What are this agent's strengths and limitations?
- The current FC edition packages five task families, the AgentRL Controller, Freebase, and Redis in one Docker Compose deployment.
- The original benchmark spans eight materially different environments: OS, DB, KG, digital card games, lateral-thinking puzzles, house-holding, web shopping, and web browsing.
- The v0.2 documentation provides an explicit worker/controller/assigner execution path plus standard and lite configurations.
- The README provides approximate startup times and per-worker memory needs for all eight original tasks, helping with capacity planning.
- The FC quick-start documentation does not specify model credentials, a Controller invocation, or a command that submits evaluation jobs.
- The webshop environment needs roughly 16GB of RAM to start; the older table also lists about 15GB per worker.
- The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.
- knowledgegraph requires separate Freebase data and service setup; the older documentation describes its online dependency as unstable.
- The older version pins older scientific-Python dependencies and recommends Python 3.9 for reliable installation.
How does this agent compare with similar options?
For the original benchmark, AgentBench uses published ALFWorld, WebShop, and Mind2Web datasets for House-Holding, Web Shopping, and Web Browsing, while describing its other five domains as newly created. The README also introduces VisualAgentBench, a separate evaluation project for visual foundation agents across embodied, GUI, and visual-design environments.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AgentBench This agent | Insufficient evidence | Self-hosted serviceFree + model costs | ★ 3.8k | 9mo ago | Python | OpenAI API |
| ToolBench & ToolLLaMA | 32 · Major gaps | CLIFree + model costs | ★ 5.8k | 1y ago | Python | OpenAI API |
| Mnemosyne | 59 · Major gaps | CLIFree | ★ 3.6k | 1d ago | Python | Codex · Claude Code |
| Graphiti: Temporal Context Graph Engine for AI Agents | 50 · Major gaps | Library / SDKFree + model costs | ★ 33k | 1d ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Insufficient evidence: no documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on self-consistency, dependency availability, or failure messages. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on audience and scenarios, capability boundaries, trigger precision, or environment fit. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on information architecture, install notes, naming stability, examples and FAQ, known limitations, license, versioning/changelog, or maintenance responsibility. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on output usability, marginal value, or cost-benefit. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on claim traceability, cross-source corroboration, or fact-inference separation. All criteria unsupported, hence 0.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- This repository is a benchmark framework, not a directly usable Agent product; its purpose is to evaluate LLM agents, not to act as an agent itself.
- The repository provides no security or trust documentation, such as permission management, data flow transparency, or sensitive data handling.
- Dependencies are not pinned to exact versions, and no security audit or vulnerability mitigation is provided.
- Publisher identity is unverified, and no clear maintenance responsibility or update path is provided.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/agentbench)<a href="https://followagents.com/en/agents/agentbench"><img src="https://followagents.com/badge/agentbench.svg" alt="FARS review"></a>FAQ
Does AgentBench FC require a paid model API?
configs/agents/openai-chat.yaml.Can I run it without Docker?
Why does the knowledge-graph task need extra setup?
./virtuoso_db/virtuoso.db or an adjusted Compose mount; v0.2 can use a configured SPARQL endpoint.