AgentBench
A multi-environment benchmark for measuring how language models execute multi-turn tasks.
- Source repo
- THUDM/AgentBench
- Stars
- ★ 3.8k
- Last updated
- 9mo ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 54/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
- Not a fit if
- Individuals wanting a ready-to-run tool without Docker and data-service setup
- Laptops or small machines with under 16GB of RAM
- Teams needing long unattended runs despite the alfworld memory leak
- Maintenance
- No commits for 6+ monthslast commit 2026-02-08+1.0 ★/day over 7 days
- Source review
- 54/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
AgentBench is a benchmark for evaluating large language models operating as autonomous task executors. The current repository contains AgentBench FC, a function-calling version integrated with AgentRL. Its Docker Compose deployment runs an AgentRL Controller, task workers, a Freebase service, and Redis for five task families: alfworld, dbbench, knowledgegraph, os_interaction, and webshop. The README also documents the original v0.2 benchmark, which has eight environments and Dev/Test splits. That older workflow uses task workers, a controller, and an assigner, with an OpenAI Chat configuration as the documented example. It fits research or model-evaluation teams that need self-hosted, repeatable environment-based testing rather than a general-purpose end-user assistant.
For FC, docker compose -f extra/docker-compose.yml up starts the AgentRL Controller, five task-worker types, the Freebase service required by knowledgegraph, and Redis for container allocation. dbbench uses mysql:8, while os_interaction requires locally built local-os/default, local-os/packages, and local-os/ubuntu images. knowledgegraph requires Freebase data at ./virtuoso_db/virtuoso.db, unless the mount point in extra/docker-compose.yml is changed. In the older workflow, python -m src.client.agent_test checks an agent configuration, python -m src.start_task -a starts workers, and python -m src.assigner launches evaluation. The process produces model results on benchmark test tasks, with full results presented through the project leaderboard.
- A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
- A researcher reproducing LLM-as-Agent experiments can use the original v0.2 Dev/Test splits, task services, and assigner workflow.
- An engineer with an OpenAI API key who wants to validate a
gpt-3.5-turbo-0613setup can runsrc.client.agent_test. - An individual researcher with constrained capacity can use the v0.2 lite start and assignment presets, which run one worker per task.
- A team needing a local knowledge-graph evaluation service can set up Freebase and point
configs/tasks/kg.yamlto its own SPARQL endpoint.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
docker pull mysql:8pip install -r requirements.txt
- Config files
configs/agents/openai-chat.yamlextra/docker-compose.ymlconfigs/start_task_lite.yamlconfigs/assignments/lite.yaml- Runtime
- Python 3.9
- Install notes cover
- Linux · Docker
How do you install or deploy this agent?
The current FC setup requires Docker and Docker Compose. Prepare images with docker pull mysql:8, then run docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles, docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles, and docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles. Download and extract the Freebase-Setup data, then place the database at ./virtuoso_db/virtuoso.db. The README does not document FC model credentials or a command for submitting an evaluation job. For v0.2, it recommends a conda Python 3.9 environment followed by pip install -r requirements.txt, then entering an OpenAI API key in configs/agents/openai-chat.yaml.
How do you use this agent?
Start the current FC stack with docker compose -f extra/docker-compose.yml up. It starts the Controller, five task-worker types, Freebase, and Redis; if Redis 7+ already runs on the machine, its Compose service may be omitted. For a first complete v0.2 evaluation, run python -m src.start_task -a, wait for task services to become ready, then run python -m src.assigner. The lite equivalents are python -m src.start_task -a --config configs/start_task_lite.yaml and python -m src.assigner --config configs/assignments/lite.yaml.
What are this agent's strengths and limitations?
- The current FC edition packages five task families, the AgentRL Controller, Freebase, and Redis in one Docker Compose deployment.
- The original benchmark spans eight materially different environments: OS, DB, KG, digital card games, lateral-thinking puzzles, house-holding, web shopping, and web browsing.
- The v0.2 documentation provides an explicit worker/controller/assigner execution path plus standard and lite configurations.
- The README provides approximate startup times and per-worker memory needs for all eight original tasks, helping with capacity planning.
- The FC quick-start documentation does not specify model credentials, a Controller invocation, or a command that submits evaluation jobs.
- The webshop environment needs roughly 16GB of RAM to start; the older table also lists about 15GB per worker.
- The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.
- knowledgegraph requires separate Freebase data and service setup; the older documentation describes its online dependency as unstable.
- The older version pins older scientific-Python dependencies and recommends Python 3.9 for reliable installation.
How does this agent compare with similar options?
For the original benchmark, AgentBench uses published ALFWorld, WebShop, and Mind2Web datasets for House-Holding, Web Shopping, and Web Browsing, while describing its other five domains as newly created. The README also introduces VisualAgentBench, a separate evaluation project for visual foundation agents across embodied, GUI, and visual-design environments.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AgentBench This agent | 54 · Major gaps | Self-hosted serviceFree + model costs | ★ 3.8k | 9mo ago | Python | OpenAI API |
| ToolBench & ToolLLaMA | 46 · Major gaps | CLIFree + model costs | ★ 5.8k | 1y ago | Python | OpenAI API |
| Yuxi – Self-Hosted Multi-Tenant Knowledge Agent Platform | 85 · Good | Self-hosted serviceFree + model costs | ★ 7.7k | today | Python | OpenAI API |
| Mnemosyne | 71 · Some gaps | CLIFree | ★ 3.6k | today | Python | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The README asks users to configure an OpenAI API key and describes Docker services, Redis, Freebase data, and model calls, giving partial visibility into data flows and external effects. The sample does not show permission isolation, protected secret storage, sensitive-data handling, confirmation, or rollback, so these receive limited credit. Dependencies are version-constrained, but there is no evidence of security auditing or vulnerability response. The README lists the paper, upstream datasets, and citation details, supporting some attribution; publisher identity remains unknown.
The README and configuration guides provide setup instructions, identify an unstable external KG service, WebShop memory needs, and an Alfworld resource leak; the workflow shows lightweight configuration validation. However, the README's old/new version framing is not clearly tied to this pinned revision, and the English config guide describes agent/task fields while the Chinese guide says agents/tasks. This weakens consistency. Pinned dependencies and external-service requirements are documented, but failure diagnosis guidance is limited.
The benchmark covers varied settings including operating systems, databases, knowledge graphs, shopping, and household tasks, with configuration, extension guidance, and a lite concurrency preset. Its role as an LLM-agent evaluation benchmark rather than a general-purpose production agent can be inferred from the task list. Users must configure models and tasks themselves, and trigger and environment-fit details are mostly left to configuration documentation, with no detailed compatibility matrix in the sample.
The README, configuration guide, and references to entrance and extension guides provide a usable information structure. Installation steps, a Python recommendation, Docker image setup, resource estimates, and several known limitations are documented. The Apache-2.0 license text is complete. The README describes project evolution and older version tags, but the sample gives no clear maintenance owner or commitment; configuration field names differ between the Chinese and English guides.
The benchmark spans multiple interactive environments and provides Dev/Test splits, launch commands, configuration examples, and result displays, making it useful for model comparison and experiment setup. Cost evidence covers some resource and startup estimates, but does not adequately quantify the overall burden of multiple services, external data downloads, model API calls, or high-memory tasks, so cost-benefit receives limited credit.
The README provides a paper citation, results entry points, environment names, resource estimates, and configuration and launch paths that make several claims traceable. The sampled config guides and CI workflow corroborate only part of the configuration-validation story; independent result-verification material is not shown. Resource limits and service instability are stated explicitly, but the sample does not systematically separate measured facts from broader claims.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Startup involves Docker services, external Freebase data, and a model API key; confirm how data, network access, and secrets are handled before following the instructions.
- The README describes both a newer function-calling version and older AgentBench versions, while the config guides disagree on field names; verify the entry points and config format for the pinned revision.
- WebShop requires about 16 GB of RAM, and the README reports that Alfworld workers may continue consuming memory and disk space.
FAQ
Does AgentBench FC require a paid model API?
configs/agents/openai-chat.yaml.Can I run it without Docker?
Why does the knowledge-graph task need extra setup?
./virtuoso_db/virtuoso.db or an adjusted Compose mount; v0.2 can use a configured SPARQL endpoint.Is it practical on a laptop?
Is AgentBench open source?
Is AgentBench safe to use?
Is AgentBench still maintained?
Who makes AgentBench?
Thanks — we use this to improve the page.