AgentBench

A multi-environment benchmark for measuring how language models execute multi-turn tasks.

Source repo
THUDM/AgentBench
Stars
★ 3.8k
Last updated
9mo ago
License
Apache-2.0
Primary language
Python

At a glance

How it runs
Self-hosted serviceCLI
Works with
Portable with changesOpenAI API
Cost
Free software; you pay for model usage
Setup effort
High · needs real infrastructure
You'll need
DockerDocker ComposePython 3.9CondaMySQL 8Redis 7+Shell / CLINetwork accessLocal filesystem
Typical use
A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
Not a fit if
  • Individuals wanting a ready-to-run tool without Docker and data-service setup
  • Laptops or small machines with under 16GB of RAM
  • Teams needing long unattended runs despite the alfworld memory leak
Maintenance
No commits for 6+ monthslast commit 2026-02-08+1.0 ★/day over 7 days

What does this agent do, and when should you use it?

AgentBench is a benchmark for evaluating large language models operating as autonomous task executors. The current repository contains AgentBench FC, a function-calling version integrated with AgentRL. Its Docker Compose deployment runs an AgentRL Controller, task workers, a Freebase service, and Redis for five task families: alfworld, dbbench, knowledgegraph, os_interaction, and webshop. The README also documents the original v0.2 benchmark, which has eight environments and Dev/Test splits. That older workflow uses task workers, a controller, and an assigner, with an OpenAI Chat configuration as the documented example. It fits research or model-evaluation teams that need self-hosted, repeatable environment-based testing rather than a general-purpose end-user assistant.

For FC, docker compose -f extra/docker-compose.yml up starts the AgentRL Controller, five task-worker types, the Freebase service required by knowledgegraph, and Redis for container allocation. dbbench uses mysql:8, while os_interaction requires locally built local-os/default, local-os/packages, and local-os/ubuntu images. knowledgegraph requires Freebase data at ./virtuoso_db/virtuoso.db, unless the mount point in extra/docker-compose.yml is changed. In the older workflow, python -m src.client.agent_test checks an agent configuration, python -m src.start_task -a starts workers, and python -m src.assigner launches evaluation. The process produces model results on benchmark test tasks, with full results presented through the project leaderboard.

  1. A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
  2. A researcher reproducing LLM-as-Agent experiments can use the original v0.2 Dev/Test splits, task services, and assigner workflow.
  3. An engineer with an OpenAI API key who wants to validate a gpt-3.5-turbo-0613 setup can run src.client.agent_test.
  4. An individual researcher with constrained capacity can use the v0.2 lite start and assignment presets, which run one worker per task.
  5. A team needing a local knowledge-graph evaluation service can set up Freebase and point configs/tasks/kg.yaml to its own SPARQL endpoint.

Key setup facts

Pulled from the install and usage notes below. The project README has the current values.

Install / run
  • docker pull mysql:8
  • pip install -r requirements.txt
Config files
configs/agents/openai-chat.yaml extra/docker-compose.yml configs/start_task_lite.yaml configs/assignments/lite.yaml
Runtime
Python 3.9
Install notes cover
Linux · Docker

How do you install or deploy this agent?

The current FC setup requires Docker and Docker Compose. Prepare images with docker pull mysql:8, then run docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles, docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles, and docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles. Download and extract the Freebase-Setup data, then place the database at ./virtuoso_db/virtuoso.db. The README does not document FC model credentials or a command for submitting an evaluation job. For v0.2, it recommends a conda Python 3.9 environment followed by pip install -r requirements.txt, then entering an OpenAI API key in configs/agents/openai-chat.yaml.

How do you use this agent?

Start the current FC stack with docker compose -f extra/docker-compose.yml up. It starts the Controller, five task-worker types, Freebase, and Redis; if Redis 7+ already runs on the machine, its Compose service may be omitted. For a first complete v0.2 evaluation, run python -m src.start_task -a, wait for task services to become ready, then run python -m src.assigner. The lite equivalents are python -m src.start_task -a --config configs/start_task_lite.yaml and python -m src.assigner --config configs/assignments/lite.yaml.

What are this agent's strengths and limitations?

Pros
  • The current FC edition packages five task families, the AgentRL Controller, Freebase, and Redis in one Docker Compose deployment.
  • The original benchmark spans eight materially different environments: OS, DB, KG, digital card games, lateral-thinking puzzles, house-holding, web shopping, and web browsing.
  • The v0.2 documentation provides an explicit worker/controller/assigner execution path plus standard and lite configurations.
  • The README provides approximate startup times and per-worker memory needs for all eight original tasks, helping with capacity planning.
Limitations
  • The FC quick-start documentation does not specify model credentials, a Controller invocation, or a command that submits evaluation jobs.
  • The webshop environment needs roughly 16GB of RAM to start; the older table also lists about 15GB per worker.
  • The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.
  • knowledgegraph requires separate Freebase data and service setup; the older documentation describes its online dependency as unstable.
  • The older version pins older scientific-Python dependencies and recommends Python 3.9 for reliable installation.

How does this agent compare with similar options?

For the original benchmark, AgentBench uses published ALFWorld, WebShop, and Mind2Web datasets for House-Holding, Web Shopping, and Web Browsing, while describing its other five domains as newly created. The README also introduces VisualAgentBench, a separate evaluation project for visual foundation agents across embodied, GUI, and visual-design environments.

Key facts side by side with the most closely related agents.

Agent Source review Form / cost Stars Updated Language Full support on
AgentBench This agent 54 · Major gaps Self-hosted serviceFree + model costs ★ 3.8k 9mo ago Python OpenAI API
ToolBench & ToolLLaMA 46 · Major gaps CLIFree + model costs ★ 5.8k 1y ago Python OpenAI API
Yuxi – Self-Hosted Multi-Tenant Knowledge Agent Platform 85 · Good Self-hosted serviceFree + model costs ★ 7.7k today Python OpenAI API
Mnemosyne 71 · Some gaps CLIFree ★ 3.6k today Python Codex · Claude Code

How does FollowAgents rate this agent?

FollowAgents source review · FARS-2.1
Major gaps
54/ 100 5-point scale 2.7 / 5
Trust 10/29
Reliability 5/14
Adaptability 14/18
Convention 11/18
Effectiveness 10/13
Verifiability 4/8
Why each dimension lost points
Trust10 / 29 · 1.7/5

The README asks users to configure an OpenAI API key and describes Docker services, Redis, Freebase data, and model calls, giving partial visibility into data flows and external effects. The sample does not show permission isolation, protected secret storage, sensitive-data handling, confirmation, or rollback, so these receive limited credit. Dependencies are version-constrained, but there is no evidence of security auditing or vulnerability response. The README lists the paper, upstream datasets, and citation details, supporting some attribution; publisher identity remains unknown.

Reliability5 / 14 · 1.8/5

The README and configuration guides provide setup instructions, identify an unstable external KG service, WebShop memory needs, and an Alfworld resource leak; the workflow shows lightweight configuration validation. However, the README's old/new version framing is not clearly tied to this pinned revision, and the English config guide describes agent/task fields while the Chinese guide says agents/tasks. This weakens consistency. Pinned dependencies and external-service requirements are documented, but failure diagnosis guidance is limited.

Adaptability14 / 18 · 3.9/5

The benchmark covers varied settings including operating systems, databases, knowledge graphs, shopping, and household tasks, with configuration, extension guidance, and a lite concurrency preset. Its role as an LLM-agent evaluation benchmark rather than a general-purpose production agent can be inferred from the task list. Users must configure models and tasks themselves, and trigger and environment-fit details are mostly left to configuration documentation, with no detailed compatibility matrix in the sample.

Convention11 / 18 · 3.1/5

The README, configuration guide, and references to entrance and extension guides provide a usable information structure. Installation steps, a Python recommendation, Docker image setup, resource estimates, and several known limitations are documented. The Apache-2.0 license text is complete. The README describes project evolution and older version tags, but the sample gives no clear maintenance owner or commitment; configuration field names differ between the Chinese and English guides.

Effectiveness10 / 13 · 3.8/5

The benchmark spans multiple interactive environments and provides Dev/Test splits, launch commands, configuration examples, and result displays, making it useful for model comparison and experiment setup. Cost evidence covers some resource and startup estimates, but does not adequately quantify the overall burden of multiple services, external data downloads, model API calls, or high-memory tasks, so cost-benefit receives limited credit.

Verifiability4 / 8 · 2.5/5

The README provides a paper citation, results entry points, environment names, resource estimates, and configuration and launch paths that make several claims traceable. The sampled config guides and CI workflow corroborate only part of the configuration-validation story; independent result-verification material is not shown. Resource limits and service instability are stated explicitly, but the sample does not systematically separate measured facts from broader claims.

Risks and how to mitigate them
  • Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
  • Startup involves Docker services, external Freebase data, and a model API key; confirm how data, network access, and secrets are handled before following the instructions.
  • The README describes both a newer function-calling version and older AgentBench versions, while the config guides disagree on field names; verify the entry points and config format for the pinned revision.
  • WebShop requires about 16 GB of RAM, and the README reports that Alfworld workers may continue consuming memory and disk space.
Evidence confidence: Low Reviewed Nov 08, 2026 Reviewed revision d1e4a10db08c
See the full review method →
Maintain this project? Add the FARS badge to your README

The badge shows the current review score and verdict and links back to this page. It refreshes about once an hour after a re-review.

FARS

[![FARS review](https://followagents.com/badge/agentbench.svg)](https://followagents.com/en/agents/agentbench)
<a href="https://followagents.com/en/agents/agentbench"><img src="https://followagents.com/badge/agentbench.svg" alt="FARS review"></a>

FAQ

Does AgentBench FC require a paid model API?
The FC quick start does not document model credentials or billing. The older example requires an OpenAI API key in configs/agents/openai-chat.yaml.
Can I run it without Docker?
The FC quick start uses Docker Compose, and the older workflow also requires Docker plus images for dbbench and os-std.
Why does the knowledge-graph task need extra setup?
It depends on Freebase data and a Freebase service. FC expects the database at ./virtuoso_db/virtuoso.db or an adjusted Compose mount; v0.2 can use a configured SPARQL endpoint.
Is it practical on a laptop?
v0.2 includes a one-worker-per-task lite preset, but webshop has substantial memory demand. The FC README estimates roughly 16GB of RAM just to start webshop.
Is AgentBench open source?
Yes. The source code is on GitHub (THUDM/AgentBench) under the Apache-2.0 licence.
Is AgentBench safe to use?
FollowAgents' FARS review (Nov 08, 2026) scored AgentBench 10/29 on trust and safety. The review did not find these safety controls in the source: rollback or recovery path. Before giving it real accounts or data: Back up first, or work on a git branch or snapshot, so its changes can be undone.
Is AgentBench still maintained?
The GitHub repository has had no commits since 2026-02-08, more than six months ago. That can mean the project is finished or abandoned; check its issues before relying on it.
Who makes AgentBench?
AgentBench is published on GitHub by the account THUDM. FollowAgents has not verified which person or company is behind that account.
Was this page helpful?
View on GitHub ↗ Install ↓