AgentBench
A multi-environment benchmark for measuring how language models execute multi-turn tasks.
Per-dimension scores and reasoning
Insufficient evidence: no documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on self-consistency, dependency availability, or failure messages. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on audience and scenarios, capability boundaries, trigger precision, or environment fit. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on information architecture, install notes, naming stability, examples and FAQ, known limitations, license, versioning/changelog, or maintenance responsibility. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on output usability, marginal value, or cost-benefit. All criteria unsupported, hence 0.
Insufficient evidence: no documentation on claim traceability, cross-source corroboration, or fact-inference separation. All criteria unsupported, hence 0.
- This repository is a benchmark framework, not a directly usable Agent product; its purpose is to evaluate LLM agents, not to act as an agent itself.
- The repository provides no security or trust documentation, such as permission management, data flow transparency, or sensitive data handling.
- Dependencies are not pinned to exact versions, and no security audit or vulnerability mitigation is provided.
- Publisher identity is unverified, and no clear maintenance responsibility or update path is provided.
What does this agent do, and when should you use it?
AgentBench is a benchmark for evaluating large language models operating as autonomous task executors. The current repository contains AgentBench FC, a function-calling version integrated with AgentRL. Its Docker Compose deployment runs an AgentRL Controller, task workers, a Freebase service, and Redis for five task families: alfworld, dbbench, knowledgegraph, os_interaction, and webshop. The README also documents the original v0.2 benchmark, which has eight environments and Dev/Test splits. That older workflow uses task workers, a controller, and an assigner, with an OpenAI Chat configuration as the documented example. It fits research or model-evaluation teams that need self-hosted, repeatable environment-based testing rather than a general-purpose end-user assistant.
For FC, docker compose -f extra/docker-compose.yml up starts the AgentRL Controller, five task-worker types, the Freebase service required by knowledgegraph, and Redis for container allocation. dbbench uses mysql:8, while os_interaction requires locally built local-os/default, local-os/packages, and local-os/ubuntu images. knowledgegraph requires Freebase data at ./virtuoso_db/virtuoso.db, unless the mount point in extra/docker-compose.yml is changed. In the older workflow, python -m src.client.agent_test checks an agent configuration, python -m src.start_task -a starts workers, and python -m src.assigner launches evaluation. The process produces model results on benchmark test tasks, with full results presented through the project leaderboard.
- A model-evaluation team comparing candidates across database, operating-system, knowledge-graph, and web-shopping environments can deploy the FC workers.
- A researcher reproducing LLM-as-Agent experiments can use the original v0.2 Dev/Test splits, task services, and assigner workflow.
- An engineer with an OpenAI API key who wants to validate a
gpt-3.5-turbo-0613setup can runsrc.client.agent_test. - An individual researcher with constrained capacity can use the v0.2 lite start and assignment presets, which run one worker per task.
- A team needing a local knowledge-graph evaluation service can set up Freebase and point
configs/tasks/kg.yamlto its own SPARQL endpoint.
What are this agent's strengths and limitations?
- The current FC edition packages five task families, the AgentRL Controller, Freebase, and Redis in one Docker Compose deployment.
- The original benchmark spans eight materially different environments: OS, DB, KG, digital card games, lateral-thinking puzzles, house-holding, web shopping, and web browsing.
- The v0.2 documentation provides an explicit worker/controller/assigner execution path plus standard and lite configurations.
- The README provides approximate startup times and per-worker memory needs for all eight original tasks, helping with capacity planning.
- The FC quick-start documentation does not specify model credentials, a Controller invocation, or a command that submits evaluation jobs.
- The webshop environment needs roughly 16GB of RAM to start; the older table also lists about 15GB per worker.
- The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.
- knowledgegraph requires separate Freebase data and service setup; the older documentation describes its online dependency as unstable.
- The older version pins older scientific-Python dependencies and recommends Python 3.9 for reliable installation.
How do you install or deploy this agent?
The current FC setup requires Docker and Docker Compose. Prepare images with docker pull mysql:8, then run docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles, docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles, and docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles. Download and extract the Freebase-Setup data, then place the database at ./virtuoso_db/virtuoso.db. The README does not document FC model credentials or a command for submitting an evaluation job. For v0.2, it recommends a conda Python 3.9 environment followed by pip install -r requirements.txt, then entering an OpenAI API key in configs/agents/openai-chat.yaml.
How do you use this agent?
Start the current FC stack with docker compose -f extra/docker-compose.yml up. It starts the Controller, five task-worker types, Freebase, and Redis; if Redis 7+ already runs on the machine, its Compose service may be omitted. For a first complete v0.2 evaluation, run python -m src.start_task -a, wait for task services to become ready, then run python -m src.assigner. The lite equivalents are python -m src.start_task -a --config configs/start_task_lite.yaml and python -m src.assigner --config configs/assignments/lite.yaml.
How does this agent compare with similar options?
For the original benchmark, AgentBench uses published ALFWorld, WebShop, and Mind2Web datasets for House-Holding, Web Shopping, and Web Browsing, while describing its other five domains as newly created. The README also introduces VisualAgentBench, a separate evaluation project for visual foundation agents across embodied, GUI, and visual-design environments.
FAQ
Does AgentBench FC require a paid model API?
configs/agents/openai-chat.yaml.Can I run it without Docker?
Why does the knowledge-graph task need extra setup?
./virtuoso_db/virtuoso.db or an adjusted Compose mount; v0.2 can use a configured SPARQL endpoint.