AppWorld Agent Benchmark Environment
A controllable world of 9 apps and ~100 people for benchmarking function-calling and interactive coding agents with state-based evaluation.
Per-dimension scores and reasoning
The README frames AppWorld as a no-consequence sandbox and includes Code Execution Safety and Agent Development Restrictions sections, showing awareness of permission and execution risk. However, the bulk of the implementation (apps, tests, data generation) ships as encrypted .bundle files, so a static review cannot verify actual permission boundaries, least privilege, or confirmation requirements for sensitive operations. The dependency list pins fastapi-login, freezegun, cryptography and others with explicit rationale, which is verifiable dependency hygiene, so dependency_security scores 2; there is no SBOM, no vulnerability scanning config, and no committed lockfile, so not 3. Data-flow transparency and sensitive-data handling are only conceptual (supervisor can expose passwords, addresses, payment cards) with no concrete minimization or redaction policy, so 1 each. Rollback only mentions world.close() for resource release, with no state rollback or recovery from destructive defaults, so 1. Source attribution is solid: full Apache-2.0 LICENSE plus license and authors/maintainers in pyproject, so 2.
pyproject pins key dependencies with dated rationale, and CI covers Python 3.11-3.14, Windows, lowest-direct resolution, bundle-check, and multiple verify tasks, giving adequate evidence for self-consistency and dependency availability, 2 each. Failure messages are thin: README suggests filing an issue on verify failure and offers python -m appworld.cli as a fallback, but there is no structured error taxonomy or diagnostics doc, so 1.
Audience and scenarios cover researchers, agent developers, MCP clients, and terminal agents with multiple documented entry paths, so 2. Capability boundaries are reasonably clear: 9 apps, 457 APIs, full ground truth for train/dev but only evaluation programs for test, so 2. Trigger precision is weak: CLI subcommands and --help are documented, but when an agent should call complete_task or consider a task done lacks a precise trigger spec, so 1. Environment fit is good with conda, source install, Docker, serverless TestClient, and APPWORLD_ROOT configuration, so 2.
Information architecture is clear with a complete anchored README TOC, so 2. Install notes cover pip, source, Git LFS, encrypted bundle unpacking, data download, and verification, which is thorough, so 3. Naming stability is good across package, CLI, and module paths, but the version is 0.2.0.dev0 with Development Status Alpha, so 2. Examples and FAQ include TLDR snippets, a notebook, and collapsible Q&A, but not a systematic FAQ, so 2. Known limitations are explicitly disclosed: encrypted bundles, withheld test setup/solutions, and training-contamination risk, so 2. License is the full Apache-2.0 text, so 3. Versioning/changelog is only a version string in pyproject with no CHANGELOG, so 1. Maintenance responsibility has maintainers and contribution guidance, but publisher identity is unverified, so 2.
Output usability is adequate: world.execute, evaluate().report(), and leaderboard pack/unpack produce actionable artifacts, so 2. Marginal value is clear as a reproducible interactive-coding agent benchmark with a controllable world, so 2. Cost-benefit is reasonable but non-trivial: data download, bundle unpacking, and verification runs are required in exchange for standardized evaluation, so 2.
claim_traceability: README claims (paper, website, leaderboard, CI badges) trace to concrete files and links, so 2. cross_source_corroboration: README, pyproject, CI workflows, and tests are broadly consistent, but the core implementation is inside encrypted bundles and cannot be cross-checked, so 1. fact_inference_separation: the docs distinguish released from withheld content, but some capability claims (sandbox safety, no-consequence execution) are assertions rather than verifiable facts, so 1.
- Core implementation, apps, and tests ship as encrypted .bundle files, so a static review cannot verify real permission boundaries, sandbox isolation strength, or sensitive-data handling; validate in an isolated environment before deployment.
- The supervisor app can expose passwords, addresses, and payment cards, yet no explicit redaction, minimization, or access-audit policy is visible; harden before connecting real data.
- Version is 0.2.0.dev0 marked Alpha with no CHANGELOG, so interfaces and data formats may change; pin versions for production use.
- Publisher identity is unverified; maintenance responsibility and update paths rest only on in-repo maintainer metadata and must not be treated as evidence of reliability or safety.
What does this agent do, and when should you use it?
AppWorld is an ACL'24 Best Resource Paper from the Stony Brook NLP group, packaged as both an execution environment and a benchmark. The AppWorld Engine implements 9 day-to-day apps (Amazon, Spotify, Venmo, and more) over 457 APIs and 100+ database tables, populated with the digital activities of roughly 100 related simulated people. The AppWorld Benchmark adds natural, diverse, challenging autonomous tasks, each defined by a Supervisor, an Instruction, and an Initial State. An agent solves a task by interactively writing code inside an IPython-based shell and calling APIs, then signals completion through the Supervisor app's complete_task API. Evaluation is database-state-based rather than process-based, producing TGC/SGC aggregates plus per-task reports. Tasks are split into train, dev, test_normal, and test_challenge, and the package ships most implementation code inside encrypted .bundle files to reduce benchmark contamination.
AppWorld loads the apps and database state for a task_id, sets the task date, and exposes a stateful IPython shell through world.execute(...). Inside it, APIs can be called functionally as apis.{app_name}.{api_name}(**parameters) or via the requester object in REST form, and variables from earlier executions (such as an access_token) persist across calls like a Jupyter notebook. The environment supports save_state()/load_state() checkpointing and, by default, enforces raise_on_unsafe_syntax and raise_on_unsafe_execution guards against destructive modules and functions. Every execution and API call is logged to logs/environment_io.md and logs/api_calls.jsonl, and final database deltas are saved as jsonl under the experiment output directory. Running appworld evaluate executes database-state unit tests and writes aggregate (tgc, sgc, num_tasks, num_scenarios) and per-task pass/fail reports with assertion traces. A newer MCP layer exposes the same APIs as MCP tools via appworld serve mcp http or stdio, reachable from GUI clients and agent frameworks through world.mcp.call_tool or the familiar world.execute path.
- Agent research labs that need a standardized, stateful, reproducible benchmark for function calling and interactive coding, developing on train/dev and reporting only aggregate test scores.
- Teams doing multi-step RL or agent training that want to parallelize worlds or use save_state/load_state checkpoints for rollouts.
- Developers evaluating terminal coding agents such as Codex or Gemini, wiring them into AppWorld through the MCP server and the evaluating_terminal_agents guide.
- App or API designers who want to stress-test agent affordances by adding new apps and task generators using the bundled guides.
- Instructors and demo presenters who want an interactive walkthrough via appworld play alongside the task and API explorers.
What are this agent's strengths and limitations?
- Evaluation is database-state-based rather than process-based, so agents can write Python, use a different HTTP client, or even another language, as long as they can make HTTP calls.
- By default it runs FastAPI's TestClient in-process, so no server needs to be managed; first task load takes 4-5 seconds and subsequent loads average under 0.5 seconds.
- Both an environment server and an API server can be served over HTTP, optionally with --docker so the sandbox only sees experiments/outputs and data directories.
- Native MCP support exposes all AppWorld APIs over a standard protocol with HTTP and STDIO transports, and existing agents can toggle MCP proxying with about one line of code.
- The train/dev/test_normal/test_challenge split comes with explicit agent-development restrictions that reduce the risk of contamination and unfair comparisons.
- Most implementation code (apps, tests, data, task generation) lives in encrypted .bundle files and is not browsable in plain text on GitHub, so local unpacking via appworld install --repo is required.
- For test_normal and test_challenge, only evaluation programs are released; the initial-state setup and official solutions are withheld, so task construction cannot be fully reproduced.
- The maintainers explicitly ask that code or data extracted from .bundle files not be posted online in plain text or images, which constrains redistribution.
- It requires Python 3.11+, and source installs add git LFS plus an extra install --repo step; Docker isolation needs a Docker daemon (or Podman).
- Integrating external MCP agent frameworks bypasses world.execute, so you must call world.save() yourself, and one API server process handles only one task state at a time, requiring separate ports for parallel agents.
How do you install or deploy this agent?
Python 3.11+ is required; a conda environment works well: conda create -n appworld python=3.11.0 -y && conda activate appworld. Install the package with pip install appworld, then run appworld install to unpack the encrypted .bundle code in site-packages, and finally appworld download data. To install from source, run git lfs install first (some files are tracked in Git LFS), then git clone https://github.com/StonyBrookNLP/appworld, pip install -e ., and appworld install --repo to unpack bundles in the current directory. Data lands in the APPWORLD_ROOT, which defaults to the current directory and can be overridden with --root, the APPWORLD_ROOT environment variable, or a .env file. Validate the setup with appworld verify tests and appworld verify tasks. If the appworld command is missing, substitute python -m appworld.cli.
How do you use this agent?
Minimal flow: after installation, use from appworld import AppWorld, load_task_ids, iterate over load_task_ids("test_challenge"), and for each task open with AppWorld(task_id=task_id, experiment_name="sample") as world. Read world.task.instruction, call APIs such as apis.spotify.login(...) via world.execute, and finish by calling apis.supervisor.complete_task() (optionally with an answer). To use bundled agents instead, install appworld-agents and run appworld run auto --agent-name {AGENT_NAME} --model-name {MODEL_NAME} --dataset-name test_challenge. Evaluate with appworld evaluate {experiment_name} {dataset_name}; overall reports go to experiments/outputs/{experiment_name}/evaluations/ and per-task reports to each task's evaluation/report.md. For MCP, start appworld serve apis --port 9000, then appworld serve mcp http --remote-apis-url http://localhost:9000 --port 10000 (or the stdio variant), and generate client config with python scripts/generate_mcp_config.py. When running multiple agents, give each its own API server or port so task state does not collide.