OSWorld
Benchmark multimodal computer-use agents on open-ended tasks in real desktop environments.
Per-dimension scores and reasoning
Evidence shows: the repository does not provide explicit mechanisms for permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. README mentions Google account and proxy configuration but does not explain how data is processed or protected. Therefore, all trust criteria score 0.
Evidence shows: multiple test files exist (e.g., test_gpt54_infeasible.py, test_maestro_minimax_provider.py, test_muse_spark_infeasible_keywords.py), indicating some consistency testing. Dependencies are listed in pyproject.toml and requirements.txt but without version pinning or integrity checks. Failure messages are mentioned in README (e.g., Docker cleanup command) but not systematically. Thus, self-consistency, dependency availability, and failure messages each score 1.
Evidence shows: README details multiple deployment scenarios (VMware, VirtualBox, Docker, Modal, Daytona, AWS) and provides quick start and experiment guides, so audience and scenarios score 2. Capability boundaries are partially described (e.g., need for Google account and proxy) but not all limitations are listed, so capability boundaries score 1. Trigger precision is addressed with command-line examples but not detailed trigger conditions, so score 1. Environment fit is supported across platforms but without detailed system requirements, so score 2.
Evidence shows: README is well-structured with sections for installation, quick start, experiments, evaluation, FAQ, etc., so information architecture scores 2. Installation notes are detailed, including dependency installation and platform-specific steps, score 2. Naming stability is defined in pyproject.toml but no naming convention documentation, score 1. Examples and FAQ provide multiple example commands and common questions, score 2. Known limitations are partially mentioned (e.g., macOS no KVM) but not comprehensive, score 1. License is Apache-2.0, score 2. Versioning and changelog: README has update log but no formal CHANGELOG, score 1. Maintenance responsibility: maintainer emails listed but scope not clear, score 1.
Evidence shows: output usability is addressed with results saved as screenshots, actions, and videos, and show_result.py script, score 2. Marginal value is high as a benchmark framework with multiple model and platform support, score 2. Cost-benefit: no cost estimates or performance data provided, score 1.
Evidence shows: claim traceability is partial with paper and website references but no specific data sources, score 1. Cross-source corroboration is limited to internal tests, no external validation, score 1. Fact-inference separation is not explicit; README mixes facts and inferences without clear labeling, score 1.
- The repository lacks permission management or user confirmation mechanisms; running agents may execute arbitrary actions, use with caution.
- Dependencies are not version-pinned, posing supply chain risks.
- README mentions Google account and proxy configuration but does not explain data privacy protections.
- No rollback mechanism is provided; erroneous actions may have irreversible consequences.
What does this agent do, and when should you use it?
OSWorld is a benchmark and execution environment for multimodal agents handling open-ended tasks in real computer environments. Its `DesktopEnv` runs Ubuntu or Windows virtual machines through providers including VMware, VirtualBox, Docker, Modal, Daytona, and AWS. After implementing the agent interface, users run tasks with `run.py` or `scripts/python/run_multienv.py`; the documented baseline example uses screenshot observations and GPT-4o. Runs write screenshots, actions, and task-completion videos to a result directory, while `show_result.py` reports success statistics. The repository also includes manual task examination and a process for verified public leaderboard evaluation.
It installs the environment, initializes DesktopEnv, and selects a provider_name plus a VM path or provider-specific configuration. An agent receives observations such as screenshot and performs actions on tasks in evaluation_examples; run.py runs one environment and scripts/python/run_multienv.py runs multiple environments in parallel. The run stores screenshots, actions, and videos in result_dir, then show_result.py reports domain, category, and overall success rates; --detailed prints domain-level score/total values. scripts/python/manual_examine.py can manually execute, record, and verify a selected domain and example ID.
- A research team comparing screenshot-driven computer-use models on a shared task suite in Ubuntu or Windows virtual machines.
- An agent developer who has implemented the OSWorld agent interface and needs a single-environment regression evaluation with
run.py. - An evaluation engineer who needs parallel desktop-task runs and can use Docker, AWS, or another documented provider.
- A maintainer investigating a problematic LibreOffice or other domain task by reproducing and recording it with
manual_examine.py. - An institution seeking a maintainer-verified result for the public leaderboard.
What are this agent's strengths and limitations?
- Executes agents in real desktop virtual machines instead of offering only static examples or text-only evaluation.
- Uses one
DesktopEnvacross VMware, VirtualBox, Docker, Modal, Daytona, and AWS providers. - Supports single-environment and parallel multi-environment runs, preserving screenshots, actions, and videos for trajectory review.
- Includes domain, category, and overall reporting in
show_result.py, plus a manual tool for inspecting individual tasks.
- Local operation requires Python plus virtualization software or Docker; the Docker route is recommended for hosts with KVM support, while macOS hosts generally do not support KVM.
- The documentation warns that VirtualBox may have limited parallelism and limited macOS-on-Apple-chip support.
- Some tasks require Google account access, OAuth2.0 setup, or proxy configuration; missing setup can cause task failures and lower scores.
- Verified public leaderboard evaluation is not self-service: it requires arranging a run with maintainers and supplying a disclosable implementation and report, or sharing monitoring data and trajectories from a trusted institution.
How do you install or deploy this agent?
Use Python 3.10 or later, then run:
git clone https://github.com/xlang-ai/OSWorld
cd OSWorld
pip install -r requirements.txtFor desktop or bare-metal use, install VMware Workstation Pro; use VMware Fusion on Apple Silicon, configure vmrun, and verify it with vmrun -T ws list. VirtualBox is an optional alternative. To install only the environment without benchmark tasks, run pip install desktop-env. For the Docker provider, the README recommends checking KVM support on Linux with egrep -c '(vmx|svm)' /proc/cpuinfo and installing Docker.
How do you use this agent?
For a minimal check, run python quickstart.py. With a VMware VM path, run python quickstart.py --provider_name vmware --path_to_vm "path/to/your/vm.vmx". Before the documented GPT-4o baseline, set export OPENAI_API_KEY='changeme', then run python run.py --provider_name vmware --path_to_vm Ubuntu/Ubuntu.vmx --headless --observation_type screenshot --model gpt-4o --sleep_after_execution 3 --max_steps 15 --result_dir ./results --client_password password. The Docker parallel example uses scripts/python/run_multienv.py with --num_envs 10; inspect detailed results with python show_result.py --detailed.
How does this agent compare with similar options?
VMware is the primary path described for desktops, laptops, and bare-metal machines; VirtualBox is an alternative but may have weaker parallelism and Apple-chip macOS support. Docker targets servers or users avoiding VMware/VirtualBox and is recommended with KVM; because macOS hosts generally lack KVM support, the documentation advises VMware on macOS.
FAQ
Can I evaluate my own agent?
run.py or a parallel run script, and execute the appropriate benchmark command.What credentials does the GPT-4o baseline need?
OPENAI_API_KEY; OPENAI_BASE_URL can point to a custom OpenAI-compatible endpoint. Some tasks additionally need a Google account and OAuth2.0 configuration.What commonly causes failures or lower scores?
What does a run save?
result_dir is specified, the run saves screenshots, agent actions, and videos of task completion. show_result.py summarizes the scores.