OSWorld
Benchmark multimodal computer-use agents on open-ended tasks in real desktop environments.
- Source repo
- xlang-ai/OSWorld
- Stars
- ★ 3.3k
- Last updated
- 3d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 33/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A research team comparing screenshot-driven computer-use models on a shared task suite in Ubuntu or Windows virtual machines.
- Not a fit if
- macOS hosts: the Docker route needs KVM, which macOS generally does not support
- Teams wanting self-service leaderboard listing: verified runs require maintainer scheduling and disclosure
- Source review
- 33/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
OSWorld is a benchmark and execution environment for multimodal agents handling open-ended tasks in real computer environments. Its DesktopEnv runs Ubuntu or Windows virtual machines through providers including VMware, VirtualBox, Docker, Modal, Daytona, and AWS. After implementing the agent interface, users run tasks with run.py or scripts/python/run_multienv.py; the documented baseline example uses screenshot observations and GPT-4o. Runs write screenshots, actions, and task-completion videos to a result directory, while show_result.py reports success statistics. The repository also includes manual task examination and a process for verified public leaderboard evaluation.
It installs the environment, initializes DesktopEnv, and selects a provider_name plus a VM path or provider-specific configuration. An agent receives observations such as screenshot and performs actions on tasks in evaluation_examples; run.py runs one environment and scripts/python/run_multienv.py runs multiple environments in parallel. The run stores screenshots, actions, and videos in result_dir, then show_result.py reports domain, category, and overall success rates; --detailed prints domain-level score/total values. scripts/python/manual_examine.py can manually execute, record, and verify a selected domain and example ID.
- A research team comparing screenshot-driven computer-use models on a shared task suite in Ubuntu or Windows virtual machines.
- An agent developer who has implemented the OSWorld agent interface and needs a single-environment regression evaluation with
run.py. - An evaluation engineer who needs parallel desktop-task runs and can use Docker, AWS, or another documented provider.
- A maintainer investigating a problematic LibreOffice or other domain task by reproducing and recording it with
manual_examine.py. - An institution seeking a maintainer-verified result for the public leaderboard.
How do you install or deploy this agent?
Use Python 3.10 or later, then run:
git clone https://github.com/xlang-ai/OSWorld
cd OSWorld
pip install -r requirements.txtFor desktop or bare-metal use, install VMware Workstation Pro; use VMware Fusion on Apple Silicon, configure vmrun, and verify it with vmrun -T ws list. VirtualBox is an optional alternative. To install only the environment without benchmark tasks, run pip install desktop-env. For the Docker provider, the README recommends checking KVM support on Linux with egrep -c '(vmx|svm)' /proc/cpuinfo and installing Docker.
How do you use this agent?
For a minimal check, run python quickstart.py. With a VMware VM path, run python quickstart.py --provider_name vmware --path_to_vm "path/to/your/vm.vmx". Before the documented GPT-4o baseline, set export OPENAI_API_KEY='changeme', then run python run.py --provider_name vmware --path_to_vm Ubuntu/Ubuntu.vmx --headless --observation_type screenshot --model gpt-4o --sleep_after_execution 3 --max_steps 15 --result_dir ./results --client_password password. The Docker parallel example uses scripts/python/run_multienv.py with --num_envs 10; inspect detailed results with python show_result.py --detailed.
What are this agent's strengths and limitations?
- Executes agents in real desktop virtual machines instead of offering only static examples or text-only evaluation.
- Uses one
DesktopEnvacross VMware, VirtualBox, Docker, Modal, Daytona, and AWS providers. - Supports single-environment and parallel multi-environment runs, preserving screenshots, actions, and videos for trajectory review.
- Includes domain, category, and overall reporting in
show_result.py, plus a manual tool for inspecting individual tasks.
- Local operation requires Python plus virtualization software or Docker; the Docker route is recommended for hosts with KVM support, while macOS hosts generally do not support KVM.
- The documentation warns that VirtualBox may have limited parallelism and limited macOS-on-Apple-chip support.
- Some tasks require Google account access, OAuth2.0 setup, or proxy configuration; missing setup can cause task failures and lower scores.
- Verified public leaderboard evaluation is not self-service: it requires arranging a run with maintainers and supplying a disclosable implementation and report, or sharing monitoring data and trajectories from a trusted institution.
How does this agent compare with similar options?
VMware is the primary path described for desktops, laptops, and bare-metal machines; VirtualBox is an alternative but may have weaker parallelism and Apple-chip macOS support. Docker targets servers or users avoiding VMware/VirtualBox and is recommended with KVM; because macOS hosts generally lack KVM support, the documentation advises VMware on macOS.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| OSWorld This agent | 33 · Major gaps | CLIFree + model costs | ★ 3.3k | 3d ago | Python | OpenAI API |
| Windows Agent Arena | 35 · Major gaps | CLIFree + model costs | ★ 915 | 10d ago | Python | OpenAI API |
| ClawBench Browser Agent Benchmark | 48 · Major gaps | CLIFree + model costs | ★ 1.2k | 1d ago | Python | Codex · Claude Code |
| Telegram Search | 58 · Major gaps | Self-hosted serviceFree + model costs | ★ 4.1k | 1d ago | TypeScript | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: the repository does not provide explicit mechanisms for permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. README mentions Google account and proxy configuration but does not explain how data is processed or protected. Therefore, all trust criteria score 0.
Evidence shows: multiple test files exist (e.g., test_gpt54_infeasible.py, test_maestro_minimax_provider.py, test_muse_spark_infeasible_keywords.py), indicating some consistency testing. Dependencies are listed in pyproject.toml and requirements.txt but without version pinning or integrity checks. Failure messages are mentioned in README (e.g., Docker cleanup command) but not systematically. Thus, self-consistency, dependency availability, and failure messages each score 1.
Evidence shows: README details multiple deployment scenarios (VMware, VirtualBox, Docker, Modal, Daytona, AWS) and provides quick start and experiment guides, so audience and scenarios score 2. Capability boundaries are partially described (e.g., need for Google account and proxy) but not all limitations are listed, so capability boundaries score 1. Trigger precision is addressed with command-line examples but not detailed trigger conditions, so score 1. Environment fit is supported across platforms but without detailed system requirements, so score 2.
Evidence shows: README is well-structured with sections for installation, quick start, experiments, evaluation, FAQ, etc., so information architecture scores 2. Installation notes are detailed, including dependency installation and platform-specific steps, score 2. Naming stability is defined in pyproject.toml but no naming convention documentation, score 1. Examples and FAQ provide multiple example commands and common questions, score 2. Known limitations are partially mentioned (e.g., macOS no KVM) but not comprehensive, score 1. License is Apache-2.0, score 2. Versioning and changelog: README has update log but no formal CHANGELOG, score 1. Maintenance responsibility: maintainer emails listed but scope not clear, score 1.
Evidence shows: output usability is addressed with results saved as screenshots, actions, and videos, and show_result.py script, score 2. Marginal value is high as a benchmark framework with multiple model and platform support, score 2. Cost-benefit: no cost estimates or performance data provided, score 1.
Evidence shows: claim traceability is partial with paper and website references but no specific data sources, score 1. Cross-source corroboration is limited to internal tests, no external validation, score 1. Fact-inference separation is not explicit; README mixes facts and inferences without clear labeling, score 1.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository lacks permission management or user confirmation mechanisms; running agents may execute arbitrary actions, use with caution.
- Dependencies are not version-pinned, posing supply chain risks.
- README mentions Google account and proxy configuration but does not explain data privacy protections.
- No rollback mechanism is provided; erroneous actions may have irreversible consequences.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
:package: What’s New? VM + Cloud support! - New VirtualBox to allow free VM snapshots - AWS Support w/ optimized interface mappings and instance types - Fixes for reported annotation bugs - Support new models from Gemini 1.5-Pro, Llama-3, and Qwen - Add more detailed README on proxy and so. Contributors Thanks to all the contributors who have helped with this release! <a href="https://github.com/xlang-ai/OSWorld/graphs/contributors"> <img src="https://contrib.rocks/image?repo=xlang-ai/OSWorld" /> </a>View on GitHub ↗ -
Release notes excerpt
The first release of the [OSWorld](https://os-world.github.io/) environment See our [README](https://github.com/xlang-ai/OSWorld/blob/main/README.md) for details! Contributors Thanks to all the contributors who have helped with this release! <a href="https://github.com/xlang-ai/OSWorld/graphs/contributors"> <img src="https://contrib.rocks/image?repo=xlang-ai/OSWorld" /> </a>View on GitHub ↗
FAQ
Can I evaluate my own agent?
run.py or a parallel run script, and execute the appropriate benchmark command.What credentials does the GPT-4o baseline need?
OPENAI_API_KEY; OPENAI_BASE_URL can point to a custom OpenAI-compatible endpoint. Some tasks additionally need a Google account and OAuth2.0 configuration.What commonly causes failures or lower scores?
What does a run save?
result_dir is specified, the run saves screenshots, agent actions, and videos of task completion. show_result.py summarizes the scores.