OSWorld 2.1 Computer-Use Agent Benchmark

least_privilege: README states some tasks require sudo inside the VM and default credentials are user/osworld-public-evaluation; this is a known benchmark design, but no least-privilege configuration or isolation guidance is provided, score 1. user_confirmation: the setup-osworld prompt asks before cloud spend, DNS, SSH, or secret steps, but this is an example prompt rather than an enforced mechanism, and evaluation itself consumes cloud resources, score 1. data_flow_transparency: README explains gated Hugging Face task/asset distribution and OSWORLD_FILE_BASE_URL, but does not systematically describe screenshots, trajectories, or model API data flows, score 1. sensitive_data_handling: sample scripts require AWS keys, Anthropic API key, GitLab private token, etc.; README only warns that a shared GitLab token is risky and offers no secret-management or redaction scheme, score 1. dependency_security: pyproject.toml has a very large dependency set, mostly unpinned, with only a few ~= or == constraints, and no lockfile or vulnerability-scan evidence, score 1. external_effects: evaluation creates cloud VMs, downloads images, and calls external model APIs; README gives cost/resource warnings but no automatic cleanup or blast-radius control, score 1. rollback: README offers version switching, backing up docker_vm_data, and --clean, but no evaluation-failure or state rollback mechanism, score 1. source_attribution: LICENSE carries XLANG NLP Lab copyright, and README provides paper citation, acknowledgements, and data-source links, score 2.

self_consistency: tests test_release_name_is_consistent_across_components and test_release_source_resolution verify release-manifest component tags, and README warns against mixing releases, score 2. dependency_availability: dependencies come from PyPI, gated Hugging Face datasets, Docker images, AWS AMI, and a local packages/osworld-ffmpeg path; some require manual approval or private credentials, so availability has external dependency risk, score 1. failure_messages: tests show ClaudeCodeExecutionError with error_kind and cc_ask returning structured outcome on error results, giving reasonably clear failure information, score 2.

audience_and_scenarios: README targets researchers, evaluators, and agent developers, covering Docker/AWS scenarios, score 2. capability_boundaries: it clearly states only Docker and AWS images are provided, other providers require migration, and main is development code, score 2. trigger_precision: setup-osworld and migrate-osworld-agent skills are triggered by natural-language prompts with no precise trigger conditions or parameter validation, score 1. environment_fit: requires Python >=3.12, uv, KVM, cloud credentials, self-hosted websites, and GitLab; requirements are clear but heavy, score 2.

information_architecture: README is well structured with updates, version matrix, setup, evaluation, FAQ, and citation, score 2. install_notes: provides uv sync --frozen, task/asset downloads, and provider setup steps, score 2. naming_stability: release tags such as osworld-v2.1 are consistent and tested, score 2. examples_and_faq: includes multi-env Claude, Qwen, and manual-examination examples plus FAQ, score 2. known_limitations: README mentions possible task failures from proxy/network issues and that main is development code, but lacks a systematic known-limitations list, score 1. license: full Apache-2.0 text is present and pyproject.toml declares it consistently, score 3. versioning_changelog: benchmark_releases manifests, version matrix, and update log are present, score 3. maintenance_responsibility: README lists current maintainer emails and a public evaluation process, but no long-term maintenance commitment or response-time expectation, score 2.

output_usability: evaluation outputs include result directories, a trajectory viewer, and manual-examination scripts, but require environment setup, score 2. marginal_value: as a long-horizon computer-use agent benchmark with gated tasks/assets and multi-provider support, it has clear value for the evaluation ecosystem, score 2. cost_benefit: running requires cloud VMs, model APIs, self-hosted websites, and GitLab, imposing high cost and operational burden, score 1.

claim_traceability: README claims map to benchmark_releases manifests and test assertions, but some claims such as 'bug-fix release' are not elaborated, score 2. cross_source_corroboration: README, pyproject.toml, LICENSE, and tests corroborate version, license, and dependency-source claims, score 2. fact_inference_separation: README is mostly operational instructions and does not clearly separate verified facts from inference, and test coverage is limited, score 1.