Windows Agent Arena
Benchmark multimodal desktop agents in a reproducible Windows 11 environment, locally or at Azure ML scale.
Per-dimension scores and reasoning
Evidence shows: repository from Microsoft official organization, but no specific implementation or documentation for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. Deductions: these criteria are not supported in static review, hence score 0.
Evidence shows: README and config files provide installation and run steps, but no documentation for error handling or failure messages. Deductions: self-consistency partially supported (steps consistent), dependency availability partially supported (dependencies listed), but failure messages missing.
Evidence shows: README clearly targets researchers and developers, provides local and Azure deployment scenarios, and explains capability boundaries (e.g., BYOA). Deductions: trigger precision (e.g., command parameters) partially supported, but not all boundary conditions detailed.
Evidence shows: README has clear structure, includes installation notes, examples, FAQ, known limitations (e.g., KVM requirement), MIT license, version update log (updates section), and maintenance responsibility (contribution guide). Deductions: naming stability not explicit, version changelog incomplete.
Evidence shows: README provides output result display method (show_results.py), and explains marginal value (e.g., parallelization) and cost-benefit (cost table in FAQ). Deductions: these are static descriptions, not verified by actual runs.
Evidence shows: README cites paper and project page, providing traceable claims. Deductions: cross-source corroboration limited, fact-inference separation unclear.
- Static review cannot verify actual runtime performance; all claims about performance, cost, and security need independent verification.
- The repository does not provide security details like least privilege or data flow transparency; users must assess risks themselves during deployment.
- Availability and security of dependencies (e.g., Docker, Azure) need user confirmation.
What does this agent do, and when should you use it?
Windows Agent Arena (WAA) is a Windows platform for testing and benchmarking multimodal desktop AI agents. It is built around a Docker-hosted Windows 11 VM and a golden snapshot; a Python server inside the VM receives and executes agent commands. The repository includes the Navi baseline agent, with screen-element detection modes using Omniparser, accessibility trees, WebParse, GroundingDINO, and TesseractOCR. Evaluations can run locally on WSL or Linux, or be parallelized through Azure Machine Learning compute instances. Local results can be displayed with `show_results.py`, while downloaded Azure outputs can be summarized as a Markdown table with `show_azure.py`.
It builds windowsarena/winarena:latest with scripts/build-container-image.sh, then uses scripts/run-local.sh --prepare-image true to prepare a Windows 11 golden image containing required programs and the Windows-side Python command server. Running ./run-local.sh evaluates Navi on benchmark tasks; --som-origin supports mixed-omni, omni, oss, a11y, and mixed-oss, while --a11y-backend supports uia and win32. For cloud runs, scripts/run_azure.py reads experiments.json, creates Azure Compute Instance VMs for unfinished experiments, launches one ML Training Job per VM, and disposes the VMs after completion. Bring-your-own agents belong under src/win-arena-container/client/mm_agents and must implement predict() and reset() in agent.py.
- A desktop-agent researcher who needs repeatable comparisons across Windows benchmark tasks and screen-understanding configurations.
- A developer with Docker and WSL or Linux who wants to run the Navi baseline on a local Windows VM.
- An Azure ML team that needs to distribute a full benchmark across multiple workers to reduce turnaround time.
- An agent developer who wants to plug a custom desktop agent into an existing evaluation environment by implementing
predict()andreset(). - A research group comparing Omniparser, accessibility-tree, and OSS detection combinations on Windows GUI tasks.
What are this agent's strengths and limitations?
- Provides a repeatable Windows 11 evaluation environment through a golden image, Docker container, and an in-VM command server.
- Supports Azure ML multi-worker execution; the documentation describes reducing results for hundreds of tasks from days to minutes.
- Navi exposes several concrete screen-understanding combinations, including a mixed Omniparser and UIA accessibility-tree mode.
- Supports bring-your-own agents through a documented
agent.pycontract requiringpredict()andreset().
- Initial local setup requires an approximately 6GB Windows 11 ISO and produces an approximately 30GB golden image; automated preparation takes about 20 minutes.
- Operation depends on Docker, Python 3.9, OpenAI or Azure OpenAI credentials, and a local WSL or Linux deployment path.
- Azure parallel deployment requires an Azure subscription, Azure ML workspace, storage account, and compute that supports nested virtualization.
- The documentation warns that disabling KVM acceleration causes performance issues and is not recommended for local benchmark runs.
How do you install or deploy this agent?
Prerequisites are a running Docker daemon, Python 3.9, and an OpenAI or Azure OpenAI API key. Clone and install dependencies: git clone https://github.com/microsoft/WindowsAgentArena.git, cd WindowsAgentArena, and pip install -r requirements.txt. Create config.json at the repository root with OPENAI_API_KEY, or with AZURE_API_KEY and AZURE_ENDPOINT. Run docker pull windowsarena/winarena-base:latest, then from scripts run ./build-container-image.sh. Download the Windows 11 Enterprise Evaluation ISO (90-day, English/United States), rename it to setup.iso, place it in src/win-arena-container/vm/image, and run ./run-local.sh --prepare-image true to prepare the golden image.
How do you use this agent?
After preparing the golden image, run ./run-local.sh from scripts; the running Windows VM is visible at http://localhost:8006. For the configuration identified as best-performing in the documentation, run ./run-local.sh --gpu-enabled true --som-origin mixed-omni --a11y-backend uia. When the local run finishes, go to src/win-arena-container/client and run python show_results.py --result_dir <path_to_results_folder>. For Azure parallel execution, add AZURE_SUBSCRIPTION_ID, AZURE_ML_RESOURCE_GROUP, and AZURE_ML_WORKSPACE_NAME to config.json, configure experiments.json, then run python run_azure.py --experiments_json "experiments.json".
How does this agent compare with similar options?
WAA uses the original benchmark task framework from OSWorld and Docker infrastructure from Dockur. Its distinct documented role is a Windows-focused, scalable desktop-agent evaluation path with Azure ML parallel execution.
FAQ
What resources are needed for the first local run?
--ram-size and --cpu-cores.Is Azure required?
Which model credentials are required?
gpt-4-1106-vision-preview.Can I evaluate my own agent?
src/win-arena-container/client/mm_agents and ensure its agent.py implements predict() and reset().