Windows Agent Arena
Benchmark multimodal desktop agents in a reproducible Windows 11 environment, locally or at Azure ML scale.
What does this agent do, and when should you use it?
Windows Agent Arena (WAA) is a Windows platform for testing and benchmarking multimodal desktop AI agents. It is built around a Docker-hosted Windows 11 VM and a golden snapshot; a Python server inside the VM receives and executes agent commands. The repository includes the Navi baseline agent, with screen-element detection modes using Omniparser, accessibility trees, WebParse, GroundingDINO, and TesseractOCR. Evaluations can run locally on WSL or Linux, or be parallelized through Azure Machine Learning compute instances. Local results can be displayed with `show_results.py`, while downloaded Azure outputs can be summarized as a Markdown table with `show_azure.py`.
It builds windowsarena/winarena:latest with scripts/build-container-image.sh, then uses scripts/run-local.sh --prepare-image true to prepare a Windows 11 golden image containing required programs and the Windows-side Python command server. Running ./run-local.sh evaluates Navi on benchmark tasks; --som-origin supports mixed-omni, omni, oss, a11y, and mixed-oss, while --a11y-backend supports uia and win32. For cloud runs, scripts/run_azure.py reads experiments.json, creates Azure Compute Instance VMs for unfinished experiments, launches one ML Training Job per VM, and disposes the VMs after completion. Bring-your-own agents belong under src/win-arena-container/client/mm_agents and must implement predict() and reset() in agent.py.
- A desktop-agent researcher who needs repeatable comparisons across Windows benchmark tasks and screen-understanding configurations.
- A developer with Docker and WSL or Linux who wants to run the Navi baseline on a local Windows VM.
- An Azure ML team that needs to distribute a full benchmark across multiple workers to reduce turnaround time.
- An agent developer who wants to plug a custom desktop agent into an existing evaluation environment by implementing `predict()` and `reset()`.
- A research group comparing Omniparser, accessibility-tree, and OSS detection combinations on Windows GUI tasks.
What are this agent's strengths and limitations?
- Provides a repeatable Windows 11 evaluation environment through a golden image, Docker container, and an in-VM command server.
- Supports Azure ML multi-worker execution; the documentation describes reducing results for hundreds of tasks from days to minutes.
- Navi exposes several concrete screen-understanding combinations, including a mixed Omniparser and UIA accessibility-tree mode.
- Supports bring-your-own agents through a documented `agent.py` contract requiring `predict()` and `reset()`.
- Initial local setup requires an approximately 6GB Windows 11 ISO and produces an approximately 30GB golden image; automated preparation takes about 20 minutes.
- Operation depends on Docker, Python 3.9, OpenAI or Azure OpenAI credentials, and a local WSL or Linux deployment path.
- Azure parallel deployment requires an Azure subscription, Azure ML workspace, storage account, and compute that supports nested virtualization.
- The documentation warns that disabling KVM acceleration causes performance issues and is not recommended for local benchmark runs.
How do you install or deploy this agent?
Prerequisites are a running Docker daemon, Python 3.9, and an OpenAI or Azure OpenAI API key. Clone and install dependencies: git clone https://github.com/microsoft/WindowsAgentArena.git, cd WindowsAgentArena, and pip install -r requirements.txt. Create config.json at the repository root with OPENAI_API_KEY, or with AZURE_API_KEY and AZURE_ENDPOINT. Run docker pull windowsarena/winarena-base:latest, then from scripts run ./build-container-image.sh. Download the Windows 11 Enterprise Evaluation ISO (90-day, English/United States), rename it to setup.iso, place it in src/win-arena-container/vm/image, and run ./run-local.sh --prepare-image true to prepare the golden image.
How do you use this agent?
After preparing the golden image, run ./run-local.sh from scripts; the running Windows VM is visible at http://localhost:8006. For the configuration identified as best-performing in the documentation, run ./run-local.sh --gpu-enabled true --som-origin mixed-omni --a11y-backend uia. When the local run finishes, go to src/win-arena-container/client and run python show_results.py --result_dir <path_to_results_folder>. For Azure parallel execution, add AZURE_SUBSCRIPTION_ID, AZURE_ML_RESOURCE_GROUP, and AZURE_ML_WORKSPACE_NAME to config.json, configure experiments.json, then run python run_azure.py --experiments_json "experiments.json".
How does this agent compare with similar options?
WAA uses the original benchmark task framework from OSWorld and Docker infrastructure from Dockur. Its distinct documented role is a Windows-focused, scalable desktop-agent evaluation path with Azure ML parallel execution.