Agent S
A GUI automation framework that turns screenshots and model decisions into computer actions.
- Source repo
- simular-ai/Agent-S
- Stars
- ★ 13k
- Last updated
- 15d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 41/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A developer automating multi-step GUI work on a single-monitor Linux, macOS, or Windows machine.
- Not a fit if
- Users on multi-monitor setups; the agent is designed for single-monitor screens
- Teams unwilling to let the agent run arbitrary Python/Bash with the launching user's permissions
- Source review
- 41/100 · Major gaps 5 safety controls not found
What does this agent do, and when should you use it?
Agent S is an open-source framework for autonomous computer interaction, with Agent S3 presented as the current version in the README. Its documented setup combines a main generation model, an OSWorldACI grounding agent, and an optional LocalEnv. The main model plans actions while the grounding model maps them to executable screen coordinates. It exposes both the agent_s CLI and a Python SDK centered on AgentS3 and OSWorldACI; the SDK example captures a screenshot with PyAutoGUI and executes returned Python action code. The project states support for Linux, macOS, and Windows, but is designed for a single-monitor display. It documents main-model paths for Azure OpenAI, Anthropic, Gemini, OpenRouter, and vLLM, while requiring users to configure a separate grounding endpoint.
A caller supplies a task instruction and a screenshot. In the SDK example, pyautogui.screenshot() captures the display, the image is converted to PNG bytes, and AgentS3.predict(instruction, observation) receives those bytes in observation["screenshot"]. AgentS3 uses the configured generation engine, while OSWorldACI uses a separate grounding engine plus grounding_width and grounding_height to produce executable actions; the example runs the result with exec(action[0]). The agent_s CLI accepts the main-model provider and model plus the grounding provider, URL, model, and coordinate resolution. With --enable_local_env, the agent can use call_code_agent to run Python and Bash locally for file, data, system-automation, or coding tasks.
- A developer automating multi-step GUI work on a single-monitor Linux, macOS, or Windows machine.
- A researcher deploying Agent S3 in OSWorld to evaluate computer-use tasks.
- A team with an OpenAI, Anthropic, Gemini, OpenRouter, or vLLM inference path that needs a separate UI grounding model for screen-coordinate actions.
- An application developer who needs to capture the current screen in Python, send an instruction such as “Close VS Code,” and run the returned action.
- A user in a trusted environment who wants the agent to process CSVs, files, or scripts through the optional local coding environment.
How do you install or deploy this agent?
Install the released package with pip install gui-agents. For editable development after cloning the repository, run pip install -e .. Install Tesseract; the macOS command shown is brew install tesseract, and the README says PyTesseract requires it. Configure at least one main-model credential, for example export OPENAI_API_KEY=<YOUR_API_KEY>; ANTHROPIC_API_KEY and HF_TOKEN are also documented. You must additionally provide a grounding-model provider, URL, model name, and output coordinate resolution.
How do you use this agent?
For UI-TARS-1.5-7B grounding, run: agent_s --provider openai --model gpt-5-2025-08-07 --ground_provider huggingface --ground_url http://localhost:8080 --ground_model ui-tars-1.5-7b --grounding_width 1920 --grounding_height 1080. UI-TARS-72B uses --grounding_width 1000 --grounding_height 1000. For Python integration, construct OSWorldACI(env=local_env, platform=current_platform, engine_params_for_generation=engine_params, engine_params_for_grounding=engine_params_for_grounding), then AgentS3(engine_params, grounding_agent, platform=current_platform), and call agent.predict(). Add --enable_local_env or create LocalEnv() only in a trusted environment, because it enables arbitrary local Python and Bash execution.
What are this agent's strengths and limitations?
- It separates the main reasoning model from the grounding model, allowing a general model to be paired with a dedicated UI coordinate model.
- It offers both the
agent_sCLI and a Python SDK usingAgentS3andOSWorldACI. - The README explicitly documents Linux, macOS, and Windows support, plus an OSWorld deployment path.
- The optional
LocalEnvextends GUI interaction with local code, file, and data operations.
- A working deployment needs a separate grounding provider, endpoint, model, and coordinate resolution; a main-model API key alone is insufficient.
- The documented design assumes a single monitor, so multi-monitor use is outside the stated setup.
- LocalEnv executes arbitrary Python and Bash with the launching user's permissions, and the README limits it to trusted environments.
- Although several main-model providers are listed, the recommended configuration uses a specific main model and UI-TARS grounding model; alternate combinations require user validation.
How does this agent compare with similar options?
The README claims that Agent S2 outperformed OpenAI CUA/Operator and Anthropic Claude 3.7 Sonnet Computer-Use on the benchmarks it names, and that Agent S3 reached 72.6% on OSWorld with Behavior Best-of-N. It does not provide a workflow-, deployment-, or cost-level comparison with those alternatives.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Agent S This agent | 41 · Major gaps | CLIFree + model costs | ★ 13k | 15d ago | Python | OpenAI API · Claude API |
| TARS Multimodal Operator | 30 · Major gaps | CLIFree + model costs | ★ 40k | 28d ago | TypeScript | Claude API |
| Skales | 36 · Major gaps | Desktop appFreemium | ★ 2.2k | 3d ago | — | ChatGPT · OpenAI API · Claude API |
| LingShu | 71 · Some gaps | Desktop appFree + model costs | ★ 11 | 3mo ago | Rust | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence: README explicitly warns that the local coding environment executes arbitrary code and recommends use in trusted environments, showing limited consideration of least privilege (least_privilege=1). However, no user confirmation mechanism (user_confirmation=0), data flow transparency (data_flow_transparency=0), sensitive data handling (sensitive_data_handling=0), or dependency security (dependency_security=0). External effects: README warns about code execution risks but no rollback mechanism (rollback=0). Source attribution: README provides paper citations and author info (source_attribution=1). Deductions: lack of security mechanisms and transparency.
Evidence: README and code examples are consistent, providing CLI and SDK usage, good self-consistency (self_consistency=2). Dependency availability: requirements.txt lists dependencies but no version pinning or integrity checks (dependency_availability=1). Failure messages: test_providers.py checks config errors but not runtime failures (failure_messages=1). Deductions: dependencies not pinned, limited failure handling.
Evidence: README identifies target audience (researchers, developers) and use cases (OSWorld, WindowsAgentArena, AndroidWorld), with multi-platform support (audience_and_scenarios=2). Capability boundaries: mentions single monitor limitation and local env risks (capability_boundaries=2). Trigger precision: detailed CLI parameters but no explicit trigger conditions (trigger_precision=1). Environment fit: supports Linux, macOS, Windows with platform-specific deps (environment_fit=2). Deductions: trigger conditions unclear.
Evidence: README has clear structure with TOC, installation, usage, citations (information_architecture=2). Install notes detailed with pip and extra deps (install_notes=2). Naming stability: version history but no naming conventions (naming_stability=1). Examples and FAQ: CLI and SDK examples but no FAQ (examples_and_faq=2). Known limitations: mentions single monitor and local env risks but not comprehensive (known_limitations=1). License: Apache-2.0 full text (license=3). Versioning/changelog: README updates but no separate CHANGELOG (versioning_changelog=2). Maintenance responsibility: no maintainers or contribution guidelines (maintenance_responsibility=1). Deductions: missing FAQ, full limitations, maintenance info.
Evidence: Output usability: CLI and SDK usage, output is executable code (output_usability=2). Marginal value: claims to surpass human performance but no independent verification (marginal_value=2). Cost-benefit: requires multiple API keys and grounding model, high cost, no cost analysis (cost_benefit=1). Deductions: cost-benefit not quantified.
Evidence: README provides paper links and citations, traceable (claim_traceability=2). Cross-source corroboration: multiple benchmark results but no independent reproduction (cross_source_corroboration=1). Fact-inference separation: README distinguishes results and inferences but some claims not clearly labeled (fact_inference_separation=1). Deductions: lack of independent verification and clear separation.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- The local coding environment executes arbitrary code; use only in trusted environments and consider sandboxing.
- Requires multiple API keys and a grounding model, which can be costly; evaluate budget.
- Dependencies are not pinned, posing supply chain risks.
- No user confirmation mechanism; automated actions may cause unintended changes.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
Full Changelog: https://github.com/simular-ai/Agent-S/compare/v0.3.1...v0.3.2View on GitHub ↗ -
Release notes excerpt
Full Changelog: https://github.com/simular-ai/Agent-S/compare/v0.3.0...v0.3.1View on GitHub ↗ -
Release notes excerpt
What's Changed * Fixed open action for Windows desktop. by @wehos in https://github.com/simular-ai/Agent-S/pull/130 * Implement a Ctrl+C signal handler to pause for debug purposes by @tageniu in https://github.com/simular-ai/Agent-S/pull/126 * S3 🧠🤓🤯 by @alckasoc in https://github.com/simular-ai/Agent-S/pull/142 New Contributors * @wehos made their first contribution in https://github.com/simular-ai/Agent-S/pull/130 * @tageniu made their first contribution in https://github.com/simular-ai/Agent-S/pull/126 …View on GitHub ↗
FAQ
What credentials and services are required?
OPENAI_API_KEY, ANTHROPIC_API_KEY, and HF_TOKEN.Can it execute code on my machine?
--enable_local_env is enabled or LocalEnv is used in the SDK. That mode can run Python and Bash with the same permissions as the user who launched the agent.