SwiftAutoGUI Desktop Automation
Automate macOS interfaces with Swift or sagui, and let a vision model observe and act toward a task goal.
- Source repo
- NakaokaRei/SwiftAutoGUI
- Stars
- ★ 94
- Last updated
- 18d ago
- License
- MIT
- Primary language
- Swift
- FA score
- 60/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformClaude Code · OpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- macOS developers who need mouse, typing, or keyboard-shortcut automation inside a Swift app can compose the Action API.
- Not a fit if
- Users who need to run it on macOS versions earlier than 27.0
- Teams that need native UI control without Accessibility or Screen Recording permissions
- Source review
- 60/100 · Some gaps
What does this agent do, and when should you use it?
SwiftAutoGUI is a Swift library for macOS automation, with a sagui command-line tool available through Homebrew. It handles mouse and keyboard input, screenshots, image recognition, and AI agents. Its Agent follows an Observe → Think → Act loop, using a vision-capable LLM to inspect the screen and act until it reaches a goal or iteration limit. Developers can compose Action calls in Swift or invoke the Agent API; the CLI also offers deterministic native UI and Chromium CDP commands. The optional SwiftAutoGUIBrowser product connects to an existing Chromium debugging session and keeps browser actions on the browser backend without native input fallback. It requires macOS 27.0+ and Swift 6.2+; native UI automation requires system permissions, and cloud models require provider credentials.
Import SwiftAutoGUI into a Swift program or install sagui with Homebrew. Call Action.leftClick, Action.write, and Action.keyShortcut for individual operations, or execute an array of Actions as a sequence. The Agent uses a vision-capable LLM to observe the screen and act in a loop toward a goal, for example with sagui agent "Open Safari and search for Swift"; the CLI defaults to an on-device Apple model and also supports the OpenAI provider. With screen context enabled, the Agent can target step-local Accessibility identifiers; actions return structured results, and a UI change triggers a fresh observation. For browser automation, add SwiftAutoGUIBrowser, connect to a running Chromium CDP session, observe the page, and act using browser roles, names, or domain constraints.
- macOS developers who need mouse, typing, or keyboard-shortcut automation inside a Swift app can compose the Action API.
- macOS users who want to complete a desktop task from a natural-language goal can run
sagui agentto observe and act on the screen. - Developers automating an existing Chrome, Edge, or other Chromium debugging session can add the optional SwiftAutoGUIBrowser product.
- Claude Code users who want to control native macOS apps or Chromium pages can install the plugin and use its corresponding skills.
- Teams automating browser navigation or clicks can set an allowed-domain list and provide an authorization policy for cross-origin navigation and downloads.
How do you install or deploy this agent?
Native macOS automation requires macOS 27.0+. Add SwiftAutoGUI from Swift Package Manager in Package.swift:
.dependencies: [
.package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
]Add the product to your target dependencies. To install the CLI with Homebrew:
brew install NakaokaRei/tap/saguiFor browser automation, add the optional SwiftAutoGUIBrowser product and start Chromium with a dedicated user-data directory and remote debugging port. Grant Accessibility permission to the terminal or app running native automation; Claude Code native control also requires Screen Recording permission. OpenAI use requires OPENAI_API_KEY.
How do you use this agent?
In a Swift 6.2+ project, import SwiftAutoGUI and configure an Agent with a model and goal:
import SwiftAutoGUI
let backend = AutomationModels.openAI(apiKey: "sk-...", model: "gpt-5.6-sol")
let agent = Agent(model: backend, maxIterations: 15, visionMode: .automatic)
let result = try await agent.run(goal: "Open Safari and search for Swift")
print("Completed: \(result.completed), Steps: \(result.iterationsUsed)")The CLI can run keyboard and mouse commands directly:
sagui key list
sagui key shortcut return
sagui mouse click --x 100 --y 200The CLI defaults to the on-device Apple model. To choose OpenAI, set OPENAI_API_KEY and run:
sagui agent "Open Safari and search for Swift" --provider openaiFor browser automation, first start a Chromium debugging session, then connect to its loopback endpoint with a navigation allowlist. Cross-origin navigation and downloads require a BrowserActionAuthorizing implementation.
What are this agent's strengths and limitations?
- Offers both a Swift library and a
saguiCLI, so teams can embed automation or run it from a terminal. - The Agent supports
always,automatic, andnevervision modes;automaticcan omit screenshots when actionable Accessibility elements are available. - The optional SwiftAutoGUIBrowser product keeps Chromium CDP actions separate from native macOS automation, with no fallback to system input.
- Browser navigation can use a domain allowlist, and stale DOM elements fail safely instead of clicking a saved coordinate.
- Requires macOS 27.0+ and Swift 6.2+, excluding older macOS setups.
- Native UI control requires Accessibility permission; Claude Code native control also needs Screen Recording permission.
- OpenAI use needs an API key, and the README does not state model-call pricing.
- Browser automation depends on a running Chromium remote-debugging session; cross-origin navigation and downloads need an additional authorization implementation.
How does this agent compare with similar options?
The README says the project is inspired by pyautogui and describes SwiftAutoGUIBrowser as an optional Chromium CDP automation product.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| SwiftAutoGUI Desktop Automation This agent | 60 · Some gaps | CLIFree + model costs | ★ 94 | 18d ago | Swift | Claude Code · OpenAI API |
| macOS Harness | 55 · Major gaps | CLIFree + model costs | ★ 1k | 21d ago | Python | Codex · Claude Code |
| Ghost OS | Insufficient evidence | CLIFree | ★ 1.7k | 7mo ago | Swift | Claude Code |
| AIHawk – AI Browser Agent | 59 · Major gaps | MCP serverFree + model costs | ★ 32k | 5d ago | TypeScript | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The README says native automation requires Accessibility and Screen Recording permissions and distinguishes browser-only CDP use. Browser navigation can be restricted by domain, cross-origin navigation and downloads are denied without authorization, and the browser backend does not fall back to native input. This supports scores of 2 for least privilege and external effects. However, the native Agent can act autonomously and the materials do not describe general per-step confirmation, so user_confirmation scores 1. The README explains OpenAI key input, PCC eligibility, and that cloud fallback must be explicitly selected, supporting data_flow_transparency at 2. It gives no key-protection or screenshot and screen-content handling policy, so sensitive_data_handling scores 1. Dependency security scores 1: workflows and a pinned Apple Chat Completions adapter are mentioned, but there is no dependency risk review or mitigation evidence. UI changes stop the remaining action batch and trigger a new observation, and stale elements fail safely, but no general recovery mechanism is shown; rollback scores 1. The README names an inspiration and a contributor, but source attribution and publisher maintenance identity are limited, so source_attribution scores 1.
The README gives concrete details about the vision-mode default, removal of the old reasoning-effort option, provider conditions, and re-observation after UI changes. Some cross-product consistency cannot be established from the supplied materials, so self_consistency scores 2. README instructions cover SPM, Homebrew, and the optional browser product, but the SPM example uses the master branch and there is no evidence of dependency availability or locking policy; dependency_availability scores 1. Structured action results, safe failure for stale elements, and workflows that capture test failure details support failure_messages at 2; the score is limited because a complete error reference or user-facing troubleshooting guide is not supplied.
The materials cover a Swift library, CLI, Claude Code plugin, native macOS automation, and Chromium CDP, supporting audience_and_scenarios at 2; they do not provide a fuller audience or selection guide. The browser-only backend explicitly rejects native actions, requires authorization for cross-origin navigation and downloads, and distinguishes local from cloud models, earning capability_boundaries 3. Goals are explicit command inputs and provider and vision-mode options are documented, so trigger_precision scores 2; finer trigger conditions or interaction-confirmation settings are not shown. macOS 27 and Swift 6.2 are specified, along with PCC eligibility and the dedicated-profile requirement for Chrome 136+, supporting environment_fit at 2; the supported environment is specific but narrow.
The README is organized into requirements, installation, examples, browser backend, Agent, and plugin sections, and links to DocC, supporting information_architecture at 2; full API detail requires the external documentation. Installation notes cover SPM, Homebrew, permissions, and browser setup, earning install_notes 2; coverage of all platforms and troubleshooting is not shown. SwiftAutoGUI, SwiftAutoGUIBrowser, and sagui are used consistently in the main examples, so naming_stability scores 2; no complete naming policy is provided. Examples cover basic actions, the Agent, browser use, and CLI commands, supporting examples_and_faq at 2; evidence of a FAQ is limited. Browser constraints, permission requirements, PCC conditions, and a removed option are documented, earning known_limitations 2; no comprehensive limitations inventory is shown. The LICENSE file contains the complete MIT terms, so license scores 3. The supplied revision-pinned materials contain no release policy or changelog, so versioning_changelog scores 1. One contributor is listed, but the publisher is unverified and maintenance ownership or update commitments are unclear, so maintenance_responsibility scores 1.
The Agent combines screen observation, Accessibility elements, action execution, and result reporting; the CLI also offers direct commands for deterministic operations, supporting output_usability at 2. Native UI and Chromium semantic automation, structured action results, and an optional on-device model provide clear utility, so marginal_value scores 2; benefits depend on granting system permissions and having a suitable model and task. Cost_benefit scores 2: the README documents an on-device default and cloud options and makes browser support optional, but provides no quantified cost or benefit evidence.
The README gives concrete steps for installation, permissions, provider selection, and browser policy, while workflows list build, test, and documentation jobs, supporting claim_traceability at 2; the supplied materials do not include actual run results. The README, LICENSE, and workflows independently support installation, licensing, and CI configuration, so cross_source_corroboration scores 2; cross-source confirmation of product behavior is limited. The README presents the demo, feature descriptions, limitations, and workflow configuration as distinct material, supporting fact_inference_separation at 2; some feature claims remain README assertions that cannot be independently confirmed from the supplied files.
- The native Agent can control the mouse and keyboard and observe the screen; check the target applications and action scope before granting Accessibility and Screen Recording permissions.
- The materials do not specify complete retention and transmission policies for API keys, screenshots, or screen content; avoid sending sensitive screens when using a cloud provider.
- The installation example uses the master branch; the supplied materials do not show a pinned release or dependency security review.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * Publish BrowserAgent documentation on GitHub Pages by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/126 * chore: migrate pbxproj to xcproj by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/127 * Unify AI automation with Foundation Models language models by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/129 Full Changelog: https://github.com/NakaokaRei/SwiftAutoGUI/compare/0.30.0...1.0.0View on GitHub ↗ -
Release notes excerpt
What's Changed * Add dedicated Chromium CDP browser backend by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/123 * Add deterministic browser commands and Claude Code Skill by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/124 * Add real-browser integration CI by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/125 Full Changelog: https://github.com/NakaokaRei/SwiftAutoGUI/compare/0.29.0...0.30.0View on GitHub ↗ -
Release notes excerpt
What's Changed * Add semantic element targeting and verified Agent execution by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/119 * Fix tiled image recognition Metal crash by @NakaokaRei in https://github.com/NakaokaRei/SwiftAutoGUI/pull/121 Full Changelog: https://github.com/NakaokaRei/SwiftAutoGUI/compare/0.28.0...0.29.0View on GitHub ↗
FAQ
Can I use the library without installing the CLI?
Which permissions does native UI automation need?
Does the Agent require OpenAI's cloud model?
Can the browser Agent control native macOS apps?
Are cross-origin navigation and downloads enabled by default?
BrowserActionAuthorizing implementation and are denied without one.