Ghost OS
Give MCP agents structured control of native macOS apps and reusable learned workflows.
- Source repo
- ghostwright/ghost-os
- Stars
- ★ 1.7k
- Last updated
- 7mo ago
- License
- MIT
- Primary language
- Swift
- FA score
- 53/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specificClaude Code
- Cost
- Free, no paid service needed
- Setup effort
- Low · running in minutes
- You'll need
- Typical use
- A macOS developer using Claude Code who needs an agent to fill Gmail fields and send a report email.
- Not a fit if
- Teams needing cross-platform desktop automation; runtime is macOS 14+ only
- Environments where Accessibility, Screen Recording, or Input Monitoring permissions cannot be granted
- Setups that must stay fully offline without downloading the local vision model
- Source review
- 53/100 · Major gaps
What does this agent do, and when should you use it?
Ghost OS is a Swift MCP server that gives AI clients computer-use capabilities on a local Mac through stdio. It reads the macOS accessibility tree for structured application and UI-element data, with an optional local ShowUI-2B vision model for cases where accessibility data is insufficient. Its 29 tools cover UI inspection, mouse and keyboard actions, window control, screenshots, visual grounding, and recipe management. It can also observe a user performing a task through a CGEvent tap and return an accessibility-enriched action sequence for a parameterized, replayable JSON recipe. The documented deployment boundary is macOS 14+ and the documented clients include Claude Code, Cursor, VS Code, and other MCP clients.
An MCP client can call ghost_context for the current app, window, URL, focused element, and interactive elements, and use ghost_state, ghost_find, ghost_read, ghost_inspect, or ghost_element_at to inspect the UI. It operates apps with ghost_click, ghost_type, ghost_press, ghost_hotkey, ghost_hover, ghost_long_press, ghost_drag, ghost_scroll, ghost_focus, and ghost_window, while ghost_wait waits for UI changes. When the AX tree cannot adequately describe a web or dynamic interface, ghost_ground and ghost_parse_screen use the local ShowUI-2B model for visual grounding. Workflows can be installed with ghost_recipe_save, examined with ghost_recipe_show, run with parameter substitution through ghost_run, and removed with ghost_recipe_delete. Learning mode uses ghost_learn_start to observe user actions, ghost_learn_stop to return the enriched sequence, and ghost_learn_status to report recording state.
- A macOS developer using Claude Code who needs an agent to fill Gmail fields and send a report email.
- A researcher who repeatedly searches arXiv and downloads papers, and wants to save the completed process as a parameterized recipe.
- An operations user moving between Slack, Finder, and other native Mac applications as part of a recurring desktop workflow.
- A team lead who wants to review the JSON steps of a proven desktop workflow before sharing it with colleagues.
- A user automating a dynamic web app whose target control is not sufficiently exposed through the accessibility tree and needs local visual grounding.
How do you install or deploy this agent?
Ghost OS supports macOS 14+. Install and configure it with Homebrew:
brew install ghostwright/ghost-os/ghost-os
ghost setupAccording to the documentation, ghost setup configures permissions, MCP, recipes, and the vision model. To build from source:
git clone https://github.com/ghostwright/ghost-os.git
cd ghost-os
swift build
.build/debug/ghost setupSource builds require Swift 6.2+. No account or API key requirement is documented. Accessibility and Screen Recording permissions are required; learning mode additionally requires Input Monitoring.
How do you use this agent?
After ghost setup, use the Ghost OS MCP server from an MCP-capable client. Start with ghost_context to inspect the active UI. For example, run an installed recipe with ghost_run recipe:"gmail-send" params:{"recipient":"[email protected]","subject":"Hello","body":"World"}. To learn a workflow, call ghost_learn_start task_description:"send email in Gmail", perform the task manually, call ghost_learn_stop, then save the returned workflow with ghost_recipe_save. Run ghost doctor to check permissions, MCP configuration, recipes, AX-tree access, and vision components.
What are this agent's strengths and limitations?
- Uses the macOS accessibility tree for structured UI data rather than relying solely on screenshots, with a local ShowUI-2B fallback for visual grounding.
- Stores workflows as inspectable JSON recipes and runs them with parameter substitution through
ghost_run. - Covers native-app control, including windows, keyboard and pointer actions, UI reading, and recipe execution rather than browser-only automation.
- Exposes its capabilities through MCP over stdio and explicitly documents Claude Code alongside other MCP clients.
- Its core runtime is macOS 14+, so it is not a drop-in cross-platform desktop automation option.
- Adoption requires Accessibility and Screen Recording permissions; workflow learning also requires Input Monitoring.
- Interfaces with weak accessibility data depend on the local ShowUI-2B vision fallback; the diagnostic example lists the model at 3.0 GB.
- The documented recipe-synthesis example uses Claude, while equivalent configuration and results for other models are not documented.
How does this agent compare with similar options?
The repository compares Ghost OS with Anthropic Computer Use, OpenAI Operator, and OpenClaw. It positions Ghost OS around an accessibility tree plus a local VLM, native macOS app control, and JSON workflow recipes; its table describes Anthropic Computer Use and OpenAI Operator as screenshot-based and OpenClaw as browser-DOM-based.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Ghost OS This agent | 53 · Major gaps | CLIFree | ★ 1.7k | 7mo ago | Swift | Claude Code |
| SwiftAutoGUI Desktop Automation | 46 · Major gaps | CLIFree + model costs | ★ 94 | 21d ago | Swift | Claude Code · OpenAI API |
| Open Computer Use (MCP desktop service) | 79 · Good | CLIFree + model costs | ★ 2.9k | 1d ago | Swift | Codex · Claude Code |
| Clawd Cursor | 72 · Some gaps | MCP serverFree + model costs | ★ 407 | today | TypeScript | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The product requests broad Mac accessibility, screen recording, and input monitoring access, and can click, type, and run recipes; evidence of least-privilege controls is limited (1). Learning mode says the user manually demonstrates a task and requires Input Monitoring, but an example recipe sends email without documenting per-action confirmation (1). The README says data stays local and the vision model runs locally, supporting data-flow transparency (2), but retention and deletion of recordings, recipes, and screen data are not specified, so sensitive-data handling is thin (1). The README mentions the local vision model and CLAUDE.md names AXorcist, but a full dependency review or security mitigations are not shown (1). External effects include sending email; examples are visible and recipes are JSON, but pre-action confirmation is not documented (1). Recipes can be reviewed and deleted, while action rollback and recovery are not adequately described (1). AXorcist is attributed, but attribution for other sources and a maintainer identity path are unclear (1).
The tool count and interface are mostly consistent: README says 29 and the CLAUDE.md categories also total 29. However, ghost_parse_screen is explicitly called an unimplemented stub, which conflicts with the README's feature presentation (1). AXorcist has a minimum version and the sidecar dependency problems and version range are documented, but complete dependency pinning and availability safeguards are not shown (1). The code says model load failures are logged and returned as failures; the guide lists diagnostics and an error-suggestion convention, providing partial failure support (2).
The product targets macOS users with MCP clients and names Claude Code, Cursor, VS Code, and native-app scenarios, though guidance for beginners and organizational use is limited (2). Documentation distinguishes AX, visual grounding, and placeholder detection, and notes limits such as Chrome tabs, giving reasonably clear capability boundaries (2). Tool names and examples indicate when to use recipes or visual fallback, but selection rules are only partly specified (2). macOS 14+, Swift 6.2+, the Apple Silicon vision model, and required system permissions are documented, giving useful environment-fit guidance (2).
The README, developer guide, tool tables, architecture, and build instructions provide a clear structure, though a complete API reference is not shown (2). Homebrew, manual, and source installation steps are concrete and mention required permissions (2). Tool and recipe names are consistent, and version-consistency rules are documented (2). The README gives interaction and diagnostic examples, but no FAQ coverage is shown (2). CLAUDE.md lists known limitations, sidecar fragility, and troubleshooting advice (2). The repository includes full MIT license text (3). Version updates, a release checklist, and three-location version consistency are described, but the sample shows only the current and previous release summaries, limiting changelog depth (2). A contribution guide and PR process are mentioned, but no clear release owner or maintainer contact path is shown (1).
The 29 MCP tools, readable JSON recipes, and diagnostics can produce actionable results, but the parse_screen stub reduces completeness (2). Combining native macOS accessibility actions with reusable workflows provides clear task-specific value (2). Installation documents the multi-gigabyte model and permission costs, while evidence quantifying time, resource use, and reliability benefits is limited (1).
The tool list, named source files, dependency notes, and recipe descriptions make many capability claims traceable, though some promotional claims lack supporting material in this packet (2). The README, developer guide, and sidecar source corroborate architecture, version, and vision behavior, with a discrepancy around the placeholder feature (2). Documentation often labels implemented, placeholder, or permission-dependent behavior, but the boundary between promotional claims and established facts is not always clear (2).
- The tool can observe and operate native apps, including Input Monitoring recording and recipes that send email; review permissions, recorded content, and each recipe's steps before use.
- ghost_parse_screen is explicitly marked as an unimplemented stub in the developer guide; the vision sidecar also has documented dependency-version compatibility issues.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/ghost-os)<a href="https://followagents.com/en/agents/ghost-os"><img src="https://followagents.com/badge/ghost-os.svg" alt="FARS review"></a>Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v2.2.1 Ghost OS v2.2.1
Release notes excerpt
Complete recipe JSON schema documentation and verify-before-save workflow. What changed: - Documented the full recipe JSON schema in GHOST-MCP.md (all 10 action types, all params, target/Locator fields, wait conditions) - Added verify-before-save flow: agents must test recipes end-to-end before saving - Fixed codesign issue in release binary (linker-signed adhoc broke on copy) - Updated README: star count, line count Agents can now author recipes correctly on the first attempt. Previously required reverse-engineering the schema from bundled recipes. Upgrade: ``` …View on GitHub ↗ -
v2.2.0 Ghost OS v2.2.0
Release notes excerpt
Self-Learning Recipes Ghost OS can now watch you work and learn from it. Say "watch me send an email" -- Ghost OS observes every click, keystroke, and app switch with full accessibility context. Then Claude synthesizes your actions into a replayable recipe. New tools | Tool | Purpose | |------|---------| | ghost_learn_start | Begin recording user actions | | ghost_learn_stop | Stop recording, return enriched action sequence | | ghost_learn_status | Check recording state and stats | 29 tools total (was 26). How it works ``` You: "Watch me send an email in Gmail" …View on GitHub ↗ -
v2.1.2 Ghost OS v2.1.2
Release notes excerpt
Pin vision sidecar dependencies for reliable ghost_ground. What changed: - Pin mlx-vlm==0.1.15 (with --no-deps) and transformers==4.48.3 - transformers>=4.49 requires PyTorch for Qwen2VL video processor, breaking MLX-only environments - ghost setup now recreates the Python venv on upgrade (version stamp) - ghost doctor flags incompatible transformers versions - Vision sidecar diagnoses dependency errors at startup Upgrade: ``` brew upgrade ghost-os ghost setup ``` ghost setup will recreate the Python environment with pinned deps. Verify with ghost doctor. Fixes #7View on GitHub ↗
FAQ
Does Ghost OS require a paid account or API key?
What applications can it control?
Which macOS permissions are needed?
What happens when an accessibility tree does not expose the target UI?
ghost_ground or ghost_parse_screen for local ShowUI-2B visual grounding. ghost doctor reports the vision component status.Can a workflow be reviewed before it runs?
ghost_recipe_show displays their full steps and configuration before ghost_run executes them.Thanks — we use this to improve the page.