Ghost OS
Give MCP agents structured control of native macOS apps and reusable learned workflows.
What does this agent do, and when should you use it?
Ghost OS is a Swift MCP server that gives AI clients computer-use capabilities on a local Mac through stdio. It reads the macOS accessibility tree for structured application and UI-element data, with an optional local ShowUI-2B vision model for cases where accessibility data is insufficient. Its 29 tools cover UI inspection, mouse and keyboard actions, window control, screenshots, visual grounding, and recipe management. It can also observe a user performing a task through a CGEvent tap and return an accessibility-enriched action sequence for a parameterized, replayable JSON recipe. The documented deployment boundary is macOS 14+ and the documented clients include Claude Code, Cursor, VS Code, and other MCP clients.
An MCP client can call ghost_context for the current app, window, URL, focused element, and interactive elements, and use ghost_state, ghost_find, ghost_read, ghost_inspect, or ghost_element_at to inspect the UI. It operates apps with ghost_click, ghost_type, ghost_press, ghost_hotkey, ghost_hover, ghost_long_press, ghost_drag, ghost_scroll, ghost_focus, and ghost_window, while ghost_wait waits for UI changes. When the AX tree cannot adequately describe a web or dynamic interface, ghost_ground and ghost_parse_screen use the local ShowUI-2B model for visual grounding. Workflows can be installed with ghost_recipe_save, examined with ghost_recipe_show, run with parameter substitution through ghost_run, and removed with ghost_recipe_delete. Learning mode uses ghost_learn_start to observe user actions, ghost_learn_stop to return the enriched sequence, and ghost_learn_status to report recording state.
- A macOS developer using Claude Code who needs an agent to fill Gmail fields and send a report email.
- A researcher who repeatedly searches arXiv and downloads papers, and wants to save the completed process as a parameterized recipe.
- An operations user moving between Slack, Finder, and other native Mac applications as part of a recurring desktop workflow.
- A team lead who wants to review the JSON steps of a proven desktop workflow before sharing it with colleagues.
- A user automating a dynamic web app whose target control is not sufficiently exposed through the accessibility tree and needs local visual grounding.
What are this agent's strengths and limitations?
- Uses the macOS accessibility tree for structured UI data rather than relying solely on screenshots, with a local ShowUI-2B fallback for visual grounding.
- Stores workflows as inspectable JSON recipes and runs them with parameter substitution through `ghost_run`.
- Covers native-app control, including windows, keyboard and pointer actions, UI reading, and recipe execution rather than browser-only automation.
- Exposes its capabilities through MCP over stdio and explicitly documents Claude Code alongside other MCP clients.
- Its core runtime is macOS 14+, so it is not a drop-in cross-platform desktop automation option.
- Adoption requires Accessibility and Screen Recording permissions; workflow learning also requires Input Monitoring.
- Interfaces with weak accessibility data depend on the local ShowUI-2B vision fallback; the diagnostic example lists the model at 3.0 GB.
- The documented recipe-synthesis example uses Claude, while equivalent configuration and results for other models are not documented.
How do you install or deploy this agent?
Ghost OS supports macOS 14+. Install and configure it with Homebrew:brew install ghostwright/ghost-os/ghost-osghost setup
According to the documentation, ghost setup configures permissions, MCP, recipes, and the vision model. To build from source:git clone https://github.com/ghostwright/ghost-os.gitcd ghost-osswift build.build/debug/ghost setup
Source builds require Swift 6.2+. No account or API key requirement is documented. Accessibility and Screen Recording permissions are required; learning mode additionally requires Input Monitoring.
How do you use this agent?
After ghost setup, use the Ghost OS MCP server from an MCP-capable client. Start with ghost_context to inspect the active UI. For example, run an installed recipe with ghost_run recipe:"gmail-send" params:{"recipient":"[email protected]","subject":"Hello","body":"World"}. To learn a workflow, call ghost_learn_start task_description:"send email in Gmail", perform the task manually, call ghost_learn_stop, then save the returned workflow with ghost_recipe_save. Run ghost doctor to check permissions, MCP configuration, recipes, AX-tree access, and vision components.
How does this agent compare with similar options?
The repository compares Ghost OS with Anthropic Computer Use, OpenAI Operator, and OpenClaw. It positions Ghost OS around an accessibility tree plus a local VLM, native macOS app control, and JSON workflow recipes; its table describes Anthropic Computer Use and OpenAI Operator as screenshot-based and OpenClaw as browser-DOM-based.