Clawd Cursor
Compiles the screen into one addressable UI map so any AI agent can safely drive the real desktop — local, cross-platform, model-agnostic, with every action verified.
- Source repo
- AmrDab/clawdcursor
- Stars
- ★ 406
- Last updated
- today
- License
- MIT
- Primary language
- TypeScript
- FA score
- 74/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformCodex · Claude Code
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A developer using Claude Code or Cursor who wants the agent to operate native apps like Outlook or Notepad that expose no API or CLI
- Not a fit if
- macOS users unwilling to grant Accessibility and Screen Recording permissions
- Linux hosts where system packages like tesseract-ocr and python3-gi can't be installed
- Source review
- 74/100 · Some gaps
What does this agent do, and when should you use it?
Clawd Cursor is a local MCP server that gives any tool-calling agent — Claude Code, Cursor, Windsurf, Zed, OpenAI Codex — safe control of the real desktop: clicking, typing, reading the screen, and opening apps. Its core innovation is the UI State Compiler, which fuses the accessibility tree with OCR into a confidence-scored map where every element carries a stable el_NN id; the agent acts by element id instead of pixel coordinates, dropping to the expensive screenshot/vision tier only for canvas-only apps. Every call routes through a single safety.evaluate() chokepoint (allow / confirm / block), and consequential actions accept an expect parameter that re-checks the live screen, returning a DEVIATION when reality didn't match. It runs on Windows, macOS, and Linux, exposes stdio and HTTP MCP transports, and ships 7 compound tools alongside 98 granular primitives. While an agent drives, an on-screen banner with a blinking red dot appears and can be double-clicked as a kill switch.
At runtime, compile_ui fuses the accessibility tree (free) with OS-level OCR (cheap) into a confidence-scored element map with stable el_NN ids; find_button / find_field locate targets by meaning. Agents call seven compound tools — computer, accessibility, window, system, browser, task, batch — each with an action enum of roughly 10–20 verbs (click, type, key, invoke, open_app, etc.), or the 98 granular primitives directly. Every call first passes the safety.evaluate() gate with three tiers: Allow (executes immediately), Confirm (sends, deletes, purchases, sensitive apps pause for human approval), and Block (Ctrl+Alt+Del-class sequences refused outright). Actions carrying expect re-read the live screen after a short settle window and return DEVIATION if state didn't obey. A sparse a11y tree triggers system.detect_webview to switch Electron/WebView2 apps to CDP-based browser.* tools; canvas-only apps fall back to screenshot plus coordinate click as a last resort. The deployment boundary is fully local: stdio MCP spawned by the editor host, or clawdcursor agent serving HTTP MCP on 127.0.0.1:3847 with Bearer-token auth — no telemetry by default.
- A developer using Claude Code or Cursor who wants the agent to operate native apps like Outlook or Notepad that expose no API or CLI
- Business-process automation over legacy desktop software, e.g. finding a form field by name and typing into it instead of brittle pixel-coordinate scripts
- Privacy-sensitive setups leveraging per-OS OCR (Apple Vision on macOS, Windows.Media.Ocr on Windows, Tesseract on Linux) so screen content never leaves the machine
- Hybrid orchestration teams that plan with an expensive model (Claude) and delegate grunt work via task() to the built-in loop on a cheaper LLM
- Auditable ops automation: every step passes one safety gate, with email, banking, and password-manager apps auto-elevated to human confirmation
- Integrators running headless agents (Agent SDK, custom loops) against the same tool catalog over HTTP MCP at 127.0.0.1:3847
How do you install or deploy this agent?
Requires Node.js 20+. Global install (recommended — pinnable and inspectable on disk):
npm i -g clawdcursor
clawdcursor consent --accept # one-time desktop-control consent (required)
clawdcursor grant # macOS only: approve Accessibility + Screen RecordingPer-OS prerequisites: Windows installs clean (sharp and @nut-tree-fork/nut-js ship prebuilt binaries); macOS needs Xcode Command Line Tools (xcode-select --install) for screenshots/vision; Linux needs tesseract-ocr, python3-gi, gir1.2-atspi-2.0, plus ydotool on Wayland.
Add to Claude Code:
claude mcp add clawdcursor -s user -- clawdcursor mcp --compactAdd to OpenAI Codex (~/.codex/config.toml):
[mcp_servers.clawdcursor]
command = "clawdcursor"
args = ["mcp", "--compact"]Cursor / Windsurf / Claude Desktop: add {"mcpServers":{"clawdcursor":{"command":"clawdcursor","args":["mcp","--compact"]}}} to the host's MCP config; Zed uses the context_servers key. Or install the bundled plugin in one step: claude plugin marketplace add AmrDab/clawdcursor && claude plugin install clawdcursor@clawdcursor.
How do you use this agent?
Once installed, the editor host spawns clawdcursor mcp over stdio on demand. Issue natural-language tasks directly, e.g. "open Outlook and reply to the latest email from Sarah." Headless/daemon mode: run clawdcursor agent (HTTP MCP on 127.0.0.1:3847, Bearer token at ~/.clawdcursor/token, POST JSON-RPC to /mcp), or clawdcursor agent --no-llm for the tool surface only. Example calls:
computer({ action: "key", combo: "mod+s" }) // Cmd+S / Ctrl+S, resolved per-OS
accessibility({ action: "invoke", name: "Send" }) // click by name, not pixels
window({ action: "open_app", name: "Outlook" })
task({ instruction: "open Notepad and type hello" }) // delegate to the built-in loopCollapse known step sequences with batch({steps:[…]}) — each step is still perceived and safety-gated. Diagnose with clawdcursor doctor (configures the LLM for agent mode), clawdcursor status (readiness check), or clawdcursor stop (stop all modes).
What are this agent's strengths and limitations?
- Cheapest-tier-first perception: accessibility tree (free) → OCR (cheap) → screenshot (expensive, the only tier that puts pixels in model context), so token cost tracks task difficulty
- Unique reactive verification: pass expect on a consequential action and it re-checks the live screen, reporting DEVIATION instead of hollow success
- Single safety chokepoint: stdio, HTTP, and built-in loop calls all pass safety.evaluate() (allow/confirm/block); sensitive apps auto-elevate to human confirmation and the agent cannot bypass it
- Human always informed: a desktop-control banner with blinking red dot appears while an agent drives — double-click to stop
- Fully local, model-agnostic, cross-OS: with a local model nothing leaves the machine, and one MCP config works with any MCP host
- macOS requires Accessibility permission (mandatory) and Screen Recording (optional), plus Xcode Command Line Tools for the screenshot/vision tier — real first-run friction
- Linux depends on system packages npm cannot install (tesseract-ocr, python3-gi, gir1.2-atspi-2.0, plus ydotool on Wayland)
- Perception quality depends on each OS's native stacks; dark or non-standard UIs with sparse a11y trees force tier-climbing, and canvas-only apps are limited to coordinate clicks
- Verification and safety gating add round-trips; batch collapses calls but still re-perceives and gates each step
- Requires Node.js 20+ and a recommended global install (rather than auto-fetching via npx), an extra operational constraint in locked-down environments
How does this agent compare with similar options?
The README contrasts Clawd Cursor with several named alternatives: Windows-MCP and Terminator (peer desktop MCP servers); browser-use and Playwright MCP (browser-only); OmniParser and UI-TARS (vision-centric parsing approaches needing a screenshot in model context for every observation); and Anthropic computer-use (Claude-only, screens sent to a sandbox/cloud). Its stated differentiators: any desktop app (not just the web), cross-OS, vision-model-free perception (a11y + OCR fusion), self-verified actions, one safety gate, any model/vendor, MCP-native, and local-only.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Clawd Cursor This agent | 74 · Some gaps | MCP serverFree + model costs | ★ 406 | today | TypeScript | Codex · Claude Code |
| Open Computer Use (MCP desktop service) | 44 · Major gaps | CLIFree + model costs | ★ 2.7k | 5d ago | Swift | Codex · Claude Code |
| AIHawk – AI Browser Agent | 59 · Major gaps | MCP serverFree + model costs | ★ 32k | today | TypeScript | Codex · Claude Code |
| macOS Harness | 55 · Major gaps | CLIFree + model costs | ★ 965 | 9d ago | Python | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: single safety.evaluate() chokepoint (allow/confirm/block), HTTP bound to 127.0.0.1 with bearer-token auth, screen text wrapped in <untrusted-screen-content> tags, log redaction of password fields, explicit confirm/block tier table, revocable one-time consent (--revoke), uninstall cleanup path, MIT with named author. Deductions: the safety gate and platform adapter are described mainly in the README; gate implementation code is not in the sample; the granted desktop-wide permissions (Accessibility, screen recording, synthetic input) inherently exceed least privilege — scored 2.
CI matrix covers 3 OS × 2 Node versions with lint, typecheck, npm audit gate, coverage ratchet and perf smoke; the sampled tests (pidfile, orphan teardown, credentials) are high quality with honest skip rationales. Deductions: the '1,000+ tests' claim cannot be verified from the sample; a known native-module segfault on headless Linux teardown is admitted in test comments.
Clear target audience (MCP agent users), honest 'last-mile fallback' positioning, cross-platform matrix (Windows/macOS/Linux/Wayland) with per-OS prerequisites and known limits (ydotool on Wayland, headless Linux segfault). Deductions: trigger precision and capability boundaries rest on narrative rather than a specification file.
README is well structured (install/how-it-works/tools/platforms/safety/architecture/CLI); package. metadata is complete (repository, bugs, bin, files, engines, overrides); full MIT LICENSE, SECURITY.md with reporting channels, CHANGELOG referenced, version-sync script. Deductions: CHANGELOG and SKILL.md contents are not in the sample; maintenance responsibility is only indirectly evidenced by CI config; publisher is unverified.
A11y+OCR fusion, stable el_NN addressing, expect-based verification with DEVIATION reporting, and tiered perception to control token cost give clear marginal value over screenshot-first approaches, with an honest comparison table. Deductions: token/performance benefits are asserted, not demonstrated; cost-benefit depends on scenario (Node 20+, native deps per OS).
Claims mostly trace to concrete mechanisms (safety gate, consent file, pidfile tests, CI workflows consistent with README); test comments candidly record failed attempts and skip rationale; accepted audit advisories (issue #117) are disclosed. Deductions: performance/cost/reliability conclusions are inference; static review cannot execute them, hence low confidence.
- This tool requires desktop-wide OS permissions (Accessibility, Screen Recording, synthetic input); any vulnerability gives an attacker the same control as the logged-in user — only enable it for agents you trust on that machine.
- The allow/confirm/block behavior of the safety gate is described mainly in the README; the gate implementation was not in the reviewed sample. Inspect the SafetyLayer in src/core yourself and verify there is no bypass path before deployment.
- The HTTP-mode bearer token lives at ~/.clawdcursor/token; watch for leakage via logs, error responses, or local web pages (CORS), and periodically confirm the port binds only to 127.0.0.1.
- CI currently accepts two moderate advisories (unpatched file-type infinite loop, ip-address XSS); re-check the status of these transitive dependencies before production use.
- A known native-module segfault exists on headless-Linux teardown and Wayland support depends on external ydotool; run clawdcursor status to self-check before relying on it.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
Fixed - **Windows: find, wait_for and invoke no longer match single letters.** An element whose name sits inside the request matched however short it was: a lone "C" from a PDF tab matched "User code 0" and "MCP Registry". That reverse match now needs a real part of the request (3+ characters, at least half its length), close to macOS and Linux, which only match the forward way.View on GitHub ↗ -
Release notes excerpt
Added - A pause ends the moment the user's step is done. pause_for_user (MCP) and the built-in agent's ask_user take done_when: what is true on screen once the user has done their part (the same checks as expect). clawdcursor checks it every 1.5 s and continues by itself, so the prompt goes from blue back to amber with no Resume click. A done_when that is already true is rejected. Resume still works. Changed - Agents ask the user for one step only. Tool descriptions, the MCP instructions and SKILL.md now say: hand over only what the user must do …View on GitHub ↗ -
Release notes excerpt
Fixed - set_value with no field name or role types nothing. It fell back to typing into the window's largest editable field (seen live: the browser's address bar) and reported Set "" = … chars. It now fails and says to name the field, pass a compile_ui reference, or use type. - **macOS: a failed screen capture at startup no longer breaks every tool (#244).** With no screenshot helper and no Screen Recording grant, the startup probe failed and took window, key and accessibility down with it. The screen size now comes from NSScreen; only screenshots and OCR …View on GitHub ↗