OpenGUI Mobile Operator
An Android GUI framework that lets AI inspect screens, plan steps, and operate real mobile apps.
Per-dimension scores and reasoning
Evidence shows: README mentions user authorization for USB debugging, AccessibilityService, etc., but does not explicitly state least privilege; user confirmation steps exist (e.g., authorization actions) but not systematic; data flow description (e.g., screenshots, task dispatch) exists but not transparent enough; sensitive data handling (e.g., API keys) mentioned but not detailed; dependency security not mentioned; external effects (e.g., remote control) described but risks not assessed; rollback mechanism not mentioned; source attribution (e.g., Core-Mate) mentioned but not verified. Deductions: lack of implementation details and evidence.
Evidence shows: README and code structure are consistent, but there are inconsistencies (e.g., README mentions 'Qwen 3.6 Plus' which may not exist); dependency availability not clear; failure messages mentioned (e.g., error reporting) but not detailed. Deductions: insufficient evidence for dependencies and failure handling.
Evidence shows: README clearly identifies target users (developers, AI agents) and use cases (Android automation); capability boundaries described (e.g., requires Android device, permissions); trigger methods (CLI, IM) explained but not precise; environment fit (e.g., Docker, local) mentioned. Deductions: insufficient detail on trigger precision and boundaries.
Evidence shows: Information architecture clear (README, docs, skills); installation notes exist (start.sh, get-started.md); naming stability not clear; examples and FAQ limited; known limitations listed; license clear (BUSL-1.1); versioning/changelog not provided; maintenance responsibility not clear. Deductions: insufficient evidence for naming stability, examples, changelog, maintenance.
Evidence shows: Output usability (structured results) described; marginal value (e.g., cost savings) mentioned; cost-benefit (e.g., model cost comparison) explained. Deductions: lack of actual runtime data.
Evidence shows: Claims in README (e.g., features, performance) have some code references but not fully mapped; cross-source verification insufficient; facts and inferences not clearly separated. Deductions: insufficient claim-code correspondence and independent verification.
- This repository uses BUSL-1.1 license, not OSI-approved open source, production use requires commercial license.
- Models mentioned in README (e.g., Qwen 3.6 Plus) may not exist, verify.
- Remote control features (e.g., Discord, REST API) may introduce security risks, configure carefully.
What does this agent do, and when should you use it?
OpenGUI is a mobile GUI agent framework for Android phones and emulators that understands live screens and operates app interfaces. Its backend uses a Plan Supervisor for task state, an Executor Graph for screenshot, vision, action, and call-user loops, and a Summarizer for structured results. The Android client keeps a standby connection through StandbySocketManager.kt and executes actions through GestureService.kt using AccessibilityService. Tasks can arrive through a local CLI, REST API, or Feishu, Telegram, and Discord dispatch paths to standby devices. The repository also documents bootstrap and remote-control workflows for Claude Code and Codex, with separate routing for planning and VLM execution.
A submitted goal is broken into executable steps by Plan Supervisor. Executor Graph reads the current Android UI and runs screenshot, vision, action, and call-user loops against live device state; execution review and retry feed back into supervision, and Summarizer returns a structured result. server/apps/backend/src/common/ws/standby.gateway.ts handles standby dispatch, StandbySocketManager.kt maintains the Android connection, and GestureService.kt executes Android-side gestures. Operators can list devices with pnpm opengui -- devices --json, dispatch work with pnpm opengui -- do "..." --json, and inspect a run with pnpm opengui -- status <executionId> --json.
- A developer who needs an attached Android device to inspect its current app screen and return a summary.
- A researcher collecting recent topic-related posts from X, Reddit, or Hacker News through a real Android phone.
- An operations team dispatching Android jobs to standby devices from Feishu, Telegram, Discord, or a REST API.
- An automation team running repetitive Android app workflows and receiving a structured execution result.
- A Codex or Claude Code user who wants guided local setup, APK build, installation, and debugging while handling only phone-side approvals.
What are this agent's strengths and limitations?
- It is designed around a main graph plus an executor subgraph, with Plan Supervisor retaining task state, continuation, and review rather than only a short interactive loop.
- The device path is explicit: StandbySocketManager.kt maintains a standby connection and GestureService.kt performs actions through Android AccessibilityService.
- It provides local CLI, REST, Feishu, Telegram, and Discord task-entry paths, enabling standby phones to act as remotely dispatched workers.
- Planning and VLM execution can be routed separately across documented provider paths including Claude, GPT, Gemini, Kimi, MiniMax, and compatible models.
- It requires an Android phone or emulator, USB debugging, and AccessibilityService permissions.
- Execution quality depends on the selected model, app UI, network state, and task duration; long-running reliability still needs more real-world testing.
- It is not yet an always-on operating-system assistant; tasks are manually triggered or launched through configured channels.
- The repository uses BUSL-1.1, so production or commercial use, hosted services, and commercial-product integration require a separate Core-Mate license.
How do you install or deploy this agent?
Prepare an Android phone or emulator, enable USB debugging and AccessibilityService, and provide model API keys for real task execution. Start the backend with cd server then ./start.sh; start the Android client with cd client then ./start.sh. The backend setup includes Docker-based PostgreSQL and Redis setup. For Discord dispatch, configure DISCORD_BOT_TOKEN; when it is empty, the backend starts without Discord.
How do you use this agent?
After the backend and client are running, from server run pnpm opengui -- devices --json to inspect available devices, pnpm opengui -- do "Observe the current Android screen and summarize what you see" --json to submit a first task, and pnpm opengui -- status <executionId> --json to track it. Codex and Claude Code can also use skills/open-gui-bootstrap/SKILL.md for setup or skills/open-gui-remote-control/SKILL.md for local-CLI Android task dispatch.
How does this agent compare with similar options?
Against the repository's described typical phone-agent demo, OpenGUI uses a main graph plus executor subgraph, manages task state in the backend, and includes Android standby connections with Feishu, Telegram, Discord, REST, and local dispatch paths.