mobile-use: Automate Your Phone with Natural Language
An open-source AI agent that controls real Android/iOS apps using natural language, from sending messages to extracting structured data.
Evidence shows: The repository does not provide any explicit permission model, user confirmation mechanism, data flow transparency, sensitive data handling, dependency security audit, external effect control, rollback mechanism, or source attribution. All trust criteria are unsupported, hence all scores are 0.
Evidence shows: README and pyproject.toml provide basic consistency information such as version, dependency list, and test configuration, but no detailed error handling or recovery mechanisms. Dependency list is complete but no availability guarantee. Failure messages: README provides some troubleshooting but not comprehensive. Hence each score is 1.
Evidence shows: README clearly identifies target audience (developers and users) and multiple use cases (natural language control, data scraping), score 2. Capability boundaries: only mentions limited support for games, not detailed, score 1. Trigger precision: provides CLI examples but no trigger conditions or precision, score 1. Environment fit: supports multiple devices and platforms but no detailed configuration options, score 2.
Evidence shows: README is well-structured with sections for installation, usage, architecture, etc., score 2. Install notes are detailed including Docker and manual setup, score 2. Naming stability: project name and version are clear but no API stability statement, score 1. Examples and FAQ: multiple examples and troubleshooting provided, score 2. Known limitations: only mentions limited game support, score 1. License: Apache-2.0, score 2. Versioning and changelog: version number present but no changelog, score 1. Maintenance responsibility: authors listed but no explicit maintenance responsibility, score 1.
Evidence shows: Output usability: provides structured output examples (e.g., JSON), score 2. Marginal value: provides unique features (e.g., data scraping), score 2. Cost-benefit: no performance or cost data, score 1.
Evidence shows: README claims 100% on AndroidWorld benchmark but no verifiable test results or reproduction steps, score 1. Cross-source corroboration: provides paper and benchmark links but no independent verification, score 1. Fact-inference separation: README mixes facts and inferences without clear distinction, score 1.
- No permission model or user confirmation mechanism provided, may execute high-risk actions.
- Dependency list includes many third-party libraries but no security audit or vulnerability scanning.
- Benchmark claims in README lack reproducible test steps.
What does this agent do, and when should you use it?
mobile-use is a powerful open-source AI agent that controls your Android or iOS device using natural language. It understands commands and interacts with the UI to perform tasks, from sending messages to navigating complex apps. The agent is UI-aware, using accessibility tree data to navigate (though it has limited effectiveness with games). It supports data scraping, structuring extracted information into formats like JSON based on natural language descriptions. It is extensible and customizable, supporting multiple LLM providers including OpenAI, Google, xAI, OpenRouter, and MiniMax. mobile-use claims to be the first agentic framework to achieve 100% on the AndroidWorld benchmark. Deployment is flexible: you can use their cloud platform, run a Docker quickstart for Android, or set up from source for development.
mobile-use takes natural language commands and translates them into device operations. It connects to Android devices via ADB or iOS simulators via IDB, retrieves the UI hierarchy, and performs actions like taps and swipes. The main entry point is python ./minitap/mobile_use/main.py "<command>" [--output-description <format>], which can format output as structured data like JSON. It supports configuration through llm-config.override.jsonc to switch providers, and environment variables like OPENAI_API_KEY, ANTHROPIC_API_KEY, or MINIMAX_API_KEY. The system is built on a multi-agent architecture that decomposes tasks for better accuracy.
- A user on an Android phone asks the agent to open Gmail and list unread emails with sender and subject lines.
- A QA tester automates a series of UI actions on an iOS simulator using natural language, without writing code.
- A data analyst extracts specific fields (e.g., product prices, contact info) from an app and outputs them as JSON.
- A developer debugging their Android app guides the AI to perform a specific sequence of taps on an emulator.
- A non-technical business user automates repetitive data collection tasks on a mobile app.
- A researcher replicates the 100% accuracy result on the AndroidWorld benchmark for their own experiments.
What are this agent's strengths and limitations?
- First to achieve 100% accuracy on the AndroidWorld benchmark, demonstrating superior agentic task decomposition.
- Supports multiple LLM providers (OpenAI, Google, xAI, OpenRouter, MiniMax) and OpenAI-compatible endpoints, avoiding vendor lock-in.
- Offers a Docker quickstart script for simple deployment on Android.
- Supports both Android and iOS simulators, expanding testing scenarios.
- Provides a cloud platform and a clean CLI for easy adoption.
- Physical iOS devices are not supported yet, only simulators.
- Limited effectiveness with games due to lack of accessibility tree data.
- Requires external LLM API keys, incurring costs and network dependency.
- Docker quickstart is Android-only; iOS requires manual setup on macOS.
- Setup requires technical knowledge including ADB, environment variables, and package management.
How do you install or deploy this agent?
- Clone the repository:
git clone https://github.com/minitap-ai/mobile-use.git && cd mobile-use. 2. Copy.env.exampleto.envand add your API keys. 3. (Optional) Copyllm-config.override.template.jsonctollm-config.override.jsoncto customize models. 4. Install [uv](https://github.com/astral-sh/uv). 5. Create a virtual environment:uv venv. 6. Activate it (macOS/Linux:source .venv/bin/activate, Windows:.venv\Scripts\activate). 7. Install dependencies:uv sync. For Android, you need [ADB](https://developer.android.com/studio/releases/platform-tools) and USB debugging enabled. For iOS simulators (macOS only), install [Xcode](https://developer.apple.com/xcode/) and [fb-idb](https://fbidb.io/docs/installation/) viabrew install idb-companion.
How do you use this agent?
First, connect your device. For Android: connect via USB with USB debugging enabled, or launch an emulator. For iOS: launch a simulator and ensure idb_companion is in your PATH. Then run a command, e.g., python ./minitap/mobile_use/main.py "Go to settings and tell me my current battery level". To extract structured data, add --output-description, e.g., python ./minitap/mobile_use/main.py "Open Gmail, find all unread emails, and list their sender and subject line" --output-description "A JSON list of objects, each with 'sender' and 'subject' keys". If no model is configured, you will be prompted to choose one. Accept any ADB connection prompts on your device.
FAQ
Which LLM providers are supported?
Can I use it on a physical iPhone?
What are the costs involved?
How do I troubleshoot connection issues?
adb shell ip addr show up to find the interface and pass --interface to the script.