AppAgent
A multimodal framework that operates Android apps through screen-level actions.
- Source repo
- TencentQQGYLab/AppAgent
- Stars
- ★ 6.9k
- Last updated
- 1y ago
- License
- MIT
- Primary language
- Python
- FA score
- Insufficient evidence
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A mobile QA researcher with a connected Android device can explore an app with
learn.pybefore running related app tasks throughrun.py. - Not a fit if
- Users without an Android device or emulator
- Individuals unwilling to pay per GPT-4V request
- Source review
- Insufficient evidence 8 safety controls not found
What does this agent do, and when should you use it?
AppAgent is an LLM-based multimodal framework for operating Android applications with simplified actions such as taps and swipes. Its workflow has exploration and deployment phases: it first learns an app autonomously or from a human demonstration, then executes a task. During learning, it generates and saves documentation for the UI elements it interacts with, creating a knowledge base for deployment. learn.py handles exploration or demonstrations, while run.py performs tasks using a selected documentation base. It runs against an Android device connected through adb or an Android Studio emulator, rather than relying on app back-end access.
In learn.py, the user selects autonomous exploration or human demonstration, then supplies an app name and task description. Autonomous exploration attempts the task, reflects on prior actions, and generates documentation for explored elements; human demonstration captures phone screenshots, applies numeric tags to interactive elements, and learns from the actions and targets entered by the user. In run.py, the user selects an app, a documentation base, and a task description; the agent then executes the task with that documentation, or can run without it with no guaranteed success rate. The model configuration in config.yaml calls GPT-4V or Qwen, and adb connects the framework to an Android device or emulator.
- A mobile QA researcher with a connected Android device can explore an app with
learn.pybefore running related app tasks throughrun.py. - A researcher studying GUI-agent learning can choose
human demonstration, perform a similar workflow manually, and generate UI-element documentation. - A developer without an Android handset can try a smartphone GUI agent using an Android Studio emulator and installed APKs.
- An experimenter comparing vision-language models for smartphone operation can switch between GPT-4V and
qwen-vl-maxinconfig.yaml.
How do you install or deploy this agent?
Install Python 3 and Android Debug Bridge (adb). Connect an Android device with USB debugging enabled; alternatively, install Android Studio and use its Device Manager emulator. From the repository directory, run:
pip install -r requirements.txtConfigure a model in the root config.yaml. For GPT-4V, provide an eligible OpenAI API key. For Qwen, create an Alibaba Cloud DashScope API key, set DASHSCOPE_API_KEY, and change MODEL from OpenAI to Qwen.
How do you use this agent?
Run python learn.py, choose autonomous exploration or human demonstration, and enter the app name and task description. After learning finishes, run python run.py, enter the app name, choose a documentation base, and provide the task. Running without existing documentation is allowed, but success is not guaranteed.
What are this agent's strengths and limitations?
- Its two-phase design turns autonomous exploration or human demonstrations into reusable UI-element documentation for later tasks.
- It operates apps with screen-level actions such as tapping and swiping, without requiring app back-end access.
- It supports both autonomous exploration and guided human demonstrations, allowing different levels of user involvement.
- It supports GPT-4V and
qwen-vl-max, and documents a path for custom models through a new class inscripts/model.py.
- It requires adb plus either an Android device with USB debugging or an Android Studio emulator, limiting operation to Android environments.
- GPT-4V requires an eligible paid OpenAI API key; the project estimates about $0.03 per request/response pair.
- The README reports poorer AppAgent performance from
qwen-vl-maxthan from GPT-4V. - The agent can run without a prior documentation base, but success is not guaranteed, and generated documentation may need manual review and revision.
How does this agent compare with similar options?
The project presents qwen-vl-max as an alternative multimodal model to GPT-4V: it was described as free to use at the time of the README, but with poorer AppAgent performance than GPT-4V.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| AppAgent This agent | Insufficient evidence | CLIFree + model costs | ★ 6.9k | 1y ago | Python | OpenAI API |
| OpenGUI Mobile Operator | 45 · Major gaps | CLIFree + model costs | ★ 1.8k | 19d ago | HTML | Codex · Claude Code · OpenAI API · Claude API |
| Mobilerun | 43 · Major gaps | CLIFree + model costs | ★ 10k | 1d ago | Python | OpenAI API · Claude API |
| Damru — Android-Native Stealth Browser Automation | 42 · Major gaps | CLIFree | ★ 356 | 4mo ago | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Insufficient evidence: The repository does not provide any documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on self-consistency, dependency availability, or failure messages. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on target audience, capability boundaries, trigger precision, or environment fit. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on information architecture, installation notes, naming stability, examples or FAQ, known limitations, license, versioning, or maintenance responsibility. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on output usability, marginal value, or cost-benefit. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on claim traceability, cross-source corroboration, or fact-inference separation. All criteria are unmet, hence score 0.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository lacks any security or privacy documentation; assess risks before use.
- No dependency security audit is provided; dependencies may have unknown vulnerabilities.
- No rollback mechanism is provided; actions may be irreversible.
- No data flow transparency is documented; user data may be sent to third-party APIs.