AppAgent
A multimodal framework that operates Android apps through screen-level actions.
Per-dimension scores and reasoning
Insufficient evidence: The repository does not provide any documentation on permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on self-consistency, dependency availability, or failure messages. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on target audience, capability boundaries, trigger precision, or environment fit. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on information architecture, installation notes, naming stability, examples or FAQ, known limitations, license, versioning, or maintenance responsibility. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on output usability, marginal value, or cost-benefit. All criteria are unmet, hence score 0.
Insufficient evidence: No documentation on claim traceability, cross-source corroboration, or fact-inference separation. All criteria are unmet, hence score 0.
- The repository lacks any security or privacy documentation; assess risks before use.
- No dependency security audit is provided; dependencies may have unknown vulnerabilities.
- No rollback mechanism is provided; actions may be irreversible.
- No data flow transparency is documented; user data may be sent to third-party APIs.
What does this agent do, and when should you use it?
AppAgent is an LLM-based multimodal framework for operating Android applications with simplified actions such as taps and swipes. Its workflow has exploration and deployment phases: it first learns an app autonomously or from a human demonstration, then executes a task. During learning, it generates and saves documentation for the UI elements it interacts with, creating a knowledge base for deployment. `learn.py` handles exploration or demonstrations, while `run.py` performs tasks using a selected documentation base. It runs against an Android device connected through adb or an Android Studio emulator, rather than relying on app back-end access.
In learn.py, the user selects autonomous exploration or human demonstration, then supplies an app name and task description. Autonomous exploration attempts the task, reflects on prior actions, and generates documentation for explored elements; human demonstration captures phone screenshots, applies numeric tags to interactive elements, and learns from the actions and targets entered by the user. In run.py, the user selects an app, a documentation base, and a task description; the agent then executes the task with that documentation, or can run without it with no guaranteed success rate. The model configuration in config.yaml calls GPT-4V or Qwen, and adb connects the framework to an Android device or emulator.
- A mobile QA researcher with a connected Android device can explore an app with
learn.pybefore running related app tasks throughrun.py. - A researcher studying GUI-agent learning can choose
human demonstration, perform a similar workflow manually, and generate UI-element documentation. - A developer without an Android handset can try a smartphone GUI agent using an Android Studio emulator and installed APKs.
- An experimenter comparing vision-language models for smartphone operation can switch between GPT-4V and
qwen-vl-maxinconfig.yaml.
What are this agent's strengths and limitations?
- Its two-phase design turns autonomous exploration or human demonstrations into reusable UI-element documentation for later tasks.
- It operates apps with screen-level actions such as tapping and swiping, without requiring app back-end access.
- It supports both autonomous exploration and guided human demonstrations, allowing different levels of user involvement.
- It supports GPT-4V and
qwen-vl-max, and documents a path for custom models through a new class inscripts/model.py.
- It requires adb plus either an Android device with USB debugging or an Android Studio emulator, limiting operation to Android environments.
- GPT-4V requires an eligible paid OpenAI API key; the project estimates about $0.03 per request/response pair.
- The README reports poorer AppAgent performance from
qwen-vl-maxthan from GPT-4V. - The agent can run without a prior documentation base, but success is not guaranteed, and generated documentation may need manual review and revision.
How do you install or deploy this agent?
Install Python 3 and Android Debug Bridge (adb). Connect an Android device with USB debugging enabled; alternatively, install Android Studio and use its Device Manager emulator. From the repository directory, run:
pip install -r requirements.txtConfigure a model in the root config.yaml. For GPT-4V, provide an eligible OpenAI API key. For Qwen, create an Alibaba Cloud DashScope API key, set DASHSCOPE_API_KEY, and change MODEL from OpenAI to Qwen.
How do you use this agent?
Run python learn.py, choose autonomous exploration or human demonstration, and enter the app name and task description. After learning finishes, run python run.py, enter the app name, choose a documentation base, and provide the task. Running without existing documentation is allowed, but success is not guaranteed.
How does this agent compare with similar options?
The project presents qwen-vl-max as an alternative multimodal model to GPT-4V: it was described as free to use at the time of the README, but with poorer AppAgent performance than GPT-4V.