Open-AutoGLM Phone Agent
Automate your phone with natural language: multimodal screen understanding and hands-free operation.
Evidence shows: README explicitly mentions built-in sensitive operation confirmation mechanism and supports manual takeover (Take_over), with custom callback examples, so user_confirmation scores 2. sensitive_data_handling scores 2 because documentation states that on sensitive pages (payment, password, banking) screenshots turn black and automatically request manual takeover, indicating protection of sensitive data. least_privilege scores 1 because documentation requires enabling USB debugging (secure settings) etc., but does not explain least privilege principle or permission levels. data_flow_transparency scores 1 because documentation explains screenshot and ADB control flow but not detailed data flow and storage. dependency_security scores 1 because requirements.txt lists dependencies but lacks version pinning or security audit info. external_effects scores 1 because documentation warns for research only but does not detail potential external impacts. rollback scores 0 because no rollback or undo mechanism is mentioned. source_attribution scores 1 because README provides citation papers but does not clearly state code origin or contributors.
self_consistency scores 2 because commands, configs, and API examples in README are consistent with no obvious contradictions. dependency_availability scores 2 because dependency list is clear and model download links and third-party service options are provided. failure_messages scores 2 because documentation provides solutions for common issues like device not found, text input not working, etc.
audience_and_scenarios scores 2 because documentation targets developers and researchers with multiple usage scenarios (CLI, Python API, remote debugging). capability_boundaries scores 2 because supported apps and actions are listed, and limitations (like sensitive page takeover) are explained. trigger_precision scores 2 because tasks are triggered by natural language and parameters like --list-apps are provided. environment_fit scores 2 because Android, HarmonyOS, and iOS are supported with environment preparation guides.
information_architecture scores 2 because README is well-structured with sections for installation, configuration, usage, examples, etc. install_notes scores 2 because detailed installation steps and dependency instructions are provided. naming_stability scores 2 because project name and API naming are consistent. examples_and_faq scores 2 because multiple examples and FAQ are provided. known_limitations scores 2 because documentation mentions limitations like sensitive page takeover and Windows encoding issues. license scores 2 because full Apache-2.0 license text is provided. versioning_changelog scores 1 because no version history or changelog is provided. maintenance_responsibility scores 1 because maintenance responsibility or contribution guidelines are not clearly stated.
output_usability scores 2 because output format is clear with verbose mode output examples. marginal_value scores 2 because it provides value in automating phone operations and supports many apps. cost_benefit scores 2 because both third-party service and local deployment options are provided, allowing users to choose based on cost.
claim_traceability scores 2 because claims in README (like supporting 50+ apps) are supported by specific lists. cross_source_corroboration scores 1 because only README is relied upon, no other sources for verification. fact_inference_separation scores 2 because documentation distinguishes facts (like supported apps) from inferences (like expected output).
- This framework requires ADB debugging permissions, which may pose security risks; ensure use only on trusted devices.
- Documentation lacks version history or changelog; monitor project updates.
- Dependencies are not version-pinned, posing potential supply chain risks.
- Remote debugging feature may increase attack surface; use with caution.
What does this agent do, and when should you use it?
Open-AutoGLM is an open-source phone agent framework that controls Android or HarmonyOS devices via ADB or HDC. It uses a vision-language model to understand screen content and a planning module to execute tasks described in natural language. The system supports multimodal perception, automated action generation, confirmation for sensitive operations, and human takeover. It includes remote debugging capabilities and both CLI and Python API. The repository provides models (AutoGLM-Phone-9B and Multilingual variant) downloadable from Hugging Face or ModelScope, plus integration with third-party model services like Zhipu BigModel. It supports over 50 mainstream Chinese Android apps and 60+ HarmonyOS apps out of the box. The project is intended for research and learning, with an explicit non-commercial clause.
Reads natural language instructions (e.g., 'Open Meituan and search for hotpot restaurants nearby'), sends the current screen screenshot to an AutoGLM vision-language model (e.g., AutoGLM-Phone-9B) which outputs a structured action sequence, then executes those actions (tap, swipe, type, launch) on the device through ADB or HDC, looping until the task is complete. It provides a CLI (main.py) with interactive and single-task modes, a Python API (PhoneAgent class), custom callbacks for confirmation and takeover, remote device connection (adb connect/hdc tconn), multi-device selection (--device-id), and configurable system prompts (--lang en/cn). It integrates with Midscene.js for cross-platform automation and offers deployment guides for vLLM/SGLang and third-party services.
- Developers automating end-to-end test flows on Android apps, such as sign-up, login, and purchase flows.
- Individuals who want to delegate tasks to their phone, like comparing product prices across apps and placing an order.
- QA engineers running repeated regression tests on HarmonyOS devices using HDC.
- Researchers reproducing or extending state-of-the-art mobile GUI agents based on AutoGLM.
- Business users integrating phone-based operations into their workflows, like collecting data from WeChat and filling forms.
- Remote operators controlling unattended Android devices over Wi-Fi for automation.
What are this agent's strengths and limitations?
- Cross-platform support for Android and HarmonyOS using standard ADB/HDC tools.
- Open-source models (AutoGLM-Phone-9B) available for local deployment, plus third-party API options.
- Built-in confirmation and human takeover for sensitive operations, balancing autonomy and safety.
- Remote debugging over Wi-Fi/network without USB, enabling unattended scenarios.
- Pre-configured support for 50+ Chinese Android apps and 60+ HarmonyOS apps.
- Requires manual setup of ADB/HDC and ADB Keyboard, which may be daunting for non-technical users.
- Dependency on either paid API services or high-end GPU (24GB+ VRAM) for local model hosting.
- Model services are tied to AutoGLM family; no documented alternative model adapters.
- License restricts to research and learning, prohibiting commercial or illegal use—may not suit enterprise deployment.
- HarmonyOS support requires specific HDC tools and HarmonyOS NEXT, a limited subset of devices.
How do you install or deploy this agent?
- Clone the repo: git clone https://github.com/zai-org/Open-AutoGLM.git && cd Open-AutoGLM 2. Install dependencies: pip install -r requirements.txt && pip install -e . 3. Prepare device: Enable developer mode and USB debugging on Android 7.0+ or HarmonyOS device, connect via USB, and install/activate ADB Keyboard on Android. 4. Configure a model service: either use a third-party service like Zhipu BigModel (--base-url https://open.bigmodel.cn/api/paas/v4 --model autoglm-phone) or ModelScope (--base-url https://api-inference.modelscope.cn/v1 --model ZhipuAI/AutoGLM-Phone-9B), or deploy locally with vLLM/SGLang.
How do you use this agent?
Once a model service is running, execute a task: python main.py --base-url http://localhost:8000/v1 --model autoglm-phone-9b "Open WeChat and send a message to File Transfer saying: deployment successful". Use interactive mode by omitting the task argument. Use --lang en for English prompts, --device-type hdc for HarmonyOS, and --device-id to target a specific device. Python API: from phone_agent import PhoneAgent; agent = PhoneAgent(model_config); result = agent.run("Open Taobao and search for wireless earbuds").