MobileGym
A programmable mobile simulator for deterministic GUI-agent evaluation and online RL training.
- Source repo
- Purewhiter/mobilegym
- Stars
- ★ 794
- Last updated
- 26d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 50/100 · Major gaps
At a glance
- Works with
- Portable with changes
- You'll need
- Typical use
- A mobile-agent researcher comparing models on WeChat, Alipay, or cross-app tasks with identical initial state for every trial.
- Main limitation
- The apps are research surrogates rather than live integrations, so the platform cannot validate authentication, backend behavior, or payments in real production apps.
- Source review
- 50/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
MobileGym is a browser-hosted Android-like simulation platform for mobile GUI-agent research. It combines a React/Vite simulator, a Python-and-Playwright benchmark runtime in bench_env, and an online RL pipeline in mobilegym-rl. The environment is exposed as structured JSON that can be read, patched, snapshotted, restored, and cloned; task judges evaluate resulting state and side effects programmatically. The repository includes 28 simulated apps, 416 parameterized task templates, and a 256-task test split, while its apps never connect to real services, accounts, or funds. It fits teams seeking repeatable evaluation and scalable browser rollouts, not teams that need automation against real Android apps or live consumer accounts.
The frontend composes a simulated phone from apps/, system/, and os/; bench_env drives it through Playwright using __SIM__, __OS__, and __SIM_INPUT__ to receive screenshots and issue tap, type, swipe, back, home, wait, drag, and complete actions. A task class supplies description, JSON state injection in setup, and deterministic check_goals() or get_answer() logic; python -m bench_env.run can execute individual tasks, suites, splits, parallel evaluations, pass@k, and retries. The agent package includes autoglm, uitars, venus, gui_owl, generic, generic_v2, mai_ui, and human adapters, and completed runs can be inspected in run_explorer.html with screenshots, action annotations, prompts, and model responses. mobilegym-rl/ contains the online RL training code.
- A mobile-agent researcher comparing models on WeChat, Alipay, or cross-app tasks with identical initial state for every trial.
- An RL team cloning task state into many browser instances to run parallel GRPO-style rollouts.
- A benchmark maintainer adding a simulated-app task with setup state injection and a deterministic check_goals() judge.
- A model-integration engineer evaluating a GUI model through an existing AutoGLM, UI-TARS, UI-Venus, GUI-Owl, or unified-JSON adapter.
- A developer using human mode to replay a task, debug its judge, and inspect the resulting trajectory in a browser.
How do you install or deploy this agent?
Prerequisites: Node.js >=22, Python >=3.11, a shell, and network access. Run: git clone https://github.com/Purewhiter/mobilegym.git; cd mobilegym; npm install; pip install -r bench_env/requirements.txt; playwright install chromium. The optional companion dataset is about 1.9 GB: curl -L -o mobilegym-data.tar.gz https://github.com/Purewhiter/mobilegym/releases/download/data-v0.1.0/mobilegym-data-v0.1.0.tar.gz; tar -xzf mobilegym-data.tar.gz. Simulator keys are optional; the canonical benchmark does not require credentials.
How do you use this agent?
First run npm run build && npm run preview -- --port 4173, then verify the setup in manual mode: python -m bench_env.run --task-id wechat.ReadMyWxid --agent human --env-url http://localhost:4173. List tasks with python -m bench_env.run --list. For the full test split: python -m bench_env.run --split test --parallel 8 --env-url http://localhost:4173 --agent autoglm --model-name autoglm-phone-9b. For higher concurrency, install nginx, run npm run build and ./scripts/server/start_nginx_gateway.sh, then use https://localhost:4180 as the environment URL.
What are this agent's strengths and limitations?
- It manages the full environment as structured JSON, enabling snapshot, restore, injection, and cloning rather than relying only on screenshots or accessibility trees.
- Tasks use programmatic check_goals() / get_answer() judges that can assess both outcomes and unexpected side effects without VLM judging.
- Its browser-hosted instances are designed for parallel rollouts; the README documents resource and startup figures for 256 instances on one server.
- Apps, OS functionality, benchmark tasks, and model adapters are separated into explicit repository layers, with documented locations for new apps and adapters.
- The apps are research surrogates rather than live integrations, so the platform cannot validate authentication, backend behavior, or payments in real production apps.
- Local use requires Node.js, Python, and Playwright Chromium; high-concurrency operation additionally requires nginx and may require Linux inotify tuning.
- The companion dataset is about 1.9 GB and the data/content license is CC BY-NC 4.0, which may limit commercial adoption.
- A new model must fit an existing adapter schema or require an adapter implemented and registered under bench_env/agent/.
How does this agent compare with similar options?
Compared with real-device or conventional emulator workflows, MobileGym is designed around writable, resettable, cloneable JSON state and deterministic state-diff evaluation. Its README identifies AppWorld, WebArena, VisualWebArena, AndroidWorld, AndroidLab, and A3 as inspirations.
Key facts side by side with the most closely related agents.
| Agent | Source review | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|
| MobileGym This agent | 50 · Major gaps | ★ 794 | 26d ago | Python | — |
| Tongyi DeepResearch | 49 · Major gaps | ★ 20k | 6mo ago | Python | — |
| MARO Resource Optimization Platform | 44 · Major gaps | ★ 925 | 2y ago | Python | — |
| RLinf RL Infrastructure | 35 · Major gaps | ★ 5.3k | 4d ago | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: repository declares Apache-2.0 license, but no security policy or vulnerability reporting channel; dependencies are not pinned, posing supply chain risk; simulator claims not to connect to real services, but no detailed data flow transparency; rollback mechanism exists (state snapshot/reset), but no user confirmation mechanism; external effects are limited to sandbox, but no detailed explanation. Deductions: lack of user confirmation, insufficient data flow transparency, dependency security not fully addressed.
Evidence shows: README and test files provide task definitions and judge logic, but no complete error handling documentation; dependencies are not pinned, potentially affecting reproducibility; failure messages are not clearly documented. Deductions: dependency availability not fully guaranteed, failure messages unclear.
Evidence shows: README provides multiple use cases (exploration, evaluation, training), and explains capability boundaries (e.g., parallelism limits, inotify limits); trigger conditions (commands) are clear; environment fit (Node, Python versions) is documented. Deductions: capability boundaries not detailed enough, environment fit not covering all platforms.
Evidence shows: README provides detailed information architecture (table of contents, documentation map), clear installation instructions, rich examples, known limitations mentioned, license clear, but no version changelog, maintenance responsibility not clear. Deductions: missing version changelog, maintenance responsibility unclear.
Evidence shows: high output usability (multiple output formats), high marginal value (solves problems real devices cannot), reasonable cost-benefit (lightweight, parallel). Deductions: cost-benefit lacks specific data.
Evidence shows: claims in README (e.g., performance data) lack traceable verification methods, cross-source verification insufficient, facts and inferences not clearly separated. Deductions: claims lack traceability, facts and inferences mixed.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Dependencies are not pinned, posing supply chain risk.
- Lack of user confirmation mechanism may affect security.
- Data flow transparency is insufficient, needs further explanation.
- Version changelog missing, maintenance responsibility unclear.