ESP32 iPhone Operator
Automate a real iPhone with an ESP32-S3 and a vision model, without a computer at runtime.
- Source repo
- onehumanbeing/esp32-iphone-agent
- Stars
- ★ 3.2k
- Last updated
- 1d ago
- License
- MIT
- Primary language
- C
- FA score
- 58/100 · Major gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- An iPhone user who wants a repeating alarm can ask the agent to open Clock, enter a time, select repeat days, and save it.
- Not a fit if
- Users who need landscape app automation
- Users who need non-ASCII text input
- Users without cellular access for model requests
- Source review
- 58/100 · Major gaps
What does this agent do, and when should you use it?
This phone-control agent runs across an iPhone app and an ESP32-S3 board. The board receives the iPhone’s AirPlay screen mirror and acts as a USB HID mouse and keyboard; the app receives the video over BLE and sends screenshots to an OpenAI-compatible vision model. Its planner and actor carry out actions such as tapping, swiping, drawing, typing, pressing keys, going home, or finishing a task. The board provides its own Wi-Fi hotspot, while model requests use the phone’s cellular connection. A Mac with Xcode is needed to flash the board and build the app initially, but not to run tasks.
Plug the ESP32-S3 into an iPhone, pair it over Bluetooth in the app, join the board’s Wi-Fi hotspot, and select “ESP32 Agent” in Control Center for screen mirroring. The app decodes the H.264 stream forwarded over BLE and sends screenshots to the configured vision endpoint. Its planner and actor execute actions such as taps, swipes, drawing, ASCII keyboard input, key presses, or returning to the Home Screen. After each action, the app waits for the display to settle and checks for changes; if a connection drops, it pauses and resumes at the same step. Runs are stored on the phone for step-by-step replay and export of screenshots, actions, and annotations.
- An iPhone user who wants a repeating alarm can ask the agent to open Clock, enter a time, select repeat days, and save it.
- A user who repeatedly navigates iOS Settings can ask the agent to open Settings and reach a specified page.
- A person who wants to automate an iPhone without a computer at runtime can use the app with an ESP32-S3 plugged into the phone.
- A mobile automation developer can replay task steps and export screenshots, actions, and annotations for analysis.
- A user running a self-hosted vision model can configure an endpoint such as vLLM, SGLang, llama.cpp, or Ollama.
How do you install or deploy this agent?
Initial setup requires a Mac: install arduino-cli, ESP32 core 3.3.x, and NimBLE-Arduino 2.5.1, then flash the board. Copy the Xcode configuration example, set the development team and app Bundle ID, and use Xcode to install the IPhoneAgent scheme on the iPhone. XcodeGen generates the project from ios/project.yml. A free Apple ID can sign the app, with re-signing required every 7 days; one-tap hotspot joining requires the Hotspot Configuration capability available with a paid developer account.
How do you use this agent?
Plug the board into the iPhone and open ESP32 Agent. Follow setup to pair over Bluetooth, join the board’s hotspot, enable Screen Mirroring from Control Center, and turn on AssistiveTouch for pointer calibration. Enter a vision model endpoint and key, type a task, and press Start.
What are this agent's strengths and limitations?
- AirPlay receiving and USB HID control run on the ESP32-S3, so no computer is needed while tasks run.
- The app accepts any OpenAI-compatible chat API that accepts images, including cloud and self-hosted endpoints.
- Run records stay on the phone and support step replay plus export of screenshots, actions, and annotations.
- A lost board or mirroring connection pauses the run and resumes from the same step.
- Requires an iPhone with iOS 18 or later, an ESP32-S3 with native USB and PSRAM, and a Mac with Xcode for initial flashing and app building.
- BLE throughput is about 300–500 kbps, limiting the mirror to about 480×1040 at up to 5 fps; steps take a few seconds.
- The app uses a private iOS suspend API, which the project says must be replaced before App Store release.
- Only ASCII text and portrait coordinates are supported; the phone lock and some passcode, Apple Pay, and DRM screens interrupt or block mirroring.
How does this agent compare with similar options?
Compared with VersaiOS, this project does not need a computer running UxPlay and supports multi-step instructions. Compared with ja1chen/actell, it does not depend on Developer Mode or a Developer Disk Image. Compared with boboabw/phone-agent, it also brings the screen mirror through AirPlay on the board.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| ESP32 iPhone Operator This agent | 58 · Major gaps | Desktop appFree + model costs | ★ 3.2k | 1d ago | C | OpenAI API |
| mobile-use: Automate Your Phone with Natural Language | 33 · Major gaps | CLIFree + model costs | ★ 3.3k | today | Python | OpenAI API · Claude API |
| AppAgent | 0 · Major gaps | CLIFree + model costs | ★ 6.9k | 1y ago | Python | OpenAI API |
| Nuphus Desktop Automation MCP | 72 · Some gaps | MCP serverFree + model costs | ★ 326 | 5d ago | Rust | Claude.ai · OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
The README says the app sends screen images to an OpenAI-compatible vision model and performs touch and keyboard actions; it says the model key is stored in Keychain. SECURITY.md identifies BLE, hotspot, AirPlay receiver, and unstarted-action risks. Starting a task requires user action and the run can be stopped, but the evidence does not show fine-grained permission controls, screenshot handling or retention controls, or action rollback, so those areas lose points. README and LICENSE attribute MIT licensing and the GPL-3.0 exception for the AirPlay-derived code. No dependency vulnerability review is shown.
The README describes pausing on disconnection and resuming at the same step. CI covers iOS, firmware, and legacy server builds or tests; legacy tests also check parse errors and missing-board errors. This supports ordinary reliability, but the supplied material does not show specific user-facing failure messages for the current iOS and firmware paths, so failure messaging receives only partial credit.
The README targets users seeking to automate a real iPhone and lists hardware, iOS, model endpoint, and setup requirements. It clearly identifies unsupported protected screens, locked-phone behavior, landscape apps, and non-ASCII input. Tasks are triggered by user-entered or saved tasks, and boundaries and environment requirements are reasonably clear, though scenarios and trigger conditions are mostly described at a high level.
The repository layout, quick start, example task, related documentation, and limitations are documented. The MIT license and GPL-3.0 exception for AirPlay code are clear. The current architecture and legacy tag and branch are identified, but no formal changelog is supplied. Maintenance responsibility is only partly conveyed through contribution and security-report channels, while publisher identity is unverified, so those scores remain limited.
The product combines mirroring, vision-model planning, and USB HID actions into an iPhone automation workflow, with step replay and trace export. The README also discloses low bandwidth, low frame rate, model latency, and reliance on cellular data. The supplied material does not quantify success rates or cost-benefit, so full marks are not justified.
The README gives concrete hardware, installation, data-flow, and limitation details, while CI and legacy test files support some engineering claims. However, demonstrations, compatibility, and current-version behavior are mostly asserted in the README, with limited independent corroboration among the supplied files. The documentation distinguishes the current and legacy versions and states known limitations; it does not present unexecuted tests as verified results.
- The agent controls a real phone through USB HID. Use it only with devices and accounts you are allowed to operate, watch its actions, and keep the stop control available.
- Screen images are sent to the configured vision-model endpoint; consider whether screenshots may contain personal or sensitive information when using a cloud endpoint.
- The supplied material says landscape apps, non-ASCII input, some protected screens, and mirroring while locked are unsupported; performance depends on BLE bandwidth and cellular connectivity.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v0.1.0 v0.1.0 — AirPlay + ESP32 USB HID (computer in the middle)
Release notes excerpt
First tagged release: the original architecture, before the next big redesign. - Eyes: the iPhone mirrors its screen over AirPlay to UxPlay on a Mac or Linux computer. - Hands: a ~$7 ESP32-S3 plugs into the iPhone's USB port as a mouse + keyboard (AssistiveTouch pointer), connected to the computer over Wi-Fi. - Brain: any OpenAI-compatible vision model (demo: Qwen3.8-Flash-Next on two DGX Sparks). - No Mac required, no developer account, no jailbreak. Known limits: portrait apps only, ASCII-only typing, ~1.3 s per step. …View on GitHub ↗