What Agents Can Do

Once iphone-use is installed, you say what you want done on the phone and the agent does it: it opens the app by name, reads the screen as text, taps, types and scrolls, and checks each step. It works in apps that have no API, with the sign-ins already on the phone.

Example requests

What the agent can do

• Open any app by name (launch_app {"app": "Health"}), without hunting for its icon.
• Read the screen as text. The phone's accessibility tree comes back as a list of elements, so the agent reasons over labels and values rather than pixels. A screenshot is there when pixels matter.
• Type long text in one step, up to 20,000 characters, CJK included.
• Read a whole list. It reads a list across pages and says whether it reached the end.
• Work on screens hidden from capture. Banking and payment apps black out screenshots; the agent drives them by their elements, and the screenshot shows a labelled wireframe instead of a blank picture.
• Read the Home Screen layout. phone_home_layout reads the home pages, Dock, folders and widget stacks without opening folders or taking screenshots.
• Act in batches. Up to 24 steps go in one call with waits between screens, and the reply carries the screen the batch ended on.
• Reuse saved work. When the agent enters an app, the response lists that app's saved flows; if one does the task, it runs in one call. See Flows & replay.

Sign-ins, passcodes and codes

For logins, passcodes and Face ID the agent stops and asks you to finish on the phone. It never asks for the password or the code. If you keep the account in your own bitwarden-use vault, phone_login fills and submits the app's login form inside the daemon: the values go from the vault to the phone, the password never reaches the model, and the answer names only the entry and a masked account. A verification code the app asks for can come from an SMS (message-use), an email (mail-use) or the entry's authenticator, at most two per login.

For app developers and testers

The iphone-use-qa skill turns the agent into an exploratory tester for one app: it plans the areas to cover, explores them in batches, captures evidence for every issue (screenshot, element tree, exact steps) and writes a Markdown report. It never pays, sends, posts or deletes.

Hand the daemon one goal

Instead of driving step by step, a caller can hand the daemon a goal in plain words (POST /agent/task) and poll for the result. The daemon carries it out on its phone:

1. Saved flow first. If a saved flow matches the goal, it runs in one call; running the flow uses no model tokens.
2. Jev picks each step from the element table; a stronger assist model takes over a step when Jev is stuck.
3. Confirm gate. Before a step that would pay, send a message, delete, share or submit an order, the task stops and asks.
4. Summary. A few sentences for the person with what was done and the values read; a goal that was not verified ends as blocked.

Screen text is treated as untrusted data at every step, never as instructions. From an MCP client the same idea is phone_jev_run, which needs the model credentials configured.

Limits you set, and limits of iOS

• It cannot get past the passcode or Face ID; someone has to unlock the phone.
• App Switcher and Control Center are system gestures the runner cannot reach. System alerts are answered through their own action, not by tapping.
• force_press needs 3D Touch; on every iPhone since the XR / 11 it is refused before touching the screen. A long press (menu) works instead.
• An action whose outcome is unknown is marked retry_safe:false: the agent reads the screen again before doing anything else.

Connect your agent: MCP, HTTP API & skill. Privacy and the safety limits: Privacy & Security.