Skip to main content
With vision enabled, agents receive a screenshot of the device alongside the accessibility tree on every step.

​
Enabling vision

vision_only: true (CLI: --vision-only) is a separate mode that drops the accessibility tree and drives the device from screenshots alone.

​
The coordinate contract

With vision active, coordinates are handled automatically:
  1. The screenshot is resized to the exact dimensions the model will ground on, with a labeled coordinate grid drawn on it.
  2. The device state declares that coordinate space, and element bounds in the accessibility tree text are emitted in the same space.
  3. Coordinates the model passes to click_at, click_area, long_press_at, and swipe are converted back automatically — to device pixels on Android, to points on iOS.
  4. Coordinates outside the declared space are rejected with a corrective error and the model retries.
No configuration is needed. Element-index actions (click, long_press) are unaffected; they always tap accessibility-tree bounds.

​
Per-model effective screenshot size

Some providers downsize images server-side before the model sees them. To keep coordinates exact, mobilerun resolves the screenshot dimensions each active vision model actually grounds on and resizes to the smallest result across all recipients (manager + executor, or the fast agent): Unknown Anthropic model IDs fall back to the standard 1568 budget. The declared coordinate space always matches the image the model receives, so convert_point is exact regardless of which model is configured.

​
Capping the screenshot size

For local or undocumented vision models that downsize to a size mobilerun can’t infer (for example, certain Ollama models), set an explicit cap with agent.model_screenshot_max_side. The cap is applied on top of per-model resolution and uses the smaller of the two.
Leave this unset for the supported providers above — the per-model defaults already match what they ground on.

​
Coordinate tools and vision

swipe is always available. click_at, click_area, and long_press_at are disabled by default; enabling vision re-enables click_at automatically (non-normalized). To enable all coordinate tools:

​
Normalized coordinates

agent.use_normalized_coordinates: true makes the model emit [0-1000] coordinates instead of pixels; the coordinate contract is inactive in this mode.
An occasional “Coordinates … are outside the … coordinate space” error is the guardrail correcting the model; it retries automatically. Persistent errors suggest a model that is weak at visual grounding.