> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mobilerun.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision Mode

> Screenshots for agents, and how screen coordinates stay accurate

With vision enabled, agents receive a screenshot of the device alongside the
accessibility tree on every step.

## Enabling vision

```yaml theme={null}
agent:
  manager:
    vision: true      # planning agent sees screenshots
  executor:
    vision: true      # action agent sees screenshots
  fast_agent:
    vision: true      # direct-execution agent sees screenshots
```

```bash theme={null}
# CLI: enable for all agents at once
mobilerun run "Open settings and enable dark theme" --vision
```

```python theme={null}
# SDK
from mobilerun import MobileConfig
from mobilerun.config_manager import AgentConfig, FastAgentConfig

config = MobileConfig(agent=AgentConfig(fast_agent=FastAgentConfig(vision=True)))
```

`vision_only: true` (CLI: `--vision-only`) is a separate mode that drops the
accessibility tree and drives the device from screenshots alone.

## The coordinate contract

With vision active, coordinates are handled automatically:

1. The screenshot is resized to the exact dimensions the model will ground
   on, with a labeled coordinate grid drawn on it.
2. The device state declares that coordinate space, and element bounds in the
   accessibility tree text are emitted in the same space.
3. Coordinates the model passes to `click_at`, `click_area`, `long_press_at`,
   and `swipe` are converted back automatically — to device pixels on
   Android, to points on iOS.
4. Coordinates outside the declared space are rejected with a corrective
   error and the model retries.

No configuration is needed. Element-index actions (`click`, `long_press`) are
unaffected; they always tap accessibility-tree bounds.

### Per-model effective screenshot size

Some providers downsize images server-side before the model sees them. To
keep coordinates exact, mobilerun resolves the screenshot dimensions each
active vision model actually grounds on and resizes to the smallest result
across all recipients (manager + executor, or the fast agent):

| Model family                         | Longest edge |
| ------------------------------------ | ------------ |
| Anthropic standard (e.g. Sonnet)     | 1568 px      |
| Anthropic high-res (e.g. Opus, 4.x+) | 2576 px      |
| OpenAI, Gemini, Ollama               | 2048 px      |

Unknown Anthropic model IDs fall back to the standard 1568 budget. The
declared coordinate space always matches the image the model receives, so
`convert_point` is exact regardless of which model is configured.

### Capping the screenshot size

For local or undocumented vision models that downsize to a size mobilerun
can't infer (for example, certain Ollama models), set an explicit cap with
`agent.model_screenshot_max_side`. The cap is applied on top of per-model
resolution and uses the smaller of the two.

```yaml theme={null}
agent:
  # Cap the model-facing screenshot at 1280 px on the longest edge.
  model_screenshot_max_side: 1280
```

```python theme={null}
from mobilerun import MobileConfig
from mobilerun.config_manager import AgentConfig

config = MobileConfig(agent=AgentConfig(model_screenshot_max_side=1280))
```

Leave this unset for the supported providers above — the per-model defaults
already match what they ground on.

## Coordinate tools and vision

`swipe` is always available. `click_at`, `click_area`, and `long_press_at`
are disabled by default; enabling vision re-enables `click_at` automatically
(non-normalized). To enable all coordinate tools:

```yaml theme={null}
tools:
  disabled_tools: []
```

## Normalized coordinates

`agent.use_normalized_coordinates: true` makes the model emit `[0-1000]`
coordinates instead of pixels; the coordinate contract is inactive in this
mode.

<Note>
  An occasional "Coordinates ... are outside the ... coordinate space" error is
  the guardrail correcting the model; it retries automatically. Persistent
  errors suggest a model that is weak at visual grounding.
</Note>