> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mobilerun.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# DeviceDriver Base Class
Base class defining the interface for all device drivers.
## DeviceDriver
```python theme={null}
class DeviceDriver
```
Base class for all device drivers.
`DeviceDriver` is the base class for asynchronous device drivers in `mobilerun-core-local`. Use `supported` to check optional device operations. Most unavailable operations raise `NotImplementedError`; `input_coordinate_size()` is always available and defaults to the screenshot dimensions.
***
## Quick Reference
**Driver Methods:**
* `connect()`, `ensure_connected()`
* `tap()`, `swipe()`, `input_text()`, `press_button()`, `press_key_code()`, `drag()`
* `start_app()`, `stop_app()`, `install_app()`, `uninstall_app()`, `get_apps()`, `list_packages()`
* `screenshot()`, `input_coordinate_size()`, `get_ui_tree()`, `get_date()`
**Key Attributes:**
* `supported`: `set[str]` - Set of method names the driver implements. Check membership before calling.
* `supported_buttons`: `set[str]` - Set of button names accepted by `press_button()` (e.g. `{"back", "home", "enter"}`).
***
## How It Works
`DeviceDriver` sends commands to the device. `StateProvider` converts device data into `UIState`. Action functions receive an `ActionContext` and return an `ActionResult`.
### Imports
Import asynchronous drivers from `mobilerun-core-local`:
```python theme={null}
from mobilerun_core_local import (
AndroidDriver,
AndroidPortalHttpDriver,
IOSPortalHttpDriver,
)
from mobilerun_core_local.driver import (
RecordingDriver,
StealthDriver,
VisualRemoteDriver,
)
from mobilerun_core_local.driver.cloud import CloudDriver
```
Install `mobilerun-core-local[cloud]` to use `CloudDriver`.
***
## Common Interface
All DeviceDriver implementations may provide these methods (check `supported` set for availability):
### Lifecycle
* `connect() -> None` - Establish connection to the device
* `ensure_connected() -> None` - Connect if not already connected
### Input Actions
* `tap(x: int, y: int) -> None` - Tap at screen coordinates (pixels on Android; logical points on iOS)
* `swipe(x1: int, y1: int, x2: int, y2: int, duration_ms: float = 1000) -> None` - Swipe gesture
* `drag(x1: int, y1: int, x2: int, y2: int, duration: float = 3.0) -> None` - Drag gesture
* `input_text(text: str, clear: bool = False, stealth: bool = False, wpm: int = 0) -> bool` - Text input into focused field. `stealth` enables human-like typing delays; `wpm` sets the typing speed in words per minute (0 = instant).
* `press_button(button: str) -> None` - Press a named button (e.g. back, home, enter). Raises `ValueError` if not in `supported_buttons`.
* `press_key_code(key_code: int) -> None` - Press an integer key code.
### App Management
* `start_app(package: str, activity: str | None = None) -> str` - Launch app
* `stop_app(package: str) -> str` - Stop a running app
* `install_app(path: str, **kwargs) -> str` - Install app
* `uninstall_app(package: str) -> str` - Uninstall app
* `list_packages(include_system: bool = False) -> List[str]` - List packages
* `get_apps(include_system: bool = True) -> List[Dict[str, str]]` - Get apps with labels
### State / Observation
* `screenshot(hide_overlay: bool = True) -> bytes` - Capture screen as PNG bytes
* `input_coordinate_size(screenshot_width: int, screenshot_height: int) -> tuple[int, int]` - Return the dimensions used for input coordinates. On iOS, these can differ from screenshot dimensions.
* `get_ui_tree() -> Dict[str, Any]` - Get raw UI / accessibility tree
* `get_date() -> str` - Get device date/time
***
## StateProvider
```python theme={null}
class StateProvider:
def __init__(self, driver: DeviceDriver): ...
async def get_state(self) -> UIState: ...
```
Base class for state providers. Subclass it to support another platform. Its `supported` set lists available UI features, such as element lookup and coordinate conversion.
### AndroidStateProvider
```python theme={null}
class AndroidStateProvider(StateProvider)
```
Fetches and formats device state as a `UIState`. Set `stealth=True` to vary tap coordinates within element bounds.
***
## UIState
```python theme={null}
class UIState
```
Holds parsed UI elements for a single device state snapshot.
**Key Methods:**
* `get_element(index: int) -> Dict | None` - Recursively find an element by its index
* `get_element_coords(index: int) -> Tuple[int, int]` - Return the centre (x, y) of an element. Raises `ValueError` when element is missing or has no bounds.
* `get_element_info(index: int) -> Dict` - Return element metadata (text, className, type, child\_texts)
* `get_clear_point(index: int) -> Tuple[int, int]` - Find a tap point that avoids overlapping elements (falls back to centre)
* `convert_point(x: int, y: int) -> Tuple[int, int]` - Convert point to absolute pixels if normalized mode is active
**Key Attributes:**
* `elements` - List of parsed UI elements
* `formatted_text` - Formatted text representation of the UI tree
* `focused_text` - Text of the currently focused element
* `phone_state` - Dict with current activity, keyboard visibility, etc.
* `screen_width` / `screen_height` - Device screen dimensions
* `use_normalized` - Whether normalized coordinate mode is active
***
## ActionContext
```python theme={null}
class ActionContext
```
Everything an action function needs to interact with the device.
**Attributes:**
* `driver` - `DeviceDriver` instance for raw device I/O
* `ui` - `UIState` instance for element resolution (refreshed each step)
* `shared_state` - `MobileAgentState` for shared agent state
* `state_provider` - `StateProvider` for fetching fresh UI state
* `app_opener_llm` - LLM instance for app opening workflow (optional)
* `credential_manager` - CredentialManager instance (optional)
* `streaming` - Whether streaming is enabled
***
## ActionResult
```python theme={null}
@dataclass
class ActionResult:
success: bool
summary: str
```
Structured return type from action functions. The `summary` field describes the result.
***
## Action Functions
Action functions follow this pattern:
```python theme={null}
async def click(index: int, *, ctx: ActionContext) -> ActionResult:
"""Click the element with the given index."""
x, y = ctx.ui.get_element_coords(index)
await ctx.driver.tap(x, y)
return ActionResult(success=True, summary=f"Clicked on element at ({x}, {y})")
```
**Available actions:**
* `click(index)` - Click UI element by index
* `click_at(x, y)` - Click at screen coordinates
* `click_area(x1, y1, x2, y2)` - Click center of area defined by coordinates
* `long_press(index)` - Long press UI element by index
* `long_press_at(x, y)` - Long press at screen coordinates
* `type(text, index=None, clear=False)` - Optionally focus an indexed element, then input text (set `clear=True` to clear first)
* `type_text(text, clear=False)` - Input text into the focused field
* `type_secret(secret_id, index)` - Input a configured credential into an indexed element
* `swipe(coordinate, coordinate2, duration=1.0)` - Swipe gesture between two coordinate lists
* `system_button(button)` - Press system buttons (back, home, enter)
* `open_app(...)` - Open an Android app by name or an iOS or Visual Remote app by ID
* `wait(duration=1.0)` - Wait for a duration in seconds
* `complete(success, message)` - Mark task as finished
**Coordinate tools (`click_at`, `click_area`, `long_press_at`) are disabled by default.** Vision enables `click_at` when normalized coordinates are off. Screenshot-only mode enables all three. To enable all three with standard vision, set `disabled_tools: []` in `ToolsConfig`. See [Vision Mode](/framework/features/vision).
***
## Custom Tool Integration
### Adding Custom Tools
```python theme={null}
def my_custom_tool(param: str, **kwargs) -> str:
"""Custom tool description."""
return f"Result: {param}"
custom_tools = {
"my_custom_tool": {
"parameters": {
"param": {"type": "string", "required": True},
},
"description": "Custom tool description with usage example",
"function": my_custom_tool
}
}
agent = MobileAgent(
goal="Do something",
config=config,
custom_tools=custom_tools
)
```
***
## Driver and Platform Comparison
| Driver | Platform / transport | Supported methods and limits |
| - | - | - |
| `AndroidDriver` | Android over ADB; Portal optional | Tap, swipe, text, named buttons, integer key codes, app start/stop/install/uninstall, packages, screenshot, tree, and date. Drag is unavailable. |
| `AndroidPortalHttpDriver` | Android Portal HTTP; no ADB | Tap, swipe, text, buttons/key codes, app start/stop, packages, screenshot, and tree. A bearer token is required. Install, uninstall, date, and drag are unavailable. |
| `IOSPortalHttpDriver` | iOS through `mobilerun-ios --local` | Tap, coordinate swipe, text with clear, five named buttons, integer key codes, app start/stop/install, packages, screenshot, tree, and date. Uninstall and drag are unavailable. Input coordinates are logical iOS points. |
| `CloudDriver` | Mobilerun Cloud through the async SDK | Tap, swipe, text, Back/Home/Enter, app start, packages, screenshot, tree, and date. Drag, install, stop, uninstall, and key codes are unavailable. |
| `VisualRemoteDriver` | Screenshot/action HTTP server | The server reports its available actions and buttons when it connects. Screenshots are required; an accessibility tree is not available. |
`RecordingDriver` and `StealthDriver` support the same methods as the driver they wrap.
***
## Best Practices
### 1. Check supported methods before calling
```python theme={null}
if "get_date" in driver.supported:
date = await driver.get_date()
else:
date = "Unknown"
```
### 2. Use ActionContext for agent-level interactions
```python theme={null}
# Action functions use ctx for all device interaction
async def my_action(param: str, *, ctx: ActionContext) -> ActionResult:
x, y = ctx.ui.get_element_coords(5)
await ctx.driver.tap(x, y)
return ActionResult(success=True, summary="Done")
```
### 3. Use StateProvider for UI state
```python theme={null}
from mobilerun.tools import AndroidStateProvider
from mobilerun_core_local import AndroidDriver
# StateProvider handles fetching + parsing + retries
provider = AndroidStateProvider(driver, tree_filter=my_filter, tree_formatter=my_formatter)
ui_state = await provider.get_state()
# UIState provides element lookup
element = ui_state.get_element(5)
x, y = ui_state.get_element_coords(5)
```
***
## Error Handling
Driver methods use consistent error handling:
**Unsupported operations:**
```python theme={null}
from mobilerun_core_local import PlatformUnsupportedError
try:
await driver.uninstall_app("com.example.app")
except PlatformUnsupportedError:
print("This platform does not support uninstall")
except NotImplementedError:
print("This driver does not implement uninstall")
```
`PlatformUnsupportedError` is a subclass of `NotImplementedError`. Check `driver.supported` before calling an optional method. `press_button()` raises `ValueError` for names outside `supported_buttons`. `CloudDriver` raises `DeviceDisconnectedError` for SDK connection, timeout, and conflict failures. Other connection and authentication failures may raise connection, HTTP, or permission errors.
**ActionResult for action functions:**
```python theme={null}
result = await click(5, ctx=ctx)
if not result.success:
print(f"Action failed: {result.summary}")
```
***
## See Also
* [AndroidDriver API](/framework/sdk/adb-tools) - Android driver
* [IOSPortalHttpDriver API](/framework/sdk/ios-tools) - iOS driver
* [MobileAgent API](/framework/sdk/droid-agent) - Agent integration
* [Configuration](/framework/sdk/configuration) - Configuration reference
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.