DeviceDriver Base Class

​

DeviceDriver

class DeviceDriver

​

Quick Reference

• connect(), ensure_connected()
• tap(), swipe(), input_text(), press_button(), press_key_code(), drag()
• start_app(), stop_app(), install_app(), uninstall_app(), get_apps(), list_packages()
• screenshot(), input_coordinate_size(), get_ui_tree(), get_date()

• supported: set[str] - Set of method names the driver implements. Check membership before calling.
• supported_buttons: set[str] - Set of button names accepted by press_button() (e.g. {"back", "home", "enter"}).

​

How It Works

​

Imports

from mobilerun_core_local import (
    AndroidDriver,
    AndroidPortalHttpDriver,
    IOSPortalHttpDriver,
)
from mobilerun_core_local.driver import (
    RecordingDriver,
    StealthDriver,
    VisualRemoteDriver,
)
from mobilerun_core_local.driver.cloud import CloudDriver

​

Common Interface

​

Lifecycle

• connect() -> None - Establish connection to the device
• ensure_connected() -> None - Connect if not already connected

​

Input Actions

• tap(x: int, y: int) -> None - Tap at screen coordinates (pixels on Android; logical points on iOS)
• swipe(x1: int, y1: int, x2: int, y2: int, duration_ms: float = 1000) -> None - Swipe gesture
• drag(x1: int, y1: int, x2: int, y2: int, duration: float = 3.0) -> None - Drag gesture
• input_text(text: str, clear: bool = False, stealth: bool = False, wpm: int = 0) -> bool - Text input into focused field. stealth enables human-like typing delays; wpm sets the typing speed in words per minute (0 = instant).
• press_button(button: str) -> None - Press a named button (e.g. back, home, enter). Raises ValueError if not in supported_buttons.
• press_key_code(key_code: int) -> None - Press an integer key code.

​

App Management

• start_app(package: str, activity: str | None = None) -> str - Launch app
• stop_app(package: str) -> str - Stop a running app
• install_app(path: str, **kwargs) -> str - Install app
• uninstall_app(package: str) -> str - Uninstall app
• list_packages(include_system: bool = False) -> List[str] - List packages
• get_apps(include_system: bool = True) -> List[Dict[str, str]] - Get apps with labels

​

State / Observation

• screenshot(hide_overlay: bool = True) -> bytes - Capture screen as PNG bytes
• input_coordinate_size(screenshot_width: int, screenshot_height: int) -> tuple[int, int] - Return the dimensions used for input coordinates. On iOS, these can differ from screenshot dimensions.
• get_ui_tree() -> Dict[str, Any] - Get raw UI / accessibility tree
• get_date() -> str - Get device date/time

​

StateProvider

class StateProvider:
    def __init__(self, driver: DeviceDriver): ...
    async def get_state(self) -> UIState: ...

​

AndroidStateProvider

class AndroidStateProvider(StateProvider)

​

UIState

class UIState

• get_element(index: int) -> Dict | None - Recursively find an element by its index
• get_element_coords(index: int) -> Tuple[int, int] - Return the centre (x, y) of an element. Raises ValueError when element is missing or has no bounds.
• get_element_info(index: int) -> Dict - Return element metadata (text, className, type, child_texts)
• get_clear_point(index: int) -> Tuple[int, int] - Find a tap point that avoids overlapping elements (falls back to centre)
• convert_point(x: int, y: int) -> Tuple[int, int] - Convert point to absolute pixels if normalized mode is active

• elements - List of parsed UI elements
• formatted_text - Formatted text representation of the UI tree
• focused_text - Text of the currently focused element
• phone_state - Dict with current activity, keyboard visibility, etc.
• screen_width / screen_height - Device screen dimensions
• use_normalized - Whether normalized coordinate mode is active

​

ActionContext

class ActionContext

• driver - DeviceDriver instance for raw device I/O
• ui - UIState instance for element resolution (refreshed each step)
• shared_state - MobileAgentState for shared agent state
• state_provider - StateProvider for fetching fresh UI state
• app_opener_llm - LLM instance for app opening workflow (optional)
• credential_manager - CredentialManager instance (optional)
• streaming - Whether streaming is enabled

​

ActionResult

@dataclass
class ActionResult:
    success: bool
    summary: str

​

Action Functions

async def click(index: int, *, ctx: ActionContext) -> ActionResult:
    """Click the element with the given index."""
    x, y = ctx.ui.get_element_coords(index)
    await ctx.driver.tap(x, y)
    return ActionResult(success=True, summary=f"Clicked on element at ({x}, {y})")

• click(index) - Click UI element by index
• click_at(x, y) - Click at screen coordinates
• click_area(x1, y1, x2, y2) - Click center of area defined by coordinates
• long_press(index) - Long press UI element by index
• long_press_at(x, y) - Long press at screen coordinates
• type(text, index=None, clear=False) - Optionally focus an indexed element, then input text (set clear=True to clear first)
• type_text(text, clear=False) - Input text into the focused field
• type_secret(secret_id, index) - Input a configured credential into an indexed element
• swipe(coordinate, coordinate2, duration=1.0) - Swipe gesture between two coordinate lists
• system_button(button) - Press system buttons (back, home, enter)
• open_app(...) - Open an Android app by name or an iOS or Visual Remote app by ID
• wait(duration=1.0) - Wait for a duration in seconds
• complete(success, message) - Mark task as finished

​

Custom Tool Integration

​

Adding Custom Tools

def my_custom_tool(param: str, **kwargs) -> str:
    """Custom tool description."""
    return f"Result: {param}"

custom_tools = {
    "my_custom_tool": {
        "parameters": {
            "param": {"type": "string", "required": True},
        },
        "description": "Custom tool description with usage example",
        "function": my_custom_tool
    }
}

agent = MobileAgent(
    goal="Do something",
    config=config,
    custom_tools=custom_tools
)

​

Driver and Platform Comparison

​

Best Practices

​

1. Check supported methods before calling

if "get_date" in driver.supported:
    date = await driver.get_date()
else:
    date = "Unknown"

​

2. Use ActionContext for agent-level interactions

# Action functions use ctx for all device interaction
async def my_action(param: str, *, ctx: ActionContext) -> ActionResult:
    x, y = ctx.ui.get_element_coords(5)
    await ctx.driver.tap(x, y)
    return ActionResult(success=True, summary="Done")

​

3. Use StateProvider for UI state

from mobilerun.tools import AndroidStateProvider
from mobilerun_core_local import AndroidDriver

# StateProvider handles fetching + parsing + retries
provider = AndroidStateProvider(driver, tree_filter=my_filter, tree_formatter=my_formatter)
ui_state = await provider.get_state()

# UIState provides element lookup
element = ui_state.get_element(5)
x, y = ui_state.get_element_coords(5)

​

Error Handling

from mobilerun_core_local import PlatformUnsupportedError

try:
    await driver.uninstall_app("com.example.app")
except PlatformUnsupportedError:
    print("This platform does not support uninstall")
except NotImplementedError:
    print("This driver does not implement uninstall")

result = await click(5, ctx=ctx)
if not result.success:
    print(f"Action failed: {result.summary}")

​

See Also

• AndroidDriver API - Android driver
• IOSPortalHttpDriver API - iOS driver
• MobileAgent API - Agent integration
• Configuration - Configuration reference