AI Agent Integration

Why tauri-pilot for AI agents

No existing tool lets AI agents interact with Tauri app UIs. The gap exists because:

• Playwright doesn’t work — Playwright drives standalone browser processes (Chromium, Firefox, and its own bundled WebKit build). Tauri embeds the system WebView directly (WebKitGTK on Linux, WebKit on macOS), so there is no browser process for Playwright to attach to.
• tauri-pilot speaks a protocol optimized for LLM consumption — the accessibility tree output is text-based, compact, and structured to be read directly by a language model.
• Refs map to UI elements unambiguously — @e3 is a stable handle within a snapshot, removing the need for CSS selectors or XPath expressions.

The snapshot → interact → verify workflow

The core loop for any AI agent using tauri-pilot is: take a snapshot to discover elements, interact using refs, then verify the result.

# Step 1: Get the accessibility tree
tauri-pilot snapshot -i

# Output:
# - heading "PR Dashboard" [ref=e1]
# - textbox "Search PRs" [ref=e2] value=""
# - button "Refresh" [ref=e3]
# - list "PR List" [ref=e4]
#   - listitem "fix: resolve memory leak #142" [ref=e5]

# Step 2: Interact using refs
tauri-pilot click @e3

# Step 3: See what changed (instead of re-reading the full tree)
tauri-pilot diff -i

The diff command compares the current page with the last snapshot and returns only added, removed, and changed elements. This saves significant tokens — a typical diff after a click is 2-5 lines vs 50-100 for a full re-snapshot. Pass diff the same -i/-s/-d as that snapshot, or it refuses: a snapshot captured with other options would show every filtered element as added or removed.

The -i flag filters to interactive elements only, reducing noise in the output. That set includes native controls and unmapped hosts with draggable="true", contenteditable, an onclick handler, or tabindex.

Structured output with –json

Use the --json flag to get machine-parseable output when the agent needs to process responses programmatically.

tauri-pilot snapshot --json
tauri-pilot text @e1 --json
tauri-pilot url --json

JSON output is useful when the agent needs to extract specific values, compare states, or pass data between steps without text parsing.

Example: Automated UI testing workflow

# 1. Health check
tauri-pilot ping

# 2. Navigate to a page
tauri-pilot navigate "http://localhost:1420/settings"

# 3. Wait for page load
tauri-pilot wait --selector ".settings-form"

# 4. Snapshot to discover elements
tauri-pilot snapshot -i

# 5. Fill a form
tauri-pilot fill @e2 "new-value"

# 6. Submit
tauri-pilot click @e5

# 7. Verify success
tauri-pilot wait --selector ".success-toast"
tauri-pilot snapshot -i
tauri-pilot assert text @e1 "Settings saved"
tauri-pilot assert url "/settings"

Best practices for snapshot parsing

• Always take a fresh snapshot before interacting — refs reset on each snapshot. @e1 in one snapshot may refer to a different element in the next.
• Use diff instead of re-snapshotting — after an interaction, tauri-pilot diff -i (after snapshot -i) returns only what changed. This is much cheaper than re-reading the full tree.
• Use -i to filter interactive elements — this reduces output size and makes the tree easier to parse.
• Use -s to scope to a section — tauri-pilot snapshot -s "#sidebar" limits the tree to a subtree, further reducing noise.
• Use wait before snapshot — after navigation or interaction, wait for the page to settle before taking a snapshot to avoid acting on stale state.
• Use assert for verification — tauri-pilot assert text @e1 "Dashboard" returns exit 0 on match, exit 1 on mismatch. This replaces the three-step text @e1 → parse → compare pattern, saving a round-trip and token parsing.
• Save snapshots for multi-step workflows — tauri-pilot snapshot --save before.snap then tauri-pilot diff --ref before.snap lets you compare against any point in time.
• Record agent workflows for regression testing — wrap your interaction sequence in record start / record stop --output test.json to capture actions as a replayable script. Use replay test.json to re-run the same sequence, or replay test.json --export sh to generate a standalone shell script. Ref steps are saved with a stable selector and fingerprint, so the replay does not depend on a snapshot; if record stop lists steps with no stable locator, give those elements an id or data-testid and re-record.

# Assert examples — one-step verification
tauri-pilot assert text @e1 "Dashboard"           # exact text match
tauri-pilot assert visible @e3                     # element is visible
tauri-pilot assert value @e2 "workspace"           # input value
tauri-pilot assert count ".list-item" 5            # element count
tauri-pilot assert contains @e1 "error"            # partial text match
tauri-pilot assert url "/dashboard"                # URL substring

# Scoped snapshot example
tauri-pilot snapshot -i -s "#main-content"

Multi-window apps

For Tauri apps with multiple windows, use tauri-pilot windows to discover available windows and --window <label> to target a specific one:

# List all windows
tauri-pilot windows
# main      http://localhost:1420/        My App
# settings  http://localhost:1420/settings Settings

# Target a specific window
tauri-pilot --window settings snapshot -i
tauri-pilot --window settings fill @e2 "new-value"

# Or set via environment variable
export TAURI_PILOT_WINDOW=settings
tauri-pilot snapshot -i

Without --window, all commands target the main window (or the first available). The window flag works with every command including --follow mode and ipc.

Integration with Claude Code

tauri-pilot is designed to be used as a tool by Claude Code directly:

• Claude Code can read the snapshot output directly — it is an accessibility tree, the same representation Claude Code uses internally for UI reasoning.
• The ref system (@e1) maps directly to how Claude Code thinks about UI elements as discrete, addressable targets.
• Combine with --json for structured data extraction when the agent needs to compare values or branch on state.

A typical Claude Code session using tauri-pilot looks like this:

# Claude Code calls these as shell commands during a task
tauri-pilot ping                          # verify the app is running (not that a window is ready)
tauri-pilot snapshot -i                   # discover what's on screen
tauri-pilot fill @e2 "search query"       # interact
tauri-pilot click @e3                     # submit
tauri-pilot wait --selector ".results"    # wait for response
tauri-pilot diff -i                       # see what changed (token-efficient)
tauri-pilot snapshot -i                   # refresh refs
tauri-pilot assert text @e1 "Results"     # verify in one step

No custom integration code is needed — tauri-pilot is a CLI that Claude Code can invoke directly.

MCP Server

Agents with native Model Context Protocol support can use tauri-pilot as a stdio MCP server:

{
  "mcpServers": {
    "tauri-pilot": {
      "command": "tauri-pilot",
      "args": ["mcp"]
    }
  }
}

The MCP server exposes the same app-facing commands as structured tools, namespaced under pilot.*: pilot.snapshot, pilot.click, pilot.fill, pilot.logs, pilot.network, pilot.eval, pilot.ipc, pilot.assert_*, pilot.run, and more. pilot.run takes a scenario path or inline TOML content and returns a JSON report (ok, counts, summary, steps). A finished run including failed steps is a successful tool result with ok false; only parse, step-key, I/O, connect, and timeout failures are tool errors. tools/list advertises the prefixed names; bare names (e.g. snapshot) still resolve via tools/call for backwards compatibility. Tool calls return structured JSON content, so agents do not need to parse terminal output. pilot.screenshot is the exception: on success it returns one image content block (mimeType: "image/png", base64 data) and no structured content, so the agent sees the picture; its errors keep the usual error fields. Without selector it captures the whole scroll height, which some MCP clients reject as too large an image on long pages: pass a selector there.

A tool error the app raised keeps its fields: error holds the domain code when the app sent one (e.g. WINDOW_NOT_FOUND, or NO_WEBVIEW when the app has no window yet, else the message), next to message, the JSON-RPC rpc_code and any detail such as available_windows. If the app’s detail holds a different, non-null message or rpc_code, or an error that is not a string, that value moves to data_<key> (e.g. data_message) instead of being dropped. Other tool errors carry only error, as text.

Use global flags before mcp when an agent should target a specific app socket or window:

{
  "mcpServers": {
    "tauri-pilot": {
      "command": "tauri-pilot",
      "args": ["--socket", "/run/user/1000/tauri-pilot-com.myapp.dev.sock", "--window", "main", "mcp"]
    }
  }
}

The path is the default on a Linux desktop: replace 1000 with your user ID (id -u) and com.myapp.dev with your app’s identifier. On macOS, where $XDG_RUNTIME_DIR is usually unset, use /tmp/tauri-pilot-com.myapp.dev.sock. On Windows, use the Named Pipe, with each backslash doubled in JSON: "\\\\.\\pipe\\tauri-pilot-com.myapp.dev".