Accessibility Tree-Based AI Browser Automation Architecture with Playwright MCP

Wed, 09 Sep 2026 10:03:04 +0900

In automated web browser testing and E2E verification, implementations relying on hardcoded CSS selectors or XPath easily break due to subtle UI modifications or dynamic class name obfuscation (CSS-in-JS, Tailwind, production build minification). Furthermore, recent screenshot coordinate estimation approaches using LLMs or multimodal models are vulnerable to screen resolution discrepancies and responsive rendering variations, frequently causing non-deterministic operational errors.

To eliminate such operational friction, “Playwright MCP,” based on the Model Context Protocol (MCP) being developed within the Microsoft ecosystem, has been introduced. This article outlines the architecture of the Accessibility Tree (A11y Tree) snapshot mechanism adopted by Playwright MCP, the deployment procedures for various MCP client environments, and troubleshooting strategies during operation.

Structural Approach: Accessibility Tree Snapshot Mechanism

Rather than relying on pixel-level image recognition or decorative nested <div> tag structures, Playwright MCP provides the Accessibility Tree—interpreted by the OS and screen readers—to the LLM as context.

[ Web Page / Raw DOM Tree ]
 │
 ▼
[ Accessibility Tree Engine (Playwright) ]
 ├── Semantic role extraction (button, textbox, link)
 ├── State &amp; attribute analysis (aria-*, expanded, checked)
 └── Issuance of deterministic Element References
 │
 ▼
[ LLM Context Window (Structured text representation) ]

This architecture provides the following advantages:

Elimination of Selector Breakage: Identifies target elements based on their roles and Accessible Names, preventing script failures caused by CSS class name changes or DOM hierarchy modifications.
Token Efficiency Optimization: Reduces context window consumption by transmitting a pruned tree containing only semantic information, rather than whole-screen pixel data or extensive HTML strings to the LLM.
Deterministic Operations: A unique reference index (Element Reference) is assigned to each element, explicitly determining the target for clicks and inputs.

MCP Server Architecture and Client Configuration

Prerequisites

Node.js: 20.x or higher recommended

Client Configuration (mcpServers)

Add the following JSON block to the configuration files of MCP clients such as Claude Desktop, Cursor, Cline, or VS Code (e.g., claude_desktop_config.json or the Cursor configuration pane).

{
 "mcpServers": {
 "playwright": {
 "command": "npx",
 "args": [
 "@playwright/mcp@latest"
 ]
 }
 }
}

Execution Loop and Internal State Machine

Playwright MCP operates not as a single static command execution, but as a closed-loop state machine between the AI agent and the browser.

┌────────────────────────────────────────────────────────┐
│ AI Agent │
└───────────────────────────┬────────────────────────────┘
 │ (1) Start session / Navigation instruction
 ▼
┌────────────────────────────────────────────────────────┐
│ Browser Process (Playwright) │
└───────────────────────────┬────────────────────────────┘
 │ (2) Page load complete / Generate A11y snapshot
 ▼
┌────────────────────────────────────────────────────────┐
│ Accessibility Snapshot Generator │
└───────────────────────────┬────────────────────────────┘
 │ (3) Return semantic tree
 ▼
┌────────────────────────────────────────────────────────┐
│ AI Agent │
│ (Identify element references &amp; infer execution actions) │
└───────────────────────────┬────────────────────────────┘
 │ (4) Send operation command (Click, Fill, etc.)
 ▼
┌────────────────────────────────────────────────────────┐
│ Playwright MCP Execution Pipeline │
└───────────────────────────┬────────────────────────────┘
 │ (5) Recapture updated DOM state
 ▼
 [ Continue to next step ]

Primary Tool Interfaces Provided

The primary tool interfaces exposed by Playwright MCP to clients are as follows:

browser_navigate: Controls navigation to URLs, back/forward history traversal, and reloading.
A11y Tree Target Resolution: Dispatches Click events to elements identified on a role basis.
Semantic Form Handlers: Handles string input into form fields, checkbox toggles, and dropdown selections.
Screenshot Engine: Extracts screen captures on a full-page or per-element basis.
Storage State Persistence: Serializes session information (Cookies, localStorage) to enable authentication bypass and reuse across test scenarios.
browser_run_code_unsafe: Directly executes arbitrary Playwright scripts from the LLM (*Executes outside the sandbox; use only in isolated, trusted environments).

Troubleshooting

1. Missing Browser Binary Error

When running npx @playwright/mcp@latest, if browser binaries such as Chromium are missing in the host environment, an error occurs immediately upon process startup.

Error: browserType.launch: Executable doesn't exist at /root/.cache/ms-playwright/chromium-1155/chrome-linux/chrome
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
╚═════════════════════════════════════════════════════════════════════════╝

Resolution Steps: Pre-install browser dependencies within the host environment or container.

npx playwright install --with-deps chromium

2. Missing System Dependencies in Headless Environments

When running in a Linux server environment, missing shared libraries required for GUI rendering will cause a host system dependencies error. In this case, installing additional packages suited to the distribution is required.

sudo npx playwright install-deps

3. Verifying Deployment Verification Logs

Verify that the MCP process is correctly resident and in a state to accept the JSON-RPC protocol via standard output and the process list.

$ ps aux | grep playwright
node /usr/local/bin/npx @playwright/mcp@latest
/root/.cache/ms-playwright/chromium-1155/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --headless=new --remote-debugging-pipe

Configuration Notes

Playwright MCP is an approach that resolves the challenges of both traditional “code-based procedural testing” and “ambiguous UI manipulation via visual models.” By utilizing the Accessibility Tree as a standard interface, a browser automation pipeline based on robust semantic analysis can be constructed. For integration into production environments, properly designing the serialization management of authentication states and execution permission controls is essential.

Accessibility-Tree on K-Life Hack | Systems Architecture & DevOps