Browser-agent demos frame the problem as perception: can a model understand a webpage well enough to click the right thing?

Production failures are often more mundane. The model chose element 17, the single-page application re-rendered, and element 17 now refers to a different node. The hard problem is referential integrity across changing browser states.

Browser Use addresses that problem by constructing an action-oriented page model and retaining multiple identities for every interacted element.

Four Chrome views become one state

The current DomService uses Chrome DevTools Protocol calls rather than handing raw HTML to the model. It obtains several views in parallel:

  • DOM.getDocument for the full node tree, with pierce enabled for Shadow DOM;
  • Accessibility.getFullAXTree for semantic roles, names, and states across frames;
  • DOMSnapshot.captureSnapshot for computed styles, paint order, rectangles, and layout data;
  • viewport and device-pixel information for coordinate normalization.

It also detects JavaScript click listeners on manageable pages. Cross-origin frames require target discovery, recursive tree construction, depth limits, and coordinate translation. Same-origin iframe scroll offsets and high-DPI screenshots introduce additional coordinate systems that must agree.

The trees are joined primarily through backend DOM node IDs. An enhanced node can therefore carry structural identity, accessibility semantics, computed visibility, screen bounds, click-listener evidence, frame ownership, and absolute position.

Serialization is an action compiler

The model does not receive the full enhanced tree. DOMTreeSerializer creates a smaller representation optimized for action:

  1. filter scripts, styles, metadata, and decorative SVG children;
  2. infer interactivity from native tags, ARIA roles, accessibility properties, listeners, and pointer style;
  3. remove nodes hidden by visibility and paint-order constraints;
  4. collapse wrapper structures and propagate relevant bounds;
  5. assign compact indices to interactive elements;
  6. build selector_map[index] -> EnhancedDOMTreeNode.

The integer index is convenient for the model, but it is explicitly state-local. Browser Use keeps the richer node object because the index alone cannot survive a re-render.

Vision is a separate, optional evidence channel. The implementation captures screenshots for every step so history and cloud synchronization remain useful. Whether an image enters the LLM context depends on use_vision: true includes it every step, auto includes it when an action requests it, and false omits it from the model message.

That is more precise than “switch to vision when DOM fails.” DOM, accessibility, geometry, and screenshots provide different evidence about the same page state.

The reattachment ladder

When a historical action must be mapped onto the current page, Browser Use searches the new selector map through a five-level fallback:

  1. Exact element hash — includes the full attribute and accessibility identity.
  2. Stable hash — removes dynamic CSS classes such as focus, hover, and animation state.
  3. XPath — uses structural position in the current DOM.
  4. Accessible name — matches node type plus the name derived from the AX tree.
  5. Unique attribute — falls back to name, id, or aria-label.

The order moves from strict identity toward semantic approximation. Exact matching avoids confusing repeated labels. Stable hashing tolerates framework-generated class churn. XPath survives some attribute changes but is sensitive to reordering. Accessibility names are effective for menus and SPAs whose structure is regenerated. Unique attributes maintain compatibility with older histories.

If every level fails, the safe result is not to guess. The referent no longer exists under the available evidence.

Why screenshots are not enough

A vision model can recognize a blue “Submit” button, but it may not know which frame owns it, whether a transparent overlay intercepts the click, or whether the same-looking control represents the same object after navigation.

DOM structure provides addressability. Accessibility data provides semantic identity. Snapshot geometry provides physical clickability. Vision provides visual context. Reliable automation comes from agreement among these representations, not from declaring one of them universally superior.

Personal take

Browser Use will resonate with QA teams, operations automation, data-extraction products, and developers integrating systems that will never expose a complete API. For much of the economy, the browser remains the only universal integration layer.

The market will move from “the agent can click” to “the agent can prove what it clicked.” Applications involving jobs, purchases, finance, or healthcare need pre-action target identity and post-action state evidence, not only a natural-language trace generated by the model.

There is also a constructive social effect: browser agents reward accessible markup. Stable roles, names, labels, and states help humans who use assistive technology and machines that need semantic controls. Accessibility may shift from a compliance obligation toward a competitive machine-operability interface. A visually polished but semantically empty website will fail both groups.

Sources