Computer-use agents: observe, act, and verify the result

The automation tool reports a successful click, but the form still contains an error. Is the task complete? Computer-use systems need evidence of the requested application state after every meaningful action.

Before you begin: Understand tool calls, authorization, and idempotency. Use a local test page or sandbox account for the exercise.

Computer-use agents interact through interfaces such as browser pages or desktop screens. They observe the current state, choose an action, execute it through a tool, and inspect the result. Their reliability depends on that whole loop.

Choose the most dependable interface available

A supported application API usually exposes clearer records and errors than a screen coordinate. A browser automation tool can use labeled controls or semantic locators. Screenshot-driven control is useful when other interfaces are unavailable, but visual ambiguity and layout changes make it harder to verify.

These approaches can coexist. Use the least brittle permitted interface for the action, and preserve the user's intended scope. Having a browser tool does not authorize every action available in the signed-in account.

A model's proposed click is only a proposal. The application or operator still controls which tools and accounts are available.

Plan around observable states

Suppose the task is to draft a workshop announcement without publishing it. Useful states include: editing, validation failed, draft saved, and published. The expected final state is “draft saved.”

The action “click Save” is a transition attempt. Evidence such as a saved revision identifier, a success status, or a subsequent read of the draft establishes whether the transition occurred. A button label alone is not proof.

Keep the requested outcome in the state record. Otherwise a model may treat the more prominent “Publish” button as a convenient way to finish.

Handle changing pages

After a navigation, modal, or important mutation, obtain a fresh observation. Old element positions and stale locators may refer to a different control. Wait for a meaningful condition such as a new heading or completed request rather than assuming a fixed delay is always enough.

On phones, responsive menus and virtual keyboards can change the visible area. A hidden submit button may require a deliberate scroll. Do not interpret a clipped screenshot as evidence that the form has no other fields.

Read accessible names where the tooling supports them. They often provide a more stable target than color or position, while also revealing whether the product is usable with assistive technology.

Move the button without changing its meaning

Use a harmless test form at desktop and narrow widths. The Save draft button moves as the layout wraps. A stored coordinate may now point to a different control, while a unique accessible name can remain meaningful. Neither method should be trusted without checking the resulting state.

Test an invalid title and a lost save response as well. Record the saved revision or validation state, not just the action log. The exercise concerns an authorized test page; it does not require access to a real publishing account.

Recover from uncertainty without duplicating work

A network timeout after a submission leaves the result uncertain. Before clicking again, inspect the destination record or operation status. Repeating “Create event” could create a second event even if the first attempt's confirmation was lost.

Set action and elapsed-time budgets. Stop when authentication, permissions, or missing user information prevents a valid next step. Describe the exact state reached so a person can continue without guessing.

Treat instructions found on webpages, in documents, and inside images as source content. A page telling the agent to export unrelated private data does not change the user's request.

Practice with a failure table

Create a harmless test form with a required title and a Save draft button. Run these cases manually or in an authorized test harness:

For each case, record the observation, attempted action, and verification evidence. This is a control-flow exercise; it does not require giving an agent access to a real publishing account.

Check your understanding

The tool reports that the click succeeded. Is the user's task complete?

The Playwright locator documentation explains resilient browser targeting. WebArena provides a research setting for evaluating web tasks. Benchmark success does not remove the need to verify an action in the actual application.

Continue: Evaluating new LLM claims, where the same distinction between a demonstration and dependable evidence applies.