Computer use
Start with the observe–act–check loop
Read the computer-use overview, then identify the observation, action, and verification steps in a bounded task.
Start with: Computer use
RedLectern topic
Computer-use agents
Agents operating browsers and graphical interfaces · observations · actions · verification.
For: Users · automation builders.
After this page you can: Assess and supervise a bounded interface-based automation task.
Page reviewed
Part of: Artificial Intelligence › AI agents
About Computer-use agents
Computer-use agents operate browser and desktop interfaces through observations and input actions. An integration provides the environment, executes requested actions, and returns screenshots or other tool results for the next decision. Some integrations translate structured mouse and keyboard actions into interface input; others run code in a browser or desktop environment. The computer-use documentation describes these integration patterns.
To assess a bounded task, define the requested end state and limits before the run. Review observations and actions, then check the resulting interface state instead of relying only on the agent’s final answer. Set limits, support cancellation, and keep a person in control of consequential actions such as purchases, data transmission, destructive changes, and other hard-to-reverse actions. Evaluations can measure task completion in a defined environment; a benchmark result describes that tested setup, not any particular run.
Start here
Learn the foundations
1.Interfaces and actionsComputer use connects an agent to a browser or desktop interface.
Computer use connects an agent to a browser or desktop interface. The integration determines what the agent can observe and what actions it can request: one documented pattern returns structured mouse and keyboard actions for an application to translate into input, while another runs code in a browser or desktop environment. When assessing a task, identify the environment, available actions, and boundaries on what the agent may change. The computer-use guide recommends restricting the environment to the sites and actions needed for the task. Treat interface content as untrusted and do not let page text expand the task’s permissions.
- Computer use
A maker’s reference for browser and desktop interface automation.
Computer use
Operating a graphical interface through observations and input actions.
Try it
A task asks an agent to open a settings panel and change one option. Name one observation, one action, and one check a supervisor could use to review the run.
Observation: a screenshot showing the current settings screen. Action: open the panel and select the requested option. Check: a fresh screenshot showing that the requested option is selected.
2.Read the interface stateAn observation is the information available to the agent when it decides what to do next.
An observation is the information available to the agent when it decides what to do next. In a screenshot-based interaction, the integration can provide a current screenshot, execute a short group of actions, and return another screenshot so the agent can check the result. Before allowing the next action, review whether the observation shows the relevant part of the interface and whether the action fits the visible state. If the interface has changed or the state is uncertain, request a fresh observation. The computer-use guide describes using screenshots and other tool results to inform the next action.
- Computer use
The guide explains how screenshots and tool results can inform the next action.
Try it
After a browser agent navigates to a new page, what should a supervisor check in the next observation before the agent continues?
Check that the observation shows the new page or relevant interface state, and that the proposed next action targets something visible in that state.
3.Verify the outcome and superviseVerification asks whether the interface reached the requested state, not merely whether the agent reports success.
Verification asks whether the interface reached the requested state, not merely whether the agent reports success. Set out the expected result and task limits before a run; afterward, inspect the interface and compare it with that result. Include the observations and actions that led to the outcome in the review, and set limits and a way to cancel the run. Keep a person in control of purchases, data transmission, destructive changes, and other hard-to-reverse actions. A web-task benchmark can evaluate functional correctness of completions in its defined environment. For interface quality, automated accessibility tests catch some common issues, while Playwright’s documentation notes that many problems require manual testing.
- WebArena: A Realistic Web Environment for Building Autonomous Agents
The paper describes benchmark tasks focused on functional correctness of task completions.
- Accessibility testing
The documentation explains that automated accessibility tests find some problems, while others require manual testing.
Try it
An agent reports that it saved a changed setting. Name two pieces of evidence a supervisor should check before accepting the result.
Check a fresh observation showing the setting in its requested state, and review the actions that led to it. If the change is consequential or hard to reverse, require human confirmation.
Latest
No recent updates are available.
Books
Designing the User Interface: Strategies for Effective Human-Computer Interaction
Ben Shneiderman, Catherine Plaisant, Maxine Cohen, Steven Jacobs, Niklas Elmqvist, and Nicholas Diakopoulos · 2021
Use its interface design and evaluation coverage to assess GUI actions and outcomes. The 2021 edition predates current computer-use agents, so it offers HCI foundations rather than agent-specific methods.
Human-Computer Interaction
Alan Dix, Janet E. Finlay, Gregory D. Abowd, and Russell Beale · 2003
Use its broad HCI grounding to think about what an interface exposes, how actions map to interaction, and how outcomes can be evaluated. This 2003 edition predates current computer-use agents.
- The Design of Everyday Things: Revised and Expanded Edition
The Design of Everyday Things: Revised and Expanded Edition
Donald A. Norman · 2013
Use its account of feedback, task steps, and errors to consider whether an agent can tell what happened after an action. The revised 2013 edition predates current computer-use agents and does not teach their APIs.