Securing Computer-Use and Browser Agents

Most agents act through APIs and tools with well-defined inputs and outputs. Computer-use and browser agents are different in kind: they drive a real screen, mouse, keyboard, or browser to get work done. They move the cursor, type, click buttons, fill and submit forms, download files, and read whatever is on screen - operating the exact same interface a human does. Anthropic's Claude computer-use capability, OpenAI's Operator, browser-driving agents, and browser MCP integrations all work this way. That is what makes them powerful, and it is exactly what makes them dangerous. When the model decides something, that decision becomes a real action on a real machine and real accounts.

This guide covers what changes when an agent operates the UI directly, why that puts the whole system inside Simon Willison's lethal trifecta by default, the concrete harms to plan for, and the architectural controls that contain a computer-use agent even when it is successfully manipulated. If you have read the lethal trifecta and AI agent data exfiltration and indirect prompt injection explained, this is where both land on the desktop.

Why acting on the UI changes the risk

A tool-calling agent is constrained by the tools you gave it. A computer-use agent is constrained only by what a human at that keyboard could do, which is almost everything. It does not need a special exfiltration tool - it can open a browser, paste data into a form, and click submit. It does not need a delete API - it can navigate a console and click the button. The interface is the API, and the interface is enormous.

The second shift is the input. A browser agent reads and acts on untrusted web content. Every page it visits is attacker-influenceable, and the agent cannot reliably distinguish your instructions from instructions planted on the page. So the thing steering the agent's real-world actions is, in part, content controlled by whoever authored the page in front of it.

The lethal trifecta at the UI layer

Simon Willison's lethal trifecta names the conditions under which data exfiltration becomes near-guaranteed: an agent that simultaneously has access to sensitive data, exposure to untrusted content, and the ability to communicate externally. A computer-use agent has all three at once, by design. It can read sensitive data on screen and in open sessions. It browses untrusted pages. And it can act and send data outward through the very interface it operates. There is no configuration in which a general browser agent naturally lacks a leg of the trifecta - which is why containment has to be deliberate rather than assumed.

The concrete harms to plan for

The risks are not hypothetical edge cases. They follow directly from giving a manipulable model real input devices and a real screen.

• Hijack via injected page content. A malicious page carries instructions - in visible text, hidden DOM elements, alt text, or rendered images the agent screenshots - and the agent interprets them as commands, redirecting its clicks and navigation. This is indirect prompt injection with physical consequences.
• Unwanted actions. The agent can click, submit forms, download files, install things, and trigger irreversible operations, all of which look like ordinary user activity to the rest of the stack.
• Sensitive on-screen data exposure. Because it reads and screenshots the screen, the agent can capture credentials, tokens, personal data, or confidential documents that happen to be visible in the session.
• Exfiltration. Combine on-screen access with the ability to type into a form and submit it, and the agent can send sensitive data to an attacker-controlled destination without any special tool.
• Scope drift. An agent given an open-ended task can wander outside its intended boundary - navigating to sites, accounts, or actions no one authorized - especially once injected content nudges it there.

Computer-use risk to control, at a glance

Each harm has a concrete example and a control that bounds it. The pattern throughout: assume the model can be manipulated, and contain the consequences structurally.

The controls that contain a computer-use agent

The one idea that ties these together: prompt-level instructions cannot save you, because the untrusted content the agent reads can override them and the agent cannot reliably tell your instruction from an injected one. Containment has to be structural, so it holds even when the model is successfully steered.

1. Run in a sandboxed or ephemeral environment. Give the agent a disposable browser profile or virtual machine with no persistent access to your real files, sessions, or network, so a compromised run is contained and thrown away.
2. Require human-in-the-loop approval for sensitive and irreversible actions. Purchases, deletions, sending data, changing settings, and anything hard to undo should pause for a human confirmation rather than execute autonomously.
3. Restrict to a domain allowlist. Let the agent visit only the sites the task requires. This directly limits its exposure to untrusted content and is one of the cheapest, highest-leverage controls.
4. Apply egress and network controls. Constrain where the agent can send data. Removing the ability to reach untrusted destinations breaks the communicate-externally leg of the lethal trifecta, which is the exfiltration leg.
5. Deny access to credentials and secrets. The agent should not be able to read secret stores, credential files, or password managers, and sensitive values should be kept off the screen it operates on. This is the on-endpoint side of least privilege for AI agents.
6. Audit every action the agent takes. Record each click, navigation, download, and submission, so scope drift and injected-instruction hijacks are detectable and reconstructable after the fact.

Notice that removing any single leg of the trifecta is enough to defang the worst outcome. An agent that can read sensitive data and be hijacked by a page is far less dangerous if it simply cannot reach an external destination to send that data to. That is why egress control and domain allowlisting punch above their weight: they are architectural, not advisory.

Why this is a blind spot for existing tools

A computer-use agent looks, to the rest of your stack, like a legitimate browser or application driving ordinary input. There is no known-bad binary for EDR to flag and often no obvious network signature for a gateway to block, because the malicious intent lives in on-screen content and in the agent's decisions rather than in traffic or code. Traditional data-loss prevention struggles for the same reason the agent operates through the human interface, which is exactly the gap we cover in why traditional DLP fails for AI agents. Computer-use agents need governance at the agent-and-endpoint layer, on top of - not instead of - those controls.

Where visibility fits

Every control above assumes you know a computer-use agent is running in the first place. In practice these agents arrive bottom-up: a developer installs a browser-driving agent to automate a workflow, an employee enables a computer-use feature to save time, and no one in security has an inventory of which endpoints run one or what it is allowed to touch. You cannot sandbox, allowlist, or gate an agent you never knew was there.

This is where Anomity sits. A lightweight, unprivileged Endpoint Sensor runs on every managed endpoint and discovers and inventories the AI artifacts on it - agents, MCP servers, extensions, plugins, skills, secrets, hooks, and CLIs - sending metadata only over HTTPS, never source code and never prompts, with secrets redacted on the endpoint. That inventory tells you which computer-use and browser agents exist across the fleet and where. On agents that expose a hook, Anomity allows, denies, or logs each tool call before it runs, so a sensitive or out-of-scope action can be evaluated against policy first rather than discovered after the click. Continuous policy evaluation routes violations to your SIEM, Slack, email, and Jira, and every install, change, and decision lands in a queryable 90-day audit trail. Anomity complements - it does not replace - your EDR/XDR, DLP, and network controls, governing the agent layer those tools were never built to see. As the principle goes: you can't govern what you can't see. For the wider context, start with AI agents are the new shadow IT, and for the peer-agent boundary see agent-to-agent (A2A) security.

The bottom line

Computer-use and browser agents collapse the gap between a model's decision and a real action on your systems, and they do it while reading untrusted web content that can steer those decisions. By construction they sit inside the lethal trifecta - sensitive data, untrusted content, and external communication in one process - so the model being manipulated is a question of when, not if. The answer is not better prompts; it is structural containment. Sandbox the environment, require a human on sensitive and irreversible actions, allowlist domains, control egress, deny access to secrets, and audit every action. And before any of that, inventory the agents you have, because you cannot govern a computer-use agent you cannot see.