Cloudflare 探讨 AI 智能体带来的中间人挑战并推出可见性与访问控制功能

AI Agents now act on the web for millions of people. To a Site, an Agent can take many forms. It may be indistinguishable from a human user carrying out a task, or it might look like one visitor carrying many Users. This removes the ability for the Site to recognize, serve, and hold accountable the user or users behind the Agent, and causes the Site to face the costs when something goes wrong. This post argues that in order for Sites to maximize the access they provide to Agents, the ecosystem must develop new mechanisms to preserve the relationship between Sites and Users. Traditionally, in Security and Fraud Prevention, that requires three things: Identity, Reputation, and Accountability, allowing Sites to answer: Is the person acting being truthful about who they are? Is that person trustworthy? And who answers for the action?

“Personal AI Agents” (“Agents” for short) are automations that take instructions from a human (“User”) and pursue a goal by controlling a cloud computer or browser. They are usually hosted and maintained by an “Operator” as part of a managed service or consumer offering. Examples include Muse from Meta and dots from OpenAI, and more are launching all the time. Agents can potentially take any action online that their User could, without the limits that hold humans back. They don't tire, they run around the clock, they act in parallel, and they put up with friction no person would. Much of the web, from how Sites plan capacity to how they make money, assumes those limits.

Fundamentally, remotely hosted Agents are intermediaries. An intermediary is anything that sits between a User and a Site, so the Site sees it instead of the User. Agents are the newest kind of intermediary, but intermediaries have existed for a long time. Cloud browsers (hosted browsers that software controls on a User's behalf) and Proxies (services that relay a User's traffic, including VPNs) are intermediaries too. What has changed is the trajectory. Agentic traffic is growing rapidly and may eclipse traditional human browsing on the Internet in the near future. This forces a reckoning for anyone operating a Site today. Historically, a Site could simply block intermediary traffic and require Users to disable the intermediary service. That stops working when the majority of traffic is agentic. Once blocking is no longer viable, Site Operators are left with the problem that intermediaries have always created. The Site loses direct information about the User and must decide whether to trust the intermediary without it.

This post covers Agents hosted somewhere other than the User's device. Local agents (e.g. OpenClaw) or Agents running in the User's own browser, on their own device, are out of scope of the primary topic addressed by this post (see below, "On open source and local Agents").

This post rests on one principle:

The Intermediation Principle: The business relationship between Users and Sites, and each participant’s security responsibilities, must be upheld regardless of intermediation.

A Site needs new tools to recognize, serve, and hold accountable a User acting through an Agent just as it could if that User had visited directly. The Agent, and the Operator behind it, must be recognizable and accountable too, because an Agent can do what no User could. Agents are able to scale human access almost without limit, so they create new risks, even for behavior that was previously benign.

The intermediary problem

Meet Ezra. Ezra works as a personal assistant for people in her city. Her best client is Amy, a local CEO, but she also works for Bob, Charlie, and Debra. She runs their errands: groceries, appointments, paperwork, and flights. Ezra blends into the crowd.

The corner grocer used to know Amy. He knew what she bought and set aside what he thought she'd like. Now only Ezra comes in, carrying orders for four people, and the grocer can't tell whose preferences are whose. He doesn't want to turn Ezra away, because she brings him a lot of business.

Then Bob asks Ezra to buy him an iPhone. A week later he calls the store, says he never authorized the charge, and keeps the phone. To win the dispute, the store has to prove Bob asked for it. But the only person the store ever saw was Ezra.

Then Charlie gives Ezra a list of passwords: "I forgot mine, but it's one of these." Ezra patiently tries each one at Charlie's bank. The bank can see an attack happening, but the only person it can ban is Ezra. Banning Ezra also bans Amy, Bob, and Debra.

None of these business owners has a problem with Ezra herself. Their problem is that they can only see Ezra, not the person she's acting for. Swap Ezra for an AI Agent and the businesses for websites, and you have the web today.

A website owner wants sufficient visibility both to prevent bad behavior and to effectively monetize or get value from their Site. But as with Ezra, much of that visibility goes away with AI Agents. Below is a list of actions that a website owner might want to address that were previously solved, but are made more challenging by allowing Agents.

In nearly every challenge above, the Site bears the primary cost. That outcome follows from how trust works when an intermediary is involved.

Our vision

Our aim is an Internet where all Users have equal or greater access to websites than they had before. Before Agents, a Site was typically able to mitigate these risks with a collection of techniques, including fingerprinting the User’s device, session, and history. Users could still browse using private browsing mode, and we believe Agents should be able to do that as well. But they probably won’t be able to log in to a bank without additional verification.

None of this requires an Agent to act like a human or a human to be present for every action. It requires that Sites have the means to obtain or require information about the User to hold them accountable, when applicable, with or without an Agent.

What people have tried, and why they fall short

A common proposal is for Agents to pass a Site's CAPTCHA through to the User to solve. That proves some human is present at that moment (“human in the loop”, or HITL), and nothing more. HITL is primarily a mechanism to prevent mistakes from misalignment. Misalignment is an issue between the Operator and the User, and resolving it does not resolve the majority of the problems above. For example, it can't tell Bob from anyone else (Traffic commingling), and it says nothing about who the Agent acts for the rest of the time. Additionally, Agents exist to remove friction, and HITL adds back that friction, so Operators are incentivized to remove or bypass it for the User.

Another common proposal is to use tightly scoped access controls. Agents are built to do anything a User could, so in practice they ask for, and get, permission to do everything. A scope of "everything" tells the Site nothing. And even a narrow scope only says what the Agent was allowed to do. It doesn't say who answers when the Agent gets it wrong, which is the question the Site actually needs answered.

What the web needs

The web needs tools and protocols that allow Sites and Agents to collaborate to determine what behaviors and actions are allowed, as well as express preferences and policies from Sites to Agents, all in a way that can be reasonably supported by all Sites and Agents. Fortunately, work is already underway along all of these fronts, and many, if not all, of these can be addressed by applying existing standards.

Protocols

Agents and Sites can engage in explicit protocols for solving some of the problems above. For instance, MOLE is one upcoming approach that aims for rate limiting without User-specific identifiers, allowing Sites to transition away from implicit identities and towards explicit protocols for accomplishing the same goal. This is similar to the x402 protocol, which aims to enable payments on the web while remaining privacy-preserving. We are actively exploring ways in which we can leverage existing protocols for addressing other problems, as well as exploring designing new protocols when existing solutions fall short.

Identities

When discussing Agent Identity, Web Bot Auth (WBA) is our preferred method. WBA allows an Agent to sign its requests, which provides a secure way for an Agent to identify itself (and, implicitly, the Operator). We must still define a mechanism that supports multiple intermediaries: when Agent A uses Agent B inside Browser C, the Site needs to be able to know this, so that the Site can correctly ensure that access by Agent A is correctly understood and controlled, for example, in order to prevent Policy circumvention.

Other types of identifying information, such as session, account, or even User-specific IDs, might be necessary, depending on the context. We are working with the relevant standards to explore ways to communicate such identifying information.

Let’s take a look into types of Identity in regards to a User. Ultimately, Identity is a spectrum that trades various degrees of information about a user, while ideally practicing principles of data minimization. Identity provides us a mechanism to answer “who are you?” in pursuit of answering the follow-up questions of: “are you who you say you are?” and “can we trust you?”.

In many cases, identifying information isn’t needed, and privacy-preserving methods for moderating access to agents, such as through Privacy Pass and MOLE, can help us answer “can we trust you?” without needing to answer “who are you?”. Private Access Tokens was one early way in which we tried to establish trust without identifying the user, and MOLE is intended to be the successor to this technology However, there are cases where graduated forms of disclosure along this spectrum would be needed, and you can generally characterize them in the following ways:

Nothing - Users may act on the internet while bringing no identifiers to the table.
Session - Sites require some way of linking requests in the same session to a single agent. This allows the Site to establish reputation for a given session and act on it accordingly. For example, Poppy has a notion of a session ID for exactly this purpose.
Account - Sites require a way of attributing requests to the same User across sessions. This might be a stable identifier assigned by an Operator or third party to a User that persists across the account with the Operator, such as a single Agent visiting the same Site every day using the same identifier. An example use case this would enable is: “Bind your Site account to your Agent account”.
Unique - Some Sites might require a unique, per-User identifier provisioned by a third party that persists across Operators, accounts, Agents, and sessions. This may be useful for high risk actions, or for Sites that need to provide limited access to an expensive resource or that have very restrictive obligations or compliance requirements. Government IDs, third party identity verification, and decentralized technologies like Proof of Human are all examples of technologies that can provide this.

Ultimately, Sites determine what information they need for clients to answer “is the entity behind this request allowed to do this action at this time?” In many cases, protocols already exist, or are being developed, to convey this information: WBA for conveying an Agent’s platform identity, MOLE for conveying User reputation, OAuth for conveying account identities, and so on.

Claims

Beyond identities, some actions require a fact about the real person, such as being over 18 or being the cardholder (Identity verification breaks, Friendly fraud). Often the Operator verified this at sign-up or a third party already has this information.

In order to allow Agents access to Sites or actions with these kinds of restrictions, we need mechanisms for Sites to request this information, and for Operators or third parties to provide this information.

Site Agent access policy

Today, an Agent infers a Site’s rules by trying something and interpreting the Site response (usually, whether it gets blocked). That wastes resources on both sides, and in some cases makes honest Agents appear to be malicious. Robots.txt says where a crawler may go, but isn’t fully suited to express what Agents may do.

Sites need a machine-readable way to publish what actions Agents are allowed to do. The Internet Engineering Task Force (IETF) is actively working on some preference expression mechanisms for this, primarily in the AI Preferences Working Group. However, the focus of the group is on expressing preferences for use (such as training, inference, and search), and there is still not a group focused on the larger need for preference expression mechanisms discussed here.

On open source and local Agents

Agents that are not run as an “Intermediary” (e.g. on-device Agents, or locally hosted Agents such as OpenClaw) raise different concerns. Primarily, they look like (and are) automation, and they have no single party to build reputation or hold accountability with.

There are many solutions to this, one of which is to ask these Agents to provide a Unique ID. However, addressing this is not the initial focus of Trusted Agent Delegation.

What’s next

Real-world solutions to the problems that Sites face won’t emerge from fragmented, vendor-specific mechanisms. They require open, interoperable standards that are rooted in real world experience. We have been, and continue to actively collaborate across the ecosystem, with major Operators, identity verification companies, payment providers, merchants, publishers, proxy providers, and cloud browsing companies, to align operational requirements directly with active work on these problems in the IETF and elsewhere

Our focus is on bridging the gap between running code and emerging specifications across working groups and activities like Web Bot Auth, AIPREFs, OAuth, WIMSE, and MOLE. We are doing this, in part, by co-authoring and refining open protocols like WBA and MOLE and building and deploying implementations of these protocols at scale.

We are also announcing the Trusted Agent Delegation (TAD) coalition. The goal of TAD is to bring a group of representatives across the ecosystem to have discussions on the overall problem set that was put forth in this document, examine all existing protocols, and then to bring more well-defined problems or problems to the IETF to work through, as needed. We intend to hold side meetings at the IETF and workshops in the future to facilitate discussion.

If you are building agents, agent runtimes, running an edge or cloud platform, or managing commercial or compliance applications or infrastructure, we need you to get involved in these efforts too. Check out this page for more information on how to participate.

What you can do today

An open, interoperable foundation is where we're headed, starting with TAD. But you don't have to wait for new protocols to start regaining visibility and control. As of this week, Cloudflare customers can take three concrete steps:

See which Agents are on your site, and what they're doing. Security Analytics now breaks out Personal Assistants as their own traffic category, and a new widget lists every Agent visiting your site, how many requests each one sends, the paths it hits, and the errors it runs into. Unexplained automated traffic becomes traffic you can reason about. And for the broader picture of AI crawlers and bots affect your business, including how much each Operator crawls versus the traffic it sends back, Enterprise Bot Management customers can use Business Insights.

Set access on your terms, per Agent. Using AI Crawler and Agent access policies, you can welcome the Agents that bring you business and restrict the ones that don't — allow an Agent across your site, block it only on pages with ads, or block it entirely. Your Search and Training policies stay independent.

Tie an Agent back to the user it acts for — and act on it. For sites using Account Abuse Protection (early access), every login and sign-up now shows whether a personal Agent made the request, and which one. When an attacker abuses a shared Agent on one account, you can mitigate that account — with a WAF rule on its Hashed User ID — without blocking the Agent for every other legitimate user behind it. This is a first, concrete answer to the Traffic commingling challenge described above: you can finally see past the Agent to the User, and respond accordingly.

These are the first building blocks of the vision, already in production. More will follow as the standards mature.

And if your company has a more restrictive position on Agents, we would love to hear it. Reach out and share your problems with us directly. We are looking to help build an Internet where site owners have visibility and control, but also where Agents have the maximum access that is reasonable within those controls. You can reach out to your Customer Support team, or email us at crawlercontrols@cloudflare.com.