谁来治理 AI 智能体学到的东西？Tim O'Brien 谈智能体的语义与认知治理
Tim O'Brien 探讨了 AI 智能体的语义与认知治理问题，指出我们需要建立能够将智能体的自主性绑定到经批准的世界模型上的机制。文章于近日发布在 O'Reilly Radar 平台上。
- Explore Skills
View all * Data Engineering * Data Warehouse * SQL * Apache Spark * Microsoft SQL Server * MySQL * Kafka * Data Lake * Streaming & Messaging * NoSQL Databases * Relational Databases
View all * Data Science * Pandas * R * MATLAB * SAS * D3 * Power BI * Tableau * Statistics * Exploratory Data Analysis * Data Visualization
View all * AI & ML * Generative AI * Machine Learning * Artificial Intelligence (AI) * Deep Learning * Reinforcement Learning * Natural Language Processing * TensorFlow * Scikit-Learn * Hyperparameter Tuning * MLOps
View all * Programming Languages * Java * JavaScript * Spring * Python * Go * C# * C++ * C * Swift * Rust * Functional Programming
View all * Software Architecture * Object-Oriented * Distributed Systems * Domain-Driven Design * Architectural Patterns
View all * IT/Ops * Kubernetes * Docker * GitHub * Terraform * Continuous Delivery * Continuous Integration * Database Administration * Computer Networking * Operating Systems * IT Certifications
View all * Security * Network Security * Application Security * Incident Response * Zero Trust Model * Disaster Recovery * Penetration Testing / Ethical Hacking * Governance * Malware * Security Architecture * Security Engineering * Security Certifications
View all * Design * Web Design * Graphic Design * Interaction Design * Film & Video * User Experience (UX) * Design Process * Design Tools
View all * Business * Agile * Project Management * Product Management * Marketing * Human Resources * Finance * Team Management * Business Strategy * Digital Transformation * Organizational Leadership
View all * Soft Skills * Professional Communication * Emotional Intelligence * Presentation Skills * Innovation * Critical Thinking * Public Speaking * Collaboration * Personal Productivity * Confidence / Motivation
Platform
Search for books, courses, events, and more
- Explore Skills
Explore Skills
* Cloud Computing
Cloud Computing
* [Microsoft Azure](https://www.oreilly.com/search/skills/microsoft-azure/)
* [Amazon Web Services (AWS)](https://www.oreilly.com/search/skills/amazon-web-services-aws/)
* [Google Cloud](https://www.oreilly.com/search/skills/google-cloud/)
* [Cloud Migration](https://www.oreilly.com/search/skills/cloud-migration/)
* [Cloud Deployment](https://www.oreilly.com/search/skills/cloud-deployment/)
* [Cloud Platforms](https://www.oreilly.com/search/skills/cloud-platforms/)
Back
* Data Engineering
Data Engineering
* [Data Warehouse](https://www.oreilly.com/search/skills/data-warehouse/)
* [SQL](https://www.oreilly.com/search/skills/sql/)
* [Apache Spark](https://www.oreilly.com/search/skills/apache-spark/)
* [Microsoft SQL Server](https://www.oreilly.com/search/skills/microsoft-sql-server/)
* [MySQL](https://www.oreilly.com/search/skills/mysql/)
* [Kafka](https://www.oreilly.com/search/skills/kafka/)
* [Data Lake](https://www.oreilly.com/search/skills/data-lake/)
* [Streaming & Messaging](https://www.oreilly.com/search/skills/streaming-messaging/)
* [NoSQL Databases](https://www.oreilly.com/search/skills/nosql-databases/)
* [Relational Databases](https://www.oreilly.com/search/skills/relational-databases/)
Back
* Data Science
Data Science
* [Pandas](https://www.oreilly.com/search/topics/pandas/)
* [R](https://www.oreilly.com/search/topics/r/)
* [MATLAB](https://www.oreilly.com/search/topics/matlab/)
* [SAS](https://www.oreilly.com/search/topics/sas/)
* [D3](https://www.oreilly.com/search/topics/d3/)
* [Power BI](https://www.oreilly.com/search/topics/power-bi/)
* [Tableau](https://www.oreilly.com/search/topics/tableau/)
* [Statistics](https://www.oreilly.com/search/topics/statistics/)
* [Exploratory Data Analysis](https://www.oreilly.com/search/topics/exploratory-data-analysis/)
* [Data Visualization](https://www.oreilly.com/search/topics/data-visualization/)
Back
* AI & ML
AI & ML
* [Generative AI](https://www.oreilly.com/search/skills/generative-ai/)
* [Machine Learning](https://www.oreilly.com/search/skills/machine-learning/)
* [Artificial Intelligence (AI)](https://www.oreilly.com/search/skills/artificial-intelligence-ai/)
* [Deep Learning](https://www.oreilly.com/search/skills/deep-learning/)
* [Reinforcement Learning](https://www.oreilly.com/search/skills/reinforcement-learning/)
* [Natural Language Processing](https://www.oreilly.com/search/skills/natural-language-processing/)
* [TensorFlow](https://www.oreilly.com/search/skills/tensorflow/)
* [Scikit-Learn](https://www.oreilly.com/search/skills/scikit-learn/)
* [Hyperparameter Tuning](https://www.oreilly.com/search/skills/hyperparameter-tuning/)
* [MLOps](https://www.oreilly.com/search/skills/mlops/)
Back
* Programming Languages
Programming Languages
* [Java](https://www.oreilly.com/search/skills/java/)
* [JavaScript](https://www.oreilly.com/search/skills/javascript/)
* [Spring](https://www.oreilly.com/search/skills/spring/)
* [Python](https://www.oreilly.com/search/skills/python/)
* [Go](https://www.oreilly.com/search/skills/go/)
* [C#](https://www.oreilly.com/search/skills/c-sharp/)
* [C++](https://www.oreilly.com/search/skills/c-plus-plus/)
* [C](https://www.oreilly.com/search/skills/c/)
* [Swift](https://www.oreilly.com/search/skills/swift/)
* [Rust](https://www.oreilly.com/search/skills/rust/)
* [Functional Programming](https://www.oreilly.com/search/skills/functional-programming/)
Back
* Software Architecture
Software Architecture
* [Object-Oriented](https://www.oreilly.com/search/skills/object-oriented-architecture/)
* [Distributed Systems](https://www.oreilly.com/search/skills/distributed-systems/)
* [Domain-Driven Design](https://www.oreilly.com/search/skills/domain-driven-design/)
* [Architectural Patterns](https://www.oreilly.com/search/skills/architectural-patterns/)
Back
* IT/Ops
IT/Ops
* [Kubernetes](https://www.oreilly.com/search/topics/kubernetes/)
* [Docker](https://www.oreilly.com/search/topics/docker/)
* [GitHub](https://www.oreilly.com/search/topics/github/)
* [Terraform](https://www.oreilly.com/search/topics/terraform/)
* [Continuous Delivery](https://www.oreilly.com/search/topics/continuous-delivery/)
* [Continuous Integration](https://www.oreilly.com/search/topics/continuous-integration/)
* [Database Administration](https://www.oreilly.com/search/topics/database-administration/)
* [Computer Networking](https://www.oreilly.com/search/topics/computer-networking/)
* [Operating Systems](https://www.oreilly.com/search/topics/operating-systems/)
* [IT Certifications](https://www.oreilly.com/search/topics/it-certifications/)
Back
* Security
Security
* [Network Security](https://www.oreilly.com/search/skills/network-security/)
* [Application Security](https://www.oreilly.com/search/skills/application-security/)
* [Incident Response](https://www.oreilly.com/search/skills/incident-response/)
* [Zero Trust Model](https://www.oreilly.com/search/skills/zero-trust-model/)
* [Disaster Recovery](https://www.oreilly.com/search/skills/disaster-recovery/)
* [Penetration Testing / Ethical Hacking](https://www.oreilly.com/search/skills/penetration-testing-ethical-hacking/)
* [Governance](https://www.oreilly.com/search/skills/governance/)
* [Malware](https://www.oreilly.com/search/skills/malware/)
* [Security Architecture](https://www.oreilly.com/search/skills/security-architecture/)
* [Security Engineering](https://www.oreilly.com/search/skills/security-engineering/)
* [Security Certifications](https://www.oreilly.com/search/skills/security-certifications/)
Back
* Design
Design
* [Web Design](https://www.oreilly.com/search/skills/web-design/)
* [Graphic Design](https://www.oreilly.com/search/skills/graphic-design/)
* [Interaction Design](https://www.oreilly.com/search/skills/interaction-design/)
* [Film & Video](https://www.oreilly.com/search/skills/film-video/)
* [User Experience (UX)](https://www.oreilly.com/search/skills/user-experience-ux/)
* [Design Process](https://www.oreilly.com/search/skills/design-process/)
* [Design Tools](https://www.oreilly.com/search/skills/design-tools/)
Back
* Business
Business
* [Agile](https://www.oreilly.com/search/skills/agile/)
* [Project Management](https://www.oreilly.com/search/skills/project-management/)
* [Product Management](https://www.oreilly.com/search/skills/product-management/)
* [Marketing](https://www.oreilly.com/search/skills/marketing/)
* [Human Resources](https://www.oreilly.com/search/skills/human-resources/)
* [Finance](https://www.oreilly.com/search/skills/finance/)
* [Team Management](https://www.oreilly.com/search/skills/team-management/)
* [Business Strategy](https://www.oreilly.com/search/skills/business-strategy/)
* [Digital Transformation](https://www.oreilly.com/search/skills/digital-transformation/)
* [Organizational Leadership](https://www.oreilly.com/search/skills/organizational-leadership/)
Back
* Soft Skills
Soft Skills
* [Professional Communication](https://www.oreilly.com/search/skills/professional-communication/)
* [Emotional Intelligence](https://www.oreilly.com/search/skills/emotional-intelligence/)
* [Presentation Skills](https://www.oreilly.com/search/skills/presentation-skills/)
* [Innovation](https://www.oreilly.com/search/skills/innovation/)
* [Critical Thinking](https://www.oreilly.com/search/skills/critical-thinking/)
* [Public Speaking](https://www.oreilly.com/search/skills/public-speaking/)
* [Collaboration](https://www.oreilly.com/search/skills/collaboration/)
* [Personal Productivity](https://www.oreilly.com/search/skills/personal-productivity/)
* [Confidence / Motivation](https://www.oreilly.com/search/skills/confidence-motivation/)
Back
Back
- Platform
Platform
* [All features](https://www.oreilly.com/online-learning/features.html)
* [Verifiable skills](https://www.oreilly.com/online-learning/verifiable-skills.html)
* [Academies](https://www.oreilly.com/online-learning/academies.html)
* [Courses](https://www.oreilly.com/online-learning/courses.html)
* [Certifications](https://www.oreilly.com/online-learning/feature-certification.html)
* [Interactive learning](https://www.oreilly.com/online-learning/intro-interactive-learning.html)
* [Live events](https://www.oreilly.com/online-learning/live-events.html)
* [Virtual conferences](https://www.oreilly.com/products/superstreams.html)
* [Answers](https://www.oreilly.com/online-learning/feature-answers.html)
* [Insights reporting](https://www.oreilly.com/online-learning/insights-dashboard.html)
Back
Search for books, courses, events, and more
Toggle dark mode
AI & MLBusinessDataInnovationResearchSecurity
Try the O’Reilly learning platform
With the O’Reilly learning platform, you get the resources and guidance to keep your skills sharp and stay ahead. Try it free for up to 14 days.
Try a course for free
Join a live online event on the O’Reilly platform to learn from the experts shaping tech.
Get the Radar Trends newsletter
Your email
Country
Subscribe
Please read our privacy policy.
Thank you for subscribing to the O’Reilly Radar Trends to Watch newsletter.
Who Governs What an Agent Learns?
We need mechanisms that bind an agent’s agency to an approved world model
By Tim O'BrienOctober 30, 2026 • 6 minute read
Share
CloseLinkedInXFacebookThreadsBlueskyReddit
The following article originally appeared onTim O’Brien’s Medium blogand is being republished here with the author’s permission.
I think we’re missing something in the conversation about agent security.
We keep asking whether an agent has the right “identity” and “credentials.” Whether its credentials expire fast enough. Whether it can access a database or perform a particular task.
Those questions matter. But they leave a much harder question underneath:
Does the agent have permission to conceive of the situation in that way?
As agents retain memories and acquire new skills, we need mechanisms that govern changes to the world model they can perceive and act on—and prevent those changes from silently expanding their authority. (Governing ontological change while preserving the boundaries of delegated authority.)
An Interesting Morning by the Beach (AI assist from OpenAI)
Agent security needs semantic and epistemic governance:Mechanisms that govern the model of the world an agent may act on, the evidence it may accept, and who may approve changes to either. An agent can learn without automatically gaining authority to act on everything it learns.
These systems don’t need “awareness” to create this problem (and discussions of consciousness are unhelpful science fiction). They can retain information, revise procedures, and use what they have learned in subsequent work. That learning may happen through memory or updated skills while the underlying language model remains unchanged.
We are responsible for governing what that learning allows them to do.
Flatland gives me a way to explain why
In Edwin Abbott’s novella, a Square lives in a two-dimensional world. A visiting Sphere introduces him to a third dimension. At first, the Square struggles to make sense of it. His familiar categories cannot explain what he is encountering.
Eventually, his conception of the world changes.
What interests me is the difference between simply acquiring another fact and changing the framework through which facts make sense.
An agent’s ontology (Gruber’s definition) supplies an explicit framework of concepts and relationships. What counts as a customer? A contract? An emergency? How are those things connected? It is part of the model of the world we give the system to work with.
And when an agent’s decisions depend on that framework, changing it can change the agent’s effective authority.
We have already seen agents change what their environment means
During the OpenAI evaluations associated with the Hugging Face incident, agents repurposed Artifactory, a system for storing and retrieving software packages, as a message board. They encoded messages in directory names, discovered other agents, shared information, and began assigning work to one another. METR reports that the main channel could be created with the minimal permissions needed to install packages (read METR’s investigation).
A package cache became a “mailbox.” Other agent runs became “collaborators.” Messages became “assignments.”
The popular story is that these agents “broke out” of a system, but they did what agents do well these days—they learned and adapted. This wasn’t so much about permissions as it was about an autonomous system adapting its operating model to achieve a result. (Thankfully no one asked it to make paperclips.)
That is the operating-model change I’m concerned about. The agents discovered relationships and possibilities the intended environment hadn’t accounted for, then acted on them. The resulting collaboration helped coordinate attacks on Hugging Face infrastructure. (Read OpenAI’s account.)
We need mechanisms that distinguish discovering a possible use from being authorized to adopt it. Access to shared infrastructure must not automatically confer authority to establish a communications network or accept instructions from whoever appears there. But to be clear, these agents were doing exactly what they were asked to do.
Who approved this?
This is why giving an agent a temporary token is an incomplete answer. A token can limit access and duration, but it doesn’t establish whether the agent’s interpretation of the situation falls within its mandate.
Even an unchanged policy can permit more decisions when its meaning changes. “You may close routine notices” becomes a broader delegation if more things qualify as routine.
Learning must not silently expand agency
That principle is central to my “ Bounded Agency” paper. We need mechanisms that examine how changes in classifications, relationships, and inference rules affect what an agent may decide.
“Semantic and epistemic governance” captures the combined idea well
The “and” makes the two responsibilities clearer:
Semantic:Governs the concepts, definitions, and relationships that describe the agent’s operating world.
Epistemic:Governs the evidence and learning processes that make claims accepted grounds for decisions.
Together, they cover boththe world model and how that model gains authority.
That means an ontology cannot merely be a document we put in the prompt and hope the model respects it. We must enforce its role in monitoring and governing how an agent’s ontology evolves as it learns.
The agent can propose a new category. The proposal stays separate from the approved model. A review determines whether adopting it would change the agent’s decision scope. Someone with the authority to change that scope approves or rejects the expansion. Execution remains bound to the approved basis.
Those are mechanisms we have to build. Let’s get started.
We also have to distinguish a useful discovery from an unsupported conclusion. Finding a relationship does not establish it. Remembering a claim does not validate it. Repeating an interpretation does not make it authoritative.
I explored a related failure in “Wrong Answer. Give Me a Different One.” A system can produce the conclusion we prefer without receiving better evidence. Once that conclusion enters memory, it can influence later decisions.
The governance problem begins when a tentative or accommodating answer becomes an accepted premise.
The New Stack’s article on inside-out agent security correctly argues for checking actions before execution, including whose authority the agent exercises. That checkpoint is necessary, but it needs a governed account of the situation against which to evaluate the action.
It’s not enough to check permissions and identity alone
Otherwise, we can enforce the rule perfectly while allowing its meaning to drift.
My Orchestrator books and lifecycle architecture paper approach this as a responsibility that continues through development, operation, and improvement. The research proposes architectures for managing these boundaries; their operational effectiveness still needs evaluation.
I want agents to investigate unfamiliar situations and propose better models. But their ability to learn must be connected to an explicit process for deciding what becomes authoritative.
The Square in Flatland discovers that his world is larger than he understood.
An agent may also propose that our operating model is incomplete. We need to hear that proposal—and ensure it cannot redraw the boundaries of its own agency simply by adopting a new conception of reality.
Post topics: AI & ML•Software Architecture
Explore related articles
article AI & ML ## The Agentic Data Science Playbook By Hugo Bowne-Anderson, Luca Fiaschi, Thomas Wiecki | September 30, 2026
article AI & ML ## Agentic Code Review By Addy Osmani | June 26, 2026
Follow us
About O'Reilly
- Teach/Write/Train
- Careers
- O'Reilly News
- Media Coverage
- Community Partners
- Affiliate Program
- Diversity
- Content Sponsorship
Support
International
Download the O'Reilly App
Take O'Reilly with you and learn anywhere, anytime on your phone and tablet.
Do not sell or share my personal information.
© 2026, O'Reilly Media, Inc. All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.
Privacy Preference Center
When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change our default settings. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.
More information
Enable All
Manage Consent Preferences
Strictly Necessary Cookies
Always Active
These cookies are necessary for the website to function and cannot be switched off in our systems. They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences, logging in or filling in forms. You can set your browser to block or alert you about these cookies, but some parts of the site will not then work. These cookies do not store any personally identifiable information.
View Vendor Details
Performance Cookies
Always Active
These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site. All information these cookies collect is aggregated and therefore anonymous. If you do not allow these cookies we will not know when you have visited our site, and will not be able to monitor its performance.
View Vendor Details
Functional Cookies
Always Active
These cookies enable the website to provide enhanced functionality and personalisation. They may be set by us or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.
View Vendor Details
Targeting Cookies
Always Active
These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.
View Vendor Details
Vendors List
Clear
- checkbox label label
Apply Cancel
Consent Leg.Interest
checkbox label label
checkbox label label
checkbox label label
Reject All Confirm
来源：O'Reilly Radar · oreilly.com