用“第二使用者测试”检验学生项目：生成式 AI 时代如何证明理解

Explore Skills
- Cloud Computing

View all * Data Engineering * Data Warehouse * SQL * Apache Spark * Microsoft SQL Server * MySQL * Kafka * Data Lake * Streaming & Messaging * NoSQL Databases * Relational Databases

View all * Data Science * Pandas * R * MATLAB * SAS * D3 * Power BI * Tableau * Statistics * Exploratory Data Analysis * Data Visualization

View all * AI & ML * Generative AI * Machine Learning * Artificial Intelligence (AI) * Deep Learning * Reinforcement Learning * Natural Language Processing * TensorFlow * Scikit-Learn * Hyperparameter Tuning * MLOps

View all * Programming Languages * Java * JavaScript * Spring * Python * Go * C# * C++ * C * Swift * Rust * Functional Programming

View all * Software Architecture * Object-Oriented * Distributed Systems * Domain-Driven Design * Architectural Patterns

View all * IT/Ops * Kubernetes * Docker * GitHub * Terraform * Continuous Delivery * Continuous Integration * Database Administration * Computer Networking * Operating Systems * IT Certifications

View all * Security * Network Security * Application Security * Incident Response * Zero Trust Model * Disaster Recovery * Penetration Testing / Ethical Hacking * Governance * Malware * Security Architecture * Security Engineering * Security Certifications

View all * Design * Web Design * Graphic Design * Interaction Design * Film & Video * User Experience (UX) * Design Process * Design Tools

View all * Business * Agile * Project Management * Product Management * Marketing * Human Resources * Finance * Team Management * Business Strategy * Digital Transformation * Organizational Leadership

View all * Soft Skills * Professional Communication * Emotional Intelligence * Presentation Skills * Innovation * Critical Thinking * Public Speaking * Collaboration * Personal Productivity * Confidence / Motivation

View all

Search for books, courses, events, and more

Explore Skills

Explore Skills

*   Cloud Computing

Cloud Computing

    *   [Microsoft Azure](https://oreilly.com/search/skills/microsoft-azure/)
    *   [Amazon Web Services (AWS)](https://oreilly.com/search/skills/amazon-web-services-aws/)
    *   [Google Cloud](https://oreilly.com/search/skills/google-cloud/)
    *   [Cloud Migration](https://oreilly.com/search/skills/cloud-migration/)
    *   [Cloud Deployment](https://oreilly.com/search/skills/cloud-deployment/)
    *   [Cloud Platforms](https://oreilly.com/search/skills/cloud-platforms/)

Back

*   Data Engineering

Data Engineering

    *   [Data Warehouse](https://oreilly.com/search/skills/data-warehouse/)
    *   [SQL](https://oreilly.com/search/skills/sql/)
    *   [Apache Spark](https://oreilly.com/search/skills/apache-spark/)
    *   [Microsoft SQL Server](https://oreilly.com/search/skills/microsoft-sql-server/)
    *   [MySQL](https://oreilly.com/search/skills/mysql/)
    *   [Kafka](https://oreilly.com/search/skills/kafka/)
    *   [Data Lake](https://oreilly.com/search/skills/data-lake/)
    *   [Streaming & Messaging](https://oreilly.com/search/skills/streaming-messaging/)
    *   [NoSQL Databases](https://oreilly.com/search/skills/nosql-databases/)
    *   [Relational Databases](https://oreilly.com/search/skills/relational-databases/)

Back

*   Data Science

Data Science

    *   [Pandas](https://oreilly.com/search/topics/pandas/)
    *   [R](https://oreilly.com/search/topics/r/)
    *   [MATLAB](https://oreilly.com/search/topics/matlab/)
    *   [SAS](https://oreilly.com/search/topics/sas/)
    *   [D3](https://oreilly.com/search/topics/d3/)
    *   [Power BI](https://oreilly.com/search/topics/power-bi/)
    *   [Tableau](https://oreilly.com/search/topics/tableau/)
    *   [Statistics](https://oreilly.com/search/topics/statistics/)
    *   [Exploratory Data Analysis](https://oreilly.com/search/topics/exploratory-data-analysis/)
    *   [Data Visualization](https://oreilly.com/search/topics/data-visualization/)

Back

*   AI & ML

AI & ML

    *   [Generative AI](https://oreilly.com/search/skills/generative-ai/)
    *   [Machine Learning](https://oreilly.com/search/skills/machine-learning/)
    *   [Artificial Intelligence (AI)](https://oreilly.com/search/skills/artificial-intelligence-ai/)
    *   [Deep Learning](https://oreilly.com/search/skills/deep-learning/)
    *   [Reinforcement Learning](https://oreilly.com/search/skills/reinforcement-learning/)
    *   [Natural Language Processing](https://oreilly.com/search/skills/natural-language-processing/)
    *   [TensorFlow](https://oreilly.com/search/skills/tensorflow/)
    *   [Scikit-Learn](https://oreilly.com/search/skills/scikit-learn/)
    *   [Hyperparameter Tuning](https://oreilly.com/search/skills/hyperparameter-tuning/)
    *   [MLOps](https://oreilly.com/search/skills/mlops/)

Back

*   Programming Languages

Programming Languages

    *   [Java](https://oreilly.com/search/skills/java/)
    *   [JavaScript](https://oreilly.com/search/skills/javascript/)
    *   [Spring](https://oreilly.com/search/skills/spring/)
    *   [Python](https://oreilly.com/search/skills/python/)
    *   [Go](https://oreilly.com/search/skills/go/)
    *   [C#](https://oreilly.com/search/skills/c-sharp/)
    *   [C++](https://oreilly.com/search/skills/c-plus-plus/)
    *   [C](https://oreilly.com/search/skills/c/)
    *   [Swift](https://oreilly.com/search/skills/swift/)
    *   [Rust](https://oreilly.com/search/skills/rust/)
    *   [Functional Programming](https://oreilly.com/search/skills/functional-programming/)

Back

*   Software Architecture

Software Architecture

    *   [Object-Oriented](https://oreilly.com/search/skills/object-oriented-architecture/)
    *   [Distributed Systems](https://oreilly.com/search/skills/distributed-systems/)
    *   [Domain-Driven Design](https://oreilly.com/search/skills/domain-driven-design/)
    *   [Architectural Patterns](https://oreilly.com/search/skills/architectural-patterns/)

Back

*   IT/Ops

IT/Ops

    *   [Kubernetes](https://oreilly.com/search/topics/kubernetes/)
    *   [Docker](https://oreilly.com/search/topics/docker/)
    *   [GitHub](https://oreilly.com/search/topics/github/)
    *   [Terraform](https://oreilly.com/search/topics/terraform/)
    *   [Continuous Delivery](https://oreilly.com/search/topics/continuous-delivery/)
    *   [Continuous Integration](https://oreilly.com/search/topics/continuous-integration/)
    *   [Database Administration](https://oreilly.com/search/topics/database-administration/)
    *   [Computer Networking](https://oreilly.com/search/topics/computer-networking/)
    *   [Operating Systems](https://oreilly.com/search/topics/operating-systems/)
    *   [IT Certifications](https://oreilly.com/search/topics/it-certifications/)

Back

*   Security

Security

    *   [Network Security](https://oreilly.com/search/skills/network-security/)
    *   [Application Security](https://oreilly.com/search/skills/application-security/)
    *   [Incident Response](https://oreilly.com/search/skills/incident-response/)
    *   [Zero Trust Model](https://oreilly.com/search/skills/zero-trust-model/)
    *   [Disaster Recovery](https://oreilly.com/search/skills/disaster-recovery/)
    *   [Penetration Testing / Ethical Hacking](https://oreilly.com/search/skills/penetration-testing-ethical-hacking/)
    *   [Governance](https://oreilly.com/search/skills/governance/)
    *   [Malware](https://oreilly.com/search/skills/malware/)
    *   [Security Architecture](https://oreilly.com/search/skills/security-architecture/)
    *   [Security Engineering](https://oreilly.com/search/skills/security-engineering/)
    *   [Security Certifications](https://oreilly.com/search/skills/security-certifications/)

Back

*   Design

Design

    *   [Web Design](https://oreilly.com/search/skills/web-design/)
    *   [Graphic Design](https://oreilly.com/search/skills/graphic-design/)
    *   [Interaction Design](https://oreilly.com/search/skills/interaction-design/)
    *   [Film & Video](https://oreilly.com/search/skills/film-video/)
    *   [User Experience (UX)](https://oreilly.com/search/skills/user-experience-ux/)
    *   [Design Process](https://oreilly.com/search/skills/design-process/)
    *   [Design Tools](https://oreilly.com/search/skills/design-tools/)

Back

*   Business

Business

    *   [Agile](https://oreilly.com/search/skills/agile/)
    *   [Project Management](https://oreilly.com/search/skills/project-management/)
    *   [Product Management](https://oreilly.com/search/skills/product-management/)
    *   [Marketing](https://oreilly.com/search/skills/marketing/)
    *   [Human Resources](https://oreilly.com/search/skills/human-resources/)
    *   [Finance](https://oreilly.com/search/skills/finance/)
    *   [Team Management](https://oreilly.com/search/skills/team-management/)
    *   [Business Strategy](https://oreilly.com/search/skills/business-strategy/)
    *   [Digital Transformation](https://oreilly.com/search/skills/digital-transformation/)
    *   [Organizational Leadership](https://oreilly.com/search/skills/organizational-leadership/)

Back

*   Soft Skills

Soft Skills

    *   [Professional Communication](https://oreilly.com/search/skills/professional-communication/)
    *   [Emotional Intelligence](https://oreilly.com/search/skills/emotional-intelligence/)
    *   [Presentation Skills](https://oreilly.com/search/skills/presentation-skills/)
    *   [Innovation](https://oreilly.com/search/skills/innovation/)
    *   [Critical Thinking](https://oreilly.com/search/skills/critical-thinking/)
    *   [Public Speaking](https://oreilly.com/search/skills/public-speaking/)
    *   [Collaboration](https://oreilly.com/search/skills/collaboration/)
    *   [Personal Productivity](https://oreilly.com/search/skills/personal-productivity/)
    *   [Confidence / Motivation](https://oreilly.com/search/skills/confidence-motivation/)

Back

Platform

Platform

*   [All features](https://oreilly.com/online-learning/features.html)
*   [Verifiable skills](https://oreilly.com/online-learning/verifiable-skills.html)
*   [Academies](https://oreilly.com/online-learning/academies.html)
*   [Courses](https://oreilly.com/online-learning/courses.html)
*   [Certifications](https://oreilly.com/online-learning/feature-certification.html)
*   [Interactive learning](https://oreilly.com/online-learning/intro-interactive-learning.html)
*   [Live events](https://oreilly.com/online-learning/live-events.html)
*   [Superstreams](https://oreilly.com/products/superstreams.html)
*   [Answers](https://oreilly.com/online-learning/feature-answers.html)
*   [Insights reporting](https://oreilly.com/online-learning/insights-dashboard.html)

Back

Search for books, courses, events, and more

Toggle dark mode

AI & ML Business Data Innovation Research Security

Try the O’Reilly learning platform

With the O’Reilly learning platform, you get the resources and guidance to keep your skills sharp and stay ahead. Try it free for up to 14 days.

Start trial

Try a course for free

Join a live online event on the O’Reilly platform to learn from the experts shaping tech.

See what’s coming soon

Get the Radar Trends newsletter

Your email

Country

Please read our privacy policy.

Thank you for subscribing to the O’Reilly Radar Trends to Watch newsletter.

Radar>Topics>AI & ML

The Second-User Test

AI can make the first version work. What happens when someone else tries to use it?

Image 1: Kaiyao Duan By Kaiyao DuanOctober 20, 2026 • 7 minute read

CloseLinkedIn X Facebook Threads Bluesky Reddit

In my work with G7–G12 students at Force Academy, a research initiative in Chongqing, I sometimes take a project that appears to be finished and give another student responsibility for using, checking, or modifying it. The handoff is a test of understanding: Can the work still make sense when its creator is no longer there to translate it? The original author knows what every variable means, why a particular decision was made, which tool was used, and which parts can safely be changed. The second student has only what the project makes visible.

Generative AI makes the handoff more revealing because it lowers the cost of producing a convincing first version. Students can generate code, analysis, prose, visualizations, and prototypes far more quickly than they could a few years ago. In a classroom, however, a project is useful only insofar as it shows what the student understands and can defend. AI can supply missing code, smooth over incomplete reasoning, or turn a rough idea into fluent prose, so polish is no longer strong evidence of understanding by itself.

A concrete example came from a student paper on AI and labor economics, Verification Responsibility in AI-Applicable Occupations. The paper asked what remains economically important when AI makes candidate outputs cheap but a person still has to check those outputs, decide whether they are safe to use, and remain responsible for accepting them.

The lead student developed the conceptual model and most of the analytical workflow. A second student was given a narrower job: rerun the main analyses independently, record the versions of the source data, document how occupations in one dataset were linked to occupations in another, and compare the reported tables and figures with the underlying outputs. What looked like procedural verification quickly became a substantive review of the study.

The first problem was reproducibility. The analysis ran on the original machine, but too much of the workflow depended on choices that lived in one student’s memory. To reproduce it elsewhere, the team had to record the data transformations, write down the rules for linking occupational datasets, and keep intermediate results so another person could see how the final numbers had been produced rather than simply rerunning a script and trusting the output.

The more consequential problem was in the measurement design. An early version of the study used some of the same autonomy and leadership survey items in two different measures: verification responsibility and adaptive capacity. The software had calculated the relationship between those measures correctly, but part of the apparent relationship could have been created by the fact that the two measures shared some of the same inputs. In other words, the code was doing what it had been asked to do; the question was whether the quantities being compared had been defined cleanly enough.

The team rebuilt the measures so they no longer shared survey items, removed ambiguous items, and revised the workflow so source versions, cross-dataset links, exclusions, intermediate files, and reported results could be traced. Later versions added alternative specifications and robustness checks. What began as “please rerun this” ended up forcing a redesign of the study.

The handoff changed how I read student work. A second user does not merely test whether the files have been packaged neatly; they ask the awkward questions the creator may have stopped noticing. Why is a particular variable there? Which conclusion depends on an assumption that was never made explicit? In this project, those questions reached back into the measurement design.

The educational value of the handoff comes from the fact that schoolwork and production software have different ends. Professional developers learn a great deal after software leaves their hands because they have to deliver it, explain it, support it, maintain it, and respond when someone uses it in an unexpected way. Student projects usually disappear when a course ends, and that is fine; the artifact is not the goal. What matters is importing some of the learning that comes from delivery and maintenance: explaining decisions, supporting someone else’s use, and revising the work when private context is no longer available.

At Force Academy, the exact task depends on the project. For a computational model, another student may change a parameter or replace one rule and then ask what should remain stable. For an AI-supported workflow, the job may be to move it to a different model or environment. For a data analysis, the second student may try to reconstruct one result from the recorded source files and transformations. When that transfer fails, I want the original student to decide what the other person actually needed to know and what should be changed in the project, rather than simply standing beside them and explaining it.

I use the word “handles” for the visible points of entry that make that possible. A handle gives another person something concrete to inspect or change without first receiving a private briefing. It might be a clearly named parameter, an explicit assumption, a test that records an earlier failure, a replaceable component, a retained intermediate dataset, or a prompt whose important context is written down instead of buried in a private chat history.

Handles also make disagreement more productive. An exposed assumption can be challenged; a component with a clear boundary can be modified or replaced without rebuilding the whole project; and a numerical result linked to its source data and transformations can be checked rather than accepted on authority.

Outside school, AI systems expose the same weakness. A prompt, workflow, skill, or agent configuration may look portable because it can be copied, while the conditions that make it work remain elsewhere: tool access, file structure, model behavior, private memory, naming conventions, or small decisions the creator never needed to document. Moving the artifact often reveals how much of the working system was never inside the artifact in the first place.

A first successful run is therefore weak evidence. It tells me that a particular combination of person, model, tools, data, and context produced an output once. A stronger test is whether another person can reconstruct what matters, distinguish assumptions from implementation details, and make a change without destroying the logic of the work.

Tim O’Reilly’s idea of an architecture of participation gives me the broader frame. A technical artifact becomes more useful when other people can enter it without negotiating privately with its creator. In software, that usually means interfaces, conventions, documentation, modularity, and replaceable components. In data analysis, the same principle shows up as recorded source versions, visible transformations, retained intermediate results, explicit assumptions, and enough analytical logic for someone else to rerun or challenge the work.

For education, the milestone I care about is whether the student understands the work well enough to make some of that understanding available to someone else. A second student who can use, question, and extend the project gives me evidence that a polished first run cannot.

AI expands what one person can build, including what a teenager can attempt without a large technical team. The expanded scope is a real gain, but it also makes successful generation easier to confuse with understanding. When I want to know what a student really understands, I now learn more from the first serious handoff than from the first successful run.

Post topics: AI & ML

About O'Reilly

Support

International

Download the O'Reilly App

Take O'Reilly with you and learn anywhere, anytime on your phone and tablet.

Do not sell or share my personal information.

Image 5: O'Reilly Logo

Privacy Preference Center

When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change our default settings. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.
More information

Enable All

Manage Consent Preferences

Strictly Necessary Cookies

Always Active

These cookies are necessary for the website to function and cannot be switched off in our systems. They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences, logging in or filling in forms. You can set your browser to block or alert you about these cookies, but some parts of the site will not then work. These cookies do not store any personally identifiable information.

View Vendor Details‎

Performance Cookies

Always Active

These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site. All information these cookies collect is aggregated and therefore anonymous. If you do not allow these cookies we will not know when you have visited our site, and will not be able to monitor its performance.

View Vendor Details‎

Functional Cookies

Always Active

These cookies enable the website to provide enhanced functionality and personalisation. They may be set by us or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.

View Vendor Details‎

Targeting Cookies

Always Active

These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

View Vendor Details‎

Vendors List

Clear

checkbox label label

Apply Cancel

Consent Leg.Interest

checkbox label label
checkbox label label
checkbox label label

Reject All Confirm