James Shore 解析如何评估 AI 对软件开发的真实回报
James Shore 撰文探讨了企业该如何评估人工智能在软件开发中的真实投资回报。文章分析了衡量 AI 赋能开发成效的关键维度与潜在挑战，帮助团队厘清实际效益。
- Explore Skills
View all * Data Engineering * Data Warehouse * SQL * Apache Spark * Microsoft SQL Server * MySQL * Kafka * Data Lake * Streaming & Messaging * NoSQL Databases * Relational Databases
View all * Data Science * Pandas * R * MATLAB * SAS * D3 * Power BI * Tableau * Statistics * Exploratory Data Analysis * Data Visualization
View all * AI & ML * Generative AI * Machine Learning * Artificial Intelligence (AI) * Deep Learning * Reinforcement Learning * Natural Language Processing * TensorFlow * Scikit-Learn * Hyperparameter Tuning * MLOps
View all * Programming Languages * Java * JavaScript * Spring * Python * Go * C# * C++ * C * Swift * Rust * Functional Programming
View all * Software Architecture * Object-Oriented * Distributed Systems * Domain-Driven Design * Architectural Patterns
View all * IT/Ops * Kubernetes * Docker * GitHub * Terraform * Continuous Delivery * Continuous Integration * Database Administration * Computer Networking * Operating Systems * IT Certifications
View all * Security * Network Security * Application Security * Incident Response * Zero Trust Model * Disaster Recovery * Penetration Testing / Ethical Hacking * Governance * Malware * Security Architecture * Security Engineering * Security Certifications
View all * Design * Web Design * Graphic Design * Interaction Design * Film & Video * User Experience (UX) * Design Process * Design Tools
View all * Business * Agile * Project Management * Product Management * Marketing * Human Resources * Finance * Team Management * Business Strategy * Digital Transformation * Organizational Leadership
View all * Soft Skills * Professional Communication * Emotional Intelligence * Presentation Skills * Innovation * Critical Thinking * Public Speaking * Collaboration * Personal Productivity * Confidence / Motivation
Platform
Search for books, courses, events, and more
- Explore Skills
Explore Skills
* Cloud Computing
Cloud Computing
* [Microsoft Azure](https://oreilly.com/search/skills/microsoft-azure/)
* [Amazon Web Services (AWS)](https://oreilly.com/search/skills/amazon-web-services-aws/)
* [Google Cloud](https://oreilly.com/search/skills/google-cloud/)
* [Cloud Migration](https://oreilly.com/search/skills/cloud-migration/)
* [Cloud Deployment](https://oreilly.com/search/skills/cloud-deployment/)
* [Cloud Platforms](https://oreilly.com/search/skills/cloud-platforms/)
Back
* Data Engineering
Data Engineering
* [Data Warehouse](https://oreilly.com/search/skills/data-warehouse/)
* [SQL](https://oreilly.com/search/skills/sql/)
* [Apache Spark](https://oreilly.com/search/skills/apache-spark/)
* [Microsoft SQL Server](https://oreilly.com/search/skills/microsoft-sql-server/)
* [MySQL](https://oreilly.com/search/skills/mysql/)
* [Kafka](https://oreilly.com/search/skills/kafka/)
* [Data Lake](https://oreilly.com/search/skills/data-lake/)
* [Streaming & Messaging](https://oreilly.com/search/skills/streaming-messaging/)
* [NoSQL Databases](https://oreilly.com/search/skills/nosql-databases/)
* [Relational Databases](https://oreilly.com/search/skills/relational-databases/)
Back
* Data Science
Data Science
* [Pandas](https://oreilly.com/search/topics/pandas/)
* [R](https://oreilly.com/search/topics/r/)
* [MATLAB](https://oreilly.com/search/topics/matlab/)
* [SAS](https://oreilly.com/search/topics/sas/)
* [D3](https://oreilly.com/search/topics/d3/)
* [Power BI](https://oreilly.com/search/topics/power-bi/)
* [Tableau](https://oreilly.com/search/topics/tableau/)
* [Statistics](https://oreilly.com/search/topics/statistics/)
* [Exploratory Data Analysis](https://oreilly.com/search/topics/exploratory-data-analysis/)
* [Data Visualization](https://oreilly.com/search/topics/data-visualization/)
Back
* AI & ML
AI & ML
* [Generative AI](https://oreilly.com/search/skills/generative-ai/)
* [Machine Learning](https://oreilly.com/search/skills/machine-learning/)
* [Artificial Intelligence (AI)](https://oreilly.com/search/skills/artificial-intelligence-ai/)
* [Deep Learning](https://oreilly.com/search/skills/deep-learning/)
* [Reinforcement Learning](https://oreilly.com/search/skills/reinforcement-learning/)
* [Natural Language Processing](https://oreilly.com/search/skills/natural-language-processing/)
* [TensorFlow](https://oreilly.com/search/skills/tensorflow/)
* [Scikit-Learn](https://oreilly.com/search/skills/scikit-learn/)
* [Hyperparameter Tuning](https://oreilly.com/search/skills/hyperparameter-tuning/)
* [MLOps](https://oreilly.com/search/skills/mlops/)
Back
* Programming Languages
Programming Languages
* [Java](https://oreilly.com/search/skills/java/)
* [JavaScript](https://oreilly.com/search/skills/javascript/)
* [Spring](https://oreilly.com/search/skills/spring/)
* [Python](https://oreilly.com/search/skills/python/)
* [Go](https://oreilly.com/search/skills/go/)
* [C#](https://oreilly.com/search/skills/c-sharp/)
* [C++](https://oreilly.com/search/skills/c-plus-plus/)
* [C](https://oreilly.com/search/skills/c/)
* [Swift](https://oreilly.com/search/skills/swift/)
* [Rust](https://oreilly.com/search/skills/rust/)
* [Functional Programming](https://oreilly.com/search/skills/functional-programming/)
Back
* Software Architecture
Software Architecture
* [Object-Oriented](https://oreilly.com/search/skills/object-oriented-architecture/)
* [Distributed Systems](https://oreilly.com/search/skills/distributed-systems/)
* [Domain-Driven Design](https://oreilly.com/search/skills/domain-driven-design/)
* [Architectural Patterns](https://oreilly.com/search/skills/architectural-patterns/)
Back
* IT/Ops
IT/Ops
* [Kubernetes](https://oreilly.com/search/topics/kubernetes/)
* [Docker](https://oreilly.com/search/topics/docker/)
* [GitHub](https://oreilly.com/search/topics/github/)
* [Terraform](https://oreilly.com/search/topics/terraform/)
* [Continuous Delivery](https://oreilly.com/search/topics/continuous-delivery/)
* [Continuous Integration](https://oreilly.com/search/topics/continuous-integration/)
* [Database Administration](https://oreilly.com/search/topics/database-administration/)
* [Computer Networking](https://oreilly.com/search/topics/computer-networking/)
* [Operating Systems](https://oreilly.com/search/topics/operating-systems/)
* [IT Certifications](https://oreilly.com/search/topics/it-certifications/)
Back
* Security
Security
* [Network Security](https://oreilly.com/search/skills/network-security/)
* [Application Security](https://oreilly.com/search/skills/application-security/)
* [Incident Response](https://oreilly.com/search/skills/incident-response/)
* [Zero Trust Model](https://oreilly.com/search/skills/zero-trust-model/)
* [Disaster Recovery](https://oreilly.com/search/skills/disaster-recovery/)
* [Penetration Testing / Ethical Hacking](https://oreilly.com/search/skills/penetration-testing-ethical-hacking/)
* [Governance](https://oreilly.com/search/skills/governance/)
* [Malware](https://oreilly.com/search/skills/malware/)
* [Security Architecture](https://oreilly.com/search/skills/security-architecture/)
* [Security Engineering](https://oreilly.com/search/skills/security-engineering/)
* [Security Certifications](https://oreilly.com/search/skills/security-certifications/)
Back
* Design
Design
* [Web Design](https://oreilly.com/search/skills/web-design/)
* [Graphic Design](https://oreilly.com/search/skills/graphic-design/)
* [Interaction Design](https://oreilly.com/search/skills/interaction-design/)
* [Film & Video](https://oreilly.com/search/skills/film-video/)
* [User Experience (UX)](https://oreilly.com/search/skills/user-experience-ux/)
* [Design Process](https://oreilly.com/search/skills/design-process/)
* [Design Tools](https://oreilly.com/search/skills/design-tools/)
Back
* Business
Business
* [Agile](https://oreilly.com/search/skills/agile/)
* [Project Management](https://oreilly.com/search/skills/project-management/)
* [Product Management](https://oreilly.com/search/skills/product-management/)
* [Marketing](https://oreilly.com/search/skills/marketing/)
* [Human Resources](https://oreilly.com/search/skills/human-resources/)
* [Finance](https://oreilly.com/search/skills/finance/)
* [Team Management](https://oreilly.com/search/skills/team-management/)
* [Business Strategy](https://oreilly.com/search/skills/business-strategy/)
* [Digital Transformation](https://oreilly.com/search/skills/digital-transformation/)
* [Organizational Leadership](https://oreilly.com/search/skills/organizational-leadership/)
Back
* Soft Skills
Soft Skills
* [Professional Communication](https://oreilly.com/search/skills/professional-communication/)
* [Emotional Intelligence](https://oreilly.com/search/skills/emotional-intelligence/)
* [Presentation Skills](https://oreilly.com/search/skills/presentation-skills/)
* [Innovation](https://oreilly.com/search/skills/innovation/)
* [Critical Thinking](https://oreilly.com/search/skills/critical-thinking/)
* [Public Speaking](https://oreilly.com/search/skills/public-speaking/)
* [Collaboration](https://oreilly.com/search/skills/collaboration/)
* [Personal Productivity](https://oreilly.com/search/skills/personal-productivity/)
* [Confidence / Motivation](https://oreilly.com/search/skills/confidence-motivation/)
Back
Back
- Platform
Platform
* [All features](https://oreilly.com/online-learning/features.html)
* [Verifiable skills](https://oreilly.com/online-learning/verifiable-skills.html)
* [Academies](https://oreilly.com/online-learning/academies.html)
* [Courses](https://oreilly.com/online-learning/courses.html)
* [Certifications](https://oreilly.com/online-learning/feature-certification.html)
* [Interactive learning](https://oreilly.com/online-learning/intro-interactive-learning.html)
* [Live events](https://oreilly.com/online-learning/live-events.html)
* [Virtual conferences](https://oreilly.com/products/superstreams.html)
* [Answers](https://oreilly.com/online-learning/feature-answers.html)
* [Insights reporting](https://oreilly.com/online-learning/insights-dashboard.html)
Back
Search for books, courses, events, and more
Toggle dark mode
AI & MLBusinessDataInnovationResearchSecurity
Try the O’Reilly learning platform
With the O’Reilly learning platform, you get the resources and guidance to keep your skills sharp and stay ahead. Try it free for up to 14 days.
Try a course for free
Join a live online event on the O’Reilly platform to learn from the experts shaping tech.
Get the Radar Trends newsletter
Your email
Country
Subscribe
Please read our privacy policy.
Thank you for subscribing to the O’Reilly Radar Trends to Watch newsletter.
Are You Getting Your Money’s Worth From AI?
By James ShoreOctober 28, 2026 • 7 minute read
Share
CloseLinkedInXFacebookThreadsBlueskyReddit
The following article was originally published on James Shore’sSuccessful Softwareblog and is being republished here with the author’s permission.
Software development has emerged as the biggest beneficiary of AI automation. One CIO I talked to recently, with 170 people in his organization, measured a 2.1x increase in productivity from AI.
That productivity came at a cost. His top users of AI each spent $21,000 on tokens per month. Another engineering leader spends six to eight thousand dollars per engineer per month.
This is not a trivial amount of money. Eventually, it has to come from somewhere, and it will probably come from your personnel budget. Now we’re talking about real consequences for real people’s lives. But there are also real business consequences for not acting.
If you’re going to make those choices—and somebody is going to—they need to be good ones. So you need to know more than the costs; you need to know the benefits too. Are you getting a good return on your investment in AI? Is there another way of using AI that would be better?
How do you know what you’re getting? Anecdotal reports abound, but the reasoning behind those anecdotes are often, to put it politely, not ideal. Some claims are outright fiction.
Given the stakes, that’s a problem.
In this essay series, I’ll show you how to assess the impact of AI on software development in your organization. It’s in four parts plus an epilogue:
- Why, and what not to do(this essay)
- Measuring delivery speed
- Measuring and modeling unintended consequences
- Modeling and forecasting business outcomes
- Epilogue: Notes for your CFO
Why you need to quantify impact
Engineering leaders everywhere are feeling pressure to do more with AI. But which approach is best? There are a lot of people with strident opinions about what to do, but not a lot of consensus, and even less hard data.
Even if there were hard data, it’s not one-size-fits-all. A solution that works best for an internal IT development shop isn’t necessarily the one that works best for a startup, and the approach that’s best for a new product isn’t necessarily the one that’s best for a crafty legacy system.
Faster approaches aren’t necessarily better, either. If you get twice as much done per person, but spend so much money on tokens that you have to cut your staff in half, have you actually seen any ROI? You might actually be better off with a more modest speed boost if it keeps your costs down.
You can’t make these trade-offs from a gut feeling alone. In a famous METR study, engineers thought AI made them 49% faster than it actually did.1 You have to measure impact to know what you’re actually getting.
Don’t neglect costs
Costs are much easier to measure than impact, but they’re not without their own challenges. The biggest one is subsidies. In the fight to gain market share, big AI firms are offering substantial discounts. One engineering leader told me each team member was spending thousands of dollars in tokens at the cost of a $200 subscription. Anthropic and OpenAI are constantly offering discounts and other promotions. At some point, these subsidies will end, and prices will go up.
Or will they? Open-weight models and local inference are exerting downward pressure on token costs. Dedicated inference chips could make tokens cheaper. Routers promise to direct your prompts to the cheapest model for the job. Models could require fewer tokens to accomplish the same task as they become more capable. All of these factors could actually cause prices to go down.
When you calculate ROI, don’t assume today’s costs are the ones you should model. Whether you think prices will go up, down, or remain the same, make your assumptions explicit, and consider modeling both optimistic and pessimistic scenarios.
One more wrinkle: Adopting AI isn’t a matter of flipping a switch. People will need time to learn how to apply it. As with any big shift, change management isn’t to be underestimated. Productivity often goes down during a big change before it goes up. This adds costs too, and means early measurements can be misleading.
Thinking for the long term
Part 2 of this series shows how to measure delivery speed. That’s only part of the picture. If you produce twice as many features, but double your maintenance costs, your team’s overall productivity spikes, then drops like a rock.
Engineer burnout is a concern too. An engineering leader using spec-driven development told me, “Every team is reporting burnout.” I’ll state the obvious: Burned out people don’t produce great work. They don’t push back when they should, they don’t report problems, and eventually they leave, taking valuable institutional knowledge with them.
Part 3 shows how to measure these unintended consequences of AI use, including techniques you can use to forecast future problems.
Finally, delivering software value faster is the point. Doubling your bug-fix rate isn’t going to yield the same results as doubling your rate of competitive new features. For that matter, delivering poorly thought-out features “just because we can” could actually make your product worse.
Part 4 shows how to model, forecast, and measure business outcomes.
Don’t measure lines of code
First, what not to do: Don’t take the easy way out.
First on the no-go list: lines of code. Engineering leaders have known for more than half a century that lines of code don’t correspond to productivity. All else being equal, a feature implemented in 1,000 lines of code will be cheaper to maintain and less bug-prone than a feature implemented in 10,000 lines of code.
AI is notorious for its verbosity. If you measure lines of code, you’re measuring slop, not productivity. Don’t do that. Please.
Don’t measure pull requests either
Pull requests (PRs) are nearly as bad. They’re checkpoints where an engineer asks for their code to be reviewed, and they’re 100% arbitrary. An engineer (or AI) can choose to make a 10-line pull request or a 10,000-line pull request. It can be a feature, a bug fix, or just a tiny code improvement. There’s no inherent correlation between number of pull requests and productivity.
It’s true that engineers will generally produce PRs that are consistent and logical. So, in a sense, a stream of PRs does reflect a certain pace of development, and an increase in PRs can reflect an increase in productivity.
But AI upends that status quo. It produces massive amounts of code very quickly, which can overwhelm engineers’ ability to keep up. Without rigorous engineer oversight, PRs no longer reflect a consistent pace of development, which means that PR counts are no longer a reliable metric.
And, of course, when people think PRs are being used to measure productivity, they’ll be tempted to game the system and produce more PRs than they really need to.
Measure approaches, not people
That brings us to Goodhart’s Law: “Once a measure becomes a target, it ceases to be a good measure.” My favorite book on this topic is Robert D. Austin’s Measuring and Managing Performance in Organizations:
Gradually, measures fall (or, more accurately, are pushed) out of synchronization with true performance, as workers succumb to pressures to take shortcuts. Measured performance trends upward; true performance declines sharply. In this way, the measurement system becomes dysfunctional.
—Robert D. Austin
Don’t establish a performance measurement program and leave it running. Instead, use these metrics to determine how well a particular approach to AI is working. You can repeat the measurements periodically, but whatever you do, don’t use them to measure the performance of teams or individuals. Measure the approach, learn from it, and stop measuring.
Enough about what not to do. In this series, we’re going to quantify AI’s impact, and we’re going to do it right. It starts with measuring delivery speed. That’s next, in part 2.
Footnote
- Participants rated themselves as 20% faster, but were measured as being 19% slower. This study was published prior to the big uplift in coding capability that started with Anthropic’s Opus 4.5, so you shouldn’t take it to mean that AI isn’t capable. But you should take it to mean productivity self-assessments aren’t the way to go. ↩︎
Post topics: AI & ML
Explore related articles
podcast AI & ML ## Is 2016 the year you let robots manage your money? By undefined | undefined
article AI & ML ## How to Get from AI-Assisted to AI Native By undefined | undefined
Follow us
About O'Reilly
- Teach/Write/Train
- Careers
- O'Reilly News
- Media Coverage
- Community Partners
- Affiliate Program
- Diversity
- Content Sponsorship
Support
International
Download the O'Reilly App
Take O'Reilly with you and learn anywhere, anytime on your phone and tablet.
Do not sell or share my personal information.
© 2026, O'Reilly Media, Inc. All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.
Close Previous
Next
Privacy Preference Center
When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change our default settings. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.
More information
Enable All
Manage Consent Preferences
Strictly Necessary Cookies
Always Active
These cookies are necessary for the website to function and cannot be switched off in our systems. They are usually only set in response to actions made by you which amount to a request for services, such as setting your privacy preferences, logging in or filling in forms. You can set your browser to block or alert you about these cookies, but some parts of the site will not then work. These cookies do not store any personally identifiable information.
View Vendor Details
Performance Cookies
Always Active
These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site. All information these cookies collect is aggregated and therefore anonymous. If you do not allow these cookies we will not know when you have visited our site, and will not be able to monitor its performance.
View Vendor Details
Functional Cookies
Always Active
These cookies enable the website to provide enhanced functionality and personalisation. They may be set by us or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.
View Vendor Details
Targeting Cookies
Always Active
These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.
View Vendor Details
Vendors List
Clear
- checkbox label label
Apply Cancel
Consent Leg.Interest
checkbox label label
checkbox label label
checkbox label label
Reject All Confirm
来源：O'Reilly Radar · oreilly.com