The question this answers: why does an AI pilot that worked stop showing results when you roll it out, and what do the firms that see a return count?
Bottom line: pilots are measured on the task; businesses are paid on units they already count. When a pilot scales, the task gain can disappear while a different number moves. The firms that report a return track a business measure and its cost from the start, and someone owns it. Most do neither.
This week’s three findings
1. Most firms cannot measure AI’s return, and even auditors find no baseline
The number: 29% of firms in Anphabe’s Vietnam survey can clearly measure the return on AI, and 4% say its value met or beat their expectations, although 90% have put AI into at least one workflow. Self-reported and second-hand, from a press report of the 18 November launch; the report’s sample size is not stated, so treat the shares as indicative.
What we saw: the gap is not a Vietnamese one. Only 10% of the 100-plus chief procurement officers in Hackett’s sourcing index consistently measure agentic AI performance (second-hand, cited by a procurement-software vendor). Two government audits make it concrete. Australia’s audit office found the health department’s AI fraud model contributed to 8 possible fraud matters worth about A$5.2 million, but the department never tracked what the system cost to build or run and did no cost-benefit analysis. The UK trade department’s own evaluation of its in-house AI assistant collected no baseline, used no control group and left value for money out of scope; self-reported time savings were positive for drafting and literature reviews and negative for data analysis.
So what: a firm that cannot say what its AI costs cannot say what it returns, however good the pilot looked. The Australian case shows the trap: the model found real money, but because nobody tracked its cost or separated its share from five other models, no one can say whether it paid. For a founder-led firm this is cheaper to fix than it sounds. You need two lines in a spreadsheet, kept from the first week: what the AI costs per month, and the business number it should move.
Do this:
- For each AI tool in use, write down its monthly cost (licences, people, vendor fees) and who keeps that number up to date.
- Before any new pilot, record today’s value of the business number it should move, and keep a comparison group that does not get the tool.
2. The pilot gain can vanish at scale while a different number moves
The number: at Boston Medical Center, an 80-physician pilot of ambient documentation shortened the time to close notes, but that gain largely disappeared after the tool was rolled out to all clinicians, while the share of visits coded at levels 4 and 5 rose from 61% to about 71% (company-reported, as told to Becker’s Hospital Review).
What we saw: the firms with usable numbers measure at a unit the business already counts. Airbnb’s support cost per booking fell more than 17% year on year, which its chief executive credited largely to its AI assistant, which now resolves about 40% of support issues without a person (company-reported, second-hand). Rush University System for Health counts about $3.3 million of net revenue found by AI pre-bill chart review since March, and a $1 million outsourcing contract ended (company-reported, second-hand).
So what: pilot teams pick early users and measure the task the tool was bought for. At scale the users change, and the value, if any, lands somewhere the pilot was not looking. That is not a failure of the tool; it is a failure of the measure. Cost per booking, revenue per chart or coding level are numbers the business already reports, so a change in them is hard to argue with and easy to own.
Do this:
- For every pilot, name the business unit measure (cost per order, revenue per case, days to cash) alongside the task measure, and track both after roll-out.
- Plan a check three months after roll-out, not only at the end of the pilot.
3. The firms that report a return track growth and name an owner
The number: in Wharton’s 2026 survey of about 1,400 people at large US firms, 28% of managers report strongly positive return on AI, although 80% report some positive return (self-reported).
What we saw: in the same survey, 58% of firms with a newly named chief AI officer report strongly positive return against 8% without one, and 35% of leaders tracking at least one growth measure do, against 13% tracking efficiency alone. Both are correlations, not proof of cause. Training shows the same gap between leaders and teams: 78% of managers say AI training is adequate, against 56% of the entry and mid-level employees they train.
Track growth outcomes alongside hours saved and maintain visibility into spending.
So what: a chief AI officer is probably a sign of firms that already took AI seriously, not the cause of their return. The useful part is what the title does: it puts one person in charge of the number and its cost. A founder-led firm does not need the title. It needs one named person who owns each AI initiative’s business measure after the pilot team moves on, and a growth measure next to the hours saved. The 22-point gap on training is the other warning: the people who decide think the team is ready more often than the team does.
Do this:
- Name one owner for each AI initiative’s business result, not for the tool.
- Add one growth measure (revenue, orders, conversion) to every AI review that now reports only hours saved.
- Ask the team, not the managers, whether the training was enough.
The surprise of the week
You would expect: AI gains to go mostly to large firms, since our library shows small firms adopt later and see less.
We found: a study of small online sellers on a large Chinese e-commerce platform found that adopting the platform’s built-in generative AI raised monthly sales by 51.6%, mostly through 47% more orders, with the largest gains for the smallest firms. It is an NBER working paper using observational data with statistical controls, not a randomised trial, and it reports sales, not profit.
What it means: small firms gain where AI comes built into a tool they already use and pay for, and the result shows up in a number they already count: orders. That is the opposite of a standalone pilot with its own measure. Singapore’s figures show how far adoption has to go: 23.4% of its small firms use AI, against 70.4% of large local firms (second-hand).
Your next 30 days
If you run a company:
- List every AI tool you pay for with its monthly cost and the person who keeps that figure.
- For each live pilot, write the business unit measure next to the task measure, with today’s value.
- Name one owner per initiative for the result after roll-out.
- Book a three-month check after each roll-out, on the same measures.
If you run an executive programme: Wharton finds managers and their teams 22 points apart on whether AI training is adequate.
- Have participants bring one live AI initiative and name its unit measure, its monthly cost and its owner, then defend them to a panel.
- Use the Boston Medical Center case: ask cohorts where the value went when the pilot gain disappeared.
Where Civico fits
- For CEOs and COOs: the AI Transformation Studio is a 12 to 16 week programme in which your leaders build their own AI initiatives on their own work, and a Pitch Day ends it, where your leadership funds the strongest initiatives for a first stage or stops them. Companies start with a half-day scoping workshop. Tell us the initiative.
- For programme directors: Vietnam Speed is a four-day AI venture residential in which EMBA and executive cohorts build, test and pitch a real AI venture or initiative, with founders, operators and investors in the room. Talk to us about an edition.
- For anyone checking the numbers: every figure above links to its source, and the rows are in the Use cases view.
Method and limits
This week 25 rows and 6 funding rounds were added and 8 existing rows were rebuilt on primary sources; last week’s Insight was corrected where the companies’ own documents differed. Survey figures are self-reported; the Wharton comparisons are correlations. Second-hand in this note: the Anphabe figures (a press report; sample not stated), the Hackett figure (cited by a vendor), the Airbnb, Rush and Boston Medical Center figures (trade-press reports of what executives said) and the Singapore figures. Primary: the Wharton report, the Australian audit, the UK evaluation and the NBER paper. The health-system examples are large US systems and may not carry over to other sectors.