← All insights
Insight

Why AI pilot gains fade at scale

A hospital's pilot gain vanished at scale, an auditor found no costs tracked, and 29% of Vietnamese firms can measure AI's return. What winners count.

22 Nov 2026

The Insight in 7 slides

Scroll through, then download the deck to share with your team.

Civico Insight

Why AI pilot gains fade at scale

A hospital's pilot gain vanished at scale, an auditor found no costs tracked, and 29% of Vietnamese firms can measure AI's return. What winners count.

Civicocivicoventures.com/insights/2026-11-22-pilot-gains-fade-at-scale/
02 / 07
29%
of Vietnamese firms surveyed can clearly measure AI's return, though 90% use it

4% say its value met or beat their expectations. Anphabe, 18 Nov 2026, self-reported, as reported by Báo Pháp luật; sample not stated.

CivicoAnphabe, The Next Productivity Leap
03 / 07
Where AI shows a return, in Wharton's 2026 survey
Firms with a newly namedchief AI officer58%Firms without one8%Leaders tracking agrowth measure35%Leaders trackingefficiency only13%
Firms with a newly named chief AI officer58%Firms without one8%Leaders tracking a growth measure35%Leaders tracking efficiency only13%

Share reporting strongly positive AI return. Self-reported and correlational, not proof that either causes the return. Wharton Human-AI Research and GBK Collective, about 1,400 people at large US firms, July 2026.

Civicocivicoventures.com/insights/2026-11-22-pilot-gains-fade-at-scale/
04 / 07

Track growth outcomes alongside hours saved and maintain visibility into spending.

Wharton Human-AI Research and GBK Collective
Civicocivicoventures.com/insights/2026-11-22-pilot-gains-fade-at-scale/
05 / 07
What the pilot measured, and what scale showed
THE PILOT80 physiciansNotes closed fasterTHE ROLL-OUTNote-closing gain largely goneVisits coded at levels 4-5 up from61% to about 71%
THE PILOT80 physiciansNotes closed fasterTHE ROLL-OUTNote-closing gain largely goneVisits coded at levels 4-5 up from 61% toabout 71%

Boston Medical Center ambient documentation, as reported by Becker's Hospital Review, 20 Nov 2026; company-reported.

Civicocivicoventures.com/insights/2026-11-22-pilot-gains-fade-at-scale/
06 / 07
Before you scale a pilot
  1. Which number the business already counts will move, and what is it today?
  2. What does the AI cost to build and run, per month, and who tracks it?
  3. Who owns the result after the pilot team hands it over?
Civicocivicoventures.com/insights/2026-11-22-pilot-gains-fade-at-scale/
Where Civico fits

Bring a real problem.

The AI Transformation Studio takes a leadership team’s own AI initiatives to a fund-or-stop decision in 12 to 16 weeks. Vietnam Speed, our AI Venture Residential, puts executive cohorts on a real venture in four days.

civicoventures.com
Civicocivicoventures.com/insights/2026-11-22-pilot-gains-fade-at-scale/
1 / 7Share on LinkedIn

The question this answers: why does an AI pilot that worked stop showing results when you roll it out, and what do the firms that see a return count?

Bottom line: pilots are measured on the task; businesses are paid on units they already count. When a pilot scales, the task gain can disappear while a different number moves. The firms that report a return track a business measure and its cost from the start, and someone owns it. Most do neither.

This week’s three findings

1. Most firms cannot measure AI’s return, and even auditors find no baseline

The number: 29% of firms in Anphabe’s Vietnam survey can clearly measure the return on AI, and 4% say its value met or beat their expectations, although 90% have put AI into at least one workflow. Self-reported and second-hand, from a press report of the 18 November launch; the report’s sample size is not stated, so treat the shares as indicative.

What we saw: the gap is not a Vietnamese one. Only 10% of the 100-plus chief procurement officers in Hackett’s sourcing index consistently measure agentic AI performance (second-hand, cited by a procurement-software vendor). Two government audits make it concrete. Australia’s audit office found the health department’s AI fraud model contributed to 8 possible fraud matters worth about A$5.2 million, but the department never tracked what the system cost to build or run and did no cost-benefit analysis. The UK trade department’s own evaluation of its in-house AI assistant collected no baseline, used no control group and left value for money out of scope; self-reported time savings were positive for drafting and literature reviews and negative for data analysis.

90%of Vietnamesefirms surveyeduse AI in atleast oneworkflow(Anphabe)29%can clearlymeasure itsreturn4%say its value metor beatexpectations10%of chiefprocurementofficersconsistentlymeasure agenticAI (Hackett)
90%of Vietnamese firmssurveyed use AI in atleast one workflow(Anphabe)29%can clearly measureits return4%say its value met orbeat expectations10%of chief procurementofficers consistentlymeasure agentic AI(Hackett)
Anphabe, 18 Nov 2026, as reported by Báo Pháp luật (self-reported, sample not stated); Hackett Autonomous Sourcing Index as cited in ISM, Nov 2026.

So what: a firm that cannot say what its AI costs cannot say what it returns, however good the pilot looked. The Australian case shows the trap: the model found real money, but because nobody tracked its cost or separated its share from five other models, no one can say whether it paid. For a founder-led firm this is cheaper to fix than it sounds. You need two lines in a spreadsheet, kept from the first week: what the AI costs per month, and the business number it should move.

Do this:

  • For each AI tool in use, write down its monthly cost (licences, people, vendor fees) and who keeps that number up to date.
  • Before any new pilot, record today’s value of the business number it should move, and keep a comparison group that does not get the tool.

2. The pilot gain can vanish at scale while a different number moves

The number: at Boston Medical Center, an 80-physician pilot of ambient documentation shortened the time to close notes, but that gain largely disappeared after the tool was rolled out to all clinicians, while the share of visits coded at levels 4 and 5 rose from 61% to about 71% (company-reported, as told to Becker’s Hospital Review).

What we saw: the firms with usable numbers measure at a unit the business already counts. Airbnb’s support cost per booking fell more than 17% year on year, which its chief executive credited largely to its AI assistant, which now resolves about 40% of support issues without a person (company-reported, second-hand). Rush University System for Health counts about $3.3 million of net revenue found by AI pre-bill chart review since March, and a $1 million outsourcing contract ended (company-reported, second-hand).

THE PILOT MEASUREDMinutes to close a note80 enthusiastic physiciansSCALE SHOWEDNote-closing gain largely goneCoding at levels 4-5 up from 61% toabout 71%
THE PILOT MEASUREDMinutes to close a note80 enthusiastic physiciansSCALE SHOWEDNote-closing gain largely goneCoding at levels 4-5 up from 61% to about71%
Boston Medical Center ambient documentation, as reported by Becker's Hospital Review, 20 Nov 2026; company-reported.

So what: pilot teams pick early users and measure the task the tool was bought for. At scale the users change, and the value, if any, lands somewhere the pilot was not looking. That is not a failure of the tool; it is a failure of the measure. Cost per booking, revenue per chart or coding level are numbers the business already reports, so a change in them is hard to argue with and easy to own.

Do this:

  • For every pilot, name the business unit measure (cost per order, revenue per case, days to cash) alongside the task measure, and track both after roll-out.
  • Plan a check three months after roll-out, not only at the end of the pilot.

3. The firms that report a return track growth and name an owner

The number: in Wharton’s 2026 survey of about 1,400 people at large US firms, 28% of managers report strongly positive return on AI, although 80% report some positive return (self-reported).

What we saw: in the same survey, 58% of firms with a newly named chief AI officer report strongly positive return against 8% without one, and 35% of leaders tracking at least one growth measure do, against 13% tracking efficiency alone. Both are correlations, not proof of cause. Training shows the same gap between leaders and teams: 78% of managers say AI training is adequate, against 56% of the entry and mid-level employees they train.

Share reporting strongly positive AI returnFirms with a newly namedchief AI officer58%Firms without one8%Leaders tracking atleast one growth measure35%Leaders trackingefficiency only13%
Share reporting strongly positiveAI returnFirms with a newly named chief AI officer58%Firms without one8%Leaders tracking at least one growth measure35%Leaders tracking efficiency only13%
Wharton Human-AI Research and GBK Collective, 2026 AI Adoption Report, about 1,400 people at US firms with 1,000+ staff, July 2026. Self-reported; correlational.
Track growth outcomes alongside hours saved and maintain visibility into spending.
Wharton Human-AI Research and GBK Collective

So what: a chief AI officer is probably a sign of firms that already took AI seriously, not the cause of their return. The useful part is what the title does: it puts one person in charge of the number and its cost. A founder-led firm does not need the title. It needs one named person who owns each AI initiative’s business measure after the pilot team moves on, and a growth measure next to the hours saved. The 22-point gap on training is the other warning: the people who decide think the team is ready more often than the team does.

Do this:

  • Name one owner for each AI initiative’s business result, not for the tool.
  • Add one growth measure (revenue, orders, conversion) to every AI review that now reports only hours saved.
  • Ask the team, not the managers, whether the training was enough.

The surprise of the week

You would expect: AI gains to go mostly to large firms, since our library shows small firms adopt later and see less.

We found: a study of small online sellers on a large Chinese e-commerce platform found that adopting the platform’s built-in generative AI raised monthly sales by 51.6%, mostly through 47% more orders, with the largest gains for the smallest firms. It is an NBER working paper using observational data with statistical controls, not a randomised trial, and it reports sales, not profit.

YOU WOULD EXPECTLarge firms gain most from AISmall firms lag on adoption andresultsWE FOUNDSmall sellers' monthly sales up51.6% after adoptingLargest gains for the smallest firms
YOU WOULD EXPECTLarge firms gain most from AISmall firms lag on adoption and resultsWE FOUNDSmall sellers' monthly sales up 51.6%after adoptingLargest gains for the smallest firms
Barwick, Fang, Liu, Xu and Yuan, NBER Working Paper 36003, November 2026; firms with annual sales below RMB 5 million; not peer reviewed.

What it means: small firms gain where AI comes built into a tool they already use and pay for, and the result shows up in a number they already count: orders. That is the opposite of a standalone pilot with its own measure. Singapore’s figures show how far adoption has to go: 23.4% of its small firms use AI, against 70.4% of large local firms (second-hand).

Your next 30 days

From pilot to a result that holds01Count the costMonthly, from weekone02Name the unitCost per order,revenue per case03Name the ownerFor the result, notthe tool04Check afterroll-outThree months on,same measure
From pilot to a result that holds01Count the costMonthly, from week one02Name the unitCost per order, revenue per case03Name the ownerFor the result, not the tool04Check after roll-outThree months on, same measure

If you run a company:

  1. List every AI tool you pay for with its monthly cost and the person who keeps that figure.
  2. For each live pilot, write the business unit measure next to the task measure, with today’s value.
  3. Name one owner per initiative for the result after roll-out.
  4. Book a three-month check after each roll-out, on the same measures.

If you run an executive programme: Wharton finds managers and their teams 22 points apart on whether AI training is adequate.

  1. Have participants bring one live AI initiative and name its unit measure, its monthly cost and its owner, then defend them to a panel.
  2. Use the Boston Medical Center case: ask cohorts where the value went when the pilot gain disappeared.

Where Civico fits

  • For CEOs and COOs: the AI Transformation Studio is a 12 to 16 week programme in which your leaders build their own AI initiatives on their own work, and a Pitch Day ends it, where your leadership funds the strongest initiatives for a first stage or stops them. Companies start with a half-day scoping workshop. Tell us the initiative.
  • For programme directors: Vietnam Speed is a four-day AI venture residential in which EMBA and executive cohorts build, test and pitch a real AI venture or initiative, with founders, operators and investors in the room. Talk to us about an edition.
  • For anyone checking the numbers: every figure above links to its source, and the rows are in the Use cases view.

Method and limits

This week 25 rows and 6 funding rounds were added and 8 existing rows were rebuilt on primary sources; last week’s Insight was corrected where the companies’ own documents differed. Survey figures are self-reported; the Wharton comparisons are correlations. Second-hand in this note: the Anphabe figures (a press report; sample not stated), the Hackett figure (cited by a vendor), the Airbnb, Rush and Boston Medical Center figures (trade-press reports of what executives said) and the Singapore figures. Primary: the Wharton report, the Australian audit, the UK evaluation and the NBER paper. The health-system examples are large US systems and may not carry over to other sectors.

Quick poll

When your last AI pilot was rolled out, did its result hold?

Anonymous. One tap.

Get the weekly Insight

One short email a week on AI and leadership, with the evidence linked. Unsubscribe any time.

The data behind this Insight

Bring a real problem.

Civico runs experiential programmes for business leaders transforming their companies with AI. Leave with a decision that holds up.

Thank you.

Follow Civico on LinkedIn for the weekly Insight and news from the residentials.