From Sandbox to Scale

AI Oversight From the Board Seat

Innovation without implementation is an expensive hobby. A gated approval process lets the board tell a promising experiment from a scalable product.

In Brief

The Problem. Most AI failures begin when a promising experiment is mistaken for a scalable product. Boards see one decision, to approve a budget, when there are six.

Why It Persists. Demonstrations run on clean, curated data, enthusiasm runs ahead of evidence, and a single approval feels decisive.

What Boards Should Do. Govern each initiative through six gates, with the board acting as patient sponsor, then skeptic, then investor, then risk overseer. Attach conditions to the final approval and record them.

The pattern is familiar. An executive returns from a conference impressed by an AI demonstration that ran on clean, curated data. A mandate follows: deploy this everywhere. Twelve months and millions of dollars later, the project has frustrated the staff it was meant to help and been quietly set aside. The cause is almost always the same: a promising experiment was treated as a scalable product.

Boards are especially exposed to this mistake. They see a single decision, to approve an artificial intelligence (AI) budget, when they should see six. Governing an AI initiative from idea to enterprise-wide use is a passage through a series of demanding gates, not a single leap of faith.

One Approval or Six

The distance between pilot and production is wide. In August 2025, researchers at the NANDA initiative of the Massachusetts Institute of Technology reported that only about five percent of custom enterprise AI tools had reached production (Claburn, 2025).

The Avancer AI Adoption at Scale Framework sets out six gates: Experimentation, Prototyping, Technical Validation, Operational Validation, Commercial Validation, and Adoption at Scale. The board's role changes at each one.

Gate	The Board's Role	The Board's Question
Experimentation and Prototyping	Patient sponsor	Is this interesting enough to merit more resources, and what did we learn last time?
Technical and Operational Validation	Skeptic	Show us that it works, on data it has not seen before.
Commercial Validation	Investor	Does this investment create more value than it consumes?
Adoption at Scale	Vigilant risk overseer	Who is accountable, what will we monitor, and when would we stop?

Gated review is not a theory. Duke Health runs its clinical algorithms through an independent review committee at four lifecycle checkpoints: impact potential, real-world readiness, real-world impact, and sustained impact. Each review ends in approval, approval with contingencies, re-review, or decline (Duke Health, n.d.). That is management's process. The board's task is to ask whether one exists and to see its results.

Gates One and Two: Experimentation and Prototyping

The work begins in a sandbox, a place of permitted failure, where the aim is learning, not profit. The board acts as a patient sponsor. It approves a modest research budget without demanding an immediate business case. The key question is not "What is the return?" It is "Is this interesting enough to merit more resources?" and "What did we learn from the last experiment?"

What Earlier Technology Waves Teach

History argues for that patience, and for many small bets rather than a few large ones. Every technology with the power to reshape an economy has arrived with a wave of excitement. In December 1996, Federal Reserve Chairman Alan Greenspan asked how anyone would know "when irrational exuberance has unduly escalated asset values" (Greenspan, 1996). Such moments of excitement are not the main event. They are the prelude.

The printing press is the long view. Its first effects fell on the scribes it displaced; its larger effects took decades to appear. European cities that adopted printing in the 1400s grew about 60 percent faster than otherwise similar cities between 1500 and 1600, although economists long found no trace of the press in aggregate statistics (Dittmar, 2011). Cities with a press by 1500 were also at least 29 percentage points more likely to be Protestant by 1600 (Rubin, 2014). No one investing in a press in 1460 was forecasting the Reformation.

The British railway boom of the 1840s followed a similar course. Parliament authorized a record 4,538 miles of new line in 1846. Investors put about GBP 250 million into railways by the end of 1850 and lost roughly a third of it. Yet the network they financed became, in the words of the mathematician Andrew Odlyzko, "the world's most extensive, modern, and effective transportation infrastructure" (Odlyzko, 2010).

The internet repeated the lesson. The speculation of the late 1990s focused on a narrow vision of online retail. Pets.com, which went public in February 2000, announced in November 2000 that it would wind down, with its shares more than 98 percent below their high (Richtmyer, 2000). Many concluded that the internet had been overhyped. They were right about the hype and wrong about the internet. In my reading, the infrastructure built during the boom carried the search, cloud, and platform businesses that followed.

The current AI boom has the same shape. Alphabet, Amazon, Meta, and Microsoft forecast about USD 650 billion in combined capital spending for 2026, an increase of about 60 percent in a year, much of it for AI computing (Day, 2026). Today's attention goes to immediate applications, such as drafting text and summarizing documents. They are useful, but the infrastructure now being built will carry uses no one has yet designed.

For a board, the lesson is twofold. First, look past the immediate applications to the strategic questions they raise. Which of our core competencies will AI devalue? What new advantages can we build with it? If a competitor masters it before we do, what is the risk to us? Second, cultivate a historically informed patience: a portfolio of small, inexpensive experiments rather than a few "bet the organization" initiatives, with the expectation that most will fail. If enthusiasm for AI cools, as it has before, that will be the moment the real work begins.

Gates Three and Four: Technical and Operational Validation

If a prototype shows promise, it enters the proving ground, where elegant code meets the reality of daily operations. Technical validation asks whether the tool performs better than the non-AI alternative, measured rigorously against the existing process. Operational validation asks a harder question. Can the tool withstand the incomplete, inconsistent data of different parts of the organization?

Skipping this gate is how a model that works on one hospital's data fails on another's. A proprietary sepsis prediction model in use at hundreds of United States hospitals shows why. When Michigan Medicine independently tested the original Epic Sepsis Model on 38,455 hospitalizations, its discrimination (the area under the curve) was 0.63. It missed 67 percent of patients with sepsis while generating alerts for 18 percent of all patients (Wong et al., 2021). A redesigned version performed better in a 2026 study across four health systems, but its accuracy varied widely by site, and only 13 to 26 percent of its alerts identified a patient who had sepsis (Wong et al., 2026). Coding habits, patient populations, and workflows differ from site to site, and so does performance.

At these gates, the board shifts from sponsor to skeptic. Its request is simple: prove that it holds up, on our data, at each site where it will run.

Gate Five: Commercial Validation

Only after clearing these hurdles should the tool face commercial validation. A system can be technically sound and operationally reliable and still be a poor investment if the cost of maintaining it exceeds its benefits. This is where the board examines the economics closely: net present value, implementation cost, maintenance and monitoring, vendor fees, and expected value.

The economics can be good when the work is done well. In a September 2026 study of 226 healthcare executives, buyers reported payback in about 12 months against the 24 months they had projected (Guerra & Kraus, 2026). The figures are self-reported and reflect uses that reached production. The question at this gate is whether this investment creates more value than it consumes.

Gate Six: Adoption at Scale

For the few initiatives that pass every test, the last stage is the most demanding. Scaling AI is mainly a human challenge, not a technical one: it requires change management, workforce training, and the trust of clinicians and staff. The board becomes a vigilant risk overseer. It demands clear accountability, monitors adoption, and ensures that ethical safeguards are in place. Its main instrument is the set of conditions it attaches to approval:

• a named executive accountable for the system's performance;
• monitoring for drift, incidents, and adoption, reported to a named committee on a schedule;
• vendor terms that bar training on the organization's data, require notice of model changes, grant audit rights, and provide indemnification;
• the rules that apply in each jurisdiction where the system will operate; and
• a stated basis for suspending or retiring the system.

Outside standards now reinforce these conditions. The Joint Commission's voluntary Responsible Use of AI in Healthcare certification launched on June 1, 2026. It asks hospitals to show governance, safeguards, monitoring, and staff education (Twenter, 2026).

The Objection: Gates Slow Us Down

The strongest objection is speed. Competitors are moving, and a board that insists on six gates may approve the right tool a year too late.

The gates are cheap where speed matters and strict where money and patients are at stake. The first two ask for little more than a modest budget and a record of what was learned; they encourage many fast experiments. The strict gates come later, when a system is about to touch thousands of patients or consume a large budget. The sepsis model is the cautionary case: adopted widely before independent validation. Skipping operational validation does not save time. It moves the cost to the bedside.

A Portfolio, Not a Wager

By breaking AI adoption into gates, a board turns a single, high-risk bet into a managed portfolio. It can end unpromising projects early, invest further in those that prove themselves, and ensure that no system reaches patients or staff until it has proven itself technically, operationally, and commercially.

The process produces a record as well. At each gate, the minutes show what the board asked, what it required, and why it proceeded. In my experience, that record is worth as much as the decision itself. It is what the board will rely on when someone later asks how a system came to be in use.

For the Board

• At which gate is each AI initiative we have approved, and who decided it could advance?
• Is our portfolio a set of small experiments or a few large wagers, and how many have we stopped?
• What evidence of operational validation have we seen, on our own data, at each site where the system will run?
• What does it cost to maintain and monitor each system, not only to build it?
• What conditions did we attach to our last AI approval, and who reports against them?
• Which of our core competencies could AI devalue, and on what basis would we suspend a deployed system?

To test a proposed deployment against these gates before the board approves it, book a conversation about a Pre-Deployment Review.

References

Claburn, T. (2025, August 18). GenAI FOMO has spurred businesses to light nearly $40 billion on fire. The Register. https://www.theregister.com/2025/08/18/generative_ai_zero_return_95_percent

Day, M. (2026, February 6). Big Tech to spend $650B this year as AI race intensifies. Data Center Knowledge. https://www.datacenterknowledge.com/hyperscalers/big-tech-to-spend-650b-this-year-as-ai-race-intensifies

Dittmar, J. E. (2011). Information technology and economic change: The impact of the printing press. The Quarterly Journal of Economics, 126(3), 1133–1172. https://doi.org/10.1093/qje/qjr035

Duke Health. (n.d.). ABCDS oversight process. Duke Health AI Governance. Retrieved October 21, 2026, from https://healthaigovernance.duke.edu/abcds-oversight/abcds-oversight-process

Greenspan, A. (1996, December 5). The challenge of central banking in a democratic society [Speech]. Annual Dinner and Francis Boyer Lecture of the American Enterprise Institute for Public Policy Research, Washington, DC, United States. Board of Governors of the Federal Reserve System. https://www.federalreserve.gov/boarddocs/speeches/1996/19961205.htm

Guerra, S., & Kraus, S. (2026, September 30). The 2026 healthcare AI ROI scorecard. Bessemer Venture Partners. https://www.bvp.com/atlas/the-2026-healthcare-ai-roi-scorecard

Odlyzko, A. (2010). Collective hallucinations and inefficient markets: The British Railway Mania of the 1840s [Preliminary version]. University of Minnesota. https://www-users.cse.umn.edu/~odlyzko/doc/hallucinations.pdf

Richtmyer, R. (2000, November 7). Pets.com at its tail end. CNNfn. https://money.cnn.com/2000/11/07/technology/pets/index.htm

Rubin, J. (2014). Printing and Protestants: An empirical test of the role of printing in the Reformation. The Review of Economics and Statistics, 96(2), 270–286. https://doi.org/10.1162/REST_a_00368

Twenter, P. (2026, June 1). Joint Commission launches certification for responsible healthcare AI use. Becker's Hospital Review. https://www.beckershospitalreview.com/quality/joint-commission-launches-certification-for-responsible-healthcare-ai-use/

Wong, A., Currey, D., Schwinne, M., Park-Egan, B., Meyer, S., Gutting, A., Cao, J., Khan, S., Dantes, R., Pan, T., Buchman, T., Singh, K., Bhavani, S. V., Lyons, P. G., Sjoding, M. W., & Tarabichi, Y. (2026). Multicenter prospective validation of an updated proprietary sepsis prediction model. JAMA Network Open, 9(2), Article e260181. https://doi.org/10.1001/jamanetworkopen.2026.0181

Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626