When the Black Box Breaks

AI Oversight From the Board Seat

A sepsis model used in hundreds of hospitals performed far worse on one health system's patients than its developer reported. The lesson for boards lies in what happens when a model meets data it was never built for.

In Brief

The Problem. A vendor model that performs well where it was built can fail on another organization's patients. When Michigan Medicine tested a sepsis prediction model used in hundreds of U.S. hospitals, it missed two of every three sepsis cases.

Why It Persists. Boards treat AI as a purchase that can be installed and left alone, accept the developer's figures, hesitate to ask basic questions, and seldom write AI risks into committee charters.

What Boards Should Do. Require independent validation on the organization's own patients before and after deployment, continuous monitoring, and a named owner who can switch the system off. Keep a record of that oversight, because courts now ask for it.

Epic Systems' sepsis prediction model was implemented at hundreds of U.S. hospitals. It scores hospitalized patients and alerts clinicians to those it judges at risk of sepsis, a condition in which hours matter. In 2021, researchers at Michigan Medicine tested the model independently on 38,455 hospitalizations. Its area under the curve was 0.63, compared with 0.76 to 0.83 in the developer's internal documentation. At the alert threshold studied, the model did not identify 67 percent of the patients who developed sepsis, while it generated alerts for 18 percent of all hospitalized patients (Wong et al., 2021).

The area under the curve measures how often a model ranks a patient who has the condition above one who does not: 1.0 is perfect, and 0.5 is no better than a coin toss. In 2024, a second Michigan team tested the same model across 77,582 hospitalizations and asked a harder question: could it warn clinicians before they had recognized sepsis themselves? Using predictions made before patients met sepsis criteria, the area under the curve was 0.62. Once predictions made after clinical recognition were excluded, it fell to 0.47, no better than chance (Kamran et al., 2024). The model looked accurate in large part because it followed what clinicians already suspected. An updated version later performed better across four health systems, but its accuracy varied from site to site, and only 13 to 26 percent of its alerts identified a patient who had sepsis (Wong et al., 2026).

The lesson for boards is direct. A vendor model validated where it was built has not been validated where the organization uses it. Before deployment, and at intervals after it, the model must be tested independently on the organization's own patients. Its performance must be monitored, and a named executive must own it, with the authority to switch it off. My argument is that this is where AI value, and AI safety, most often fail: not in the strategy, which boards scrutinize, but in the implementation, which they too often approve on trust.

The Record Becomes Evidence

When a system fails, the question soon turns to who was watching. In Estate of Lokken v. UnitedHealth Group, the families of two deceased Medicare Advantage members allege that UnitedHealthcare used nH Predict, a predictive tool, to override physicians' judgment and end coverage of post-acute care prematurely. The company disputes the allegations and maintains that physicians make the coverage decisions. The court allowed the breach of contract and good faith claims to proceed, and on March 9, 2026, a federal magistrate judge ordered discovery that reaches materials of the insurer's internal AI review board, including the identities of its members (Greenberg et al., 2026).

The allegations are unproven. The order, however, shows that the oversight record behind an AI system, including what the board and its committees asked, were told, and decided, can become evidence. Minutes that show a validation requirement, a monitoring report, and a decision to suspend a failing tool tell one story. Minutes that show a vendor presentation and an approval tell another.

Why Scaling Breaks Models

Artificial intelligence (AI) performs best on the kind of data it was built on. When an organization expands, through acquisition or through deployment to new sites, the data changes. Each acquired hospital or business unit brings its own record systems and data formats, a problem of data heterogeneity that an elegant algorithm may not be able to absorb. A model trained on the population of a large urban center can prove brittle in a rural population with different prevalent conditions. Its accuracy falls, and the clinicians it was designed to help stop trusting it.

These failures are rarely visible in the approval presentation, which shows the model's performance where it already works. From Sandbox to Scale sets out the approval gates that test a model on local data before it reaches patients. The board's question is how the model will perform where it has not yet been tested.

The same problem appears at the scale of a business. IBM spent billions of dollars acquiring health data companies, including Truven, Merge Healthcare, Explorys, and Phytel, to build Watson Health. According to STAT, IBM struggled to integrate their data and cultures into a coherent business, and the unit lost clients, missed sales targets, and was not profitable before IBM sold its data and analytics assets to Francisco Partners in January 2022 (Ross, 2022). Acquired data does not combine on its own, and a board that approves an acquisition strategy built on data should ask who will make that data work together, and at what cost.

AI Behaves Less Like a Tool Than a New Employee

The misconception at the root of many failures is that AI can be installed and left alone. Boards that treat it as a simple capital purchase, like a new fleet of servers, misunderstand both the technology and their role. Traditional software follows fixed rules; it is a predictable, depreciating asset. An AI model learns from the data it receives. It behaves less like a spreadsheet and more like a capable, unpredictable new employee, and it can develop habits that no one intended. Governing it like hardware is like supervising a star trader by checking only that the trader's computer has power, while ignoring the trader's strategy, appetite for risk, and ethics.

Consider, as an illustration, a model that screens candidates for a clinical trial. If its training data underrepresent certain groups, it may exclude eligible patients from those groups, compromising both the validity and the fairness of the trial. A hospital tool that allocates staff and beds could read a sudden rise in a common illness as routine and move resources away from a developing outbreak.

Models also drift, becoming less accurate as the world changes. A model that detects fraudulent or erroneous claims, for example, may lose effectiveness as billing patterns evolve. Governing AI therefore requires continuous monitoring, periodic revalidation, and a plan for retraining or retirement. Few health systems have that discipline in place. A 2025 survey of 233 health systems found that 88 percent used AI, but only 18 percent had both a mature governance structure and a fully formed AI strategy (Healthcare Financial Management Association, 2025).

When the Board Cannot Explain a Decision

A system that cannot be explained or audited leaves the board without an answer when a regulator, a court, or a patient asks why a decision was made. In Europe, that is now a legal question. In February 2025, the Court of Justice of the European Union ruled that a person subject to an automated decision is entitled to an intelligible explanation of the procedure and principles actually applied (Cooper et al., 2025). A reference to "the algorithm" will not satisfy that standard, or a patient.

Where Oversight Breaks Down

Directors are often reluctant to challenge proposals they do not fully understand. The hesitation is not only a fear of appearing foolish. It is often deference to complexity itself, an unspoken assumption that something so specialized must be correct. Yet these are exactly the first-principle questions the board must ask: how the algorithm was validated, how and when it will be retrained, and where an accountable person stays in the loop. Overcoming Tech Intimidation sets out how a chair can make such questions routine.

When those questions go unasked, the risk is not only a disappointing return. A poorly governed model can cause harm: biased screening in the hiring of clinical staff, inequitable allocation of new therapies, or flawed clinical decision support with life-or-death consequences.

The Objection: The Vendor Carries This Risk

The strongest objection is that this is the vendor's problem. Most healthcare organizations buy AI rather than build it, and the vendor trains, validates, and updates the model. The board's job, on this view, is to choose a reputable vendor and negotiate a sound contract.

A sound contract matters, and From Sandbox to Scale sets out the vendor terms the board should expect. But the vendor validated the model on someone else's data. The organization that deploys it owns the result at its own sites, with its own patients, and under its own license and accreditation. A contract can allocate liability. It cannot transfer the board's duty to know how the system performs where the organization uses it.

Building the Capability

Three changes help.

First, composition. Recruit directors with experience not only in technology but in the practicalities of deploying it at scale. Experience leading a difficult systems integration often teaches more about how data and workflows fail in practice than experience building new technology.

Second, committee charters. The audit and risk committees should expand their charters to cover algorithmic auditing and AI-specific risks, including model drift, data poisoning, and adversarial attacks. Where the board creates a technology committee, it needs genuine authority: its charter should reach beyond uptime and data security to model validation, testing for fairness, and the protocols for retiring a system that fails. In 2025, the National Association of Corporate Directors found that only 27 percent of public company boards had written AI oversight into committee charters (National Association of Corporate Directors, 2025).

Third, accountability. For each material AI system, the organization should record who is responsible, accountable, consulted, and informed, a record often called a RACI chart. It should also name who decides to suspend or retire the system, and on what evidence.

Mind the Governance Gap

The first wave of digital transformation was about strategy. The AI era is about execution, and a strong AI strategy is worth little without governance that can implement it and adapt it. AI already has a seat at the table; the board must decide whether it will govern that seat or be governed by it.

I would put one question to every board before it approves a plan to scale AI across new sites or acquired businesses. How will the model perform on data it has never seen?

For the Board

• Has each material vendor model been validated independently on our own patients, before deployment and since, and what evidence shows it will perform at new sites or acquired businesses?
• How do we monitor each model for drift, who reports the results to us, and would our minutes show what we asked, what we were told, and what we decided?
• Do our committee charters cover model drift, data poisoning, and adversarial attacks, and who has authority to switch off or retire a system?
• For each material AI system, who is responsible, accountable, consulted, and informed, and who is accountable if it harms a patient?
• How do we audit the data our models learn from, including data supplied by vendors?
• If a patient or regulator asked us to explain an AI-assisted decision, could we?

To test whether a model will hold up on new data before the board approves scale, book a conversation about a Pre-Deployment Review.

References

Cooper, D., Van Quathem, K., Oberschelp de Meneses, A. S., & Brazil, D. (2025, March 6). CJEU clarifies GDPR rights on automated decision-making and trade secrets. Inside Privacy. https://www.insideprivacy.com/gdpr/cjeu-clarifies-gdpr-rights-on-automated-decision-making-and-trade-secrets/

Greenberg, D. S., Heilman, K., & Shlaferman, H. Z. (2026, April 23). Federal court orders broad discovery against UHC in AI coverage denial lawsuit. ArentFox Schiff. https://www.afslaw.com/perspectives/alerts/federal-court-orders-broad-discovery-against-uhc-ai-coverage-denial-lawsuit

Healthcare Financial Management Association. (2025, August 12). Health system adoption of AI outpaces internal governance and strategy [Press release]. https://www.hfma.org/health-system-adoption-of-ai-outpaces-internal-governance-and-strategy/

Kamran, F., Tjandra, D., Heiler, A., Virzi, J., Singh, K., King, J. E., Valley, T. S., & Wiens, J. (2024). Evaluation of sepsis prediction models before onset of treatment. NEJM AI, 1(3). https://doi.org/10.1056/AIoa2300032

National Association of Corporate Directors. (2025). AI oversight activities conducted by the board [Data table]. 2025 Public Company Board Practices and Oversight Survey. Retrieved October 21, 2026, from https://www.nacdonline.org/all-governance/governance-resources/governance-surveys/surveys-benchmarking/bpo-tables/ai-oversight-activities-conducted-by-the-board

Ross, C. (2022, January 21). Once billed as a revolution in medicine, IBM's Watson Health is sold off in parts. STAT. https://www.statnews.com/2022/01/21/ibm-watson-health-sale-equity/

Wong, A., Currey, D., Schwinne, M., Park-Egan, B., Meyer, S., Gutting, A., Cao, J., Khan, S., Dantes, R., Pan, T., Buchman, T., Singh, K., Bhavani, S. V., Lyons, P. G., Sjoding, M. W., & Tarabichi, Y. (2026). Multicenter prospective validation of an updated proprietary sepsis prediction model. JAMA Network Open, 9(2), Article e260181. https://doi.org/10.1001/jamanetworkopen.2026.0181

Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626