Governing AI in Care Delivery: A Framework for Boards

Updated October 21, 2026. First published December 11, 2024.

The useful question for a health system board is not what AI can do. It is which AI serves how the organization is paid, what each kind of model demands once it is approved, and who answers for it.

In Brief

The Problem. Health systems have adopted artificial intelligence (AI) faster than they have built the governance to oversee it. Boards are asked to approve AI investments without a clear way to rank them or to judge what each will require after approval.

Why It Persists. AI proposals arrive framed by technology and vendor promises rather than by the organization's economics. A purchased or cleared tool looks finished, so the cost of validation, monitoring, and training stays out of view.

What Boards Should Do. Rank AI uses by how the organization is paid, require an oversight plan matched to each kind of model, and set the structure: charter placement, an accountable executive, an inventory, conditions on approval, and vendor terms, all recorded in the minutes.

Adoption Has Outrun Governance

Health care providers face intense pressure to improve clinical outcomes, manage costs, and improve the experience of patients and staff. AI is already part of their answer. A 2025 survey of 233 health systems found that 88 percent use AI internally, but only 18 percent reported mature AI oversight and a complete AI plan (Healthcare Financial Management Association [HFMA], 2025). The American Medical Association (AMA) reports that 81 percent of physicians used AI in 2026, up from 38 percent in 2023 (AMA, 2026).

Translating that adoption into results is not straightforward. Even when an AI tool has regulatory clearance, a provider must still ensure its safe use in the workflow, comply with privacy law, train and monitor clinicians, provide technical support, maintain and eventually replace the tool, and solve interoperability problems. Success also depends on organizational readiness, strategic alignment, and an honest view of the tool's limitations, some of which are not yet known.

My argument, drawn from work with health care clients ranging from specialty practices to integrated health systems, is that boards should not start with the technology. They should start with three lenses: how the organization is paid, what each kind of model demands in oversight, and who is accountable. Directors do not need to understand how models work. They need to know which uses deserve capital and what each approval commits the organization to.

Start With How the Organization Is Paid

Payment models carry distinct incentives, and those incentives should set AI priorities. An organization paid mainly through fee-for-service (FFS) and one paid mainly through value-based care (VBC) will rank the same AI proposals differently. Both rankings can be right.

Fee-for-service. FFS pays for each service rendered, which rewards volume. AI in this model earns its return through operational efficiency and accurate revenue capture. Examples include natural language processing that reviews clinical documentation to support accurate coding, AI that flags coding errors before claims are submitted to reduce denials, and forecasting tools that match staffing and resources to patient demand.

Value-based and prevention-oriented care. VBC ties payment to outcomes, quality, and the total cost of care. AI in this model earns its return through avoided admissions and better quality scores. Examples include predictive models that identify patients at high risk of hospitalization for early intervention, analysis that supports more personalized treatment plans, tools that share information among providers to improve care coordination, and population-level analysis that targets interventions to specific groups.

The early evidence on returns follows the same logic. A 2026 survey of 226 health care executives by Bessemer Venture Partners and Bain and Company reported an average self-reported return of 3.5 times the investment. Provider revenue cycle uses returned 4.0 times, and provider clinical uses returned 2.9 times (Guerra & Kraus, 2026). Returns that rest on volume are easier to measure than returns that rest on outcomes.

Most organizations operate under a blend of models, including bundled payments, shared savings programs, capitation, and episode-based payments. These blend elements of FFS and VBC or add incentives tied to episodes, populations, or quality measures. The principle holds across all of them. The question for the board is not whether a proposal is innovative. It is whether the proposal serves the incentives the organization faces now and expects to face over the life of the investment. A proposal that cannot answer that question is not ready for approval.

Know What Each Kind of Model Demands

Even when a provider licenses a prebuilt tool, its leaders should understand the technique behind it, because each kind of model has different data requirements, validation needs, interpretability, and maintenance demands. Those differences drive cost and risk after approval.

Kind of AI	How It Works and Typical Uses	What Oversight It Requires
Supervised learning	Learns from labeled data where the correct answer is known. Uses: detecting abnormalities in images that radiologists have labeled benign or malignant; predicting readmission from demographics, diagnoses, and prior admissions; predicting the risk of conditions such as sepsis or heart failure from vital signs, laboratory results, and history.	Validation on the organization's own patients; monitoring for drift as populations and practice change; a clinician who owns each alert.
Unsupervised learning	Finds patterns in unlabeled data. Uses: grouping patients by disease progression or treatment response; finding cost drivers in service lines from claims data; detecting unusual spikes in admissions or adverse drug events.	A clear statement of how findings will be used; review before findings drive coverage, staffing, or pricing decisions.
Natural language processing (NLP) and large language models (LLMs)	NLP is the field of systems that process human language; LLMs are a form of NLP that can generate and interpret text at scale. Uses: extracting diagnoses, symptoms, and medications from clinical notes; summarizing literature for clinicians; preparing prior-authorization requests; answering patient inquiries and scheduling; supporting coding and billing.	Human review of outputs that reach patients or payers; vendor terms on data use; monitoring for errors and fabricated content.

Other approaches, including federated learning (training models on decentralized data without pooling it), semi-supervised learning, reinforcement learning, transfer learning, and LLM-based conversational agents, show promise. In my judgment their broad clinical use remains further out, because regulatory frameworks are still evolving, clinical effectiveness evidence is thin, and organizations are not yet ready to govern them.

The common thread is that a purchased tool is not a finished tool. Regulatory clearance or validation elsewhere does not establish performance in a particular hospital, workflow, or patient population. When University of Michigan researchers tested the Epic Sepsis Model, a widely deployed vendor tool, across 77,582 hospitalizations, it discriminated well (an area under the curve of 0.87) when scored over the whole stay, but only 0.62 using data recorded before patients met sepsis criteria, and 0.47, no better than chance, once predictions made after clinicians had recognized sepsis were excluded (Kamran et al., 2024). A tool can look accurate largely because it follows what clinicians already suspect. Every significant approval should therefore carry conditions: who monitors the tool, what triggers a review, and when it returns to the board or committee.

The Standards Boards Will Be Measured Against

Providers adopt AI under guidance from many bodies, and the picture has changed since 2024. The U.S. Food and Drug Administration (FDA) had authorized more than 1,400 AI-enabled medical devices as of its March 2026 update (Reuter & Han, 2026). The Office of the National Coordinator for Health Information Technology was renamed the Assistant Secretary for Technology Policy and Office of the National Coordinator for Health Information Technology (ASTP/ONC) in July 2024 and given responsibility for AI policy at the Department of Health and Human Services (HHS) (O'Neil, 2024). Its HTI-5 proposed rule of December 29, 2025 would remove the "model card" transparency and risk management requirements for predictive decision support in certified health IT (Kraus & Brim, 2026).

Under HHS's Section 1557 nondiscrimination rule, covered entities had to review patient care decision support tools for bias by May 1, 2025, although enforcement under the current administration is uncertain (Pressly & Leeds, 2025). Voluntary frameworks fill much of the gap: the National Institute of Standards and Technology (NIST) AI Risk Management Framework (NIST, n.d.), the World Health Organization (WHO) guidance on large multi-modal models (WHO, 2024), and the International Organization for Standardization (ISO) standard ISO/IEC 42001 for AI management systems (ISO, 2023).

Two national markers now give health system boards a direct reference point. On September 17, 2025, the Joint Commission and the Coalition for Health AI (CHAI) issued guidance on responsible AI use built on seven elements: AI policies and governance structures, patient privacy and transparency, data security and data use protections, ongoing quality monitoring, voluntary reporting of AI safety-related events, risk and bias assessment, and education and training (Polsinelli, 2025). On June 1, 2026, the Joint Commission launched a voluntary Responsible Use of AI in Healthcare certification. It certifies organizations, not products, and evaluates governance, data management, risk and bias reduction, monitoring and validation of safety performance, and transparency, education, and training (Guerra, 2026).

State law is also moving. Colorado replaced its 2024 AI Act with Senate Bill 26-189, signed May 14, 2026 and effective January 1, 2027. The law requires notice before covered automated tools materially influence consequential decisions, including access to health care services. Entities covered by the Health Insurance Portability and Accountability Act (HIPAA) are exempt from many obligations but must give patients a general notice of their use of advanced technologies (Darch et al., 2026). Boards of organizations operating in several states should ask counsel for a short map of the state AI laws that apply.

Governance and Leadership Choices

AI can create revenue streams, alter existing ones, reshape roles, and raise new ethical and compliance questions. Six choices belong to the board and its leadership team.

• Strategic alignment. Link each AI initiative to the mission, the payment incentives, and the strategic plan. Investments should deliver measurable financial and clinical returns.
• Oversight structure. Decide which committee oversees AI (quality, audit and compliance, or technology) and write it into the charter. Name one executive accountable for a current AI inventory, with the owner, purpose, and risk level of each tool. Consider adding directors with backgrounds in data science, digital health, or health care regulation.
• Conditions on approval. Attach monitoring, review triggers, and a return date to every significant AI approval.
• Vendor terms. Require terms that bar training on the organization's data without consent, require notice of material model changes, and provide audit rights.
• Long-term investment. AI capability is built over years. Prioritize foundational investment in data quality, interoperability, and scalable platforms.
• Data, risk, and ethics. Strengthen expertise in data governance, privacy, and security, including HIPAA and, where it applies, the European Union General Data Protection Regulation (GDPR). Identify risks such as biased algorithms, privacy breaches, patient safety events, and lack of explainability, and assign clear accountability for each.

The board should record these decisions, and its periodic review of them, in the minutes. The board keeps its nose in and its fingers out. It asks for evidence of control; it does not manage the tools.

The Strongest Objection

The strongest objection to this framework is that clinical AI should be judged on patient benefit, not on how the organization is paid. Ranking AI by payment model, the argument goes, puts revenue ahead of care.

The objection confuses two decisions. Safety and effectiveness are conditions every AI use must meet, whatever the payment model, and the oversight requirements above apply to all of them. The payment lens answers a different question: among uses that meet those conditions, which deserve scarce capital now. A clinical tool that improves outcomes but has no path to sustainable funding under the organization's contracts is likely to be underused or abandoned. Aligning AI with incentives is how a board makes patient benefit last.

Prepare the Organization

Governance fails if clinicians do not trust or understand the tools. Physicians have said what they need. In the AMA survey, 88 percent cited safety and efficacy validation as critical, and 85 percent wanted to be consulted on or involved in adoption decisions (AMA, 2026).

Boards should ask how clinicians, managers, and frontline staff are trained, how they can report concerns about an AI tool, and how the organization will know whether AI supports clinical judgment or displaces it. Some organizations appoint a chief AI officer or embed AI specialists in clinical teams. Whatever the structure, feedback, transparent communication, and change management build the trust adoption depends on.

AI offers providers ways to improve outcomes, efficiency, revenue, and the experience of patients and staff. Realizing them requires a measured, evidence-based, and iterative approach: understand what each model demands, align uses with how the organization is paid, deploy across the care continuum with conditions attached, and govern with a clear record.

For the Board

• Which of our current and proposed AI uses serve the payment models we operate under, and which do not?
• Which committee charter assigns AI oversight, and who is the accountable executive?
• Do we have a current AI inventory with an owner and a risk level for each tool?
• What conditions did we attach to our last significant AI approval, and when does it return to us?
• Do our vendor contracts bar training on our data, require notice of model changes, and give us audit rights?
• How would we fare today against the Joint Commission's Responsible Use of AI in Healthcare certification?

Find the board question sets for health system directors in the Avancer AI Governance Navigator.

References

American Medical Association. (2026, March 12). AMA: AI usage among doctors doubles as confidence in technology grows [Press release]. https://ama-assn.org/press-center/ama-press-releases/ama-ai-usage-among-doctors-doubles-confidence-technology-grows

Darch, J. E., Brooks, P., Daudi, S., Hinckley, D., & Kaye, A. (2026, May 19). Colorado scales back AI law, with targeted implications for health care. Ropes and Gray. https://www.ropesgray.com/en/insights/alerts/2026/05/colorado-scales-back-ai-law-with-targeted-implications-for-health-care

Guerra, A. (2026, June 1). Joint Commission launches AI certification as adoption outpaces oversight. Health System CIO. https://healthsystemcio.com/2026/06/01/joint-commission-ai-certification/

Guerra, S., & Kraus, S. (2026, September 30). The 2026 healthcare AI ROI scorecard. Bessemer Venture Partners. https://www.bvp.com/atlas/the-2026-healthcare-ai-roi-scorecard

Healthcare Financial Management Association. (2025, August 14). Health system adoption of AI outpaces internal governance and strategy. https://www.hfma.org/health-system-adoption-of-ai-outpaces-internal-governance-and-strategy/

International Organization for Standardization. (2023). ISO/IEC 42001:2023 Information technology: Artificial intelligence: Management system. https://www.iso.org/standard/42001

Kamran, F., Tjandra, D., Heiler, A., Virzi, J., Singh, K., King, J. E., Valley, T. S., & Wiens, J. (2024). Evaluation of sepsis prediction models before onset of treatment. NEJM AI, 1(3). https://doi.org/10.1056/AIoa2300032

Kraus, A. D., & Brim, E. (2026, January 15). HHS proposes changes to the health IT certification program and information blocking regulations in HTI-5 proposed rule. Covington and Burling. https://www.covingtonblogs.com/2026/01/15/hhs-proposes-changes-to-the-health-it-certification-program-and-information-blocking-regulations-in-hti-5-proposed-rule/

National Institute of Standards and Technology. (n.d.). AI risk management framework. Retrieved October 21, 2026, from https://www.nist.gov/itl/ai-risk-management-framework

O'Neil, T. (2024, August 9). US Department of Health and Human Services announces a reorganization. Berkeley Research Group. https://www.thinkbrg.com/insights/publications/hss-announces-reorganization-astp-onc/

Polsinelli. (2025, October 29). Joint Commission and Coalition for Health AI release first-of-its-kind guidance on responsible AI use in healthcare. https://www.polsinelli.com/publications/tjc-chai-release-guidance-responsible-ai-use-healthcare

Pressly, D. F., & Leeds, E. I. (2025, April 17). Section 1557: Deadlines approach, but long-term prognosis unclear under executive orders. Ballard Spahr. https://www.ballardspahr.com/insights/alerts-and-articles/2025/04/section-1557-deadlines-approach-but-long-term-prognosis-unclear-under-executive-orders

Reuter, E., & Han, J. Y. (2026, May 27). AI in medtech is booming. Track new devices here. MedTech Dive. https://www.medtechdive.com/news/ai-medtech-track-new-devices-fda/748397/

World Health Organization. (2024, January 18). WHO releases AI ethics and governance guidance for large multi-modal models [News release]. https://www.who.int/news/item/18-01-2024-who-releases-ai-ethics-and-governance-guidance-for-large-multi-modal-models