Building Effective AI Governance in Health Care: What Leaders Can Learn from Duke’s Approach

Artificial intelligence is moving into health care faster than most governance models were designed to handle. Predictive analytics, ambient documentation, workflow automation, referral intelligence, and generative AI are no longer future-state concepts. They are procurement decisions, clinical workflow changes, data governance questions, and risk-management issues happening now.

That creates a hard reality for healthcare delivery organizations: if AI adoption moves faster than oversight, patient safety, operational integrity, and regulatory posture can all suffer at once.

In a recent discussion on AI governance in health care, Dr. Eric Poon of Duke Health described how his organization has built and evolved a governance structure over several years. His perspective is especially useful because it reflects a real operational setting rather than a theoretical framework. The lesson is not that one committee or one policy solves AI risk. It is that effective governance is iterative, multidisciplinary, and tightly connected to outcomes.

For healthcare executives, CISOs, CIOs, compliance leaders, and digital health teams, the discussion offers a practical blueprint: start before the model landscape stabilizes, reuse existing governance muscle where possible, and build mechanisms to test whether AI is actually delivering value.

Why AI Governance in Health Care Is Different

Most health systems already have some process for approving technology purchases, assessing vendors, and managing implementation risk. But AI introduces several complications that make a standard IT intake process insufficient.

AI changes after deployment

Traditional software often behaves consistently once implemented. AI systems may not. Performance can drift, workflows can adapt around them, and users may apply outputs in ways the original project team did not anticipate. In clinical environments, that matters because even modest shifts in performance or interpretation can affect patient care.

AI is not just a technology decision

Dr. Poon’s comments underscore a crucial point: AI governance is as much a human-systems challenge as a technical one. A model may generate a useful score or summary, but that does not guarantee clinicians or staff will use it correctly, consistently, or at all. The governance question is not simply, "Does the model work?" It is also:

  • Who sees the output?
  • What action should follow?
  • What training and accountability exist?
  • What happens when the AI is wrong?
  • Is the tool better than current practice?

These are operational design questions, not just data science questions.

Generative AI raises a different class of risk

Predictive models can often be evaluated against a measurable outcome. Generative AI is harder. It may produce several acceptable responses to the same prompt, but it can also produce misleading or inconsistent outputs. That makes validation, monitoring, and accountability more complex.

For health care, where documentation, communication, and decision support may influence treatment or access to care, this distinction matters. Governance for predictive tools cannot simply be copied and pasted onto generative AI.

The First Lesson: Don’t Wait for the Perfect Governance Model

One of the strongest themes from Duke’s experience is that governance must begin before it feels fully mature.

That may sound uncomfortable to healthcare leaders used to high-reliability environments. But the alternative is worse: a fragmented landscape where departments procure or pilot AI tools without common standards for privacy, safety, value measurement, or review.

The more durable principle is this: governance should evolve alongside the technology.

That has several implications:

  • Early governance structures should be lightweight enough to launch
  • Review criteria should mature over time
  • Governance bodies should expect to redesign themselves
  • Educational support should sit alongside control functions

In other words, health systems should not aim for a static AI governance policy. They should build an adaptive operating model.

Key Takeaways

  • Start now, not later. Waiting for a perfect framework increases the odds of unmanaged AI sprawl.
  • Reuse existing governance processes. Health systems already have technology review structures; AI governance should extend them, not replace them entirely.
  • Measure value after deployment. Approval should not be the end of oversight. Teams need evidence that a tool improves outcomes, workflow, or efficiency.
  • Treat governance as multidisciplinary. Clinical, IT, legal, compliance, security, analytics, and operational leaders all need a seat at the table.
  • Validate workflow impact, not just model performance. A strong model that lacks user adoption or action pathways delivers little value.
  • Create low-risk pilot pathways. Fast, bounded pilots can help organizations test tools without overcommitting resources.
  • Separate review speed from review rigor. Streamlining process does not require lowering standards.
  • Build different controls for different AI types. Predictive models and generative AI require different evaluation approaches.
  • Educate project teams continuously. Governance works better when people understand why requirements exist.
  • Be willing to stop weak projects. Ending low-value pilots early is a sign of maturity, not failure.

Governance Should Build on Existing Technology Oversight

A common mistake in AI strategy is treating AI as so unique that it requires an entirely separate operating system from everything that came before. Duke’s experience suggests a more practical approach.

Most organizations already have investments in:

  • IT governance committees
  • vendor review processes
  • legal and contracting workflows
  • security assessments
  • clinical operations oversight
  • analytics and quality infrastructure

Those assets should not be discarded. Instead, AI governance should extend them with additional checkpoints that account for model risk, fairness, monitoring, and post-deployment performance.

This matters especially for resource-constrained organizations. Smaller hospitals, community systems, and provider groups may not be able to launch a large standalone AI office. But they can still establish governance by layering AI-specific review questions into existing decision-making forums.

For example, an organization might add the following to its current intake process:

  • What is the intended use of the AI?
  • Is this tool advisory, administrative, or patient-facing?
  • What data does it use, and where does that data go?
  • What validation evidence exists?
  • How will success be measured locally?
  • Who owns monitoring after launch?
  • What is the shutdown plan if it underperforms?

That is often more practical than trying to build an elaborate new structure all at once.

Value Must Be Proven, Not Assumed

A standout feature of Duke’s approach is the emphasis on accountability for value after deployment.

That is a significant governance shift. In many health systems, once technology goes live and major complaints do not materialize, attention moves elsewhere. Dr. Poon argues that this is not sufficient for AI, and he is likely right.

Why post-implementation review matters more for AI

AI can look promising in demonstrations, vendor literature, or even internal pilots, yet still fail to produce meaningful operational or clinical gains at scale. Common reasons include:

  • poor workflow integration
  • alert fatigue
  • unclear user ownership
  • low trust in outputs
  • limited accuracy in local populations
  • negligible time savings after implementation overhead

Without follow-up measurement, organizations risk keeping tools that consume budget, introduce risk, and create the appearance of innovation without delivering measurable benefit.

What "value" should mean

The video does not prescribe a formal scorecard, but the logic points toward a balanced evaluation model. Depending on the use case, organizations should consider whether the AI improves:

  • patient safety
  • clinical outcomes
  • operational efficiency
  • staff burden
  • access to care
  • documentation quality
  • financial performance
  • equity and fairness

Importantly, Dr. Poon notes that teams do not necessarily need publication-grade evidence. But they do need enough data to justify continued use.

That is an important middle ground for operational leaders: governance should demand evidence, not academic perfection.

Workflow Design Is the Hidden Core of AI Governance

One of the most valuable insights in the discussion is that many teams focus on the output of the AI and neglect the people and processes around it.

That is especially dangerous in clinical settings.

A sepsis-risk model, for instance, is not useful simply because it can assign a score. Its actual value depends on whether:

  • the right clinician or team receives the signal
  • the alert appears at the right moment
  • action pathways are defined
  • users understand the confidence and limitations
  • escalation and documentation are clear
  • performance is monitored over time

This is where many AI programs stumble. They treat governance as model review instead of sociotechnical design.

For healthcare leaders, that suggests a stronger operating principle: every AI proposal should be reviewed as a workflow intervention. The model is only one component.

From Advisory Review to Formal Checkpoints

Duke’s governance journey also illustrates an evolution many organizations will likely need to make.

Initially, advisory support can be useful. It helps teams think through risks, implementation needs, and outcome measures. But as the number and complexity of AI use cases increase, informal guidance is not enough.

At some point, governance needs:

  • defined checkpoints
  • required documentation
  • review ownership
  • escalation pathways
  • decision rights
  • turnaround expectations

This shift from "advice" to "structured review" is important for two reasons.

First, scale demands consistency

Once AI requests begin arriving regularly, ad hoc reviews create bottlenecks and uneven standards. A formalized process supports fairness, speed, and auditability.

Second, health care needs evidence of due diligence

For regulated organizations, governance is not only about making better decisions internally. It also supports defensibility. If a tool contributes to a privacy issue, workflow disruption, or patient safety event, leaders need to show that the organization had a reasonable review process.

The key, however, is not to build a cumbersome mechanism that slows every initiative to a halt.

Good Governance Is Also an Education Function

A subtle but important point from the discussion is that governance works better when it teaches, not just blocks.

Many project teams proposing AI tools are not trying to avoid risk controls. They often just do not know what good AI implementation requires. That is especially true for operational or clinical sponsors who may understand the problem deeply but lack experience in model validation, data rights, or lifecycle monitoring.

Duke responded by coupling formal review with education, including office hours and transparent guidance. That approach is worth highlighting because it addresses one of the biggest causes of governance frustration: people perceive the process as bureaucracy when they do not understand the purpose behind the questions.

Educational governance can improve both speed and quality by helping teams prepare better submissions from the outset.

For health systems, this could include:

  • intake templates with examples
  • brief risk-tiering guides
  • office hours for sponsors and vendors
  • standard evaluation criteria by AI category
  • FAQ documents for legal, privacy, and security expectations

This model is especially relevant for cybersecurity and compliance leaders. Preventive education often reduces downstream review burden more effectively than stricter gates alone.

Efficiency Matters: Governance Must Scale With Demand

Early-stage AI programs may be able to review requests in occasional committee meetings. That model breaks down quickly once AI use cases expand across clinical, administrative, revenue cycle, and patient engagement domains.

Duke’s response was to redesign the process for throughput by:

  • splitting review work across smaller teams
  • moving more review activity offline
  • reducing dependence on synchronous meetings
  • enabling quicker feedback to project teams

That is an operationally important insight.

Governance programs often fail not because their standards are wrong, but because their workflow is too slow. In fast-moving AI markets, slow governance can unintentionally encourage shadow procurement or workaround behavior.

For leaders designing oversight models, the goal should be high-trust speed:

  • clear requirements
  • fast triage
  • risk-based routing
  • distributed reviewers
  • rapid feedback loops
  • documented decisions

In cybersecurity terms, this resembles modern security architecture: move from monolithic checkpoints to continuous, distributed controls.

"Fail Fast" Has a Responsible Place in Health Care

The phrase "fail fast" can sound out of place in clinical environments, where safety and reliability are paramount. But the version described in this discussion is not reckless experimentation. It is bounded, low-risk learning.

That distinction matters.

The logic is straightforward: many AI products will not create enough value to justify production rollout. If organizations require full enterprise contracting, deep integration, and lengthy review before learning basic feasibility, they will waste time and resources on weak candidates.

A mature AI program needs a way to test ideas safely and stop them quickly when they fall short.

A Practical Example: Low-Lift Piloting for Fax-Based Referrals

Dr. Poon shared a useful operational example involving a startup that proposed using AI to summarize incoming faxed referrals for specialist scheduling.

The problem was real: staff were spending significant time interpreting inbound faxes and extracting information needed to route patients appropriately.

What makes the example valuable is not the specific use case, but the governance pattern behind it.

What Duke did well

Instead of launching a heavy enterprise deployment, Duke appears to have structured the pilot around a few practical principles:

  • keep the test low risk
  • minimize technical integration
  • establish minimum security expectations
  • address key legal and data-handling issues early
  • define the pilot narrowly
  • let operational users judge whether the outputs created meaningful benefit

That is a strong governance model for administrative AI experimentation.

Why the pilot was still a success

The vendor’s technology did not perform well enough to save time in the scheduling workflow. The organization did not proceed. But that outcome was still valuable because the team learned quickly, with limited cost and contained exposure.

For healthcare cybersecurity and compliance leaders, this example is especially relevant. It shows that governance does not have to choose between two extremes:

  1. blocking innovation entirely, or
  2. allowing uncontrolled experimentation

There is a middle path: structured, low-burden pilots with defined safeguards.

What a "Skinny" Pilot Process Can Teach the Rest of the Industry

One of the most transferable ideas from the conversation is the concept of a streamlined pilot template. The exact legal and operational design was not fully specified in the video, but the principle is clear: not every AI evaluation should trigger the same level of organizational effort as a full production implementation.

That suggests a tiered framework.

Possible governance tiers

Tier 1: Concept evaluation

  • no production integration
  • limited dataset exposure
  • narrow duration
  • strict data destruction requirements
  • basic legal and security thresholds

Tier 2: Controlled pilot

  • limited operational use
  • closer workflow observation
  • additional privacy and security review
  • success metrics defined in advance

Tier 3: Production deployment

  • full governance review
  • monitoring plan
  • owner accountability
  • incident response pathway
  • contract and data-use maturity

The video does not lay out this formal model, but it strongly supports the idea of differentiated review intensity based on risk and maturity.

That is a practical recommendation for leaders trying to balance speed with control.

Generative AI Requires New Governance Questions

Duke’s experience also reflects a broader truth now facing health systems nationwide: the arrival of generative AI changed the governance challenge.

Predictive models generally answer a narrower question: what is the likelihood of a defined event? Generative systems may summarize, draft, chat, recommend, or transform content in ways that are harder to benchmark.

This means organizations need to expand their review questions.

For predictive AI, governance often focuses on:

  • performance metrics
  • local validation
  • threshold setting
  • workflow integration
  • alert burden
  • downstream outcomes

For generative AI, governance may also need to assess:

  • hallucination risk
  • prompt sensitivity
  • output variability
  • user overreliance
  • content provenance
  • human review requirements
  • role-based access
  • acceptable use boundaries

This is especially relevant for patient communications, chart summarization, coding support, and operational chat tools.

The important insight is that AI governance should be category-specific. A single checklist for all AI will likely be too vague to be useful or too rigid to be scalable.

Although the conversation centers on governance broadly, it also surfaces the importance of legal and cybersecurity collaboration.

The fax-referral pilot example makes that clear. Rather than running every exploratory use case through exhaustive enterprise review at the outset, Duke seems to have developed threshold questions that help determine whether a vendor can participate in a low-risk pilot.

For legal, privacy, and security teams, this is a critical design principle.

Instead of asking only:

  • How do we review every tool comprehensively?

Also ask:

  • How do we classify AI use cases quickly?
  • What are the minimum conditions for a safe pilot?
  • Which contract terms are non-negotiable?
  • Which data rights or retention issues should trigger escalation?
  • What security evidence is enough at pilot stage versus production stage?

This is where AI governance becomes an enterprise resilience function, not just a digital innovation process.

In many organizations, cybersecurity teams are already overburdened. A threshold-based, reusable review structure helps preserve rigor without exhausting specialized teams on early-stage evaluations that may never progress.

What Smaller and Less-Resourced Organizations Can Apply Now

Not every provider organization has Duke’s scale, informatics depth, or internal leadership bench. But the core lessons are still highly transferable.

Smaller organizations do not need to replicate Duke’s structure exactly. They can start with a lighter version built around three principles.

1. Create one front door for AI requests

Avoid departmental side deals and fragmented adoption. Even a simple intake form is better than uncontrolled experimentation.

2. Use a multidisciplinary review circle

This can be a compact group rather than a large committee. At minimum, include:

  • clinical or operational leadership
  • IT or digital leadership
  • privacy/security review
  • legal/compliance input
  • someone responsible for measuring outcomes

3. Require post-pilot evidence

Do not move a tool into routine use solely because it appears innovative or because peer organizations are discussing it. Ask what changed, what improved, and what risks remain.

For rural systems, community hospitals, and physician enterprises, disciplined simplicity may be more sustainable than elaborate governance architecture.

A Maturity Model for AI Governance in Health Care

The discussion suggests a useful way to think about AI governance maturity, even though the video does not present it formally.

Stage 1: Awareness

The organization recognizes AI is arriving and begins informal review.

Stage 2: Advisory governance

Experts help teams think through validation, workflow, and risk, but enforcement is limited.

Stage 3: Structured review

Formal checkpoints, intake processes, and role clarity are established.

Stage 4: Scalable operations

Reviews are distributed, partially asynchronous, and designed for increasing volume.

Stage 5: Lifecycle accountability

The organization not only approves AI, but also monitors whether it remains safe, fair, and worthwhile.

Many health systems are currently between stages 2 and 4. The important point is not to jump instantly to full maturity, but to move deliberately.

Conclusion

The most important message from Duke’s experience is that AI governance in health care is not a one-time policy exercise. It is an operating capability.

Strong governance does not mean slowing innovation to a crawl. It means making innovation more disciplined, measurable, and defensible. It means recognizing that a model’s technical promise is only the beginning; the real questions involve workflow design, accountability, data handling, user behavior, and ongoing value.

For healthcare leaders, the practical path forward is clear:

  • start with the governance structures you already have
  • add AI-specific checkpoints
  • educate project teams continuously
  • build low-risk pilot pathways
  • measure outcomes after deployment
  • redesign governance for speed as demand grows
  • treat generative AI as a distinct oversight challenge

Above all, do not confuse adoption with success. In health care, the standard should be higher. AI is worth pursuing only when it demonstrably improves care, operations, or resilience without introducing unacceptable risk. Governance is how organizations make that determination with discipline rather than optimism alone.

Source: "Episode 280: Building Effective AI Governance in Health Care with Dr. Eric Poon of Duke Health" - Maynard Nexsen, YouTube, Jun 23, 2026 - https://www.youtube.com/watch?v=4Questy3mn4

Related Blog Posts