AI use in healthcare is moving faster than oversight. I’d boil the problem down to this: teams are using AI in charting, clinical support, and vendor tools before many health systems know what is in use, who owns it, or how to track harm.
Here’s the short version:
- Only 16% of health systems have an enterprise AI governance plan.
- AI tool use jumped 7x in 2025.
- About 66% of U.S. physicians use AI in practice.
- Only 23% of health systems have BAAs for the third-party AI tools staff are using.
- DOJ healthcare False Claims Act recoveries hit $5.7 billion in fiscal year 2025.
What does that mean for you?
- Notes made by ambient scribes can leak PHI or add errors.
- Clinical AI can drift, bias decisions, and leave weak records for review.
- Vendors can add AI into approved software without clear notice.
- Shadow AI turns into risk no one is tracking.
If I were summarizing the fix in one line, it would be this: build one intake process, one AI inventory, one owner for each tool, and review every system before and after go-live.
A quick snapshot:
| Risk area | What goes wrong first | What to put in place |
|---|---|---|
| Documentation AI | PHI exposure, rushed note approval, billing errors | Intake review, PHI limits, note checks |
| Clinical AI | Drift, weak explainability, liability | Validation, logging, override tracking |
| Vendor AI | Hidden subprocessors, weak contract terms | AIBOM, BAA review, contract limits |
| Shadow AI | Staff use outside review | Central inventory, required approval path |
So the core issue is simple: healthcare does not just have an AI adoption boom. It has a control problem.
Healthcare AI Risk Management: Key Stats & Gaps at a Glance
Healthcare AI Governance - Risks, Compliance, and Frameworks Explained
sbb-itb-535baee
Where AI risk management problems are appearing first
The first breakdowns are showing up in documentation, clinical decision support, and third-party tools.
Ambient scribing and documentation AI
Documentation is the first pressure point, and the pace is already getting ahead of review.
Ambient scribing tools that record visits and draft notes are spreading fast. That creates immediate exposure through PHI leakage and privacy risk in voice recordings and notes [4].
The day-to-day risk often appears after the note is created. In many settings, clinicians approve AI-generated notes too fast to catch mistakes.
There’s also a data reuse issue that many organizations still haven’t dealt with. Some vendor terms allow PHI reuse for model training, which can conflict with HIPAA without patient authorization [4].
The stakes get higher when AI starts shaping clinical decisions instead of just drafting notes.
Diagnostic, triage, and clinical decision-support models
Clinical decision-support tools add model-drift risk. As patient populations change or vendors update models, performance can slip in ways normal IT monitoring won’t catch [3][7].
Explainability is another gap, and it carries direct liability risk. EHR audit trails show access, but not what the AI recommended [8].
That matters in litigation. If harm happens and the clinician’s judgment can’t be separated from a biased AI recommendation, liability risk goes up. The FDA’s 2026 CDS guidance has also narrowed exemptions for tools where clinicians can’t independently review the basis for a recommendation [3].
Without validation and traceability, clinical AI turns into a black-box risk the organization can’t defend.
Even tightly scoped internal use can start to fray once outside vendors and unsanctioned tools enter the workflow.
Third-party AI vendors and shadow AI use
Managing third-party AI risk is critical, as these vendors drive a large share of healthcare’s breach exposure. More than 700 large healthcare data breaches are reported to HHS each year, and third-party vendors are consistently the primary source [6]. The risk grows when vendors depend on subcontractors and downstream API exposure, which can create liability that many BAAs do not clearly cover.
The problem is not just unsafe AI. It’s unseen AI. In 2025, about 66% of U.S. physicians actively use AI tools in their practice, yet only 23% of health systems have BAAs in place for the third-party AI solutions their staff are actually using [7]. Vendor marketing claims are not proof of compliance.
Untracked tools turn into untracked risk.
These failure points all point to the same gap: missing governance controls.
What controls most healthcare AI programs are missing
The crisis comes down to control. AI is moving faster than inventory, validation, monitoring, and accountability can keep up. Existing cybersecurity and compliance controls were built for static data movement and human-paced decisions. AI brings drift and other moving risks into the mix [2][3]. That mismatch shows up first in inventory, validation, monitoring, and accountability.
Here’s where the biggest gaps are:
| Control Area | Current Gap | Required AI-Specific Control |
|---|---|---|
| Governance | Static policies and one-time sign-offs [9] | Centralized AI inventory; named executive accountability; NIST AI RMF alignment [10][11] |
| Privacy | Standard BAAs focused on data storage; no restrictions on model training [4] | Explicit PHI training prohibitions; BAA coverage across all downstream subprocessors [4][5] |
| Vendor Management | Reliance on "HIPAA compliant" marketing badges; no visibility into downstream subcontractors [5] | AI Bill of Materials (AIBOM); data provenance checks; AI-specific contract addendums [5][4] |
| Monitoring | Periodic audits; uptime tracking only [3] | Continuous drift detection; inference-level logging; equity/subgroup monitoring [2][9] |
| Security | Perimeter defense; static DLP rules [2] | Input/output runtime guardrails; prompt injection defense; identity-linked AI audit trails [2] |
Each of these gaps points to a control failure that healthcare teams can start fixing now.
No AI inventory or intake process
70% of healthcare organizations have established AI governance committees, but only 30% maintain an enterprise-wide AI inventory [10]. That’s a big disconnect. A committee can’t manage tools it can’t see.
And the problem gets worse when vendors slip AI into tools that were already approved. That can happen without any new procurement review at all. Over 50% of healthcare organizations have no documented method for detecting when vendors embed AI into existing products [10]. So even if a team thinks it has things under control, new AI may already be inside the stack.
What’s missing is a formal intake gate. Every new AI tool should go through security, privacy, legal, compliance, and clinical review before it touches PHI. Without that step, the inventory never gets built.
Weak pre-deployment validation and vendor due diligence
A vendor saying it is "HIPAA compliant" is just that: a vendor claim, not proof [4]. The same goes for SOC 2 or HITRUST. Those are general attestations. They do not replace a deployment-specific HIPAA risk analysis [4].
Healthcare teams need a closer look at what they’re buying and how it works. That means:
- Data provenance checks
- Bias mitigation reviews
- A verified AI Bill of Materials (AIBOM) that lists model dependencies, downstream APIs, and any fourth-party services in the data flow [5]
Contracts matter too. They shouldn’t be treated like boilerplate paperwork. They need to block PHI use for model training and make sure the BAA overrides a vendor’s standard terms of service [4][5].
Limited post-deployment monitoring and accountability
Even when a pre-deployment review happens, many programs stop paying close attention after go-live. That’s where things start to drift. 38% of healthcare organizations either share AI risk responsibility across multiple groups without clear escalation paths or have not defined ownership at all [10]. If a model starts giving inconsistent results, clinicians may have no clear place to report it.
The bigger issue is what happens during each inference. In a regulated setting, it’s not enough to say the app was secure in general. Organizations need to show what happened in a specific interaction - the prompt, the response, the model version, and the user identity [2]. That level of detail matters when something goes wrong.
There’s also a practical problem with the "human-in-the-loop" idea. On paper, it sounds safe. In a busy workflow, it can fall apart fast. If a clinician is approving 20 AI outputs per minute, that’s not meaningful oversight. It’s a rubber stamp [3].
The answer is a workflow that starts before deployment and stays in place after go-live.
How healthcare leaders can build AI governance now
The gap is fixable. Healthcare leaders need governance, risk review, and monitoring before AI goes live. What comes next is a workflow that turns those gaps into repeatable controls.
Build a multidisciplinary AI governance workflow
"Effective AI Cyber Governance integrates cybersecurity principles into the assessment, design, development, deployment, and decommissioning of AI systems." [12]
AI governance can't sit with one team. It needs input from clinical, IT, security, privacy, legal, compliance, procurement, and patients. Every use case should go through a triage intake before any PHI exposure, and every tool should have one clear owner. That owner is the person on the hook when something breaks, drifts, or needs review.
Risk-based triage helps teams move faster without giving high-impact tools a free pass. Low-risk use cases don't need the same level of scrutiny as clinical decision support, but they still need a path through review.
Once ownership is clear, the next step is a standard risk review before deployment.
Apply risk assessment and data protection by design
Use one deployment checklist that covers cybersecurity, privacy, bias, patient safety, human oversight, and regulatory exposure. The proposed 2026 HIPAA Security Rule brings AI systems that handle ePHI into scope and removes "addressable" loopholes, which makes deployment-specific risk analysis mandatory [3].
Data protection controls should be built into each AI workflow from day one. That includes:
- minimum-necessary data use
- encryption at rest and in transit
- role-based access controls
- de-identification where feasible
- contract terms that define data ownership, ban PHI use for model training, and require end-of-life data destruction [5]
The FDA's revised Clinical Decision Support guidance from January 2026 narrowed exemptions for many AI tools. On top of that, states such as California (AB 3030), Texas, and Colorado now require disclosure and meaningful human oversight for clinical AI [3]. If a risk review misses those rules, it isn't complete.
After go-live, those same controls need review again because AI behavior can shift over time.
Use continuous monitoring and scalable risk operations
Pre-deployment review is the starting line, not the finish line. Set performance and safety baselines at go-live, then put a standing review schedule in place - monthly for routine tools, and more often for clinical decision support [3].
Teams should watch for drift, version changes, and override rates after go-live. Override rates matter. Frequent overrides can point to model issues. Near-zero overrides can point to weak human review. Either way, the signal matters.
Annual risk assessments should reflect the system as it exists now, not as it looked at deployment [3].
As programs grow, teams need central workflows that automate third-party review, show fourth-party exposure, and send findings to GRC stakeholders for approval. It gets much easier to manage these controls when teams fold them into one shared operating model.
| AI Risk Area | Specific Control |
|---|---|
| Model Drift | Scheduled performance monitoring and override rate tracking [3] |
| Data Misuse / PHI Training | Contractual prohibitions on using PHI to train public or vendor models [5] |
| Shadow AI | Centralized AI inventory and mandatory intake workflow [12] |
| Algorithmic Bias | Pre-deployment validation and bias review of training data [12] |
| Unauthorized Access | Role-based access controls, encryption, and minimum-necessary data use [5] |
| Vendor / Supply Chain Risk | AI-BOM and downstream vendor visibility [5] |
| Ownership Gap | Named owner and clinician escalation path for every tool [3] |
The target is a continuous, auditable governance loop.
Conclusion: Faster governance is the path to safe AI use
The risk points are clear. What’s left is speed.
As of 2026, AI is already built into most U.S. healthcare workflows, often before governance teams can get the right controls in place. It now sits inside clinical workflows, documentation systems, and vendor platforms. In plain English: the tech moved ahead of the guardrails meant to manage it.
Those risks tend to show up in four places:
- clinical decision support tools that clinicians approve without review
- ambient scribing that feeds coding and reimbursement
- vendors that add AI without notice
- shadow AI that slips past intake
These aren’t four separate issues. They’re the same control gap showing up in different parts of the workflow. And each one can create real liability, including the kind reflected in the $556 million Kaiser Permanente settlement over unsupported diagnosis coding tied to Medicare Advantage reimbursement [1].
The structure of the answer is simple, even if the work isn’t. Governance needs to be centralized, assigned, and monitored. That means a central AI inventory, a required intake gate before PHI exposure, named owners for every tool, and regular monitoring. That gap is the risk.
Organizations that put governance in place now will be able to adopt AI faster, with less exposure and more confidence in the results. Governance is what allows healthcare to scale AI safely.
FAQs
How can a health system find all AI tools already in use?
Treat discovery as a visibility problem: build an inventory of every AI application, agent, and conversational flow.
Start by mapping the AI already built into third-party tools, especially the ones that handle protected health information or support clinical workflows. Vendors add new AI features all the time, so this can’t be a one-and-done task. Use continuous monitoring and audit trails to catch behavior changes and spot unsanctioned shadow AI.
Who should own AI risk in a healthcare organization?
AI risk should be an enterprise-wide responsibility, not something one innovation team handles on its own.
For governance to work, teams need to work together across legal, compliance, IT, information security, privacy, clinical operations, and executive leadership. If one group works in a silo, things can slip through the cracks fast.
At the same time, every AI tool needs a named owner. That person should have clear accountability, written decision rights, and set escalation paths. When issues like model drift or performance failures start to affect patient safety or compliance, that kind of ownership makes it much easier to respond.
What should be reviewed before an AI tool goes live?
Before an AI tool goes live, healthcare organizations need to look at more than the sales pitch. They should review the tool’s intended use, how it lines up with the organization’s risk appetite, and whether they have lawful authority to use the data involved, especially PHI.
They also need to check the quality of the data going in, how the model was trained, what was done to reduce bias, and whether core security controls are in place, such as encryption and audit logging. On top of that, they should confirm contracts, business associate agreements, approval pathways, and human oversight for high-risk uses.
