What Procurement Teams Need to Know About Third-Party AI Risk

If a vendor uses AI and touches PHI, I treat it as more than a standard vendor review. A SOC 2 report, a HIPAA statement, and a signed BAA do not answer the main AI questions: Does the vendor train on our data? Can the tool produce wrong outputs that affect care or billing? Who else handles the data behind the scenes?

Menlo Ventures says 22% of healthcare organizations now use domain-specific AI tools. That means procurement teams are seeing AI in more contracts, renewals, and product updates. In many cases, I may spot the risk before clinical or IT teams do.

Here’s the short version of what matters most:

• Check PHI use closely. I want clear terms on inputs, outputs, logs, retention, deletion, backups, and subprocessor access.
• Don’t rely on a BAA alone. I also look for limits on model training and “service improvement” language.
• Review model risk. Hallucinations, bias, and opaque outputs can affect notes, coding, patient messages, and care workflows.
• Map fourth parties. The vendor on the contract may rely on cloud models, subprocessors, offshore support, or embedded AI tools.
• Require notice before change. New AI features, model swaps, and data-use changes should trigger a new review.
• Use cross-team review. Legal, privacy, security, IT, and clinical leaders each see a different part of the risk.

A simple way to think about it: AI vendor risk comes down to three things - data use, model behavior, and supply-chain visibility. If I can’t get straight answers in those areas, I send the deal for deeper review before purchase or renewal.

Third-Party AI Risk: Diligence, Contracts & Exit Controls | Module 5.3

Third-Party AI Risks Procurement Teams Must Evaluate

AI vendors bring risks that standard vendor reviews often miss: PHI use, model behavior, and hidden supply-chain dependencies. Before a team signs or renews a contract, procurement needs to check all three.

The next issue isn't whether a vendor uses AI. It's what kind of risk that AI adds.

Data Exposure, PHI Handling, and Business Associate Risk

If an AI system takes in protected health information, the vendor may become a Business Associate under HIPAA and may need a valid BAA. But a BAA by itself does not cover AI-specific risk.

Most standard BAAs weren't written for AI. They often skip key questions, like whether the vendor can use PHI to train or fine-tune models, or what happens when data is shared with subprocessors. A vendor can say it's "HIPAA compliant", but if it can't back that up with security controls, governance, and model testing, that's a red flag ^[1].

HIPAA liability can also reach the covered entity. Under HIPAA, a covered entity can be held liable for a Business Associate's violation if they "knew, or by exercising reasonable diligence, should have known" of a pattern of non-compliance ^[1]. That changes the job for procurement. It isn't enough to confirm that a BAA exists. Teams need to press on data use, data sharing, and model training terms.

That makes data-use language the first contract point procurement should verify.

Model Opacity, Hallucinations, Bias, and Workflow Harm

AI failures can look normal right up until they hit the workflow. Bias and hallucinations can affect documentation, coding, and clinical decisions.

Hallucinated clinical documentation may appear valid at first glance. The problem often shows up later, when the error reaches downstream users. That's why model-level risk is hard to catch in a standard review.

Procurement teams need to check for three model risks in particular:

• Opacity
• Hallucinations
• Bias

These issues can disrupt workflow, documentation, and coding. Vendors should be able to provide QA/VV testing records that show how models were tested and how performance is tracked over time ^[2]. If they can't, that's a gap worth flagging before the contract is signed.

Those controls should be reviewed before clinical or operational use, not after deployment.

Fourth-Party, Subprocessor, and Embedded AI Risk

The vendor on the contract is often not the only party handling your data. Many AI products sit on layered supply chains that include subcontractors, offshore developers, and open-source components. As HSCC co-leads Ed Gaudet and Samantha Jacques noted:

"Healthcare organizations often lack visibility into the full scope of the AI components incorporated into third-party products and services, which are often sourced through layered supply chains." ^[1]

Those hidden dependencies create more points of failure and make standard procurement reviews less effective ^[1]. Vendors may also add or change AI features after signature, which can change how PHI is handled without procurement getting notice ^[1]. Any new AI feature, subprocessor, or workflow change should trigger a re-review.

In plain English, the signed vendor may not be the full story. Procurement needs line of sight into who touches the data, what tools sit underneath the product, and what can change later.

These risks shape the questions procurement should ask before approval, renewal, or expansion.

What to Review in AI Vendor Due Diligence Before Purchase or Renewal

Vendor due diligence helps you check data use, model behavior, security controls, and contract terms before approval or renewal. The goal is simple: decide whether a vendor can move forward, needs fixes first, or should be sent to legal, compliance, or security for deeper review.

Review Data Handling, Retention, Training Use, and Hosting Terms

Start by mapping every type of data the tool touches: PHI, operational data, de-identified data, and metadata. Then confirm the BAA clearly covers AI inputs, outputs, logs, and subprocessors.^{[3][6][7][11]} If those terms are missing or vague, you may not have a clean answer on whether the vendor can receive PHI at all.

Next, ask how long the vendor keeps PHI, logs, training data, and backups - and how deletion works for each one.^[4][7][8] A vendor saying “we delete data when no longer needed” sounds fine on the surface, but it doesn’t tell you much. You need actual retention periods and actual deletion methods.

Then verify where data is stored and processed. Ask the vendor to list every country or cloud region where your data - including logs and backups - may be stored, processed, or accessed by support staff. Cross-border processing can add HIPAA, state privacy, and contract risk, so this part matters more than many teams think.

You should also ask whether the vendor trains or fine-tunes models on customer data. Strong terms should ban the use of customer PHI for generalized model training unless your organization gives direct approval.^[7][11] If the contract gives the vendor a broad right to use data for “service improvement” or AI development, that’s a red flag.

The table below shows the difference between strong and weak terms in the areas that matter most for AI data handling:

Once you know how data is handled, move to model behavior and update controls.

Assess Model Transparency, Validation, and Security Controls

First, identify whether the tool runs on a third-party LLM, a proprietary model, or a hybrid setup. That tells you who owns the model layer, who controls changes, and where fourth-party risk may sit.^[3]

Vendors should also give you documents showing how the model was tested before launch and how performance is tracked over time. In healthcare, generic benchmark claims aren’t enough. You want validation in clinical or operational settings. Ask for pre-deployment test metrics, bias analyses across patient demographics, and proof that the vendor watches for drift or error trends after launch.

You should also require input and output filtering, interaction logging, and advance notice before model updates or new AI features go live.^[6] If the model can change without notice, your risk profile can change without warning too.

After that review, make sure the contract ties those controls down instead of leaving them as sales-call promises.

Verify Regulatory and Contract Readiness

Before approving any AI-enabled vendor, confirm that the BAA covers the AI features themselves, not just the base software platform. A BAA drafted before AI functions were added may say little - or nothing - about how PHI moves through model inputs, outputs, or logs.^[7][11]

Review the permitted-use language closely. It should spell out exactly what the vendor can do with PHI and block secondary uses, including model training, unless your organization has approved them.^[3][13] This is one of those areas where a single broad clause can undercut a lot of other contract language.

Security addenda should also name the minimum controls the vendor must meet, such as encryption, access management, incident response timelines, and logging. Those terms should line up with your own internal security rules. Breach notification language should include a specific window - often 24–72 hours for initial notice - with fuller incident reporting afterward as more facts are confirmed.^[10]

Any tool that influences diagnosis or treatment should be flagged for regulatory review before approval.^[5][9][12] The FDA has been updating its guidance on clinical decision support software, so procurement teams should route these tools for review before signature. If the regulatory status is unclear, send it to legal and compliance before the deal is signed.

sbb-itb-535baee

How to Build AI Risk Controls Into Procurement, Governance, and Contracts

Once due diligence flags AI risk, procurement needs controls that stay in place through renewal and product change. AI oversight can't stop at signature. Models change, features change, and data use can shift after the deal is done. The aim is simple: keep AI controls active across intake, review, contracting, and renewal.

Add AI Intake and Risk Tiering to the Vendor Lifecycle

Start with a short intake process that turns AI risk into a plain approve-or-escalate call. Every new vendor and every renewal should capture the AI use case, the workflow it touches, data sensitivity, the level of automation, and the effect on day-to-day work. From there, the team can assign a risk tier.

A tool that doesn't process PHI and only gives advisory outputs is low risk. A tool that processes PHI and shapes clinical decisions or takes autonomous actions is high risk. Low-risk tools can move ahead with less friction. High-risk tools should go to cross-functional review before approval. That split helps teams move faster on lower-stakes tools while giving closer review to tools with more at stake.^[14][20]

Renewals, major version upgrades, new AI modules, and expanded data use should all reopen intake. If a vendor adds a new AI module, updates its model, or changes how data is used, that should trigger a new review.^[17][19]

Use Cross-Functional Review to Reduce Blind Spots

Each team covers a different part of the risk. Security looks at controls. Privacy looks at PHI. Legal handles contract terms. Clinical leaders look at workflow safety. Procurement keeps vendor follow-up on track and helps the process keep moving. IT reviews integration needs and day-to-day resilience. Clinical and operational leaders also need to test whether the tool's outputs fit safely into actual workflows - and what staff should do when AI guidance clashes with human judgment.^{[14][16][17][20]}

One thing helps a lot here: assign each risk to a named owner so the review doesn't get stuck in procurement.

Use Contracts as an Enforceable AI Control

The contract should do more than sit in a folder. It should force a new review when the vendor changes the product. AI-specific clauses need to work as operational triggers, not just legal boilerplate.^{[15][18][19][21]}

Notice requirements and subprocessor disclosure should act as procurement stop points. Contracts should require 60–90 days' advance written notice before a vendor turns on new AI features, switches to a different underlying model, expands training use, or adds AI subprocessors. That notice should come with updated documentation and a risk summary so your team can review the change before it touches your environment. Vendors should also have to keep a current subprocessor list and notify you before adding new ones.^[18][19][21]

When a risk shows up - an incident, a bias issue, or a performance problem - the contract should spell out remediation timelines and escalation paths. For tools that affect patient care or billing, require the vendor to provide updated validation data and performance metrics whenever major model changes happen. That way, the contract becomes a live governance control. It supports transparency and gives your team enforceable remedies when the product changes.^[8][15][21]

Practical Tools Procurement Teams Can Use to Strengthen AI Vendor Decisions

An AI Risk Assessment Checklist for Healthcare Procurement

Use a standard checklist to make AI vendor review repeatable for both new deals and renewals.

The simplest way to do that is to turn your due diligence questions into a shared intake checklist. That gives procurement, legal, privacy, security, and clinical teams one common starting point instead of reinventing the process each time.

The evidence should match the vendor. A clinical documentation tool, a staff chatbot, and an analytics platform may all use AI, but they don't pose the same level or type of risk.

How Censinet Supports Third-Party AI Risk Assessment at Scale

Once the checklist is in place, teams need one system to track evidence, decisions, and renewal follow-up.

Censinet RiskOps™ gives healthcare organizations a central record for third-party relationships, including AI-enabled vendors. Teams can tag and sort vendors by AI use case, which makes it easier to spot AI exposure across clinical, operational, and administrative areas. It also helps stop untracked AI tools from slipping past procurement. Vendor questionnaires, evidence documents, BAAs, validation reports, and subprocessor lists live in one place, so teams can reuse the same materials across departments and renewals instead of sending the same request again and again.

Censinet AI™ adds an AI-assisted workflow on top of that base. It can draft AI-specific questionnaires, summarize responses, flag gaps, and send issues to the right reviewers - privacy, security, clinical leadership, or legal - based on the content and risk level. Human reviewers still make the final call on risk acceptance and escalation. Governance dashboards give executives and oversight committees a clear view of AI vendor status, open risks, and compliance alignment across the vendor portfolio.

Conclusion: Better AI Purchasing Decisions Start With Better Questions

Better AI purchasing decisions start with standard questions, clear evidence, and contract terms that can be enforced. When these checks are built into intake, due diligence, contracting, and renewal, AI risk management becomes a repeatable, steady part of how an organization buys and governs technology.

FAQs

When should AI vendors get deeper review?

Use a tiered, risk-based approach. Start by registering every AI tool early so you can build a risk register and give the governance committee a clear view of what needs attention first.

AI tools that support clinical decisions or affect care delivery need a full review. Administrative tools can go through a lighter assessment path, based on how much they affect patient safety and day-to-day reliability.

What contract terms matter most for third-party AI risk?

Procurement teams need to look past generic software language and ask for AI-specific contract terms they can enforce.

That includes:

• Business Associate Agreements that bar vendor model training on patient data without written consent
• Advance notice of material model updates, along with rollback rights if performance drops
• Audit rights, plus liability and indemnification for algorithmic errors or hallucinations

This is where a lot of teams get tripped up. A vendor may say all the right things in a demo, but if those protections aren’t written into the agreement, you may have little recourse when the model changes, makes a bad call, or uses data in ways you didn’t approve.

How should procurement handle AI changes after signing?

Treat AI management as an ongoing post-contract process, not a one-and-done approval.

That means keeping a risk register for deployed AI tools so your team can track:

• performance
• data provenance
• intended use

Your contracts should also require advance notice of material model updates. Why? Because a model that worked well last month can drift after an update. If performance drops, the organization should have the right to roll the tool back or suspend its use.

It also helps to run continuous monitoring and periodic audits. Those checks help confirm that security controls, data handling, and subcontractor use still meet compliance requirements.

Related Blog Posts

• Inside the Webinar Series Addressing Healthcare’s Most Pressing AI Risks
• The New Front Line in Healthcare Cybersecurity Is AI Governance
• Why Recent Cyber Events Are Reshaping the AI Risk Conversation in Healthcare
• Why Traditional Third-Party Risk Management Fails in the Age of AI