10 questions to ask
before you buy AI
Every major AI company now sells to hospitals. The demos look the same. These are the questions I ask when a health system brings me in to review an AI purchase.
What job does it do?
Name one workflow, one owner and one number that should improve, such as minutes per prior authorization. If nobody can name the job, stop.
Who reviews the output?
Every draft that touches a chart, a claim or a patient message needs a named human reviewer with time built into their day.
Can it write to the chart?
Read-only tools carry less risk. Anything that writes, orders or sends needs a stricter review and an audit trail.
Where does the data go?
Ask for the business associate agreement, the data flow and a written answer on whether your data trains any model.
How was it tested?
Ask for results on patients like yours, not only on a benchmark. Ask who did the testing and whether physicians were involved.
Who does it fail?
Ask how it performs across age, language, race and payer type. Ask what happens for patients who do not fit the training data.
What is the failure plan?
Decide before launch how staff report an error, who can switch the tool off and how fast.
What does it really cost?
Count licenses, integration, training, review time and the cost of errors, then compare with the time it saves.
Does the patient know?
Decide when patients are told that AI helped with their care, and write it in plain language.
Who is accountable?
Put one executive and one clinical lead on the hook for outcomes, and review the results every quarter.
How I help
I give board and leadership briefings on where AI is ready in your organization, where it is not, and how to roll it out with clinicians rather than around them. Sessions can be a keynote, a working session with your executive team, or both.
These questions are general guidance, not legal or compliance advice. Involve your compliance, privacy and legal teams before any purchase.
Written by Harvey Castro, MD, MBA, board-certified emergency physician. Last reviewed November 12, 2026.