Q&A: Successful patient engagement with AI requires ‘human in the loop’ ethos

HealthExec: Let’s start by defining our terms: What is the difference between artificial intelligence and automation? How can providers tell if an old product is simply being rebranded as AI?

Deutsch: The easiest way to tell the difference is to ask what happens when the system encounters an input it hasn't seen before. A rules engine, however sophisticated it may look, will do something predictable—and often visibly wrong—because it's following a decision tree someone wrote. Real machine learning or generative AI will generalize, for better or worse, which is exactly why it needs guardrails that a static automation tool doesn't.

A more practical—and necessary—step is to ask the vendor what AI technology is being used on the back end and exactly how the system works. That can also help determine whether the vendor operates under a signed BAA and where your data may be going. Many AI systems send patient data outside your controlled environment to third-party model providers, creating another potential risk point.

Is AI adoption in healthcare moving too quickly? When should providers feel confident an AI application is ready for clinical use?

No, AI is not moving too quickly when it comes to brick-and-mortar healthcare delivery. There's also a tremendous amount of state legislation being implemented to curb AI's use in healthcare, so in the real world of EHRs and patient engagement, adoption is actually moving quite slowly. Our clients and partners are also very resistant to AI, especially when it isn’t a “human in the loop” system.

I'd trust a tool for clinical use when three things are true: First, its behavior is reproducible; given the same input, it does the same thing every time—not a slightly different answer depending on how the model happened to weigh things that day. Second, there's a human decision point built into the workflow before anything affects patient care, not bolted on as an afterthought. Third, the vendor can show that the system has been tested and monitored over an extended period and isn't putting patient data at risk.

What types of tasks do you think lend themselves poorly to AI? In other words, where should humans have total control when it comes to care delivery?

Anything that changes the course of a patient's care needs a human in the loop, full stop. That includes triage decisions, treatment recommendations and care plans. The reality, however, is that patients are already going to AI for all of this on their own, with or without their provider knowing—Gemini, ChatGPT and Claude are answering most medical questions patients have day-to-day. Pediatricians, for example, are already seeing a drop in sick visits as parents consult AI first. Good or bad, it's happening.

Where I think AI is genuinely useful is in surfacing information a human would otherwise have to dig for, summarizing a patient's history so a provider isn't starting from zero, structuring data that used to sit in a PDF nobody reads and flagging things for a human to review rather than acting on its own judgment.

There's also a category of tasks where full human control matters for security reasons, not just clinical ones. Any AI agent with the ability to take autonomous action inside a system that touches patient data—sending something, changing a record or approving something—is a bigger security exposure than a tool that drafts something and waits for a person to approve it. I'd rather see AI recommend and a human execute than the reverse, and that's as much a security posture as it is a clinical one.

Your expertise is patient engagement. How do you apply artificial intelligence to those kinds of platforms and systems without removing that necessary human element that is core to the healthcare experience?

The goal isn't fewer humans in healthcare. It's making sure the humans involved are spending their time on the parts that actually need a human. AI can fill in the gaps. For example, the intake process—a good use for AI is to ask a patient about their medical history and reason for their visit, replacing a clipboard survey and the need for a provider to spend 10 minutes per patient asking questions. 

You can take that a step further by conducting the same intake process with an AI voice automation platform, where the patient is interviewed over the phone. That's especially useful for patients who don't have access to a computer or smartphone or who aren't comfortable completing intake forms online.

I believe we're going to see the patient engagement interface shift from the traditional patient portal or website form to conversational AI and voice automation. That will inevitably make healthcare technology more inclusive.

Finally, an important part of the patient engagement puzzle is payments—how can hospitals and health systems make that painless and not leave the people they care for with a sense of sticker shock?

Patients need to know what they're likely to owe before or at the time of the visit, which means eligibility and cost estimation need to happen upfront rather than during a back-office billing cycle. That's more of a concern in higher-acuity or specialty care, though.

In terms of making payments simpler, patients are more than ready—it’s providers that need to catch up. Patients are already used to frictionless experiences everywhere else in their lives. They tap to pay for coffee, get a text with a link when a package ships and know exactly what a Lyft will cost before getting in the car. Healthcare is one of the few places where we still ask people to wait weeks for a paper statement, and it’s a technology problem. 

The solution is largely about digitizing the entire process: completing intake online, paying co-pays during intake, receiving invoices and statements through the patient portal, and letting patients ask questions about a bill online. It's not particularly complicated. Healthcare just needs to take the steps to eliminate the paper and speed everything up.