AI adoption struggles | Partner voice | Dubious LLM advice, Cancer vaccine, OpenAI’s healthcare push, more

News and views you ought to know about: 

Study: LLM chatbots are not dieticians. Be careful trusting ChatGPT with even simple medical questions. A new survey found that the information AI doles out may be outdated, misleading or generic—even for something as simple as a healthy diet. This isn’t shocking to a doctor who spoke with VeryWellHealth who said she’d be “extremely reluctant to tell a patient to ever do something based on ChatGPT.” 

That comment comes from Ainsley MacLean, MD, a health AI consultant formerly with the MidAtlantic Kaiser Permanente Medical Group. For context, she was commenting on the results of a recent study published in Nutrients that found multiple chatbots—including Gemini, Microsoft Copilot and ChatGPT—are not good advisors on weight loss, and their answers could be misleading. The study found the LLMs can recommend healthy meal options, but they’re unable to adequately balance proteins, fats and carbohydrates—the macronutrient ratios necessary to improve healthy eating, especially when weighed against the dietary needs of an individual patient.

MacLean added that it’s necessary to question any LLMs’ results and look into when its pullable data set ends in order to improve accuracy. This includes asking AI to double-check that it’s correct. 

At the end of the day, it may be best to not rely on them at all. At least for now. 

University of Oxford turns to supercomputer for help with cancer vaccines. One of the most powerful AI-onboarded supercomputers in the U.K. is being loaned out to scientists at the University of Oxford’s Nuffield Department of Medicine as they work to develop life-saving cancer vaccines. The computer, known as "Dawn," will be tasked with analyzing tens of thousands of cancer patient records, looking to find patterns that could unlock the key to new treatments. 

In an interview with the BBC, the team said Dawn will allow them to speed up a process that manually may be impossible, given the complexity of cancer. The scientists will have access to Dawn for 10,000 hours. The supercomputer is housed at the University of Cambridge, where access is facilitated by the U.K. government and the university, based on need. 

Some facts:

• Dawn was unveiled in 2023, but heavily improved in 2024.
   
• It’s available to use for commercial projects as well as research initiatives. 
   
• It’s been used to research fusion energy and tackle the effects of climate change. 
   
• Intel and Dell had a hand in Dawn’s development. 
   
• The supercomputer is proficient at deep learning tasks. 

Generative AI company earns $50M Series B seed funding. As scheduling moves online, fewer people are answering phones at medical clinics. However, front desk staff are often left balancing phone calls and other administrative tasks, resulting in long wait times for patients who may call in with a medical question. Assort Health, an AI startup, is hoping its tech can solve the problem—and investors are betting it can. In announcing $50 million in Series B funding this week, the company is now said to be valued at $750 million, according to AutoGPT. 

Assort Health deploys voice agents to support calls with generative AI, automating tasks such as scheduling without the need for a human to pick up the phone. The kicker is that Assort’s offering is healthcare specific, meaning its technology is designed with a provider group or hospital in mind—and it adapts to real-world feedback, based on the needs of a specific office. 

Its list of clients includes dozens of specialty care clinics, including large groups and small practices alike.

OpenAI taps leadership for healthcare expansion. OpenAI, the developer of ChatGPT, has made a couple of big hires: Nate Gross, the cofounder of Doximity, and Ashley Alexander, former co-head of product at Instagram. According to Business Insider, both are taking leadership roles in the LLM developer’s healthcare business. For now, OpenAI has been mostly deploying its technology in a support capacity, mainly automating common administrative tasks at hospitals. But, founder Sam Altman has made his ambitions clear, and he wants to see GPT-5 being used more often in research and diagnostics. 

No official announcement about the future has been made, but it’s safe to say OpenAI is building a team for whatever comes next.

Some facts:

• Doximity GPT, fueled by ChatGPT, is a HIPAA-compliant service that, among other things, acts as an AI scribe. Gross helped launch the spinoff in 2023.
   
• Gross also founded Rock Health, which funds various healthcare startups. 
   
• Alexander’s background is in algorithm-driven content marketing. She worked with Meta and Instagram for 12 years. 
   
• OpenAI previously hired Karan Singal, a former Google employee who worked on its healthcare offerings. 

From AIin.Healthcare’s sibling outlets:
 

• Health Imaging: Chatbots offer patients more appropriate MRI explanations than experts, study shows 
   
• Cardiovascular Business: Health systems use AI to target patients who may have elevated Lp(a) levels 

The ambient AI playbook: Lessons from two leading health systems

At the recent CompassionIT Summit, leaders from Akron Children’s Hospital and Denver Health shared powerful lessons from rolling out ambient documentation to over 1,500 clinicians. Their biggest takeaway? Stories, not stats, drive adoption. Whether it was a heartfelt testimonial that swayed an entire department or a 60-second Nabla demo that eliminated training anxiety, the common thread was simplicity, authenticity, and clinician-centered design. Read more about the way these health systems are navigating ambient AI implementation: https://dhinsights.org/news/the-ambient-ai-playbook-lessons-from-two-leading-health-systems

The adoption of AI in healthcare settings is accelerating rapidly, and a new paper published in Artificial Intelligence in Medicine argues that clinicians may not be able to keep pace with these changes. As AI technologies are deployed in everything from diagnostics to administrative support, the study's authors express concern that educational resources are inadequate for keeping clinicians up to speed.

According to the perspective titled Unprepared and Overwhelmed: A Case for Clinician-Focused AI Education, many of the available guides lack depth and are not aligned with the realities of medical settings, such as clinics and hospitals. As it stands, educational tools are loaded with technical concepts that clinicians may be unfamiliar with, as these programs are not written or designed with real healthcare staff in mind.

The researchers, led by Nadia Siddiqui, MD, from the University of Washington School of Medicine, were inspired to examine these educational programs following the 2020 launch of the Epic Sepsis Model. The model was rapidly deployed in hundreds of hospitals nationwide—after being internally validated in only three settings to improve sepsis detection.

The system was later found to perform poorly when externally validated by informatics physicians, highlighting the need for clinicians to be viewed as stakeholders in the AI adoption process—since they are ultimately the ones responsible for ensuring these systems function as intended.

Siddiqui, et al., encourage developers to take this lesson to heart in the development of any AI system, including by ensuring that instructional materials are tailored to the average doctor or nurse.

Here are three of their key pieces of advice:

1. More case-based learning catered to specialties.

The authors recommend that educational guides be tailored to specific specialties and backed by case study examples that make the uses of AI more relatable. They note that medical schools already operate this way—applying real-world grounding to concepts through hands-on education, as opposed to purely lecture-driven teaching.

“While case-based learning may have limitations in AI education, such as a need for variation in cases and a need for flexible design, guides should include real-world AI scenarios to increase comprehension and applicability. Currently, the vast majority of guides focus on explaining AI concepts to clinicians with brief examples, as opposed to clinical cases.”

2. Informatics physicians must lead AI education.

Informatics physicians are skilled at integrating technology into clinical workflows, bridging the gap between AI adoption and patient care. Any teaching initiative should ensure their direct involvement to maximize effectiveness.

“Leveraging the expertise of clinical informaticians secures the practicality and alignment of AI/ML training in medical education with clinical decision-making, ultimately improving patient outcomes, health system efficiency and fostering responsible implementation. Just as medical education relies on cardiologists to teach about the heart and pulmonologists for the lungs, we must rely on clinical informaticians to teach about AI in healthcare.”

3. Education needs to extend beyond clinicians.

Clinicians aren’t the only stakeholders in AI adoption. Hospital administrators, IT experts and regulators will also need to navigate these systems, and any educational effort must include them. The authors suggest that educational materials emphasize how AI will integrate into existing workflows and address concerns such as data governance and liability.

Aligning expectations for all stakeholders may begin by allowing students to familiarize themselves with these tools in the classroom.

“Moreover, it should be an initiative that ultimately finds its way into the classrooms of medical education and continuing medical education with specialty-specific AI education approaches. In recent decades, medical education has integrated clinically relevant innovations, such as advances in genetics and biostatistics, equipping students with essential skills and competencies for modern practice. Similarly, education on AI tools should be integrated into curricula, preparing future physicians for their growing role in patient care.”

The full paper is available here.
 

• In other research news: 
   
  • University of California at Riverside: UC Riverside pioneers way to remove private data from AI models 
     
  • University of Colorado at Boulder: New AI tool identifies 1,000 ‘questionable’ scientific journals