AI fed baseline intake information can’t predict psychiatric outcomes
AI should not be used to predict the course of clinical depression if all it has to work with are new patients’ medical records of past diagnoses, medications, encounters and patient-reported outcomes.
The researchers who came to the negative conclusion did so after testing two AI models’ aptitudes for guiding depression care with only baseline information available in the EHR just prior to the start of treatment.
The investigation was conducted at Kaiser Permanente regional care operations in Colorado and Washington State and is slated for publication in the December edition of the Journal of Affective Disorders.
Biostatistician Yates Coley, PhD, psychiatrist Gregory Simon, MD, MPH, and colleagues note several reasons for desiring a predictive tool that could forecast disease progression from only baseline intake information:
First, this model could inform treatment planning for individual patients at the point of care. Second, a prediction model using baseline data (before provider behavior affects outcomes) could provide case-mix adjustment for comparing quality measures for mental health care, thereby enabling fair comparisons of providers or health systems. Third, the ability to predict depression psychotherapy outcomes with baseline data could improve comparative effectiveness research embedded in clinical practice.
With those potential benefits in mind, the team trained and validated two models (random forest and generalized linear regression models with variable selection) on data from more than 5,500 psychotherapy sessions.
All patients had prior diagnoses of anxiety (54%) or depression (65%) prior to the baseline visit.
Both the AI models showed poor predictive power in numerous performance metrics “despite using rich EHR data and advanced analytic techniques,” the authors report.
The disappointing showings included predictions of treatment responses as well as patient-reported outcomes.
“Health systems should proceed cautiously when considering prediction models for psychiatric outcomes using baseline intake information,” the authors write. “Transparent research should be conducted to evaluate performance of any model intended for clinical use.”
The study is available in full for free.
Dave P. has worked in journalism, marketing and public relations for more than 30 years, frequently concentrating on hospitals, healthcare technology and Catholic communications. He has also specialized in fundraising communications, ghostwriting for CEOs of local, national and global charities, nonprofits and foundations.