Machine learning models about as trustworthy as conventional means for risk prediction
When it comes to predicting an individual’s risk of suffering a heart attack or stroke over the next 10 years, machine learning models are no more consistent—and so no more reliable as guides to care pathways—than traditional statistical techniques.
The researchers making the conclusion do so after analyzing medical records from 3.6 million patients in the U.K. who were tracked by registry over a 20-year period ending in 2018.
The team looked at 19 different prediction methods, 12 of which represented some type of machine learning.
The disappointing showings were at their worst when a model in either category assumed a patient was free of any cardiovascular disease when their records were inadvertently “censored,” meaning their clinical information stopped getting updated for whatever reason.
Logistic models and commonly used machine learning models “should not be directly applied to the prediction of long-term risks without considering censoring,” the authors write in their study, published this month in The BMJ. “Survival models that consider censoring and that are explainable … are preferable.”
An example of the all-over-the-map risk scoring: A patient with a 9.5% to 10.5% risk for a cardiovascular event going by a conventional risk calculator had a 2.9% to 9.2% risk in a random forest model and a 2.4% to 7.2% risk in a neural network.
All 19 models did well enough predicting disease at the population level, but that doesn’t mean much in specific clinical settings.
“We found that the predictions of cardiovascular disease risks for individual patients varied widely between and within different types of machine learning and statistical models, especially in patients with higher risks (when using similar predictors),” the authors write. “Logistic models and the machine learning models that ignored censoring substantially underestimated risk of cardiovascular disease.”
The researchers, whose number included investigators in China and the Netherlands as well as the U.K., used cardiovascular disease for this present analysis but suggest the findings may well apply to other serious health risks.
The study is available in full for free.
Dave P. has worked in journalism, marketing and public relations for more than 30 years, frequently concentrating on hospitals, healthcare technology and Catholic communications. He has also specialized in fundraising communications, ghostwriting for CEOs of local, national and global charities, nonprofits and foundations.