JAMIA: USCD researchers propose adaptive approach to enhance CDS
Researchers from the University of California at San Diego (UCSD) proposed a data-driven approach to utilize confidence intervals (CIs) to select the most “appropriate” model from a pool of candidates to assess the individual patient's clinical condition, in hopes of improving personalized risk estimation for clinical decision support.
“Competing tools are available online to assess the risk of developing certain conditions of interest, such as cardiovascular disease,” wrote Xiaoqian Jiang, MD, division of biomedical informatics, UCSD, and colleagues. “While predictive models have been developed and validated on data from cohort studies, little attention has been paid to ensure the reliability of such predictions for individuals, which is critical for care decisions.”
Published online April 4 in the Journal of Medical Informatics Association, the patient-driven adaptive approach was compared with other strategies: the BEST model (the ideal model, which can only be achieved by access to data or knowledge of which population is most similar to the individual), CROSS model and RANDOM model selection.
When evaluated on clinical datasets, the approach significantly outperformed the CROSS model selection strategy in terms of discrimination and calibration. The method outperformed the RANDOM model selection strategy in terms of discrimination, but the improvement did not achieve significance for calibration.
“The CI may not always offer enough information to rank the reliability of predictions, and this evaluation was done using aggregation,” the authors noted. “If a particular individual is very different from those represented in a training set of existing models, the CI may be somewhat misleading.”
“In the future, we plan to use larger datasets that are increasingly being collected at healthcare institutions for predictive model building and validation,” the authors wrote. “Indeed, the problem of assessing the best result for the particular individual at hand is still an open question, as the individual gold standard for the prediction is not available.
“[W]e would like to work towards the development of better proxies for the gold standard than the ones currently available, investigate data-driven model selection for models constructed using larger datasets across multiple sites, and extend our framework to include kernel methods,” they stated, concluding that the adaptive approach has the potential to offer more reliable predictions than those offered by other heuristics for disease risk estimation of individual patients.
“Competing tools are available online to assess the risk of developing certain conditions of interest, such as cardiovascular disease,” wrote Xiaoqian Jiang, MD, division of biomedical informatics, UCSD, and colleagues. “While predictive models have been developed and validated on data from cohort studies, little attention has been paid to ensure the reliability of such predictions for individuals, which is critical for care decisions.”
Published online April 4 in the Journal of Medical Informatics Association, the patient-driven adaptive approach was compared with other strategies: the BEST model (the ideal model, which can only be achieved by access to data or knowledge of which population is most similar to the individual), CROSS model and RANDOM model selection.
When evaluated on clinical datasets, the approach significantly outperformed the CROSS model selection strategy in terms of discrimination and calibration. The method outperformed the RANDOM model selection strategy in terms of discrimination, but the improvement did not achieve significance for calibration.
“The CI may not always offer enough information to rank the reliability of predictions, and this evaluation was done using aggregation,” the authors noted. “If a particular individual is very different from those represented in a training set of existing models, the CI may be somewhat misleading.”
“In the future, we plan to use larger datasets that are increasingly being collected at healthcare institutions for predictive model building and validation,” the authors wrote. “Indeed, the problem of assessing the best result for the particular individual at hand is still an open question, as the individual gold standard for the prediction is not available.
“[W]e would like to work towards the development of better proxies for the gold standard than the ones currently available, investigate data-driven model selection for models constructed using larger datasets across multiple sites, and extend our framework to include kernel methods,” they stated, concluding that the adaptive approach has the potential to offer more reliable predictions than those offered by other heuristics for disease risk estimation of individual patients.