JAMIA: High alert overrides, low adherence hinder CDS success
 |
“Recent reports of suboptimal alert system design and implementation highlight the need for better evaluation to inform future designs,” wrote Allison B. McCoy, PhD, department of biomedical informatics at Vanderbilt University School of Medicine in Nashville, Tenn., and colleagues. The authors sought to present a comprehensive framework for evaluating the clinical appropriateness of synchronous, interruptive medication safety alerts.
Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI).
Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate.
Reviewers evaluated 487 alerts. “Inter-rater reliability for alert appropriateness was moderate; reviewers agreed for 85 percent of alerts,” the researchers found. “After resolving disagreements and reaching consensus, the reviewers selected 391 alerts as appropriate to display; the false positive alert rate was 20 percent.”
According to researchers, providers initially overrode 400 (82 percent) displayed alerts, although 228 (47 percent) of the alerted medications were modified or discontinued within 24 hours. “For medications modified or discontinued before patient discharge, the median time for response was 13 hours,” the authors wrote. “Inter-rater reliability for response appropriateness was fair; reviewers agreed to 78 percent of the alerts. After resolving disagreements and reaching consensus, reviewers adjudicated provider responses to alerts as inappropriate for 82 of 487 alerts (17 percent).”
The new evaluation model offers an effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial, the authors concluded. “Further utilization of such CDS evaluation frameworks can lead to more standardized, comparable evaluations on a widespread basis. Such frameworks could enhance the field's ability to evaluate CDS performance and impact on clinical care delivery and provide feedback to developers about optimizing CDS implementations that improve patient safety.”