JAMA: NLP strikes admin advantages
 | |
| Harvey J. Murff, MD, MPH, from the Institute for Medicine and Public Health at Vanderbilt Epidemiology Center in Nashville, Tenn. |
Harvey J. Murff, MD, MPH, of the Veterans Affairs Medical Center and Vanderbilt University, Nashville, Tenn., and colleagues conducted a study to evaluate a language processing-based approach to identify postoperative complications within a multi-hospital healthcare network using the same EMR.
“Patient safety has been particularly challenging because it relies a lot on provider incident reports where someone has to take the time to fill out a report which tends to under represent these issues,” said Murff in an exclusive interview with CMIO. “We were interested in surgical complications because it represents modifiable adverse events so some of these complications can be prevented.”
The study included 2,974 patients (median age, 64.5 years; 95 percent men) undergoing inpatient surgical procedures at six Veterans Health Administration (VHA) medical centers from 1999 to 2006. Among the outcomes measured were postoperative occurrences of acute renal failure requiring dialysis, deep vein thrombosis, pulmonary embolism, sepsis, pneumonia or heart attack identified through medical record review as part of the VA Surgical Quality Improvement Program. The researchers determined the sensitivity and specificity of the NLP approach to identify these complications and compared its performance with patient safety indicators that use discharge coding information.
“In general, for most of these cases using a free-text approach, you could find these post-surgery complications at a much higher rate,” said Murff. “You’re able to identify more than you could using the administrative codes.”
Within each analytic sample the percentage of postoperative acute renal failure requiring dialysis was 2 percent; for pulmonary embolism, 0.7 percent; for deep vein thrombosis, 7 percent; for sepsis, 16 percent; for pneumonia, 16 percent, and for heart attack, 2 percent.
The researchers found that in general, using a NLP-based approach had higher sensitivities and lower specificities than did the patient safety indicator. “The increase in sensitivity of the natural language processing-based approach compared with the patient safety indicator was more than 2-fold for acute renal failure and sepsis and over 12-fold for pneumonia. Specificities were 4 percent to 7 percent higher with the patient safety indicator method than the natural language processing approach,” the authors wrote.
“NLP correctly identified 82 percent of acute renal failure cases compared with 38 percent for patient safety indicators. Similar results were obtained for venous thromboembolism (59 percent vs. 46 percent), pneumonia (64 percent vs. 5 percent), sepsis (89 percent vs. 34 percent), and postoperative myocardial infarction (91 percent vs. 89 percent). Both NLP and patient safety indicators were highly specific for these diagnoses.”
The authors suggested that a NLP-based approach offers several advantages over administrative-code based strategies to identify healthcare quality concerns. “First is the flexibility of the approach to meet the individual institutional needs. Once documents have been processed, different approaches and query strategies to identify a specific outcome can be implemented at a relatively low programming effort using standard database query applications.
Second, as opposed to administrative codes, search strategies using daily progress notes, microbiology reports, or imaging reports could be monitored on a prospective basis. This approach could potentially identify complications while a patient is still in the hospital, which could facilitate real-time quality assurance processes. “The most important advantage over administrative codes is that it represents a clinical outcome. …Administrative codes are [typically] designed to identify things an institution might want to bill for,” said Murff. “As far as NLP context, you’re in line for identifying what the ultimate goals are [like] quality or safety issues. It’s scalable to outpatient and inpatient settings as well.”
Finally, in systems with highly integrated EMRs, prospective surveillance could be extended to the outpatient setting for individuals remaining with the healthcare system, the authors concluded.
Ashish K. Jha, MD, MPH, of the Harvard School of Public Health, Boston, wrote in an accompanying editorial that “although the promise of natural language process is substantial, its benefits will not be realized without considerable new investment in research and development.
“Murff and colleagues focused on one specific application of identifying adverse events after surgery. Dozens of permutations and combinations of syntax were tested and customized to identify the optimal strategy for finding complications in an EHR,” Jha wrote. “To realize the benefits of NLP, this kind of research will need further development not only to find better algorithms but also to investigate EHR analysis for disciplines other than surgery and optimize automated EHR searches for different types of clinicians. Although there are private-sector companies capitalizing on the benefits of NLP to help clinicians and organizations improve care delivery, the federal government can play a helpful role by funding the basic research needed to launch this field forward.”