Study: Automated quality measurement systems require refining
Despite the increasing prevalence of EHRs in healthcare settings, automated care quality measurement has yet to become a widespread reality because care guidelines are not always specified, data are not standardized and much of the information required for automation is contained in unstructured formats, according to a research report published in the June issue of the American Journal of Managed Care.
To assess whether these challenges could be overcome, researchers from the Portland, Ore.-based Kaiser Permanente Northwest Center for Health Research developed an automated quality assessment method that returned mixed results when tested in two separate healthcare systems.
Using a quality assessment system developed by the RAND, a research organization headquartered in Santa Monica, Calif., researchers identified a set of 22 measures to judge asthma care quality throughout the Kaiser Permanente Northwest system and at eight federally qualified health centers (FQHC).
Researchers then investigated intake methods at Kaiser Permanente and FQHC locations, built a quality measurement system that captured encounter-level EHR data and retroactively applied the system to 13,918 Kaiser Permanente and 1,825 FQHC patients diagnosed with persistent asthma. The quality measurement system pulled data from different areas of EHRs, including coded diagnoses, problem lists, medical histories, medication lists, laboratory tests and progress notes, before directing it to a centralized data warehouse where quality measures were computed.
By conducting a manual review of patient charts and comparing it to the quality measurement system’s performance, researchers determined that measurements did well in Kaiser Permanente’s highly integrated system, achieving an average measure accuracy of 88 percent, a mean sensitivity of 77 percent and a mean specificity of 84 percent. The system did not perform as well in the FQHCs, achieving a mean accuracy of 80 percent, a mean sensitivity of 52 percent and a mean specificity of 82 percent.
“Comprehensive and routine quality assessment requires both state-of-the-art EHR implementation and an adaptable health IT platform that enables automated measurement of complex clinical practices,” concluded the report's authors, led by Brian Hazlehurst, PhD, of the Kaiser Permanente Northwest Center for Health Research.
“We designed a system to respond to these challenges and implemented it in two diverse healthcare systems to assess outpatient asthma care. These automated measures generally performed well in the health maintenance organization setting, where clinical practice is more standardized; additional refinement is needed for health systems that encompass more diversity in clinical practice, patient population and setting.”
Read the complete research report here.
To assess whether these challenges could be overcome, researchers from the Portland, Ore.-based Kaiser Permanente Northwest Center for Health Research developed an automated quality assessment method that returned mixed results when tested in two separate healthcare systems.
Using a quality assessment system developed by the RAND, a research organization headquartered in Santa Monica, Calif., researchers identified a set of 22 measures to judge asthma care quality throughout the Kaiser Permanente Northwest system and at eight federally qualified health centers (FQHC).
Researchers then investigated intake methods at Kaiser Permanente and FQHC locations, built a quality measurement system that captured encounter-level EHR data and retroactively applied the system to 13,918 Kaiser Permanente and 1,825 FQHC patients diagnosed with persistent asthma. The quality measurement system pulled data from different areas of EHRs, including coded diagnoses, problem lists, medical histories, medication lists, laboratory tests and progress notes, before directing it to a centralized data warehouse where quality measures were computed.
By conducting a manual review of patient charts and comparing it to the quality measurement system’s performance, researchers determined that measurements did well in Kaiser Permanente’s highly integrated system, achieving an average measure accuracy of 88 percent, a mean sensitivity of 77 percent and a mean specificity of 84 percent. The system did not perform as well in the FQHCs, achieving a mean accuracy of 80 percent, a mean sensitivity of 52 percent and a mean specificity of 82 percent.
“Comprehensive and routine quality assessment requires both state-of-the-art EHR implementation and an adaptable health IT platform that enables automated measurement of complex clinical practices,” concluded the report's authors, led by Brian Hazlehurst, PhD, of the Kaiser Permanente Northwest Center for Health Research.
“We designed a system to respond to these challenges and implemented it in two diverse healthcare systems to assess outpatient asthma care. These automated measures generally performed well in the health maintenance organization setting, where clinical practice is more standardized; additional refinement is needed for health systems that encompass more diversity in clinical practice, patient population and setting.”
Read the complete research report here.