RAND: Refine performance measures to increase effectiveness
In response to a request made by the National Quality Forum (NQF), RAND attempted to determine how performance measures are being used in the healthcare industry, what factors influence their use, how institutions have changed as a result of their use and how their use could be improved.
To make those determinations, the nonprofit research organization headquartered in Santa Monica, Calif., reviewed publicly available information on 70 institutions representing a variety of healthcare stakeholders and interviewed individuals associated with 30 of them. A report detailing the research’s results was made available in December.
How are performance measures used?
RAND researchers identified possible uses for performance measures: quality improvement, public reporting, payment and accreditation, certification, credentialing, etc.
Of the 70 institutions surveyed, 48 used performance measures for quality improvement purposes, 45 for public reporting purposes, 23 for payment purposes and 11 for accreditation and accreditation-like activities. Additionally, 40 institutions used performance measures for multiple purposes, but only one used performance measures for all four possible purposes.
What factors influence the use of performance measures?
Researchers wrote that data availability was the most important factor influencing the development of performance measure sets and indicated that data availability will increase as health IT products, such as EHRs, become more advanced.
“The availability of data to construct measures, obtained either from providers or from public databases, was the single most important factor cited as either facilitating or impeding the use of measures,” researchers wrote. “Having a strong data infrastructure to allow the construction of measures allows more opportunities for measurement construction and use.
“Some organizations noted the prospect of more clinically enriched data becoming available moving forward with the advent of EHRs, but added that EHRs are not sufficiently robust at this stage to generate this information for measure construction,” they continued.
Additionally, researchers noted that institutions favored performance measure sets endorsed by organizations like NQF, but that institutions were willing to use unendorsed measure sets for areas where endorsed measure sets did not exist.
What system-wide changes have occurred as a result of performance measurement use?
RAND researchers said that only a handful of the institutions surveyed have attempted to evaluate their performance measure use and even fewer have attempted return-on-investment analysis.
Despite the lack of quantitative evidence to back it up, researchers said that several interviewees attested to a variety of changes due to performance measure use.
“Interviewees anecdotally reported a range of system changes in cost, quality, patient safety and patient experience resulting from their use of measures,” they wrote. “They noted, although they did not provide actual numbers, that they had realized costs savings from initiatives to increase the use of generic prescribing and from implementation of tiered health plan products, in which tiering was based on the use of performance measures.
“On the other hand,” they continued, “a small number of interviewees questioned the impact of their measurement efforts.”
Concerns of performance measurement skeptics included, according to researchers, quality improvement reporting’s effect on consumer decision-making and a seemingly negligible effect on outcomes.
How can performance measures be improved?
RAND researchers suggested that there is not necessarily a need for more measures, at least not immediately, but that there is more of a need to refine existing measures.
“The expression of need for more measures was counterbalanced by comments regarding the need for better alignment and prioritization of existing measures to help in selection and use of measures,” they wrote.
Researchers also suggested that organizations in the performance measure endorsement business, like NQF, should conduct more outreach and education, as well as offer more tools for end-users to increase the effectiveness of performance measures.
For example, researchers wrote, “In an effort to gain a common understanding of performance measurement and the endorsement process, some suggested NQF offer stakeholder training, potentially with medical education credit.”
RAND researchers concluded their report by making a series of suggestions to aid in the creation of and use of performance measures. Some of their suggestions included building support for the use of measures, aligning measures, establishing priorities for the use of measures and creating tools to help end-users.
The RAND report is available in its entirety here.
To make those determinations, the nonprofit research organization headquartered in Santa Monica, Calif., reviewed publicly available information on 70 institutions representing a variety of healthcare stakeholders and interviewed individuals associated with 30 of them. A report detailing the research’s results was made available in December.
How are performance measures used?
RAND researchers identified possible uses for performance measures: quality improvement, public reporting, payment and accreditation, certification, credentialing, etc.
Of the 70 institutions surveyed, 48 used performance measures for quality improvement purposes, 45 for public reporting purposes, 23 for payment purposes and 11 for accreditation and accreditation-like activities. Additionally, 40 institutions used performance measures for multiple purposes, but only one used performance measures for all four possible purposes.
What factors influence the use of performance measures?
Researchers wrote that data availability was the most important factor influencing the development of performance measure sets and indicated that data availability will increase as health IT products, such as EHRs, become more advanced.
“The availability of data to construct measures, obtained either from providers or from public databases, was the single most important factor cited as either facilitating or impeding the use of measures,” researchers wrote. “Having a strong data infrastructure to allow the construction of measures allows more opportunities for measurement construction and use.
“Some organizations noted the prospect of more clinically enriched data becoming available moving forward with the advent of EHRs, but added that EHRs are not sufficiently robust at this stage to generate this information for measure construction,” they continued.
Additionally, researchers noted that institutions favored performance measure sets endorsed by organizations like NQF, but that institutions were willing to use unendorsed measure sets for areas where endorsed measure sets did not exist.
What system-wide changes have occurred as a result of performance measurement use?
RAND researchers said that only a handful of the institutions surveyed have attempted to evaluate their performance measure use and even fewer have attempted return-on-investment analysis.
Despite the lack of quantitative evidence to back it up, researchers said that several interviewees attested to a variety of changes due to performance measure use.
“Interviewees anecdotally reported a range of system changes in cost, quality, patient safety and patient experience resulting from their use of measures,” they wrote. “They noted, although they did not provide actual numbers, that they had realized costs savings from initiatives to increase the use of generic prescribing and from implementation of tiered health plan products, in which tiering was based on the use of performance measures.
“On the other hand,” they continued, “a small number of interviewees questioned the impact of their measurement efforts.”
Concerns of performance measurement skeptics included, according to researchers, quality improvement reporting’s effect on consumer decision-making and a seemingly negligible effect on outcomes.
How can performance measures be improved?
RAND researchers suggested that there is not necessarily a need for more measures, at least not immediately, but that there is more of a need to refine existing measures.
“The expression of need for more measures was counterbalanced by comments regarding the need for better alignment and prioritization of existing measures to help in selection and use of measures,” they wrote.
Researchers also suggested that organizations in the performance measure endorsement business, like NQF, should conduct more outreach and education, as well as offer more tools for end-users to increase the effectiveness of performance measures.
For example, researchers wrote, “In an effort to gain a common understanding of performance measurement and the endorsement process, some suggested NQF offer stakeholder training, potentially with medical education credit.”
RAND researchers concluded their report by making a series of suggestions to aid in the creation of and use of performance measures. Some of their suggestions included building support for the use of measures, aligning measures, establishing priorities for the use of measures and creating tools to help end-users.
The RAND report is available in its entirety here.