Despite CMS efforts to tout its new Medicare star ratings system as a transparency move and a consumer-benefiting action, not everybody is on board yet.

The agency rolled out the Overall Hospital Quality Star Ratings system as part of its Hospital Compare website in late July, after a trial period last summer and a public comment period. The program averages 64 of the 100 measurements already enumerated on the Hospital Compare site and combines them into a single five-star rating for more than 4,000 hospitals.

“I applaud CMS for getting out of ahead of it, but I just have that cautionary side of me that says…I’ve seen people in health grades be a five star one year and one star the next. I’m a little [cautious] until I see multiple years of this data,” said Nuance Healthcare Solutions CMO Anthony Oliva, MD.

CMS says the system is meant to make it easier for Medicare patients to pick the best hospital—a one-stop shop for comparing hospitals on factors such as health outcomes for specific conditions, patient satisfaction, timeliness of care readmission rates and others.

The ratings will provide a framework “to help millions of patients and their families learn about the quality of hospitals, compare facilities in their area side-by-side, and ask important questions about care quality when visiting a hospital or other health care provider,” Kate Goodrich, MD, MHS, CMS director for clinical standards and quality, wrote in a blog post about the system’s debut.

Goodrich wrote that CMS believes in the rating system because of the input gathered during the public comment period and two separate phone calls with nearly all of the ranked providers.

But some healthcare administrators are not ready to give the system their seal of approval.

Oliva said he worries about the accuracies of the 64 measures in the first place. The metrics are measured based on coding and documentation done for payors—and there’s no way to ensure that process is uniform for all the hospitals involved, meaning there’s no way to make a universal comparison.

“Are we confident that the data being used to make those measurements is accurate?” Oliva challenged.

“There’s a natural variation in the humanness of doing coding that is a problem. As much as they try to control for that, many times it comes down to judgment of the coder as to whether they go to a code or not. Every single time that judgment is made one way or the other it impacts the outcome in some way,” Oliva said.

Oliva said he thinks it’s important physicians, hospitals and other providers continue to make sure their coding processes fit government regulations and are as accurate as possible—both for their own benefit in this new rating system and for overall accuracy and positive patient outcomes. But he said he doesn’t believe the industry has reached the kind of uniformity necessary to make billing documentation useful for clinical comparison.

Even if it had, Oliva said, this new star system still wouldn’t be perfect. Those 64 metrics are a lot to compress into an overall five-star rating.

“It is concerning that patients might make the wrong choice because they don’t know which one of those [metrics] is the driver and whether or not that is really important to them,” Oliva said.

The system might be good for a general view of which hospitals are better than others, but certain information could get lost at an individual hospital level.

“For example, let’s say it’s patient satisfaction and patient experience, and let’s say the hospital doesn’t perform well there or the provider does not perform very well there. [For me], as a consumer, that may or may not be important if I’m looking for the best cancer therapy.” Oliva explained. “If [patient experience is] the thing dropping my score, yet that organization is the best in the world at cancer care, I’ve just moved my patient away from the best provider because of something else.”

But Lauren Kennedy, director of health policy at National Partnership for Women and Families, a consumer advocacy nonprofit, said weighing different measurements gives patients a full view on what’s important to them.

“A consumer will be able to get very important info about clinical outcomes associated with a certain procedure and condition, but they’ll also be able to get info about what often matters to a lot to patients and families, about what is the patient experience of care at that hospital,” Kennedy said. “That can often make a huge different to the patient, how they felt they were treated.”

The simplicity of the system is what makes it so useful for patients and their families to make informed decisions, according to Kennedy.

“It takes very granular information and turns it into a tool that consumers are able to interpret,” she said.

And if they want to weigh one metric more heavily than others, they can dive deeper into the ratings information on the Hospital Compare website.

“You are able to drill down and see how a hospital is performing on a domain or metric that may be more important to you, so you’re not just limited to the five-star summary,” Kennedy said. 

Oliva said he understands the need for a new way for Medicare patients to choose their hospitals based on quality. He suggested the possibility of breaking down the 64 metrics into smaller categories of rankings.

“I think about my mom trying to look at 64 variables and make some judgment, so wanting to simplify this makes a lot of sense,” he said.

But for now, he won’t fully endorse this iteration of the plan until he sees more data over time. 

“I’m all for it, but I caution over-using the data to make too many decisions right now, just based on the data,” Oliva said.