The effectiveness of accountable care organizations (ACOs) may be limited by two factors, according to a new study published in Health Affairs: low numbers of enrollees attributed to participating physicians and the constant “churn” of the patient population caused by substantial physician turnover.

The study was supported by the Commonwealth Fund and led by John Hsu, MD, MBA, director of the clinical economics and policy analysis program at Massachusetts General Hospital’s Morgan Institute. Hsu and his coauthors focused solely on Partners Healthcare, the largest healthcare provider in Massachusetts and one of the largest participants in the Pioneer ACO program, with 82,000 beneficiaries within the study period of 2012 to 2014.

The authors made three hypotheses. The first was ACO beneficiary penetration “would be low given the total numbers of beneficiaries relative to the numbers of physicians.” This turned out to be correct, as the study found while most (88 percent) physicians had at least some ACO patients attributed to them, those beneficiaries accounted for less than 5 percent of a physician’s median patient panel.

“This limited ACO penetration at the physician level could mitigate the ACO’s potential to achieve its financial targets, at least for any effects mediated through physician behavior,” Hsu and his coauthors wrote. “With the small numbers, it is not surprising that the distribution of high-spending beneficiaries also is skewed such that a few physicians appeared to have the sickest beneficiaries, while many appeared to have mostly beneficiaries with modest spending.”

Their second hypothesis was there would be limited turnover among physicians, since the ACO is housed within a large, well-developed health system like Partners. In fact, the researchers found substantial turnover, with nearly half (48 percent) of physicians affiliated with the ACO for only part of the three-year study period.

This tied into the third hypothesis, that much of the growth in ACO would be tied to physician participation because patients would follow their doctors. This also turned out to be true. 49 percent of beneficiaries who joined the ACO in the second or third year so because their physician had joined the ACO, and when doctors left in those years, 90 percent of their assigned beneficiaries went with them.

“In short, the physician changes appeared to affect the composition of the ACO beneficiary population, which could have important implications for the ACO program’s overall financial impact,” Hsu and his colleagues wrote.

The authors didn’t offer any doom-and-gloom scenarios for ACOs or the Pioneer model like other policy experts. They did warn, however, that substantial turnover among physicians, along with the small number of ACO patients they tend to serve, creates an incentive for ACOs to “game” the risk pool.

“At a minimum, CMS should consider requiring that provider organizations define their structure and physician-per-beneficiary pool throughout the entire contract and regulate deviations from this structure, including changes in affiliated physicians or hospitals—for example, differentiate between genuine retirements and removals aimed at influencing the ACO beneficiary population,” Hsu and his coauthors concluded.