Medical statistician reviewing COVID AI studies: ‘Poorly reported … horrible’

As researchers have turned to AI for battling COVID-19, raising hopes and making headlines, the shortcomings of the studies have drawn the ire of numerous statistics experts.

Their main shared concern is quality.

“[It] scares a lot of us because we know that [AI] models can be used to make medical decisions,” says Maarten van Smeden, a medical statistician at University Medical Center Utrecht in the Netherlands. “If the model is bad, they can make the medical decision worse. So they can actually harm patients.”

Van Smeden offers his comments via Discover magazine, which assigned a reporter to drill into the heart of the alarm and published her article July 6.

Writer Allison Whitten reports that van Smeden is co-leader of a formidable team of international researchers who are evaluating COVID-19 machine-learning models using standardized criteria.

The project has so far recruited 40 reviewers, and they’re updating their critiques whenever a new model comes out. The BMJ is publishing the work as its first-ever “living review,” we learn.

“So far, their reviews of COVID-19 machine learning models aren’t good: They suffer from a serious lack of data and necessary expertise from a wide array of research fields,” Whitten writes.

Van Smeden tells her the COVID-19 AI studies reviewed so far are “so poorly reported that I do not fully understand what these models have as input, let alone what they give as an output. It’s horrible.”

Read it all:

Dave Pearson

Dave P. has worked in journalism, marketing and public relations for more than 30 years, frequently concentrating on hospitals, healthcare technology and Catholic communications. He has also specialized in fundraising communications, ghostwriting for CEOs of local, national and global charities, nonprofits and foundations.

Around the web

Compensation for heart specialists continues to climb. What does this say about cardiology as a whole? Could private equity's rising influence bring about change? We spoke to MedAxiom CEO Jerry Blackwell, MD, MBA, a veteran cardiologist himself, to learn more.

The American College of Cardiology has shared its perspective on new CMS payment policies, highlighting revenue concerns while providing key details for cardiologists and other cardiology professionals. 

As debate simmers over how best to regulate AI, experts continue to offer guidance on where to start, how to proceed and what to emphasize. A new resource models its recommendations on what its authors call the “SETO Loop.”