AI is moving into science’s hardest audit: studies medicine cannot reliably repeat

A consortium led by CASP founder John Moult is launching an AI-driven initiative to systematically audit published biological research, targeting the reproducibility crisis that has festered since the early 1990s. The group combines computing power with domain expertise honed over decades validating biological claims, with a focus on high-impact work across top journals where stakes are highest and validation is most critical.

The project builds directly on the CASP model that previously led to DeepMind's 2024 Nobel Prize in Chemistry, applying the same competitive-validation framework to a different problem: industrial-scale verification of published studies. If successful, this could represent the first scalable system to police the integrity of biological science at industrial volume.

The timing aligns with growing criticism of peer review's limitations, particularly in fields where results hinge on complex datasets. The real signal here is not another hand-wringing exercise but a concrete attempt to embed algorithmic checks into the publishing process itself. Early signals suggest the team will automate detection of inconsistencies in experimental design and statistical reporting, areas repeatedly flagged as culprits in reproducibility failures.

According to reports from the reproducibility research community, similar tools have already shown promise in limited trials, raising hopes that larger-scale deployment could reshape how science is vetted. The ambition is to turn retrospective review into a preventative mechanism—scanning new submissions for red flags before publication.

The operational implications extend beyond catching fraud. The system appears designed to surface subtler problems: underpowered studies, selective reporting, and statistical artifacts that slip past human reviewers. These are the chronic failures that waste billions in research funding and slow genuine therapeutic progress.

Moult's coalition faces significant hurdles. Journals must agree to cooperate, authors must accept algorithmic scrutiny, and the AI itself must prove robust against adversarial manipulation. False positives could damage careers; false negatives would erode trust. The validation challenge is meta—who validates the validators?

Yet the alternative is continuation of a broken status quo. Estimates suggest more than half of preclinical cancer studies fail replication, with costs measured in years of misdirected research and discarded drug candidates. The Moult initiative offers a structural response to a structural failure.

What distinguishes this effort is its institutional weight. CASP demonstrated that rigorous, open competition could accelerate genuine breakthroughs—AlphaFold emerged from that crucible. Applying similar competitive pressure to verification, rather than discovery, represents a conceptual shift with potentially broader impact.

The project enters a landscape already primed for change. Funding agencies increasingly demand data-sharing plans. Journals experiment with registered reports. Post-publication commentary platforms grow. Moult's system could integrate these fragmented efforts into something coherent and automatic.

The deeper question is whether scientific culture can adapt to continuous algorithmic surveillance. Reproducibility is partly technical, partly social. Tools help; incentives determine use. If the AI flags problems that editors ignore, nothing changes. If it becomes a gatekeeper, it reshapes what science gets published—and who gets to do it.