The Winner’s curse

This, my latest ‘Devil is in the Detail’ post, was partly prompted by a recent letter to an ALS and FTD scientific journal, commenting on a paper published in the same journal over a year before.

I apologise for the smoky gambling image accompanying this piece but its relevance will become clear.

The paper in focus was penned by Boddy et al. which, in 2024, examined the frailty of the ALSFRS-R scale in measuring our disease, its utility in trials when used alone and the wider challenges.

When the critical letter was published recently it highlighted the paper, which to be frankly honest had passed me by, as probably it did many?

Its appearance, though, allows me the chance to revisit many of the research challenge themes I’ve explored on my website over the years. In particular, the robustness of trial statistics. It seemed an excellent opportunity to bring together some of the key issues and common misinterpretations in one place, all grounded in a real world example.

At the heart of both the paper and the comment letter lies a shared and commendable goal: to ensure that patients are offered treatments that are not only safe but also meaningfully effective.

Our ALS/MND community has endured too many instances of premature optimism, misplaced hope, misinformation, diverted resources, and increased patient burden stemming from therapies that ultimately failed to demonstrate robust clinical benefit.

I will now jump right in!

The essence of the critique letter is that the Boddy et al. paper possibly might prevent small effect treatment candidates from both progressing through studies and potential approval.

For example the letter quotes the approval of Relyvrio and Radicava as examples of candidates that…

“…demonstrated an adequate clinical and statistical basis for approval by major regulatory agencies..”

Darke, A., & Orsulak, C. (2025). How effective does a new drug for Amyotrophic Lateral Sclerosis need to be – the patient perspective: a letter in response to “Estimating the minimum important difference in the ALSFRS-R-instrument in people living with MND” published in vol. 26, pp. 249–258. Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration, 1–2. https://doi.org/10.1080/21678421.2025.2589781

But should Radicava and Relyvrio serve as cautionary tales rather than a supporting precedent for small effect approval?

Digging deeper – Relyvrio and Radicava

Relyvrio and Radicava are two highly instructive examples of the difficulties and dangers in over-interpreting underpowered and/or short-duration trials. These two studies were not just marginally underpowered, they were significantly so!

Relyvrio was approved by the FDA based on one small phase 2 study reporting a 25% slowing in ALSFRS-R decline over six months. While the result was statistically (mathematically) significant, it emerged from a trial with a limited sample size (just 137 patients across both arms) and short duration. This raises legitimate concerns about the ability to detect true treatment effects amid the substantial variability inherent in ALSFRS-R scores.

This concern was, unsurprisingly and inevitably to many, borne out when a subsequent, larger, and longer-term trial failed to replicate the initial findings. Amylyx’s decision to withdraw the product from the market was both responsible and commendable.

This is a textbook case of the ‘Winner’s Curse’ where an underpowered study resulted in an inflated effect size that did not withstand more rigorous scrutiny. The large patient population variance in ALSFRS-R measurements, differing rates of change, and the disproportionate effect of outliers in small samples all contributed to this very real statistical phenomenon. It’s very common, if not even the norm, in underpowered and short studies.

Radicava’s approval followed a similarly constrained trial, with only about 60 participants per arm. While the oral formulation has, no doubt, reduced treatment burden, the continued absence of a robust confirmatory phase 3 trial remains deeply troubling. A third-party (Ferrer) trial of edaravone, the active ingredient, failed to demonstrate efficacy. Yet Radicava remains available in some jurisdictions. This raises important questions about the standards we apply to continued market access.

The key takeaway for anyone reading trial results is this:

A statistically significant result does not necessarily mean a correct one.

I wrote at length about the technicalities and required trial sizes in my ‘I’ll take a P, please Bob’ post.

It’s worth noting that regulators in Europe (EMA) and the UK (MHRA) declined to approve either Relyvrio (EMA statement) nor Radicava (EMA statement) at any stage, citing insufficient evidence.

This divergence in regulatory decisions invites reflection on the thresholds we set for efficacy and the consequences of lowering them.

Emerging Candidates and the Role of ALSFRS-R

The letter’s comparison of the Boddy et al. paper to early-phase results from Relyvrio and Radicava may therefore unintentionally conflate distinct issues.

While I don’t seek to defend every aspect of the Boddy et al. paper, it’s important to clarify that it does not argue against the possibility of small but real treatment effects. Rather, it contributes to a broader conversation about what constitutes a perceptible and clinically meaningful change from the patient’s perspective – an essential consideration for both trial design and regulatory evaluation.

Specifically the reference to a 3.8-point change over three months, as cited in the Boddy et al. paper, appears to serve as a measurement anchor, not a rigid benchmark for efficacy or trial duration. That said, I can understand how this might be interpreted differently, and it’s fair to scrutinise how such figures are presented and received. The broader point, that small changes in ALSFRS-R may not translate into meaningful benefit for patients, remains a valid concern, especially in the context of short or underpowered studies.

But there is a clear counterpoint isn’t there?

Would a 3.8-point or other lesser substantive reduction, over say perhaps 24 months in a well-powered trial, be precluded from approval consideration? Nothing in the paper, as I read it, rules that out.

This is not to dismiss the potential of emerging candidates such as PrimeC, CNM-Au8, or others. Rather, it is to underscore the importance of interpreting early signals with caution. Without adequately powered, longer-term trials, we risk repeating past missteps and misallocating both hope and resources.

The overwhelming conclusion is that there simply has to be more compelling evidence than what we saw with the Radicava and Relyvrio cases. I suspect, and hope, the letter’s authors would agree.

A Path Forward?

The experience with tofersen offers a more rigorous model: a well characterised patient subgroup, a plausible biological target, and extended longitudinal data.

Notably, tofersen’s approval did not hinge on ALSFRS-R changes, but on a broader evidentiary base.

The Boddy et al. paper is but one of several that urge a re-examination of the reliance on ALSFRS-R as a sole or primary endpoint. It does not argue against the approval of small-effect therapies. Rather, it calls for more robust trial designs capable of detecting meaningful, durable changes in disease trajectory.

Finally, while the letter rightly cautions against dismissing small effects, it is equally vital that we are not misled by statistical (mathematical) significance in very small samples.

The bar for approval should not be lowered in response to our very real desperation. It should be stabilised, fixed in rigorous methodology, patient centric outcomes, and reproducible evidence.

Only by adhering to these principles can we ensure that future treatments truly alter the course of ALS/MND and deliver tangible benefit to those who need it most…

We living with ALS/MND, and our families, both now and in the future deserve nothing less.

My next post will be one on the advances in research over the last decade and how they will contribute to the continued search for effective treatments.