Just like the characters in the classic Spaghetti Western, clinical trials for ALS/MND are not pure and perfect things. They are complex entities, laced with conflicting motivations, limitations and often potentially subject to the ‘foibles’ of their human designers.

This post is a frank discourse on the operation of ALS/MND trials globally at the start of 2024 and where we perhaps need to ‘go’ from here, at these truly exciting times, to efficiently expedite the most viable drug candidates. I will start off with a whistle stop tour of the major challenges our scientists face, and end with an upbeat vision of the future given recent discoveries and lessons learnt (hopefully!).

So just like TV Newsreaders often say:

“Warning – this report contains graphic statements that might cause some readers to choke on their coffee!” – all for dramatic effect and artistic licence of course!

A few years ago, a well-respected ALS/MND Research Professor said to me…

There are 3 ways to design and run a clinical trial, badly, quickly or the right way!

So that’s badly, badly or correctly?

As in many areas of life it is perhaps only after many failures that the ‘mistakes’ of what has gone before actually materialise in front of our eyes. I know that it can take myself several viewings of a movie to spot subliminal messages as well as the plot headlines. I say this because this post is not about apportioning any blame, it is simply about where we are and how we might navigate the best way forward.

Patient pressure, cost implications and inherently kind human researchers, who are genuinely trying to do the ‘right thing’ for patients, and much more can all affect a trial’s design and eventual outcome.

Nothing in this post is new news, it’s simply not aired enough (or heard or even want to be heard?). 

As we now race into 2024, we all need to do far better as a global community in communicating the complexities of research following a particularly turbulent few years with patient advocacy, especially in the USA. At the end of this post, I touch on just one area of change that we might explore to improve communication. I would appreciate your thoughts.

My ‘devil is in the detail’ series has been focused on the interpretation of clinical trials, results and advocacy and was inspired by observing such a deep and widespread misunderstanding of press releases, the repeated avalanches of misinformation, and the misinterpretation of ‘scientific’ statements and papers in the years directly following my diagnosis, now 10 years ago.

It is vital that patients and their families, including sadly those to be diagnosed in the coming few weeks and months, are able to gain access to reliable scientific information and guided down the right road.

I sincerely hope we can all do better, as letting the same environment continue will be frustrating and somewhat echo the (entertaining) movie Groundhog Day!

I will reference my series so far and will try and largely keep this post, I hope, jargon free.

Ok, on with that straight talk and it could get somewhat ugly!

I will start with…

Observation – Are pivotal* trials all too often, simply too short?

The central issue that our community appears to face is a constant over selling of many short trials and results, largely through press releases and non peer-reviewed communications.

*pivotal – a trial that is typically known (but not always) as a Phase 3. Such trials are that last step of safety and efficacy evidence gathering that, both patients and advocates hope, will be that final hurdle towards potential access, if results are positive.

I have described many aspects of trial complexity throughout my series. Notwithstanding this, there are 2 fundamental factors, that stand out, that if not addressed thoroughly in design, a trial’s quality, validity and usefulness will sadly be firmly cast in stone before even a single participant has received the first dose of placebo or experimental treatment.

They are:

  1. The number and characteristics of participants. 
  2. The length of time that a trial runs for.

Starting with the number of participants. 

This is critical, and the numbers required are far larger than many believe or are ‘led’ to believe! I have seen comments such as

“a trial needs only 30 participants to show a statistical difference and we can see it in 6 months.”

This and many other similar statements are simply complete untruths.

Understanding statistics can be a huge challenge, and I have covered in some detail in 50% chance of rain, I thought the weather report said.. and I’ll take a P, Bob.

I have also addressed, what appears as a dark art, the complex science of sizing a trial in I’ll take a P, please Bob, I am going to repeat here, although slightly differently, in this post because it’s just so important.

I will leave the world of clinical drug trials just for a moment with an analogy which I hope will help (I just love analogies).

Let’s imagine what we are testing is not a drug but whether one of the dice used in my game of backgammon is biased towards any one number.

To make this analogy directly relevant to the challenge of ALS/MND clinical trials, I want to use the same statistical criteria and levels of significance used in a typical trial during my experiment.

Specifically, I will use a significance level of 95%, ie a p of < .05 and power the test to 80%*. “What is this thing p, Lee, and what is powered?” Take a read of I’ll take a P, please Bob.

*this means that we would like an 80% chance of detecting a valid bias and in the event of the effect being detected I want to be 95% certain that the effect is not by sheer chance.

Two questions are critical in statistical planning for my suspect dice experiment. The first is what is the minimum number of rolls needed to establish, with the criteria above, if there is a meaningful* bias. The second is, possibly just as important, and it will become clear, exactly how long will my single study take me? 

*All dice will have a level of bias, perhaps with weight distribution. But in the real world, however, such ‘atomic’ bias will not be meaningful, and to all intents and purposes rolls will be fair. So my experiment will NOT be testing for just any small bias, but ‘meaningful’ bias, ie one that could mean ‘game’ disadvantage/advantage.

Using well established and accepted statistics foundations, the number of rolls can be calculated and is between about 50 and 80. I would hazard a guess that it would take me all of about 30 minutes to roll the dice, including the statistics calculation.

How is this relevant?

I wanted to give an example of a very simple problem with a very small number of well-defined outcomes (6) but also one with absolutely immediate results. We would easily know within an hour whether the dice is meaningfully biased.

Or would we?

No, we still might not! Why? The test was only powered to 80% just like our drug trials, ie there is a 20% chance that the study didn’t detect a bias (that could be there), and, in addition there is also a chance of 1/20 (5%) that if it identified bias, it would be a false flag! I could easily reduce the false flags and check for confirmation of bias by simply running the whole study, say, 50 times? I reckon I could do that within a morning. The same, absolutely, cannot be said of a 2-year clinical trial, and yet we hear results of clinical trials that are expressed so confidently!

Now returning to ALS/MND drug treatment trials, which are most often measured by the current gold standard ALSFRS-R function rating scale. The sheer variability, rate of change and precision of the scale is such that the number of participants required to show, with confidence, a valid ‘clinically significant’ effect (to the same confidence levels as in our dice experiment above) can be also calculated and is in between 300 and 600. Surprised?

And how about the time for a treatment to have an effect? It’s not immediate, is it? Our disease has taken months, if not years, to have done the variety of damage to our bodies before a trial is started. Is it not entirely reasonable to expect that any treatment might take many months, possibly a year or even longer to have a perceived, or more importantly, a measured effect?

Returning to the analogy, I like to think of the level of our scientists challenge as almost akin to validating the bias of a 200-sided dice but where each roll floats around in the air for months, if not years, before landing!

And yet time and time again our community is faced with reports of short trials (most often in press releases), with just a few participants, making vastly unjustified claims.

The scientific community knows this but for many reasons the message has not really been widely understood or heard.

Sizing and choosing a trial length are just the start of a designer’s headache!

For example, recruiting participants and assessing the risk that they might not actually complete the trial due to their own disease progression speed. And how many drop out? Even with the very best intentions at the start, a trial could be damaged severely by events beyond the control of the designers, if not considered early. This is, indeed, challenging stuff!

Given that trials don’t typically over recruit, an already minimum sized trial starts to become rather ‘presumptive’ and ‘speculative’, at best, especially with trial subgroup analysis, known as post hoc analysis. I’ll return to post analysis in just a moment as it is pretty ugly, in its claims in many cases.

So lets now talk about the ugly. Ok, so I am now going to get blunt, but this is really where we need to change and change for good in 2024.

It’s not a huge leap from the bad to a simple example of a real ugly issue that our whole community needs to be aware of, and that is the all too common and inherently flawed:

Six-month trial measuring ALSFRS-R with placebo and with just 100 participants in each arm searching for functional change.

Such a trial will *always fail/be ambiguous at best and is destined to be so right from the drawing board.

Why?

  1. This is barely time for a drug to have an effect.
  2. The number of participants is far too low within the time to statistically measure an ALSFRS-R change. A trial needs about 300 to 600 participants to show, in such a time period, statistically an effect (unless a huge effect, which is realistically, unrealistic!). 
  3. Patient variation is so just broad, that over a period of 6 months about 25% will appear to plateau anyway. In addition to plateaus, the disease heterogeneity is such that we all recognise, early on in our disease, that our progression won’t be consistent.

*Note – AMX0035 (Relyvrio) and Edaravone (Radicava) were approved by the FDA in the USA after 6-month trials, but were not only controversial, but have been rejected by the EMA in Europe which requires stronger evidence than that provided to the FDA. Both treatment candidates continue still to be only, at best, marginal prospective treatments. The trials were significantly underpowered and could, by design, only safely detect a massive change, which did not materialise in either. AMX0035 although ‘appearing’ to ‘achieve’ its primary endpoint, it did so with just 137 participants (2:1 treatment to placebo) but in reality, about 350 participants were required to show such an identified significant effect. Further, the treatment was only actually eventually approved with a controversial post hoc analysis on extended 35-month data that did not use actual trial data points but predicted placebo progression points and historical data.

Two appropriately sized and powered phase 3s (over 300 participants in each) will complete in 2024 which should shine light on whether either treatment candidate is ‘truly’ effective. The first has just this week completed, for Edaravone, which has failed. It now remains to be seen the impact on Radicava. It is also worth noting that the Phase 3 for Tudca alone (one of the components of AMX0035) will also report in 2024. Might Tudca be the effective component of the cocktail, or might neither be effective? We await the results. Given the controversy with both treatments, 2024 will probably resolve these two marginal, if any effect, candidates. 

Sadly, a number of other prominent pivotal trials have been such short 6 month studies. Certainly, our community can be led to believe that the results from such trials might justify widespread access and efficacy.

They do NOT and are a mere stepping stone. Short trials have their place, and value, but we must not believe, or promote, them to be more than they are.

For example, it’s worth noting here, the usage of the excellent and innovative Healey trial platform. Why? Remember I mentioned that a typical trial is powered to 80%? The statistics analysis plan for the Healey master protocol documents that the platform has a power of as low as just 61% for detecting a typical change. That means there is nearly a 40% chance that it won’t detect an effect even if there is an effect! I’ll let that sink in. I am not criticising the platform here, as it provides a rapid trial deployment so needed for our disease and significant trial costs reduction. However, we must not let such an excellent utility be potentially mis-used to make un-substantiated claims when used to facilitate a sponsor’s candidate experimental study. It is most definitely NOT a phase 3 platform.

I promised to mention post hoc analysis which I covered in my series, for example in Guide to surviving trial news, part 1. The simplest way to summarise the ‘ugliness’ often seen, which is essentially a misuse of post hoc analysis, is that almost every single trial in history, across all diseases, has managed to identify some ‘miracle’ optimistic characteristic from the deep torturing of data! But most fail in later, well sized and powered, trials.

We must all recognise that post analysis is not valid for efficacy testing, especially when an already small and short trial is already under powered.

Lee! This is all just too negative. What is the counter side to these bad and ugly trial truths? Please talk about the good and what ‘new’ good might we need to introduce to change the future face of trials and understand the results more clearly.

First of all, and this is good news.

All of the trial design principles and issues I have described are very well understood but not always adhered to. Please pharma, scientists and all those who represent our community be more transparent with us!

Well-designed trials are absolutely possible, right here, today. So, despite the tone of much of this post, we should all be aware we can continue in 2024 from a solid base. It goes without saying, or perhaps it does have to be, that all trial sponsors should employ a biostatistician from day 1 of any design. 

By definition, a well-designed trial, 18-to-24-months in length measuring ALSFRS-R, with significant cohort size 300 to 600 will generally be non-ambiguous in its findings. Clearly this does not appeal to patients and many advocates, but I ask – what is better one long trial or many failed shorter ones that might never actually reach any conclusion? Sounds rational?

And what is the real aim of any treatment? There is a lot of debate, but at the central heart of all disease treatments is longer survival. We see this with cancer studies, and the 5, 10 year etc survival rates of treatments. In fact with our progressive disease, longer survival most probably does match with slower disease (not conclusively though, however, as a treatment could, for eg, target respiratory failure only). Generally, however, it is accepted that with a trial powered for survival (less participants are actually needed than for ALSFRS-R measured trials) that positive results would most likely indicate a functional loss slow down as well.

Survival is relatively simple isn’t it, but yes very brutal? It’s a binary alive or dead. Again, such measures don’t appeal to we patients. However, trials sized and powered for survival can, within a 2-year period, provide unambiguous results that can really convince regulators and the regulatory process.

This 2-option trial design paradigm (ALSFRS-R function scale and survival) has really been at the heart of all our scientists trial challenges with our disease for over 25 years and it won’t go away anytime soon.

Where should/might we be going?

Clearly, our scientists need to develop and refine better ways to measure our disease, find better candidates and, and this won’t be popular, insist on and design longer pivotal trials. Recent, and not so recent, history has shown this.

Some further principles which might be our guide?

  1. Improve pre-clinical science (including targeting trials beyond genetic causal gene sub types) and utilisation of drug screening before full trials.
  2. Counterintuitively might we benefit from less, but more likely to succeed phase 3s?
  3. Use survival scaled/sized/length trials, however, unattractive?
  4. Let’s not throw the baby out with the bath water, ie ALSFRS-R (still our gold standard functional disease measure) has issues but let’s improve rather than discard.
  5. Disease progression biomarkers.

Note: You might see in publications a measure known as CAFS, which is actually preferred by regulators. It is a combined measure of both function and survival but built on base survival and ALSFRS-R measures. 

Will disease biomarkers be game changing? Most probably, but the only one we have currently, NFL still appears to take many months for positive changes in functional decline to be actually measurable. To add to this we still have, as of the start of 2024, only one trial, tofersen, that has any hard and compelling results. 

Further, we do not fully understand how changes in nerve health are reflected in changes in measured NFL. For example, how does kidney function impact NFL level? The kidneys remove NFL. How about the immune system? NFL is a protein that the immune system will try to clear. How about nerve repair? Does that lead to increases or decreases in NFL release? 

This is all hard core, ‘grunt’, science which is critical when developing fit for purpose tests such as biomarkers.

Lets not forget, one swallow does not make a summer.

To complement this post, at the recent MND Symposium in Basel at the start of December 2023, the ALS Connect panel discussed the subject of trial length in response to this question…

“Based on the time to heal for tofersen in the valor trial, is it fair to say we need to move away from the patient centric approach of shorter placebo controlled trials?”

Have a watch and listen to the answer (jump to 1 hour 27 minutes precisely) and the expanding discussion that followed. Professor Ammar Al-Chalabi goes into some detail, and far more eloquently than I have just attempted with this post.

The answer was,…..Yes.

To finish up, earlier on in this post I mentioned communications within our community and how we must improve or face continued turbulence of misinformation, through whatever cause.

I now believe that our leading researchers and scientific thought leaders, globally, are the probably the only people that could help us all move forward positively. Let’s try and move away from the human foibles, even the incredibly tempting desire to never remove hope. How? Let us attach hope to the right places and certainly not to things where, who many in the know, will know, hope does not exist.

How else might our leading researchers help us?

Might they not allow themselves to be quoted in press releases that over promote the results of a trial?

Might they even refuse to be in press releases?

Perhaps hold seminars where the challenges are laid bare?

That’s all for now folks.

What will be the subject of my next devil detail post? The chosen subject is still going around in my mind, but it’s likely to be on Bio-markers and their potential to change trials.