Getting it wrong most of the time? Comparing trialists’ choice of primary outcome with what patients and health professionals want, 2022, Treweek

Abstract
Background

Randomised trials support improved decision-making through the data they collect. One important piece of data is the primary outcome — so called because it is what the investigators decide is the most important. Secondary outcomes provide additional information to support decision-making. We were interested in knowing how important patients and healthcare professionals consider the outcomes (especially the primary outcome) measured in a selection of published trials.

Methods
The work had three stages: (1) We identified a body of late-stage trials in two clinical areas, breast cancer management and nephrology. (2) We identified the primary and secondary outcomes for these trials. (3) We randomly ordered these outcomes and presented them to patients and healthcare professionals (with experience of the clinical area), and we asked them to rank the importance of the outcomes. They were not told which outcomes trial authors considered primary and secondary.

Results
In our sample of 44 trials with 46 primary outcomes, 29 patients, one patient representative and 12 healthcare professionals together ranked the primary outcome as the most important outcome 13/46 times or 28%. Breast cancer patients and healthcare professionals considered the primary outcome to be the most important outcome for 8/21 primary outcomes chosen by trialists. For nephrology, the equivalent figure was 5/25. The primary outcome appeared in a respondent’s top 5 ranked outcomes 151/178 (85%) times for breast cancer and 225/259 (87%) times for nephrology even if the primary was not considered the most important outcome.

Conclusions
The primary outcome in a trial is the most important piece of data collected. It is used to determine how many participants are required, and it is the main piece of information used to judge whether the intervention is effective or not. In our study, patients and healthcare professionals agreed with the choice of the primary outcome made by trial teams doing late-stage trials in breast cancer management and nephrology 28% of the time.

Open access, https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-022-06348-z

 

From the abstract it looks as if the authors do not understand the process of choosing a primary outcome measure. The result looks pretty unhelpful for everyone.

 

Jo, can you elaborate? In the sense that choosing a primary outcome is not a popularity contest like voting for president or selecting Miss Universe but depends on context, the research question, state of play in the field, etc? Or...?

 

A primary outcome measure has to reflect reliability as well as directness. If a direct measure is totally unreliable it is not suited to being a primary outcome.

The whole point of distinguishing a primary outcome measure is to maximise the chances of getting a statistically significant, reliable outcome. It is designed to prevent cherry-picking and bad statistics. Readers can judge whether a statistically significant reliable primary outcome is actually of any use in terms of symptoms by looking at the secondary outcomes.

The situation is complex because it will depend on how clear it is that the primary outcome and more direct secondary outcomes are interdependent. An Alzheimer's drug might reduce brain amyloid or benefit in cognition or both. The primary outcome for a particular trial might be amyloid reduction because there might be a very objective reliable measure and if it was reduced further trials would be worthwhile even if changes in cognition were obscured by variance in results. At some stage there must be convincing evidence of a directly relevant outcome improving but things are way more complicated than equating directness with primary outcome.

Patients have right to decide what they think is the most important outcome but that is not necessarily the one to be chosen as primary outcome. It is unhelpful to suggest to them that they are the same thing; it is likely to ternate ill feeling.

 

You are missing a vital detail - it should also be highly relevant to patients quality of life. No one cares if a therapy has an effect on everything but what actually matters to patients.

 

I don't think that is right @Snow Leopard. In my phase I study of rituximab in RA the primary outcome I chose was C-reactive protein because I knew that for a small study that was open label and uncontrolled I was only going to get useful information from a highly objective measure. We have a mass of historical evidence for CRP being a good indicator that symptoms will also improve and if that is what you mean by highly relevant, OK. But no, primary outcome measures are primary for methodological reasons around statistics. For many treatments the proof of true clinical utility comes from bootstrapping up from convincing objective data on effects on process and more direct evidence of clinically relevant improvement from trials to look specifically at that. I see that they are only considering 'late-stage trials' but even these are not necessarily best given 'most relevant' primary outcomes.

I appreciate that these is a potential problem if trials are not assessing the most relevant clinical outcomes but I think it is a misconception to think that these should necessarily be the 'primary' ones. We have seen this with ME. PWME will probably argue for years what the most relevant outcome is and focus on a symptom. In previous discussions we have raised the issue that it might be much better to use something like the ACR criteria of improvement in RA, which most patients will have no understanding of the basis of, in order to get a robust measure.

I don't know much about trial sin nephrology but trials in breast cancer are very much a process of bootstrapping conclusions from a whole series of previous trials and reasonably reliable assumptions about links between objective biological changes and quality of life measures.

 

Surely quality of life should always be the primary outcome? And the outcome measures chosen to to reflect this (eg reduction in fatigue, pain, whatever). There should previously agreed and validated minimally important effect sizes. For pain or fatigue, it could be a minimally important amount of increase in activity (work, recreation, education whatever) measured both by actimetry and self-report and school/work records etc., where available.

 

I'm curious. What happens if trials are done that use an objective measure - and the objective measure is not a good indicator of how well the patient feels? In fact it is almost completely irrelevant to many patients.

Something that happens in endocrinology is the use of the TSH to tell doctors everything they think they need to know about the patient's health in relation to their thyroid. They've written articles and guidelines that all say that TSH must be kept within the reference range [Edit: for people being treated for hypo- or hyperthyroidism] otherwise the patient will suffer eventually, at some nebulous date in the future, from heart attacks, strokes, osteoporosis and death.

Patients have found, independently of doctors, that if you want a single measure of wellness, the Free T3 level is much more reliable than TSH. And patients get a few years, possibly even decades, of feeling well, which doctors seem determined not to allow them.

How can using TSH be good medicine measure for treatment, and how can it be a good objective measure for trials?

 

Surely, though medical research should ultimately relate to quality of life in the long term, there are valid research objectives that in the short term will have little direct impact on patients. For example in relation to ME, research on the underlying mechanisms of ME is likely to be essential for effective treatment but of itself does not necessarily provide a treatment, or alternatively we all want research to establish clinically useful diagnostic biomarkers, but even when biomarkers are established, for people already diagnosed this will not make significant difference of itself to their health or functional ability.

I would argue that for ME too much research has focused on improving quality of life at the cost of understanding the condition.

A major problem of the BPS research such as PACE is that it tries to demonstrate effective intervention without seriously attempting a scientific understanding of the condition. Any cursory investigation of the symptoms of ME would have demonstrated that the underlying condition can not be explained by deconditioning or false cognitions, however these researchers trial over and over again interventions based on their unevaluated beliefs and time and time again give us no meaningful results.

Even much biomedical research is chasing up drug trials, before having an understanding the underlying mechanisms by which it could work. I know it is not uncommon in medicine to have drug treatments in search of a condition, but is this the most efficient use of resources when we have such limited understanding of the condition.

 

Only if a trial is the single final arbiter of whether or not a treatment is to be judged useful. In reality trials are very rarely in that category, although for any given treatment there are often one or two trials that come close to it.

Quality of life in itself has not been used as a primary outcome in RA for 25 years. It is much more useful to use a composite measure that can be compared across relevant trials (and that does not many any trial or all trials) if the composite measure gives much more confidence that a real biological effect due to the treatment is producing clinical benefit.

The idea that trials should use what seems the most important outcome to the patient as the one chosen to bear the weight of statistical power is very simplistic. In cancer trials the key outcome for patients is usually the chance of disease free survival, or maybe just long term survival. For most trials this is not a sensible primary outcome because you want to gather reliable evidence of efficacy long before you know about long term survival - if you can. Only then can you progress towards cure in the way they did for Hodgkin's disease back in the 1970s. You keep trying different protocols and clock up the improved outcomes on short term measures in the reasonable hope that that will be reflected in survival. It is not as simple as that but it works pretty well most of the time.

There are obvious pitfalls. For instance tocilizumab is an IL-6 inhibitor for RA. IL-6 is the trigger for CRP production. So if you give an IL-6 inhibitor and CRP goes down all you can tell is that you have hit IL-6 - not that this is any use to the patient. So for tocilizumab trials it would have been sensible to remove ESR and CRP from the outcome measure once phase I studies had shown that CRP did go down.

So in answer to @Arnie Pye 's question, sure, you need to avoid objective measures that you cannot be confident will predict quality of life benefit. But that is all part of the business of designing good trials. The TSH example remains controversial and I will sidestep that.

 

I think effect sizes are for something different. They are crucial but not related to primary outcome.

Maybe we should take four scenarios with an relatively objective but indirect outcome measure O that is chosen because it seems reasonable to expect it to show an effect if the treatment is doing roughly what we think it ought to do in a way that would be specific for a clinical effect (unlike CRP for tocilizumab) and a relatively subjective but direct measure of quality of life S that is open to issues of interpretation in terms of reliability.

Make O the primary outcome.
If we have O+S+ then we are happy. But only if S+ goes above a useful effect size.
If we have O-S- we think this is time to go home.
If we have O+S- we say OK something seems to have happened but we were probably wrong to conclude that it would feed through to quality of life so we either need to go home or find out if there is some reason why the two were dissociated. Maybe the dose was wrong to be useful. Maybe other things but for definitive phase e 3 trials this is normally a treatment killer even if S- was not chosen as primary.
If we have O-S+ we know that we may have a reliability problem with S+ and need to address why there might be a dissociation and repeat in a way that does as much as possible to reduce unreliability.

Make S the primary outcome.
If we have S+O+ we are happy but only if there is a a good effect size - the same.
If we have S-O- it is time to go home as before.
If we have S-O+ then we don't really know what to make of O+ because to take note of it over any other measure would be to cherry pick. The decision to investigate whether or not there is a reason for dissociation that can be fixed is much more muddied.
If we have S+O- we are as before. Making the most relevant outcome statistically privileged probably does no useful work.

In other words the critically difficult case of O+S- is made even more difficult if O is not the primary outcome.

 

Primary outcome would depend on what is the research question

 

@Jonathan Edwards I am not currently able to read all of your explanations, but am I correct in summarizing that it is preferable for the primary outcome of early phase trials to be a biomarker, in order to verify if an intervention is effective on it (say, for instance, a drug reducing autoantibodies), and then once this is confirmed, later phases can use a QoL measure as the primary outcome?