Why are patient pleas about subtyping and stratifying in studies seemingly falling on deaf ears?

[Verity]

This question would be better posed to our researchers, who are busy enrolling all types of patients in every treatment trial they do without filtering for the appropriate "subtype"! (feel free to insert another word there if that's triggering for you)

Can you give examples of trials you think have this problem? That might help us see what you mean.

Today I read somewhere here that a member thinks they have a rheumatoid, autoimmune illness and are misdiagnosed with ME. I find this very interesting because I have a friend with a CFS diagnosis who is housebound for many years because of energy limits but whose illness looks very different from mine.

Also, she's now with a rheumatologist and feels a lot better on some drug from that area.

Are you talking about me? My symptoms look identical to standard ME, including the flu-like episodes you mentioned. I’ve spent a ton of time in ME spaces over the last seven years, and I truly cannot identify any difference. The problem is I have cutaneous lupus, and my ME symptoms improve semi-longterm after steroids in a similar manner to systemic lupus. But any ME patient hearing my symptoms without information about my steroid response or photosensitive rash would recognize them immediately. I did not improve in ME symptoms on plaquenil, if that’s what you mean. I have no clue whether to think I’m misdiagnosed, and my rheum doesn’t know either.

Anyway, distinguishing people by whether they have flu-like episodes would not rule me out. If anything, this suggests how fraught making up subcategories is when you don’t really have any evidence to go on.

Sorry if you’re talking about someone else! But I am not aware of anyone else on here in that situation who posted about it today.

 

[ChronicallyOverIt]

I think ya'll are getting hung up on subtypes. Let's forget about subtypes, or we will go back and forth about semantic category error debates all day.

New crux: trials should collect all possible data, within the limits of their funding, be it patient surveys, past drug responses, biomarkers, on-site testing, etc. and stratify results based on that data.

This is basically p-hacking no? What this quote is describing is essentially:

Post hoc analysis - Wikipedia

en.wikipedia.org

P-value hacking

{{#regex: P-value hacking, also known as data dredging, data fishing, data snooping or data butchery, is an exploitation of data analysis in order to discover patterns which would be presented as statistically significant, when in reality, there is no underlying...

embassy.science

 

[ChronicallyOverIt]

Do we have a concise and easy to read resource on trial design, I’d love to learn more and think it would help this conversation, looking for suggestions

 

[PageofME]

Can you give examples of trials you think have this problem? That might help us see what you mean.

Are you talking about me? My symptoms look identical to standard ME, including the flu-like episodes you mentioned. I’ve spent a ton of time in ME spaces over the last seven years, and I truly cannot identify any difference. The problem is I have cutaneous lupus, and my ME symptoms improve semi-longterm after steroids in a similar manner to systemic lupus. But any ME patient hearing my symptoms without information about my steroid response or photosensitive rash would recognize them immediately. I did not improve in ME symptoms on plaquenil, if that’s what you mean. I have no clue whether to think I’m misdiagnosed, and my rheum doesn’t know either.

Anyway, distinguishing people by whether they have flu-like episodes would not rule me out. If anything, this suggests how fraught making up subcategories is when you don’t really have any evidence to go on.

Sorry if you’re talking about someone else! But I am not aware of anyone else on here in that situation who posted about it today.

I'm not sure. What struck me in that patient's post was that they were assuming that they had an autoimmune illness but no diagnosis and that they fulfilled the ME/CFS criteria because of that but actually didn't not have ME but had received the diagnosis. And explicitly stated that they think they are misdiagnosed. So I think it wasn't you.

 

[Verity]

I'm not sure. What struck me in that patients post was that they were assuming that they had an autoimmune illness but no diagnosis and that they fulfilled the ME/CFS criteria because of that but actually didn't not have ME but had received the diagnosis. And explicitly stated that they think they are misdiagnosed. So I think it wasn't you.

Okay, thanks for clarifying. I am officially diagnosed with both ME and an autoimmune disease (although it’s unclear whether they’re actually just two labels for pieces of the same problem in my case) so that does not sound right. I wasn’t aware of someone else like this on the forum. I’ll have to try to find them.

 

[Liface]

Sorry for the delay all, I am bedbound/severe and wanted to think about these points more - can only use a laptop late at night when my cytokines go to sleep.

The point I was trying to make was that if someone is testing a treatment intended to address particular symptoms or comorbidities that some people with ME/CFS have, whether hives or migraines or POTS, or irritable bowel symptoms, then they need to test the treatment with samples of people who have that symptom.

If the treatment is intended to treat ME/CFS itself, for example to try to reduce or eliminate PEM episodes that everyone with ME/CFS has, then it matters less which other comorbidities the person has. There woud be less need for 'subtyping' according to which comorbidities the people have so long as they fit the criteria for ME/CFS incuding PEM.

So are you really talking about subtyping, or are you talking about co-morbidities?

Whatever we want to call it, I'm fine with - just as long as the trials are setting up hypotheses about which patients will benefit and collecting data, then stratifying or presenting whether those hypotheses were validated.

It's not triggering and I didn't author what you quoted anyway. It was Trish who asked, quite reasonably, why anyone would prescribe a drug to treat tachycardia to people without tachycardia.

I'm only arguing against crap research, and that includes trialling drugs we have no reason to think work, and setting up trials with arbitrary categories we have no reason to think differentiate meaningful subsets.

If it continues, and if patients keep putting up with it, we'll still be here in another 20 years. I've already done 50 years with ME/CFS, so you'll forgive my world-weariness. Like a lot of people here I've seen cycles of poor work being repeated over and over, and believe me it gets very boring.

Apologies, I am a bit inflammatory today looking at these ridiculous trials that were presented at the conference.

I am with you! No more crap research.

@Liface

I don’t understand what you’re suggesting.

Gathering data on what kinds of symptoms the participants have is a good idea. The same goes for blood samples and other data like steps, time spent lying down, etc. But we have no clue about how to use that data to meaningfully divide the participants into groups beforehand. And it’s not from a lack of trying - there have been quite a lot of papers published on various analyses of supposed subgroups.

Fluge and Mella analysed the data from the pilot of Dara and found that those classified as responders had more NK cells at baseline compared to those classified as non-responders. Based on this, they set a requirement of a least 125 (can’t remember the unit) for NK cells to be included in the next trial.

We have no idea if NK cells are relevant, or how they are relevant, but they chose to do it in an attempt to maximise the chance that if Dara works for some, it will show up in ResetME.

When we have a trial with a positive result, it might be easier to figure out if a measurement is relevant. But each analysis would still only generate a hypothesis, which would have to be tested in subsequent trials.

To give an example of how difficult it is to subgroup patients: they still have not been able to figure it out for MS. Some argue for differentiating between those with attacks and remissions, and those with progressive decline, but newer studies suggest that everyone progress and that each attack takes you closer to being in the progressive decline category. Maybe it’s something else.

Until we get an understanding of actual pathology, there’s no reason to hope for any subgrouping.

> Until we get an understanding of actual pathology, there’s no reason to hope for any subgrouping.

Completely agreed with you until this part. If subgrouping provides any signal (and it will), it is useful and worth doing. I feel like the reigning mentality in this thread is "It's not perfect, so we can't do it!"

The drugs we're trialing have been tried by thousands to even hundreds of thousands of patients (Low dose naltrexone, for example). We know their pharmacology and we know what they target. It should not be difficult to come up with a hypothesis about what general types of patients will benefit.

This is basically p-hacking no? What this quote is describing is essentially:

Post hoc analysis - Wikipedia

en.wikipedia.org

P-value hacking

{{#regex: P-value hacking, also known as data dredging, data fishing, data snooping or data butchery, is an exploitation of data analysis in order to discover patterns which would be presented as statistically significant, when in reality, there is no underlying...

embassy.science

The endpoints would be pre-specified.

 

[ChronicallyOverIt]

The endpoints would be pre-specified.

What end points though? We know very little about the biology in any regard, other than all patients have PEM. How do you group and form a hypothesis pre-analysis.

MCAS doesn’t seem to have a clear diagnostic. POTS has an “iffy” 30bpm cut off, as pointed out earlier. We have no clear biological signal to measure. I just don’t see how you can group people before hand clearly. If you can think of a grouping that makes sense based on a measurable and quantitative diagnostic I’d be interested

I just don’t see how you can make a hypothesis of end points on who will respond before hand. Most subtyping I’ve seen on twitter suggestions seems post-hoc

 

[ryanc97]

@Liface this pretty much explains it. It’s p hacking or data mining.

 

[Jonathan Edwards]

The endpoints would be pre-specified.

Maybe you are unaware that this does not solve the power problem, @Liface ?
The more analyses you do the weaker your statistics. If you analyse 5 subgroups your power to detect any significant result is reduced by a factor of 5 (roughly). I have designed and run many trials. You simply cannot afford to do this. If you think a drug will work on a more limited population you do your trial on that population, you do not stratify. You can post-hoc stratify for interest but not to get an efficacy result.

The subsetting craze is almost certainly driven by physicians and researchers who want to maintain unsubstantiated claims, as said above. Naltrexone probably does not work for ME/CFS but may be a rather weak painkiller. There will always be people who believe they have been helped by a drug, even a homeopathic drug with no drug in it - a hard reality that has to be taken on board on day one.

 

[PageofME]

Okay, thanks for clarifying. I am officially diagnosed with both ME and an autoimmune disease (although it’s unclear whether they’re actually just two labels for pieces of the same problem in my case) so that does not sound right. I wasn’t aware of someone else like this on the forum. I’ll have to try to find them.

You said:

This is exactly what I’m concerned by, and I say that as someone who might be one of those misdiagnosed patients. It’s pretty clear to me at this point that some patients with autoimmune diseases have symptoms that look a lot like ME...

 

[Trish]

I am wary of re-analysing trial results where there is no significant difference in outcome between those who took the drug and the placebo group to try to find a responder subgroup.

If there is a subgroup who respond to the drug that should show up as creating a better average outcome in the treatment group than the placebo group.

We have been plagued for decades by the CBT/GET proponents arguing for provision of these treatments on the grounds that 'they help some people', even though trials all show no objective or long term subjective benefit. If they really helped some people, the PACE trial would have shown much better objective outcomes on average in the GET/CBT groups, as the results of the responders would raise the group average significantly. You wouldn't need to do subgroup analysis, it would be clear from the averages.

We should see LDN and other drug trials in the same way. If the group effect is null, the treatment doesn't work for ME/CFS.

 

[Evergreen]

just as long as the trials are setting up hypotheses about which patients will benefit and collecting data, then stratifying or presenting whether those hypotheses were validated.

The problem is the assumption that some patients will benefit. We don't know that yet. Lots of people think that, for example, LDN works for lots of people with ME/CFS, because they take it, and then some of them improve. But we don't know if LDN helps anyone until we have some controlled trials. Because that reported improvement can be caused by a lot of different things - spontaneous change, expectations, regression to the mean etc. The way you find out if more than a handful of patients benefit is through a trial. Once you establish that yes, something works, you can start figuring out who benefits and who doesn't.

If subgrouping provides any signal (and it will), it is useful and worth doing. I feel like the reigning mentality in this thread is "It's not perfect, so we can't do it!"

I think it's the opposite. People are saying "LDN/other intervention doesn't even reach the lowest bar yet, so subgrouping doesn't make sense yet."

But everyone is saying that doing a trial of, for example, a GI drug just on people with ME/CFS who have GI symptoms would be fine.

The drugs we're trialing have been tried by thousands to even hundreds of thousands of patients (Low dose naltrexone, for example). We know their pharmacology and we know what they target. It should not be difficult to come up with a hypothesis about what general types of patients will benefit.

Again, the assumption is that some types of patients benefit. We don't know if anyone benefits yet, but the four (+?) trials of LDN currently going on will clarify that.

Those hypotheses are built into the trials. They're documenting various potential predictive traits, and then they'll look at whether any of those traits predict a positive outcome or a negative outcome. But if the trial does not show a positive result at group level, those potential predictors are meaningless. Because the positive outcomes - the reported improvements - are due to something other than the drug. I think a lot of people are just not open to that fact, that they can start a drug or other intervention, improve, and the improvement was not due to the drug/intervention.

I think the use of the term "responder" is unhelpful in this regard. It reinforces the idea that people who improved did so because of the treatment.

 

[MinIreland]

You can't do that because then you can select all the responders, effectively ignoring the control and introduce your own bias into the results. This is the thing to understand, we have a noisy signal you can't just select the responders to find some random post hoc grouping.

ME/CFS has a high placebo bias in it, you have to choose who you are testing and what counts as a significant result from the outset or you introduce bias. If you get a signal but its only some people then what you hope is you captured why, if you didn't then you have to guess and do another trial. But if you get no signal then there isn't a response group to go hunting for, its just the random nature of the disease as shown by the control also doing the same.

Exactly. All you could do, IF the data show certain patterns or correlations for only a subgroup of the participants, is to then create new hypothesis based on these patterns/correlations and set up new research to test the new hypothesis.

 

[MinIreland]

The problem is the assumption that some patients will benefit. We don't know that yet. Lots of people think that, for example, LDN works for lots of people with ME/CFS, because they take it, and then some of them improve. But we don't know if LDN helps anyone until we have some controlled trials. Because that reported improvement can be caused by a lot of different things - spontaneous change, expectations, regression to the mean etc. The way you find out if more than a handful of patients benefit is through a trial. Once you establish that yes, something works, you can start figuring out who benefits and who doesn't.


I think it's the opposite. People are saying "LDN/other intervention doesn't even reach the lowest bar yet, so subgrouping doesn't make sense yet."

But everyone is saying that doing a trial of, for example, a GI drug just on people with ME/CFS who have GI symptoms would be fine.


Again, the assumption is that some types of patients benefit. We don't know if anyone benefits yet, but the four (+?) trials of LDN currently going on will clarify that.

Those hypotheses are built into the trials. They're documenting various potential predictive traits, and then they'll look at whether any of those traits predict a positive outcome or a negative outcome. But if the trial does not show a positive result at group level, those potential predictors are meaningless. Because the positive outcomes - the reported improvements - are due to something other than the drug. I think a lot of people are just not open to that fact, that they can start a drug or other intervention, improve, and the improvement was not due to the drug/intervention.

I think the use of the term "responder" is unhelpful in this regard. It reinforces the idea that people who improved did so because of the treatment.

Just a little side note about LDN (as I think what follows would apply to any other 'treatment' that patients feel works for them)... What I find interesting is that patients don't ever seem to consider that, *if* LDN works for them (eg improves certain symptoms), they might fall into the placebo group.

I use LDN and I *think* it helps with sleep, but when I saw the research, I thought... Oops, might be placebo. I'm too scared to change anything in my 'regime' cause I'm quite steady at the moment.

But anyway, my point is that patients seems to think they'd never be 'prone' to placebo.

 

[Kitty]

I think the use of the term "responder" is unhelpful in this regard. It reinforces the idea that people who improved did so because of the treatment.

Yup. Drugs that actually modify a disease tend to produce significant and fairly consistent effects.

It's fine thinking about what might work for symptomatic relief, but we really need something that addresses the underlying issue.

 

[Verity]

You said:

This is exactly what I’m concerned by, and I say that as someone who might be one of those misdiagnosed patients. It’s pretty clear to me at this point that some patients with autoimmune diseases have symptoms that look a lot like ME...

I didn’t “explicitly state” that I think I’m misdiagnosed; I’m just aware of the possibility. I also didn’t say I think I have an autoimmune disease but no diagnosis; I have biopsy-proven cutaneous autoimmune disease, but it’s unclear if it’s something more than that and related to my ME symptoms somehow. If you were talking about the post you quoted here, that was a mischaracterization of what I said. I think it is really important to make sure what you’re saying is accurate if you’re going to talk about another member’s health.

Anyway, let’s return to the topic of this thread. I just didn’t want to not clear this up.

(Edited because this was way too long, and I really don’t want to get in the weeds about my health)

 

[Utsikt]

> Until we get an understanding of actual pathology, there’s no reason to hope for any subgrouping.

Completely agreed with you until this part. If subgrouping provides any signal (and it will), it is useful and worth doing. I feel like the reigning mentality in this thread is "It's not perfect, so we can't do it!"

I think if you read the response carefully you’ll find that the mentality is «it’s fatally flawed so it shouldn’t be done».

The drugs we're trialing have been tried by thousands to even hundreds of thousands of patients (Low dose naltrexone, for example). We know their pharmacology and we know what they target. It should not be difficult to come up with a hypothesis about what general types of patients will benefit.

Then perhaps some of the docs that claim to be able to make LDN work should put their money where their mouth is and do a proper controlled trial instead of continuing to prescribe unproven treatments based on beliefs.

 

[Evergreen]

I use LDN and I *think* it helps with sleep, but when I saw the research, I thought... Oops, might be placebo. I'm too scared to change anything in my 'regime' cause I'm quite steady at the moment.

Really good that you're steady. Nacul's study looks at sleep as a secondary measure, so you will hear ultimately what they found on that score. For what it's worth, I took one 1mg tablet once and slept so much more than usual for a week!

 

[Kitty]

Then perhaps some of the docs that claim to be able to make LDN work should put their money where their mouth is and do a proper controlled trial

I imagine the money is exactly why they haven't.

Proving it doesn't work is a risk that could cost them hundreds of thousands in lost income, and they'd have to go to all the trouble of finding something else. Madness.

 

[neophyte32]

Hello,
sorry I'm a bit too sick to read and absorb all the comments, but what is the actual importance of subgroups for the MECFS?

I paid quite a lot to follow the Amatica method with RNA (my results are catastrophic, of course; all the systems are malfunctioning, and this corresponds to some of my symptoms, even my cholesterol blood test results—my lipid-related genes are very bad compared to the control group) because on Twitter, many scientists swear by subgroups...

So, to summarize, what is the real story? A kind soul with a knack for summarizing... Again, sorry...