A proposal for ME Action: a commitment to evidence-based medicine

[Trish]

The point that I still don't want us to lose is that a specific scientific finding has a determinable status, a level of quality or usefulness, for which there may be proxies, like consensus or citations. The idea that a finding has a status shouldn't be controversial. But it is the issue that is implicitly or explicitly avoided in many posts. It seems to me that there is a deep discomfort and aversion to the idea that a scientific point can have a status. That's my worry - it is an open question that shouldn't exist here.

Again, I'm still looking for agreement without dispute that a scientific idea or observation can have a status.... Agreement with that first point still seems to be up in the air.

I confess I'm getting a bit tangled up trying to follow your argument, but here are a few thoughts for you to pick apart if you wish. It is very likely that I have completely missed the point.

A scientific idea, like the BPS model for ME, has a 'status' that is built on consensus within a specific group of researchers. It has a 'status' that is built on number of research papers and citations. It has a 'status' that is built on usefulness to health funding authorities. It has a 'status' that is built on the eminence of some of the researchers.

But it is also a pile of crap.

It should have very low scientific status because the research methods are so poor and the findings so weak.
............

'Status' if you want to call it that for research on basic science related to ME, such as metabolomics, should be built on robustness of findings, replicability, size of studies, etc. The 2 day CPET findings seem to be reaching that level with several centres publishing results of studies. I can't think of any other biomedical findings that are both robust and sufficiently replicated yet.

And 'status' of research on treatments should be based on the level of research evidence available,
- starting at the lowest level with anecdote: individual stories or small case studies, in which low status I would include ME specialists using what 'seems to help' their patients;
- working up through unblinded pilot studies which are only useful as an indicator that more research might be warranted;
- all the way to large scale multi centre double blinded placebo controlled trials with clear and clinically significant robust findings having the highest status.
.........

I think the problem so far with ME research is that almost all done so far is quite low status, largely because, due to lack of funding, studies have been too small and haven't been replicated, and clinical trials of treatments have only been at pilot study level if that, apart from Rituximab which didn't work.

 

[JaneL]

The point that I still don't want us to lose is that a specific scientific finding has a determinable status, a level of quality or usefulness, for which there may be proxies, like consensus or citations. The idea that a finding has a status shouldn't be controversial.

Perhaps you are thinking of something like this as far as medical evidence is concerned?



Is this what you mean by status?

But, we can't brush over that LDN, CCI or bananas have a determinable level of scientific insight available that offer a determinable status.

I’m not sure where LDN and CCI fit in terms of the above hierarchy but perhaps someone more knowledgeable than me will confirm (I’d guess near or at the bottom of the pyramid?).

 

[InfiniteRubix]

But it is also a pile of crap.

"Status" is an example of double meanings through pwME glasses, worrying that status should not equal eminence - a reasonable concern.

This is why my point is so critical. The meanings of words or concepts are not always the same to 1) a new ME researcher or a third party vs 2) loaded meanings for a pwME community steeped in injustice and rightly paranoid about the past. It allows some of us, and we might all do it sometimes or one day, to forget that scientific principles don't care about our history. History is relevant, but it is social and political, not scientific. Any word can be misinterpreted, but there will also always be some people who just don't care enough about scientific process, regardless of misinterpretation.

So I'll use your word, "crap". It's a very good one to define the spirit tbh and cut though the crap . Every scientific finding has a status and that spectrum could range from:

• Crap vs Not crap
  
• Worth further confirming its not crap
• Just an idea that needs confirmation that it's not crap

That's the principle that is being placed in doubt that is worrying me.

Also, if nearly all ME research is stuck at the "just an idea" end, so be it. We need to know this. As patients. As doctors. As funders. Etc.

 

[InfiniteRubix]

Perhaps you are thinking of something like this as far as medical evidence is concerned?

View attachment 9182

Is this what you mean by status?



I’m not sure where LDN and CCI fit in terms of the above hierarchy but perhaps someone more knowledgeable than me will confirm (I’d guess near or at the bottom of the pyramid?).

Exactly! Whatever the version or choice of terms /granularity, it is the spirit of this hierarchy that is being placed into doubt so often. That is why I have become so worried about what has been happening.

I agree that LDN, CCI or most things are not going to be high up that pyramid. I am not sure where banana fits in the list But, that's a second order issue, which I will defer to others and contribute to as and when I have a thought. But not in this thread.

Another observation, this hierarchy is for scientific findings overall, of which treatments are just a part, and would also include observed phenomena, etc. All findings are subject to some version of this hierarchy, whether we like it or not.

 

[InfiniteRubix]

"Status" is an example of double meanings through pwME glasses, worrying that status should not equal eminence - a reasonable concern.

This is why my point is so critical. The meanings of words or concepts are not always the same to 1) a new ME researcher or a third party vs 2) loaded meanings for a pwME community steeped in injustice and rightly paranoid about the past. It allows some of us, and we might all do it sometimes or one day, to forget that scientific principles don't care about our history. History is relevant, but it is social and political, not scientific. Any word can be misinterpreted, but there will also always be some people who just don't care enough about scientific process, regardless of misinterpretation.

So I'll use your word, "crap". It's a very good one to define the spirit tbh and cut though the crap . Every scientific finding has a status and that spectrum could range from:

• Crap vs Not crap
  
• Worth further confirming its not crap
• Just an idea that needs confirmation that it's not crap

That's the principle that is being placed in doubt that is worrying me.

Also, if nearly all ME research is stuck at the "just an idea" end, so be it. We need to know this. As patients. As doctors. As funders. Etc.

Perhaps you are thinking of something like this as far as medical evidence is concerned?

View attachment 9182

Is this what you mean by status?



I’m not sure where LDN and CCI fit in terms of the above hierarchy but perhaps someone more knowledgeable than me will confirm (I’d guess near or at the bottom of the pyramid?).

Exactly! Whatever the version or choice of terms /granularity, it is the spirit of this hierarchy that is being placed into doubt so often. That is why I have become so worried about what has been happening.

I agree that LDN, CCI or most things are not going to be high up that pyramid. I am not sure where banana fits in the list But, that's a second order issue, which I will defer to others and contribute to as and when I have a thought. But not in this thread.

Another observation, this hierarchy is for scientific findings overall, of which treatments are just a part, and would also include observed phenomena, etc. All findings are subject to some version of this hierarchy, whether we like it or not.

Does anyone still have any discomfort with this?

Is this issue still in the air?

It goes to the heart of CCI and MEpedia conversations, S4ME credibility and @Michiel Tack 's letter. If we don't all agree, it needs urgent clarification IMO.

 

[Trish]

I think the diagrams @JaneL posted are a great succinct summary of different levels of evidence. They may form a useful guide for MEAction and MEPedia.

But there is a big note of caution. The top of the diagram, systematic reviews, is only the highest level of evidence if it the evidence lower down the pyramid is strong. Otherwise the pyramid crumbles into a pile of crap, as we have seen with Cochrane reviews for CBT/GET with ME.

 

[InfiniteRubix]

I'd also ask @Jonathan Edwards or anyone else who should be tagged from the community (please tag!) to ensure that this position is expressing a fundamentally boring and banal point without holes.

If it has holes, they need filling. But the banal shouldn't be as disputable as it has been.

 

[InfiniteRubix]

I think the diagrams @JaneL posted are a great succinct summary of different levels of evidence. They may form a useful guide for MEAction and MEPedia.

But there is a big note of caution. The top of the diagram, systematic reviews, is only the hightest level of evidence if it the evidence lower down the pyramid is strong. Otherwise the pyramid crumbles into a pile of crap, as we have seen with Cochrane reviews for CBT/GET with ME.

The most useful pyramid needs defining. Maybe with less detail for this purpose. But the principle is remains.

 

[InfiniteRubix]

Put otherwise, which pyramid is the second order question. Too many posts seemed to dispute there was any pyramid at all. That is the entirety of my worry.

The most useful pyramid needs defining. Maybe with less detail for this purpose. But the principle is remains.

 

[Diluted-biscuit]

That pyramid needs to be shared a lot more!

 

[Hoopoe]

I think the pyramid is a dumb idea because it makes people think there is an easy method to determine what is reliable and what is not.

It reduces critical thinking when more critical thinking is needed.

The reliability of evidence depends on many things, not just a classification. There is probably also an arms race phenomenon whereby the people wishing to promote their evidence as reliable are racing against the people trying to suppress false claims. Due to this arms race it will be difficult to ever come up with a simple set of rules that determine whether something should be considered reliable or not.

The pyramid is at least useful to convey the idea that there are varying levels of reliability.

 

[JES]

Breaking down the pyramid, in terms of ME/CFS treatments:

1. Meta-analyses & systematic reviews: none on ME/CFS drugs.
2. Randomized controlled trials: should probably split further into phase 1, 2 or 3 trials. Some phase 1 trials, nothing close to phase 3 after the rituximab failure.
3-5. Cohort, case-control and cross sectional studies: on drugs used by ME/CFS patients? None that I'm aware of.
6. Animal trials & in vitro studies: some drugs tested on animal model of ME/CFS, likely of no use in humans. In vitro studies? Lots done against pathogens, also likely not of that much use.
7. Case reports, opinion papers, and letters: yeah, those exist.

Based on this, I would agree the pyramid is right now not too useful in terms of assessing ME/CFS treatments since most of the current evidence lands in the bottom two sections and nothing immediate is happening further up in the pyramid. But the point of @InfiniteRubix is important I reckon.

 

[InfiniteRubix]

I think the pyramid is a dumb idea because it makes people think there is an easy method to determine what is reliable and what is not.

It reduces critical thinking when more critical thinking is needed.

The reliability of evidence depends on many things, not just a classification. There is probably also an arms race phenomenon whereby the people wishing to promote their evidence as reliable are racing against the people trying to suppress false claims. Due to this arms race it will be difficult to ever come up with a simple set of rules that determine whether something should be considered reliable or not.

The pyramid is at least useful to convey the idea that there are varying levels of reliability.

Varying levels of reliability exist, agreed. But that is an outcome of research. Variability is a finding, or axiomatic observation at a minimum. But it's not the process.

I totally agree that pyramid might not be the best list of stages to use for a process - I defer to others about which pyramid. But a logical progression/regression of stages for a finding has to be part of our general understanding. Otherwise our contributions are not implementable at scale institutionally or commercially. They will require others to come in, not to sanity test our assumptions, but to actually provide some.

If you are deciding which treatments you want to recommend. If you're deciding which research projects you are going to invest in. If you're deciding which pharmaceuticals are a good bet commercially for R&D. Any of those scenarios require some decision to be made. If the decision isn't made systemically, with a framework of progression, with critical thinking in that process of course, how else is it to be made? It can only be made on a subjective or political basis otherwise. Policy institutions and businesses don't work that way officially, so that means that others would reinjext that discipline for us, in private.

I believe that discipline would be great to come from here, but perhaps I am being too aspirational. But, at a minimum, the idea that a research idea has a journey and can be compared or prioritised shouldn't be revolutionary. It's so banal to me when considering how to make things actionable and ensure things don't go off the rails. It's how things work.....

This is always been my worry, because I'm not just coming at it from a perspective of scientific process and probity. But also from a policy perspective. And also from a commercial perspective.

 

[Trish]

I think the pyramid is a dumb idea because it makes people think there is an easy method to determine what is reliable and what is not.

It reduces critical thinking when more critical thinking is needed.

I agree it reduces critical thinking. The Cochrane GET review is the prime example of that. Theoretically at the top of the pyramid but actually built on sand.

The reason I think the diagram is useful in looking at ME clinical trials is to demonstrate just how LOW on the pyramid everything is, apart from Rituximab.

All the other supposed 'treatments' for ME, including ones recommended by so called ME expert doctors, either fail to even reach the bottom of the pyramid and are still in the 'no evidence' circle, or are on one of the lowest levels.

 

[Louie41]

I am not a scientist; I'm a long term (29 years) pwME. I think this discussion is critically important.

When I first joined s4ME in October 2017, I felt confident that I was getting good information. That appears to have changed over the past year. Now it seems that many people are talking as if snake oil will cure us, and science based information has no more value to many commenters than personal anecdote.

One of the reasons I joined s4ME was because other fora seemed not to value science. I hope s4ME will continue to be a place where robust science-based examination of ideas will continue, and where pwME can feel confident in the information they gain on the site.

 

[Guest 2176]

I really don't think it's fair to say that other fora dont value science period. It's also an inflammatory statement. I also don't think its true.
Most people have a poor estimation of how rational and empirical they are, and cognitive bias/error is basically part of being human. So when I see any community or person make claims to be more scientific or rational than other communities I take those claims with a grain of salt.

I would say that s4me has more of a parsimonious focus on peer reviewed literature than other forums. I don't think that necessarily means its more scientific. Observation, speculation, and self experiment are all part of the scientific process. I have been frustrated sometimes about how on some other forums theres less vigorous and skeptical discussion of science than here, but I think all these forums have something to offer and having major splits between the user base in one and the other leads to less productive dialogue. Just my 2c

 

[InfiniteRubix]

This thread is worrying me profoundly - even more than it was. I am trying very hard to word that 'appropriately'. I'm worried enough to use energy that I don't have for some days now, going round in circles to make the same point in a myriad of ways in the hope that one sticks. I see it as critically important for our futures and for any of our congenital successors. Basic boring scientific 'first principles' are in doubt. This site is unique - and instead of this specific thread encouraging best principles elsewhere, it seems like the opposite is happening reverse. I would rhetorically ask why.

Humans are biased and subjective creatures. I've been trying to show that this is the problem that we are meant to be controlling - and doing it with scientific principles. The reverse seems to be happening... using the problem to undermine scientific principles. The idea that empiricism is an aspiration is not a reason to abandon empiricism.

Some cultures have the very everyday phrase of "mental masturbation". I worry that this thread is distracting from it's point, @Michiel Tack 's important letter, perhaps 'successfully'.

Nobody is telling me that I am misunderstanding what's happening here - I'd love to be wrong.

 

[InfiniteRubix]

I am not a scientist; I'm a long term (29 years) pwME. I think this discussion is critically important.

When I first joined s4ME in October 2017, I felt confident that I was getting good information. That appears to have changed over the past year. Now it seems that many people are talking as if snake oil will cure us, and science based information has no more value to many commenters than personal anecdote.

One of the reasons I joined s4ME was because other fora seemed not to value science. I hope s4ME will continue to be a place where robust science-based examination of ideas will continue, and where pwME can feel confident in the information they gain on the site.

This.

 

[Guest 2176]

Basic boring scientific 'first principles' are in doubt.

If one is to apply a level of rigor in critiquing scientific papers one should apply the same level of rigor to critiquing scientific principles--the result should end up being similar in terms of the rigor and criticality helping to greatly clarify things.

 

[Guest 2176]

If one is to apply a level of rigor in critiquing scientific papers one should apply the same level of rigor to critiquing scientific principles--the result should end up being similar in terms of the rigor and criticality helping to greatly clarify things.

Someone said on this board that the constant criticality and negativity that is part of the scientific process may be upsetting to people. I would say maybe the same is true about criticality in philosophy of science.