Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[Adrian]

I agree that it can have dreadful consequences for something that requires diversity of thought and will affect a lot of people. That's why we didn't do it that way. In case anyone's wondering: I am pretty sure I chatted with David Nunan briefly socially at a conference once. I interviewed people who reached the stage of being prospective consumer authors, but had never met any of them prior. I have never met any of the other authors (with apologies if they ever came up and said hello and I don't remember).

So perhaps you can describe the recruitment process a bit more? Prior to interviewing what did the skills and marking matrix look like? Who else was on the interview panel, what additional checks were done (for example prior to interviewing people I always try to read some papers or other material (and even look at open source proects in github!)). What style of interview (standard questions in which case share, technical presentations etc). How follow up interviews happened to deal with any potential issues or check skills areas that didn't get adaquately tested during the panel?

 

[MSEsperanza]

So Henschke was is the Cochrane systematic reviewer that is part of the ME/CFS writing team and, although he isn't personally replying to the back pain review commentator, he appears first on the list of contributors to the back pain review. The person replying to the comment appears to speak for all of the authors. And he just does not seem to understand the issue.

Apologies for not checking / mentioning the dates:

The comment is from 2000, Henschke joined the author team in 2010--

now edited my post to point to the dates:

didn't check the revisions so no idea how the author team from 2010 (when Henschke joined them) responded to it, also don't know Henschke's current stance on that.

As the reply to the comment still is linked to the review and no reply added, it still appears to be valid for the current review, though.

Again, apologies for causing extra confusion.

 

[Adrian]

I’m a bit puzzled by this. Why does it matter to engender confidence in members of the Cochrane community? Perhaps I am misunderstanding what you’re saying but are we not in agreement that the “Cochrane community” has completely failed people with ME/CFS by its inability and/or unwillingness to adequately address the issues raised by Robert Courtney, Tom Kindlon, George Faulkner and others for so long?

As I see it, if you and your team do a good job a lot of people in the Cochrane community are going to be exposed for having overseen multiple failures.

They have not just failed people with ME they have demonstrated a lack of competance in backing a review that was not up to any standard and demonstrated they don't care about accuracy by leaving that review on the books for years dispite knowing the problems. Its a systemic issue with Cochrane not just an issue with a ME/CFS review

 

[MSEsperanza]

Comment from 2000 on the review Behavioural treatment for chronic low‐back pain quoted above -

see my addition here.

I agree the editor's reply to the comment is appalling.

I think it would be good if forum members could collect some points about the review authors' work that we would like to discuss and of which it would be good to know the authors' current stance.

Editedt to add: I realize that's what we're doing already.

It would be much easier for me if we could do that in a structured manner, e.g. on a subforum or similar. Also understand if people prefer to collect and discuss in the current thread -- just won't be able to contribute much or maybe not at all in that case.

(Edited to replace quote of deleted message.)

 

[Invisible Woman]

So what do you do when people are deeply entrenched in opposed camps of people who have convinced each other without any doubt that they are right, and they each have an influence on the ground, where everyday decisions and actions are happening. Something has to shift, including in people's minds - and not just in one country. I don't believe you get there by people not talking to each other, not listening to each other, and not being seen to do so.

Some people aren't going to be convinced of something if they can't see that their point of view has been fairly and competently considered and clearly addressed. And it shouldn't be made easy for those who will just work to discredit any process that doesn't come to the conclusion they want. Critically, for the huge numbers of people who aren't in either camp but are hearing both sides, they need to be able to see for themselves that the attempts to discredit a document aren't valid.

I'm still catching up with the thread, so please forgive me if this has been raised.

Whose position is entrenched, here?

From where I'm standing it's not ours. I don't really care whether ME is eventually classed as a psychiatric, psychological, behavioural, autoimmune, or whatever disease.

Most of us here have already faced some fairly extreme discrimination and lost an awful lot of the things that mattered to us and everyone else takes for granted in life.

I don't see how asking, repeatedly, for as often and as long as is necessary, for high quality research with good sound methodology and results presented honestly and without spin.

as a community we at least want effective treatment, while we wait and support researchers working for a cure. That treatment doesn't exist at the moment because of the over reliance on spun data from research with very poor methodology.

This narrative of entrenched views and opposing camps is an extension of the unreasonable, militant activist narrative.

As a patient group we have time and again raised valid objections and queries on the science of studies. Even when our input is acknowledged as valid authors of papers have simply been able to decline to make corrections or withdraw their paper.

You make it sound as though there are two opposing and equally powerful forces with your team in the middle here. That is simply not the case.

 

[MSEsperanza]

It would be much easier for me if we could do that in a structured manner, e.g. on a subforum or similar.

Am too slow to catch up with the thread and wording my points takes too much time so apologies for my odd posting behaviour.

When re-reading through the therad a bit there are so many points others and I have made already and I think still apply.

On the other hand, ist seems that the links to reviews and comments I posted last year and re-posted today had been missed by most people back then.

That's why I think a subforum would make sense.

It would also be much easier for me if we had the space to post Hilda's spreadsheets on a therad to check which criticism the IAG is already aware of and to which we could refer if we wanted to ask the authors on their current stance on those issues.

 

[Jonathan Edwards]

Which is surely an argument to not unconditionally ignore trials with subjective outcomes in unblinded trials, but to assess whether those outcomes may have some value in the context you speak of, even if that value is to show they are not reliable.

This is a valid argument Hilda raised some time ago.

But the other side of it is this: There is no point in doing a systematic review of evidence of efficacy of a treatment unless you think you may find some reliable evidence of efficacy. If you don't find any there is no reason to suggest the therapy is used. If all the studies available fall below a minimum requirement for reliability, as for exercise for ME, then looking at them for other interesting bits of information seems pointless.

The again, it would be reasonable to say that you might want to do a systematic review, particularly for a treatment that has got itself into routine use without proper testing as non-pharmacological treatments tend to, in order to raise the possibility that you might be wanting to report that people should stop using the treatment because it causes harm. I guess that for vaccines and supplements and a variety of other things a systematic review might have detection of harm as its primary purpose. But in those cases it is usually an issue of raising harms in the context of reasonably likely or accepted benefits. For exercise for ME no such likely or accepted benefit is known. Moreover, the most usual route to documenting a problem with harm is to gather open data first and if it is worrying to look at it in a formal study, if that is justified by reliable evidence of benefit.

 

[Jonathan Edwards]

This narrative of entrenched views and opposing camps is an extension of the unreasonable, militant activist narrative.

Yes, I am rather hoping that Hilda realises she should never have raised that!

 

[Hutan]

I've had a look at some previous reviews of people on the team, and in order to accept the criticisms of PACE, they would need to consider things which they don't seem to usually consider when reviewing evidence. But I'm not sure how possible it would be for there to be (mostly) people on the team who do usually consider the kinds of criticisms made against PACE.

Why not? Research methodologists who deal mostly with drug interventions rather than behavioural interventions would probably understand the issues with PACE -- surely Cochrane have some. Expert patients who understand the methodological issues would understand the issues with PACE very well. I don't see any difficulty in having people who can deal with the sorts of issues in PACE apart from having a willingness to have those people in the team.

None of the people on the author team seem to have outright conflicts of interest, and there doesn't seem to be anything to suggest they're ideologically opposed to criticisms of GET trials or to criticisms to the universal utility of exercise, so there doesn't seem to be much reason to believe they'll outright reject those criticisms.

The Cochrane reviewers have an interest in not criticising methods that they've marked as acceptable on other reviews.

Julia Newton has an interest in not criticising methods she has used, or assumptions that she has made, in her research. Have you read much of Dr Newton's work? I actually argued before that most people understand that exercise doesn't fix everything, but Newton's paper is pretty strongly pro-exercise. I suggest there is a bit there to suggest that she is ideologically opposed to criticism of the universal utility of exercise.

It is vital that those with CFS are encouraged to increase their habitual physical activity, as this is a significant environmental factor influencing physical fitness and well-being. Evidence suggests that physical activity helps maintain physical function, delays the onset of disability, reduces mortality and extends life expectancy in normal ageing.14–16 In other fatigue associated diseases (multiple sclerosis), those who participate in regular physical exercise report better fatigue, depression and quality of life scores.

As noted below about Kay, we don't have enough information to know if she has conflicts of interest. She lists 'Navy' as a conflict of interest, so presumably she sees some potential for conflict there.

The specific inclusion of a "non-activist" patient picked for differing experiences with exercise on the author team is unusual and a bit strange, and I assume that has something to do with cochrane wanting to maintain the pretty significant amount of credibility and respect it has among people who might assume the review is pandering to "activists". I agree with other people that that's weird and not a very good look for cochrane, and may indicate some questionable views in cochrane. But Kay does seem like a fairly good pick, regardless.

How can you tell that Kay is a 'fairly good pick'? What do you know about her skills in research methodology, or explaining to others the issues with GET trials, or standing up to other members of the panel about the problems with BPS methodology? How can you be sure that she does not have an association of some sort with BPS people? How can you be sure that Kay doesn't attribute her 'high-functioning status' to the fact that she does yoga and keeps active, and thinks that people with lower levels of function need some help in the form of GET to get motivated and active too? Of course I'm speculating and she probably doesn't, but I don't think we know.

Again, apologies for causing extra confusion.

I've deleted my post about Henschke as it didn't seem quite fair given the issue with timing.

 

[Jonathan Edwards]

Technically speaking I think the philosophy behind combined use of objective and subjective measures for arthritis is that the subjective measures of pain, stiffness, swelling and tenderness (subjective either to the patient or the assessor) are primary but need to be corroborated by objective measures to ensure they have a high probability of being reliable indicators of something a treatment might be modifying.

The situation is very complex. Even the combined outcome measures are not reliable in open studies. They are largely used in blinded studies in order to guard against rather different problems from that of suggestibility, placebo effect, collusion etc. They may be more relevant to things like regression to mean and systematic shifts with time unrelated to treatment.

 

[Hilda Bastian]

Yes, I know it's on the list. That wasn't my question. I think I suggested at some point that Vink and Vink-Niese be invited to Cochranize their study to convert it into a Cochrane review. I think there are precendents for this. But of course Vink and Vink-Niese would have to agree which is by no means a given. Would that also be considered once you and the IAG and the author team have read and analysed the NICE review, the Vink Vink-Niese review and any other reviews of the evidence for exercise that already exist? The Cochrane process seems to be a process of re-inventing the wheel with many more hands on deck and potential spanners in the works (excuse shameless mixing of metaphors) than is necessary to make the simple point that the evidence that exercise is safe for people with ME/CFS and also has a useful positive effect (if objective outcomes measures are considered) is non-existent.

I was answering your first question: if it's on the list, that's the answer to the question of has it been analyzed, because everything on the list has been analyzed.

I didn't reply to the other question, because to repeat again that there will be a new protocol, prepared by the author team as announced, is so clearly redundant: that it means it won't be the one from the CDC or NICE review or any other review that already exists (including the current Cochrane review). (Vink and Vink-Niese, while a very thorough critique of the Cochrane review and re-analysis of studies, is not itself a systematic review and doesn't claim to be - no search strategy, specified methods, etc.)

 

[Jonathan Edwards]

But I'm not sure how possible it would be for there to be (mostly) people on the team who do usually consider the kinds of criticisms made against PACE.

I agree with Hutan. In fact I would go much further. Anyone not fully conversant with the reasons why PACE is no good is not competent to do this review. You don't ask people who do not understand plumbing to service a central heating system. I could muster up ten people who are conversant straight away - members of my old academic rheumatology department, including trainees.

This stuff is absolutely basic, which is why it is so shocking to hear founder members of Cochrane like Chalmers, Gotzsche, Garner or Glasziou coming out with the remarks they do.

 

[Caroline Struthers]

I was answering your first question: if it's on the list, that's the answer to the question of has it been analyzed, because everything on the list has been analyzed.

I didn't reply to the other question, because to repeat again that there will be a new protocol, prepared by the author team as announced, is so clearly redundant: that it means it won't be the one from the CDC or NICE review or any other review that already exists (including the current Cochrane review). (Vink and Vink-Niese, while a very thorough critique of the Cochrane review and re-analysis of studies, is not itself a systematic review and doesn't claim to be - no search strategy, specified methods, etc.)

Oh I didn't realise. It's only the NICE review that has not yet been analysed.

 

[Hutan]

This is a valid argument Hilda raised some time ago.

Yes, we are covering old ground on this. Of course there can be valid reasons for having subjective outcomes in a trial; having some is not a disqualifying feature of a good trial. And applying some thought to what people have measured is always a good move. I don't think anyone ever thought otherwise. We do have to use a short label when referring to the issue.

But, outcomes of surveys of fatigue or activity levels or overall wellness in GET trials aren't reliable indicators of a treatment benefit.

That's why I think a subforum would make sense.

MSESpe - Yes, a subforum could be good. The moderation team are working on what set up could help.

Anyone not fully conversant with the reasons why PACE is no good is not competent to do this review.

Yes. They are allowed a bit of time to look at it if they haven't seen it before , but if they don't have the capacity to understand the problems with it, then they should not be on the writing team.

 

[Robert 1973]

So perhaps you can describe the recruitment process a bit more? Prior to interviewing what did the skills and marking matrix look like? Who else was on the interview panel, what additional checks were done (for example prior to interviewing people I always try to read some papers or other material (and even look at open source proects in github!)). What style of interview (standard questions in which case share, technical presentations etc). How follow up interviews happened to deal with any potential issues or check skills areas that didn't get adaquately tested during the panel?

I’m a bit puzzled why none of this was included in monthly progress reports from March 2020, as decisions were being made. That is what I had assumed would happen. That way, we could have scrutinised these decisions and processes before people were appointed.

When we heard nothing for 14+ months I assumed that work had stalled. Instead, it appears that a great deal of work was going on but for some reason we weren’t being told about it. Please can you clarify why that was, Hilda?

 

[ukxmrv]

Thank you Hilda Bastien for engaging so thoroughly in this process and talking to patients here.

Last night I went to sleep dreaming I was in a Grisham courtroom drama. We have a trial "the evidence" and a jury needs to be selected. In true Grisham style we get the story of how important it is to pick the right jury and how trading and shenanigans (favourite forum word) go on behind the scenes to ensure that the right duty is selected for whatever outcome is desired by both sides.

Cochrane have gone one better by engaging you, Hilda as the "front". A champion of a certain style of consumer rights and a formidable person with lots of guts and experience. Hilda, I would like to suggest that maybe it is this old fashioned style of consumer rights dispute / evidence analysis that could potentially be the problem here. I could be wrong. However, I think it needs to be mentioned as a possibility.

We have seen a succession of Inquiries here in the UK into a range of different concerns where evidence needs to be heard and assessed. The pattern is for the first Inquiry to be controlled by the Establishment and to return a verdict that leaves people injured or affected unhappy. Then over time the Inquires become more and more focussed on the rights and the interests of the people affected as the original establishment is allowed to retire, die to disappear.

With Cochrane we are at the first stage here.

Cochrane have bought the type of court the case will be held in (the use the Grisham court room analogy), they selected someone with a firm view of how it should be done (which appears to be outmoded and wrong at least to UK sensibilities) and the jury is stacked with the wrong people.

It's no longer acceptable in England for people affected to be labelled as "activists" and for their input to be disregarded for that reason. It's recognised that unless patient interests are not being heard and addressed the problem doesn't go away. It is recognised that people affected by an issue deserve to be heard. It's no longer acceptable to put a group of people together with patient representatives that don't reflect the views and an understanding of the issues that brought this Inquiry together. It is recognised that it is normal for anyone severely affected by an issue to be informed and to speak out.

A brief look at Kay's internet presence should have raised some red flags that said wrong person and deeply "Establishment"

https://civilservice.blog.gov.uk/2019/05/03/personal-disability-stories-10-kays-story/

1. In this part she speaks about her experience of being mildly affected and exercise "Some types of exercise can be detrimental if not done the right way – I am, by necessity, more yoga than parkrun!"

The pro GET lobby use the same language. It's "exercise" not done in the "right way". Wrong person.

2. Here she says "I have always had exceptional support through Defence Medical Services.".

So what is this exceptional support on offer that the rest of us can only dream about. A ME patient would want to know. Are they truly "exceptional" offering things that are different to the original NICE Guideline and is so, why don't we know about them and have access to them.

The answer is of course that it's the same crap the rest of us are getting with more employer type support. If she is happy with her medical care then she will not understand the battle with NICE and with Cochrane that the rest of us are fighting. Wrong person.

3. She makes non-evidence based attributes to her own health problems here "Initially, I reacted physically to the increased cortisol levels firing round my body, and I ended up in an early menopause, aged 41."

We have had threads on the forum that look at the claims for cortisol and what evidence there is. Kay has obviously not seen these threads or read any medical papers on the topic. It's pure behavioural BS i.e. stress = physical problem.

Who told her this and why does she believe them? Wrong person.

4. I've also had a look through her Twitter feed and seen "likes" for particular views that are controversial areas for people with ME and pain. Personally I would not have "liked" them and I would have recognised where they come from. Wrong person.

Hilda, you should have seen all of this and recognised it.

It's looks like a disaster and deeply unprofessional, patronising and well out of date with attitudes and methods.

 

[Adrian]

I’m a bit puzzled why none of this was included in monthly progress reports from March 2020, as decisions were being made. That is what I had assumed would happen. That way, we could have scrutinised these decisions and processes before people were appointed.

When we heard nothing for 14+ months I assumed that work had stalled. Instead, it appears that a great deal of work was going on but for some reason we weren’t being told about it. Please can you clarify why that was, Hilda?

My belief is there wasn't a formal process just Hilda talking with a few people thinking they might help or I don't like them. But this is why being really transparent about approach and process is so important. And yes if the skills required and process for recruitment was published prior to recruitment then that would have allowed people to give feedback about what is needed.

 

[Adrian]

You don't ask people who do not understand plumbing to service a central heating system.

Given it is gas I suspect doing so would be illegal in the UK.

This stuff is absolutely basic, which is why it is so shocking to hear founder members of Cochrane like Chalmers, Gotzsche, Garner or Glasziou coming out with the remarks they do.

Unlike handling devices with gas those doing reviews and giving medical advice don't need to demonstrate competence.

 

[Mithriel]

Something has to shift, including in people's minds - and not just in one country. I don't believe you get there by people not talking to each other, not listening to each other, and not being seen to do so.

Nothing, no type of evidence, demonstration of error in any part of the process of research, patient experience, expert testimony, committee reports, judicial judgements, anything you can think of has changed the minds of the BPS people.

They respond to everything by making statements which are untrue, like claiming patients are terrorists then using their eminence to have special branch keep tabs on ME "activists" (that must have been boring!), making out everyone criticising their work is anti science (A committee set up by Wessley gave him an award for standing up for science against the awful ME patients).

They also use the law against patient organisations and parents who do not want BPS treatment for their children. Again, I could go on as I am trying to summarize forty years of our history.

When the PACE trial was laid bare in it's methodological, misleading awfulness they ... well they just ignored it. They just say the results showed GET and CBT are evidence based treatments for ME.

When they thought Cochrane were going to react to the problems with the last review they released a statement through Reuters stating that Cochrane had caved in to threats from patients. For the next week and every time it is revisited there are discussions of how dreadful it is that patients (small groups of activist ones of course) can influence what solid findings are accepted with heartfelt sorrow for those poor patients who won't be able to get proper treatment if they win.

Oh, and they have quiet words with editors of journals and scientists find their head of departments have had complaints.

What they never, ever do is answer the criticisms, though they claim they have done so to anyone who will listen.

If this review finds that exercise is not shown to be useful for ME I will be shocked if there is not a resurgence of newspaper reports of terrorist patients and handwringing of how far Cochrane has fallen.

 

[Adrian]

This is a valid argument Hilda raised some time ago.

But the other side of it is this: There is no point in doing a systematic review of evidence of efficacy of a treatment unless you think you may find some reliable evidence of efficacy. If you don't find any there is no reason to suggest the therapy is used. If all the studies available fall below a minimum requirement for reliability, as for exercise for ME, then looking at them for other interesting bits of information seems pointless.

Shouldn't an early stage of any review be to look at the reliability of the evidence in each study found. Looking at the strength of controls and how they may interact with the measurement system (i.e. looking for things like do the different trial arms control for the right things; would patients see treatments as equal (and blinding comes in here) and whether the measures taken will be impacted by the issues in comparing the different trial arms (which could be subjective outcomes but also other issues with even more objective measures). I realize to analyse a trial in sufficient detail to really understand the issues is a big task but it feels necessary. But I get the impression that systematic reviews don't go as far as really assessing the full data just combining things with a rough quality judgement. Maybe that would work if reports were totally honest and peer review good but that is not the case.

To my mind any protocol should specify a minimum standard for the acceptability of evidence in terms of its reliability when taking account of possible trial issues. Then the review may say 'we found no trials of sufficient standard'. I guess there is another thing that can be done of looking of secondary outcomes that may be more reliable - but these need vetting for example, it was only after analysis from patients that the PACE group admitted that they didn't do the 6mwt correctly and they are still cherry picking results in that they haven't published the step test (secondary outcome).