Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[Trish]

I assume that if a member of the appointed writing team think the review the rest of the team want to write is scientifically unsound, they can withdraw from the process and not have their names on the published review.

Similarly if the members of the IAG find themselves not heard and unable to persuade the writing team to do the job properly, they could withdraw their names, or even write a separate review.

The fact that I am contemplating this possibility, and that some of us are so concerned that the review team is unlikely to be up to the job that we agree with Hutan's suggestion that the process should be stopped, is very sad, but unsurprising.

 

[Suffolkres]

Yes, I know you are only going to be looking at the exercise stuff. Are you going to check the GRADE ratings - ie. to check if you agree that the NICE reviewers' grading of low and very low quality is correct? Are the Cochrane author team going to be looking at the NICE review as well before they write the new protocol?

https://mecfsskeptic.com/nice-guideline-committee-and-grade-methodology/

'.....Another recognizable name is Signe Agnes Flottorp, Research director of the Norwegian Institute of Public Health. Flottorp recently was at the center of controversy when she defended research into the Lightning Process, a non-medical training program that combines concepts from Neuro-Linguistic Programming, Life Coaching, and Osteopathy. Critics have denounced the Lightning Process as pseudoscience or quackery.

Other authors of the rapid response are known for their leadership in evidence-based medicine. The most important name in this regard is Canadian Professor Gordon Guyatt. Guyatt was one of the authors of the GRADE handbook....'

 

[Hilda Bastian]

I find this quite a problem, thank you for highlighting these connections Hilda.

Dr Newton is certainly not the first problematic researcher to be associated with patient charities. This is partly why we find ourselves where we are, although slowly things are improving. When we had so few researchers working on ME/CFS and there was less scrutiny of the methodology of their research I guess it was easy to be grateful for anyone who did any biological-sounding research at all.

I think those two organisations urgently need to review Newton's body of work and consider if her association with them is helpful in achieving their aims. I can certainly understand that the effective endorsement her work by these two organisations could have led Hilda to assume Dr Newton would be an acceptable choice to ME/CFS patients and not look deeper into her work. It's really unfortunate.

The credibility of those organisations might be damaged if the Cochrane exercise review ends up being anything less than a complete rejection of the BPS ME/CFS exercise trials as useful evidence. And, with the current composition of the writing team, where people like Dr Newton are effectively reviewing their own flawed work and biases, and the sole patient representative was selected largely on the basis that she has not expressed any concerns about ME/CFS exercise trials, I think we can be very sure that the Cochrane exercise review will not do what it needs to.

Hilda, I'm left with the feeling that this review process is heading towards disaster. It is highly likely to end up eroding Cochrane's credibility further, promulgating harmful stereotypes of people with ME/CFS and causing collateral damage to organisations like AfME and MERUK. I understand you may feel that the process is now set in stone, but it has become clear that the bias in Cochrane runs too deep to run a useful review. If you really want to make things better for people with ME/CFS, the approach urgently needs to be rethought.

In my previous career, I did quite a bit of work with organisations on reputational management. If I was advising Cochrane right now, I would be agreeing with @Caroline Struthers and others that the smartest move would be to withdraw the exercise review with minimal fanfare, and not attempt to replace it. The issue is just too contested for Cochrane, with its evident bias and lack of understanding of the issues, to manage successfully.

Thanks. There are people who feel exactly the same way about authors you may approve of (although I realize it's also of course possible that you think they're all unsuitable). [Edit: just realized the subsequent post makes it clear that at least one person definitely gets a tick from you.]

A group could have been assembled in which having absolutely no prior position or involvement with the condition or research related to exercise, too. That's no guarantee of a good review, either. I don't believe lack of content knowledge is an enormous asset, or that relevant biases are ever absent in anyone - and biases can certainly develop and change during the research process. Biases are always there, and therefore you need better ways to deal with it than finding people most would agree are blank slates at day zero.

Even if a so-called bias-free group of people with zero involvement or research interest were appointed, that would also result in backlash. Anything Cochrane did, or didn't do, was going to face support from some, and backlash from others. I don't believe avoidance of backlash should determine research agendas.

 

[Hilda Bastian]

I just want to add to that, that this exercise review also has the potential to harm the reputations of those associated with it, including Hilda's. I'm particularly concerned that @Medfeb, a good ME/CFS advocate, might get caught in the crossfire, being expected to work a miracle with people on the writing team and in Cochrane itself who will not change their beliefs about the effectiveness of exercise for ME/CFS, and having to defend a result that is a political compromise rather than something that is soundly based and helpful to people with ME/CFS.

A Cochrane exercise review still carries too much weight when national ME/CFS clinical care guidelines are made, for patient advocates to do anything but campaign against one that suggests that the current evidence-base supports exercise therapy as a treatment for ME/CFS.

If my own interests were front and center for me, I would never have written a word about ME/CFS. That's ineffably sad. It is literally because that is the case that I am doing it.

 

[Ariel]

I fear that having knowledge or important awareness of context is being interpreted as having "biases".

I am concerned about an approach that is giving equal weight to different "feelings" people have. The point is, who is right? Are there serious methodological problems or not, for example? One cannot take these points on board and then say, oh well, yes but there's a different view as well. Someone is wrong.

 

[Hutan]

I don't believe avoidance of backlash should determine research agendas.

I don't think anyone was suggesting that, I certainly wasn't.

What I was saying is that Cochrane does not appear to be sufficiently free of bias to be able to conduct a useful review. Therefore it is riding into a process that can only have a bad outcome for all concerned.

The point is, who is right? Are there serious methodological problems or not, for example? One cannot take these points on board and then say, oh well, yes but there's a different view as well. Someone is wrong.

This. The methodological problems are not hard to identify, or to understand if you approach them without blinkers. Please have a look, and ask us to try to explain more if you don't know what we are going on about. Understanding the flaws in the existing research is not a bias, it's knowledge, an indication of scientific competence.

Hilda, I say again, you need to put some time to actually understanding the methodological problems, as, only then can you make good choices about who to include on the writing team. This is not a case of 'oh well, there's truth on both sides.' I'm not getting any sense that you, and Cochrane, do understand that there are fundamental problems with the evidence base.

 

[Hilda Bastian]

I fear that having knowledge or important awareness of context is being interpreted as having "biases".

I am concerned about an approach that is giving equal weight to different "feelings" people have. The point is, who is right? Are there serious methodological problems or not, for example? One cannot take these points on board and then say, oh well, yes but there's a different view as well. Someone is wrong.

I agree that knowledge and awareness of context is critical, and it's an asset, not a bias. Sometimes, people need to change their minds - in the group charged with the task, and in the people who will accept or reject what they write: it's not only about what's in a GRADE table or a meta-analysis.

So what do you do when people are deeply entrenched in opposed camps of people who have convinced each other without any doubt that they are right, and they each have an influence on the ground, where everyday decisions and actions are happening. Something has to shift, including in people's minds - and not just in one country. I don't believe you get there by people not talking to each other, not listening to each other, and not being seen to do so.

Some people aren't going to be convinced of something if they can't see that their point of view has been fairly and competently considered and clearly addressed. And it shouldn't be made easy for those who will just work to discredit any process that doesn't come to the conclusion they want. Critically, for the huge numbers of people who aren't in either camp but are hearing both sides, they need to be able to see for themselves that the attempts to discredit a document aren't valid. When either of those camps say to them, "ignore that - here, I can show you why they were biased and it's just about being overtaken by a lobby, not science - don't believe them - look over here instead, this is who you should believe" - it shouldn't be super, super easy for that argument to work.
Edit: PS: Aaarghhh! I "both sides"d it!!! So hard to resist binaries... Who aren't in *any* camp but are hearing *all* sides.

 

[Hilda Bastian]

Hilda, I say again, you need to put some time to actually understanding the methodological problems, as, only then can you make good choices about who to include on the writing team. This is not a case of 'oh well, there's truth on both sides.' I'm not getting any sense that you, and Cochrane, do understand that there are fundamental problems with the evidence base.

For reference on this, here are some things I've written specific to this:

• The evidence base, the Cochrane review, and the AHRQ review in February 2019
• The amendment to the Cochrane review in October 2019
• On bias in protocols in May 2020 - which was around the time I was deep in the process of gathering, and developing an analysis of, critiques of the methodological and other issues with the Cochrane review.

People will be able to invited to let us know if there is anything we have missed.

I wouldn't say this is a case even of both sides - I think there are more than 2. And "oh well, there's truth on both sides" is not how I see this. But I do believe, there is no shortage of people who have no doubt they are right on all sides.

 

[Hilda Bastian]

Great. HAve you already looked at the Vink Vink-Niese re-analysis? https://journals.sagepub.com/doi/pdf/10.1177/2055102918805187. I think I suggested at some point that they be invited to Cochranize their work. I may not have done, but would that be considered once you and the IAG had read and analysed it? Of course they may not be willing to do that, but it seems a very logical solution.

It's number 24.

 

[BruceInOz]

I suspect that Hilda's reference to 'pounding' may be related to the insistence by most of us here that subjective outcomes in unblinded trials leads to worthless pseudoscience, and that she sees that there are those that support such trials who have differing views on this issue. I think she feels that if those differing views are excluded from the process then the final product will be open to attack from that side. Hence I think she feels that we overplay our hand, that it feels to the other side that we are the dogmatic ones who won't listen to their reason.

Hilda, if I am way off the mark here, please say so. I certainly appreciate your involvement in this forum.

So, is there a way we can increase the attention that the process will pay to the problem of subjective outcomes in unblinded trials? Should we search for good peer-reviewed articles that address this issue (for example, the Albuterol, placebo, sham acupuncture paper) and make sure they appear before the writing and advisory groups? It seems so obvious to me and, I think, most of us here, but maybe we need to support it with hard evidence.

 

[Hilda Bastian]

@Hilda Bastian, thank you for reading our posts.

Thank you - and everybody - for the time and thought you put into them. I will at times be slow getting to them, but I read them with care and appreciation for the thought and effort. Not responding doesn't mean it's ignored.

I can see it's a pain when you thought you were acting reasonably to have some of your actions called into question by some members of the 'consumer' group you have been appointed by Cochrane to communicate with, and to be faced with a flood of posts questioning some of your decisions. I accept that you are under no obligation to respond here and now to the concerns we raise.

I was a consumer advocate for decades, including leading a cross-issues national organization for a long time; I've got a highly active Twitter account and write for some very high profile magazines on highly controversial topics: I have a lot of experience of being called into question, I fully expect it, and I learn a lot from it. I believe a group of 8 authors with both Mary and Kay, is a far, far better group than the group of 6 it would be without them, which was the other option. I think they're both deeply impressive, and I have a lot of confidence in them both. I'll write more about the issue of the process in a methods report, as I've already discussed.

Research authorship: When people have jobs on offer, that can go through that process - although as we all know, that doesn't necessarily affect the outcome; and even when that occurs, it doesn't mean every one of the authorship positions were advertised. People generally sometimes put out the word they are considering authors in different ways and to different circles, and we did that, with an email address for people to send enquiries to - just as we notified that I was going to be establishing an advisory group. People and organizations certainly understood that they could propose people, because they did - and others asked if they could put their own or others' names forward, and they did. Or they just asked how was this being done. S4ME, for example, is one of the groups that asked to be considered for the IAG (to which I of course replied that they were).

How individual authors came to be on a research team is not something that's typically discussed. If there was honest transparency about that, I reckon it would mostly be "I asked some like-minded friends and/or people I wanted to be associated with and/or people I was obliged to ask".

But please, to help us all, to relieve the distress of those who feel accused, and to clear yourself of accusations of buying into the harassment narrative, please can you either be more specific about what you are referring to, or alternatively, retract the accusations of 'pounding'.

I've said that I don't think it would be helpful for me to be more specific about what I believe is counterproductive, and so I'm not going to do it. My views about effective advocacy and criticism are public in many places, but it's clear to me now that I should not mention them here. I can delete that specific post, and I'm happy to do that if that would be helpful.

I am sure we all want to make a constructive contribution, as we did to the NICE guideline draft consultation. I personally spent six weeks solidly working on our forum submission to NICE (with others) to the detriment of my health.

This matters enormously to us all. I hope we can work constructively together on it.

I hope for this too - and I'm sorry about the additional specific burden this means when it affects people's health on top of everything else. It's a major part of why I decided last year to invest heavily on capturing the things people already have said so eloquently over the years, and trying to honor it - as well as using the process to help us consider how we can reduce the burden of contributing, within the constraints we have.

Edit: fixed formatting screwup and added a missing verb and the end of an accidentally incomplete sentence.

 

[Hilda Bastian]

I suspect that Hilda's reference to 'pounding' may be related to the insistence by most of us here that subjective outcomes in unblinded trials leads to worthless pseudoscience, and that she sees that there are those that support such trials who have differing views on this issue. I think she feels that if those differing views are excluded from the process then the final product will be open to attack from that side. Hence I think she feels that we overplay our hand, that it feels to the other side that we are the dogmatic ones who won't listen to their reason.

Hilda, if I am way off the mark here, please say so. I certainly appreciate your involvement in this forum.

So, is there a way we can increase the attention that the process will pay to the problem of subjective outcomes in unblinded trials? Should we search for good peer-reviewed articles that address this issue (for example, the Albuterol, placebo, sham acupuncture paper) and make sure they appear before the writing and advisory groups? It seems so obvious to me and, I think, most of us here, but maybe we need to support it with hard evidence.

Thanks! Not about any specific issue nor about criticism or criticizing the same thing persistently. However, the "is there a way we can increase the attention...?" does go straight to the question of effectiveness.

 

[Hutan]

Thank you for the links Hilda. I have read them carefully and you are clearly aware of some of the issues.

Here are my thoughts:

(crossposted with Bruce's point about subjective outcomes...)
The main problem with all of the BPS intervention studies is the use of subjective outcomes (answers to surveys) combined with interventions that are not blinded and are explicitly designed to change how people report their symptoms. For example, 'if you feel bad, that's ok, that's normal'. None of your links show that you understand that, there's the use of 'subjective' once, but it is very much in passing. We've discussed this point earlier in the thread and my impression was that we didn't make much headway in having you understand the seriousness of that problem. It's more important than any other flaw in these BPS studies.

My concern with this is that you seem to have intentionally chosen the members of the writing team, at least the ones I know something of, to ensure that this fundamental flaw is not front and centre. As good as @Medfeb is, my impression from our earlier discussion on this is she does not seem to rate the 'subjective outcomes in unblinded studies' issue as more important than things like use of Fukuda. This is a different view to most of the people who post regularly here. The problem with that view is that it can lead people to think that these studies tell us something about a subset of people with chronic fatigue, just maybe not about ME/CFS. For example, here, what you say makes me think you think this:

This review over-estimated the strength of the evidence and its applicability, and while better, it’s still doing that. That has been having a damaging effect on people to whom the results don’t apply, but on whom the consequences of this review are visited.

What the trials actually repeatedly show is that the interventions can't be said to work any better than nothing for any group that has been included. As I have said, this flaw applies well beyond ME/CFS research. Lack of acknowledgement of the problem has probably made many Cochrane reviews flawed. The lack of a person on the writing team who deeply understands the issue with subjective outcomes in unblinded intervention trials is a large part of my concern with the composition of the team, and why I urge you to include an S4ME representative.

A similar message has been coming from multiple trials of GET – but only one particular form of pacing has been tested in a trial. The message could be fairly consistent on GET because it really does have a small amount of benefit for a minority of people (about 15% over and above regular care in PACE). Or the message could be fairly consistent because the trials have all relied on similar and deeply flawed methods. We won’t know till there are more rigorous methods to use to test these questions, and strong large trials are done that learn from the errors in the story so far.

No. Even with interventions and outcome measures explicitly designed to maximise over-statement of benefits, the PACE trial showed us that GET is not a helpful intervention, that it's not different to basically no care. There's no case for more tests of GET. Now, it's just possible that sessions of some sort of short bursts of high intensity exercise might help, something like that, or it might be a way for people to keep some fitness without incurring PEM. There's scope for more exercise research. But the concept of GET has been tested, and found to not work, for Fukuda and more stringent criteria. The ME/CFS community does not want more money wasted on studies of efforts to make us more active. Some of us are already quite active, and yet still quite sick. If you haven't already, please look at that Newton study thread that I linked to which is relevant to this.

As I’ve blogged before, ME/CFS consumer advocates have been hammering away to get this Cochrane systematic review on exercise and CFS fundamentally changed since 2015. I think they’re right. And now, they have achieved something pretty remarkable…

Gosh, hammering is effective, but pounding is not? It's so complicated. Perhaps it depends on whether one is the person or organisation being repeatedly hit?

And I’ve written before about the problem of researchers doing systematic reviews where their own studies could be included.

. Indeed. And it goes beyond people assessing their own work, as members of a tight professional circle will not want to upset their friends. And really, anyone who has made fundamental errors like subjective outcomes in unblinded trials may not be able to see, or may not want to point out, such errors in the work of others.

One of the reasons the draft NICE guideline was fairly good, even with a mix of viewpoints in the writing team, was that the grading of the evidence was done by a separate team of people with no incentive to find a particular result, one way or another. They just listened to research methodology experts like Jonathan Edwards, and systematically went about their task. (It remains to be seen if the gains in the draft can be held against the onslaught, dare I say pounding, of the BPS lobby. If the gains are lost, it will be a sad illustration of the power of the BPS industry. My best wishes to our patient reps working hard for us.)

Anyway, my understanding of the Cochrane process is that the process of grading the quality of studies won't be done by a separate unbiased team. Well, I'm not really clear what will be happening and perhaps you aren't yet either. I've appreciated @Caroline Struthers' questions about that; surely the amount of work involved means that just adopting the NICE evaluations would be a good strategy? I know you are still working out the methodology, but then that makes it all the more important that we talk about it now, before you have something that you don't want to change. But, at the moment, it's looking a lot like some people will be assessing studies that have the same flaws as their own research, and some people will be assessing studies without a solid background in research analysis.

Reading the guidance from various systematic review bodies offers several solutions that have the potential to help ensure robustness and credibility. One is to ensure that every stage of the process has hands-on involvement and oversight from people with methodological expertise who don’t have a stake in the review’s outcome.

Indeed. But that's not what we are seeing here yet. The only stake that informed patients have is getting the right answer - it's completely aligned with what Cochrane should be aiming for. And yet we don't have a single one on the writing team. The selection criteria for a patent representative of them not having engaged in the debate about ME/CFS research and not having demonstrated any methodological expertise is insulting really.

So what do you do when people are deeply entrenched in opposed camps of people who have convinced each other without any doubt that they are right, and they each have an influence on the ground, where everyday decisions and actions are happening. Something has to shift, including in people's minds - and not just in one country. I don't believe you get there by people not talking to each other, not listening to each other, and not being seen to do so.

No, the BPS proponents and the informed ME/CFS patients do not need to come together and be reconciled, each giving up some of their deeply held beliefs and reaching some happy compromise. That won't happen - too many people have been hurt and have carefully investigated the evidence on one side, and too many people have their reputations and livelihoods tied up in seeing things a certain way on the other. You only have to look at the recent rash of BPS papers with shoddy data dredged up from the back of the filing cabinet and the reaction of the BPS lobby to the draft NICE guideline to see that. And, in any case, compromise not a good outcome.

For progress, reconciliation is not required. If it is clear that the BPS proponents can not apply their tools to ME/CFS they will just continue applying them to others. If the evidence shows that a drug doesn't work and probably harms, you don't need to get the drug manufacturers in a room to sing kumbyah with patients. You just withdraw the drug.

If my own interests were front and center for me, I would never have written a word about ME/CFS. That's ineffably sad. It is literally because that is the case that I am doing it.

Your efforts are appreciated and there was a lot of excitement when we first heard that you would be involved. But, history is littered with people who thought they could come sort out things for people with ME/CFS and ended up doing harm. It's important to really hear people with ME/CFS who have put a lot of time into understanding the situation- the science and the politics. I do understand that you are probably dealing with all sorts of difficulties at Cochrane - I'm sure it can't be easy. But, I want you to know that a review done badly will be a lot worse than a review not done at all. If Cochrane is not ready, and the BPS lobby is having a lot of influence, then it would be much better to not proceed with the review.

apologies for the edits - mostly grammar

 

[Hutan]

How individual authors came to be on a research team is not something that's typically discussed. If there was honest transparency about that, I reckon it would mostly be "I asked some like-minded friends and/or people I wanted to be associated with and/or people I was obliged to ask".

While I appreciate the honesty, that's a bit sad Hilda. It reads very much as an 'old boys' club' situation, something people who aren't 'old boys' have been criticising for a long time. It's a fine approach for choosing who to go out to dinner with, but it's sad to see it applied to something that requires diversity of thought and that will affect a lot of people.

(and just to pound a bit: it underlines the point I made about not you not having included anyone who sees the issue of subjective outcomes in unblinded trials differently to you.)

 

[Sean]

So what do you do when people are deeply entrenched in opposed camps of people who have convinced each other without any doubt that they are right, and they each have an influence on the ground, where everyday decisions and actions are happening. Something has to shift, including in people's minds - and not just in one country. I don't believe you get there by people not talking to each other, not listening to each other, and not being seen to do so.

I'd be the last to claim that patients and our allies have handled all this perfectly. We have made lots of mistakes.

But there are some things that cannot be compromised on, like methodological and ethical standards.

Apart from any other reason, the implications of such compromising don't stop with this particular area of medicine. Lowering standards will inevitably compromise all other areas if allowed to go unchecked. I regard this review as very much a test case, the outcome of which will have major broader implications.

 

[Hilda Bastian]

Thank you for the links Hilda. I have read them carefully and you are clearly aware of some of the issues.

Thanks. (Though it's risky to assume that my views in June 2021 on any point are the same as they were in 2019, unless I said so. People learn things, make mistakes, and change their minds, including me.)

My concern with this is that you seem to have intentionally chosen the members of the writing team, at least the ones I know something of, to ensure that this fundamental flaw is not front and centre. As good as @Medfeb is, my impression from our earlier discussion on this is she does not seem to rate the 'subjective outcomes in unblinded studies' issue as more important than things like use of Fukuda. This is a different view to most of the people who post regularly here.

I didn't choose members of the author team to ensure that any specific methodological was given preference.

The problem with that view is that it can lead people to think that these studies tell us something about a subset of people with chronic fatigue, just maybe not about ME/CFS. For example, here, what you say makes me think you think this:


What the trials actually repeatedly show is that the interventions can't be said to work any better than nothing for any group that has been included. As I have said, this flaw applies well beyond ME/CFS research. Lack of acknowledgement of the problem has probably made many Cochrane reviews flawed.

I agree it's important to keep that clear, and I have a different view on what research is needed now than I did in 2019.

The selection criteria for a patent representative of them not having engaged in the debate about ME/CFS research and not having demonstrated any methodological expertise is insulting really.

There was no separate selection for criteria for patients as potential authors and "not having demonstrated any methodological expertise" was not a criterion.

For progress, reconciliation is not required.

I agree that reconciliation is neither required nor possible: I didn't say it was. Indeed, what I wrote makes it clear I presume people will still be making the argument - that's biased, don't look at it, look at this.

Your efforts are appreciated and there was a lot of excitement when we first heard that you would be involved. But, history is littered with people who thought they could come sort out things for people with ME/CFS and ended up doing harm. It's important to really hear people with ME/CFS who have put a lot of time into understanding the situation- the science and the politics. I do understand that you are probably dealing with all sorts of difficulties at Cochrane - I'm sure it can't be easy. But, I want you to know that a review done badly will be a lot worse than a review not done at all. If Cochrane is not ready, and the BPS lobby is having a lot of influence, then it would be much better to not proceed with the review.

Thanks. Yes, even the best-intentioned and best-skilled person can do harm. I agree with you on what a key strategy here is: it's utterly vital to really hear people with ME/CFS and their great wealth of knowledge and experience. While it took a long time, hindered by a pandemic, to get to this point, listening and reading that is a large part of what I was doing*. The opportunities for feedback on this will enable people to see where something hasn't been fully appreciated, or it's been overlooked, or new insights emerge.

* For reference here, from the report:

The re-consideration of this review is due to the commitment many people made to articulating and publishing their concerns about its contents and conclusions – as well as pushing for change and inclusion of people with ME/CFS. In particular, Robert Courtney and Tom Kindlon submitted influential major detailed comments that Cochrane published, but did not adequately respond to.

So the starting point for the update of the review was the need to deeply engage with “historical" stakeholder critiques. This part of the process provides a framework to ensure that issues are thoroughly addressed from the outset, to assist in consultations and incorporating ongoing feedback, and to systematically address the points raised.

 

[Hilda Bastian]

While I appreciate the honesty, that's a bit sad Hilda. It reads very much as an 'old boys' club' situation, something people who aren't 'old boys' have been criticising for a long time. It's a fine approach for choosing who to go out to dinner with, but it's sad to see it applied to something that requires diversity of thought and that will affect a lot of people.

(and just to pound a bit: it underlines the point I made about not you not having included anyone who sees the issue of subjective outcomes in unblinded trials differently to you.)

I agree that it can have dreadful consequences for something that requires diversity of thought and will affect a lot of people. That's why we didn't do it that way. In case anyone's wondering: I am pretty sure I chatted with David Nunan briefly socially at a conference once. I interviewed people who reached the stage of being prospective consumer authors, but had never met any of them prior. I have never met any of the other authors (with apologies if they ever came up and said hello and I don't remember).

For the IAG: I got to know George Faulkner in the discussions at Cochrane that led up to this process. For the methodologist positions on the IAG, I am going to rely on people I have known a long time, in whose expertise and integrity and listening/communication skills I have enormous confidence, who I know will be both diligent and have zero reluctance to tell me when I'm screwing up, and who I know will engender a lot of confidence in members of the Cochrane community.

 

[Snow Leopard]

One point that might not have been made yet (I haven't been able to keep up with the discussion in this thread) is that this process isn't just about our patient group.

It is a pilot programme and if successful, should in principle be used to revise or guide the development new meta analyses covering other conditions where there are similar methodological problems and ideally, improve the quality of non-pharmacological clinical trials in the future.

 

[Adrian]

What I was saying is that Cochrane does not appear to be sufficiently free of bias to be able to conduct a useful review. Therefore it is riding into a process that can only have a bad outcome for all concerned.

In general what I observe is Cochrane has very weak processes to avoid bias (or ensure competance) in its reviews. With the current review they failed to respond is issues raised by patients and then issues raised by the editor. So instead they start an entirely new process saying they will take 'consumers' into account but what this turns out to be is they appoint someone to appoint a IAG. Its not clear who came up with criteria for who would be good to be on the IAG (from the outside it seems to be something in the gift of those appointed (and hence ultimately in the gift of the editor) rather than having a strong well defined process with good definitions about roles, expectations, skills and why those roles, expectations and skills are required. Then trying to fill them through an open process with say an interview panel.

It all seems very amateur as an organization as far as I can see. Certainly not operating to processes that help understand and mitigate baises and ensure the right mix of skills exist within a team.

Personally I feel Cochrane should be shut down as I don't see them operating in a trustable way that can issue high quality reviews - it might be that some are but that seems like random luck of who is involved rather than the power of an organization. Hence the Cochrane brand is meaningless.

 

[Adrian]

I was a consumer advocate for decades, including leading a cross-issues national organization for a long time;

With healthcare I find the notion of consumer advocacy quite strange. The very work consumer hints at a expected role of patients to take what they are given and as people who need help in getting it.

It doesn't feel like empowering and educating patients who can then express optinions and be partners but rather someone steping in and saying 'what about the poor patients'. I would also point out there is a huge body of knowledge with patients and carers (including those of us with science and research backgrounds and those who are more self educated but carryout really insightful analysis. But this is often not acknowledged.