Jonathan Edwards
Senior Member (Voting Rights)
I see that West is a UCL psychologist who works on tobacco addiction. Presumably Cochrane have shown his treatment 'works'.
-
New Forum Software Installed (Still a few issues but reopening the forum) More details.
NICE and Cochrane sign collaborative agreement to deliver ‘living’ guideline recommendations
- Thread starter Sly Saint
- Start date
-
- Tags
- cochrane foi guidelines nice
Caroline Struthers
Senior Member (Voting Rights)
This group produces reviews on the effectiveness of interventions to reduce smoking. I believe they are one of the more "successful" UK groups - in that they gain kudos for Cochrane. They probably already work pretty closely with NICE, but I don't know that for sure.
Caroline Struthers
Senior Member (Voting Rights)
I think the point is NICE will move the current policy of always using their own assessors (expensive), to commissioning Cochrane for some of the work (presumably cheaper as Cochrane doesn't pay its reviewers). They will pay Cochrane something, obviously, but it won't be as expensive as having their own trained, paid (but guaranteed impartial) assessors looking at everything
Keela Too
Senior Member (Voting Rights)
So hold on a mo. If reviewers genuinely get nothing out of doing Cochrane reviews then that is exploitation plain and simple. There must be a benefit of some kind, for these reviewers to put in the huge hours that a thorough review would entail.
So if unpaid reviewers have another ulterior motive (which in my view they must) then that motive (explicitly stated or not) should immediately make them unsuitable as reviewers.
If Cochrane want to collate reviews written by volunteers with their own motivations, then that is up to them, but surely NICE are getting into very murky territory if they effectively support a zero-pay model?
Either NICE are supporting exploitation, or they acknowledge that the reviewers are gaining some other benefit from doing these reviews. If Cochrane reviewers are gaining in another undisclosed way, then NICE must surely recognise that Cochrane reviewers cannot be guaranteed to be impartial?
Edit to add: Have I over simplified the situation? What am I missing here?
So if unpaid reviewers have another ulterior motive (which in my view they must) then that motive (explicitly stated or not) should immediately make them unsuitable as reviewers.
If Cochrane want to collate reviews written by volunteers with their own motivations, then that is up to them, but surely NICE are getting into very murky territory if they effectively support a zero-pay model?
Either NICE are supporting exploitation, or they acknowledge that the reviewers are gaining some other benefit from doing these reviews. If Cochrane reviewers are gaining in another undisclosed way, then NICE must surely recognise that Cochrane reviewers cannot be guaranteed to be impartial?
Edit to add: Have I over simplified the situation? What am I missing here?
Jonathan Edwards
Senior Member (Voting Rights)
It all seems a bit muddled. Cochrane appear to have lost their major source of funding. That being the case it does not seem to make much sense for NICE to start asking Cochrane to do review work, if they already have their own people. Maybe NICE wants to cu staff but if Cochrane don't have funds how does it help to ask them to do it?
Maybe NIHR and other government funded NHS bodies think it is a waste to pay two lots of people to do reviews but then why not just let NICE do it?
The issue of payment is complicated. If systematic reviews are a good way to make sure clinical practice is evidence based then they can be seen as a legitimate part of the work of clinical academic staff. Doing reviews, like doing research or writing papers is seen as part of the way academics justify their existence - and reasonably so - part of the learning process.
But doing reviews involves two different activities. One is the data trawling and categorisation - which at NICE is done by technical staff with no conflicts of interest. The other is evaluation - which at NICE is done by people on a committee who are unpaid and who may have conflicts of interest just like at Cochrane. I think the problem with Cochrane is that who gets to do any of this is all a bit arbitrary - and indeed dependent on ulterior motives.
Maybe NIHR and other government funded NHS bodies think it is a waste to pay two lots of people to do reviews but then why not just let NICE do it?
The issue of payment is complicated. If systematic reviews are a good way to make sure clinical practice is evidence based then they can be seen as a legitimate part of the work of clinical academic staff. Doing reviews, like doing research or writing papers is seen as part of the way academics justify their existence - and reasonably so - part of the learning process.
But doing reviews involves two different activities. One is the data trawling and categorisation - which at NICE is done by technical staff with no conflicts of interest. The other is evaluation - which at NICE is done by people on a committee who are unpaid and who may have conflicts of interest just like at Cochrane. I think the problem with Cochrane is that who gets to do any of this is all a bit arbitrary - and indeed dependent on ulterior motives.
Yea I think there were some weasel words in NICEs reply i.e. assurances that NICE would review the evidence independently (of Cochrane) in accordance with the NICE guideline.
I think @Jonathan Edwards is key i.e.
The business case (hopefully released under FOI) should tease out these issues but I kind of expect the drafter avoided highlighting the fundamental flaws in the NICE/Cochrane agreement. Old advice ---- what's the answer you want? Once I know that I'll draft accordingly!
Keela Too
Senior Member (Voting Rights)
Ah okay, I overlooked the part that their time may be paid for by their daytime employer. So that raises another question.
If NICE are paying Cochrane for work done by reviewers on a third party’s payroll, does that not create another problem? As a teacher I was always told that as a college employee, materials I produced for my classes on college time belonged to the college. So how can Cochrane effectively sell materials that are not theirs to sell?
I do understand that there is fair usage etc, so in my case teachers moving jobs took their resources with them, & the resources were also left for use in the original college. It was acknowledged as a win-win to share this way. However, it would have been very frowned upon for a lecturer to privately package & sell resources produced on college time.
Are the employers who fund the Cochrane reviewers’ time, going to be consulted about the sale of those reviews by Cochrane? It’s a very messy arrangement I think.
Caroline Struthers
Senior Member (Voting Rights)
As far as I understand it, the same (paid) NICE technical staff with no conflicts do the trawling, categorisation, and the evaluation of evidence quality. The (unpaid) committee are supposed to be as free of conflict as possible, and have no connection with the technical team who present the evidence (unlike in Cochrane reviews where they are all one big happy family). The committee are presented with the evidence already quality (GRADE) evaluated. In the ME/CFS guideline, the committee also took into account the bias caused by relying on subjective outcomes in unblindable trials - not taken into account by GRADE - but I think that is unusual (unprecedented??)
I guess we'll know more if we get sight of the Agreement. From the September 1st media release and the NICE strategy statement:In their response to me NICE stated that the evaluate all reviews i.e. they do not use Cochrane reviews without reviewing them. If true then what use is the link to Cochrane i.e. if NICE has to carry out their own evaluation?
EDIT - I've posted an extract from NICEs email to me below.
Yes as a "management approach", and assuming that management neither know nor care (to be fair I've experienced that), then it makes "sense"! Bear in mind @strategist comments that with suitably subjective outcome criteria even homeopathy can be shown to work and GRADE/Cochrane looks like vehicle to prove it ---- unblinded studies with subjective outcome criteria are considered moderate quality evidence!
As per @Caroline Struthers ---------
https://www.nice.org.uk/about/who-we-are/corporate-publications/the-nice-strategy-2021-to-2026
it's clear NICE considers that there will be some synergy involved that doesn't currently exist - the example given in media release is of this Cochrane Review
https://www.cochranelibrary.com/cds...full?highlightAbstract=miscarriage|miscarriag
feeding into this Guideline
https://www.nice.org.uk/guidance/NG126
in some way that meant the Guideline was published faster/more complete (or something ?)
The key phrase seems to be "dynamic, living guideline recommendations" so presumably the value of the agreement with Cochrane is whether it measurably contributes to fulfilling that part of the NICE five year strategy.
On a different note - if there are doubts about the independence of Cochrane Reviews - does this extend to the influence of funders ? For example NIHR has an established preference for funding research into psychological therapies even where such therapies have limited application; on the back of this preference together with NIHR funding of Cochrane, does NIHR encourage funded researchers to be participants in Cochrane Reviews ?
I've no evidence to suggest this circularity but - if there was a view in NHS England and/or NHS Improvement that a particular treatment approach was desirable, and on that basis NIHR was persuaded to provide funding to research that approach, and on the basis research outcomes being presented as positive, would there be motivation from NHSE, NHSI and NIHR to load the Cochrane dice so as to influence a NICE decision in the direction favoured at the outset by NHSE/NHSI ?
Normally I'd consider such a multiplicy of organisations to preclude any viable attempt at stacking the odds but as there are no obvious Chinese Walls between NHSE, NHSI and NIHR I'm not so confident.
Jonathan Edwards
Senior Member (Voting Rights)
I don't think it is quite as simple as that. The technical team take note of the views of the committee and evidence presented to the committee in coming to a final evaluation.
rvallee
Senior Member (Voting Rights)
It clearly incentivizes people who can get BS through this way that can't pass scrutiny under a normal context. It seems to be the main appeal, frankly. Can't get a bad idea that doesn't work through peer review? Just get involved into arbitrating for yourself.
Basically people get involved because using Cochrane this way gives them an advantage. Honest researchers who don't try to polish turds don't bother, as this is what it's mainly useful for. Not everyone, but this is how the BPS ideologues are abusing it, and Cochrane appears fine with it.
on that point it is interesting to see the comment:
Dr Karla Soares-Weiser, Editor in Chief of the Cochrane Library, said: “This agreement has collaborative benefits for all involved. NICE will get the findings of Cochrane Reviews to use in their deliberations, Cochrane groups will be able to respond by producing high priority reviews, and for Cochrane review authors, it guarantees impact.
NICE and Cochrane sign collaborative agreement to deliver ‘living’ guideline recommendations | News and features | News | NICE
That grates given the amount of time and effort @Caroline Struthers has spent trying to get Cochrane to remove redundant reviews.
I think @Jonathan Edwards considers that much/everything is spin now - comes to mind when reading "dynamic, living guideline recommendations"
Yes if there was a Departmental (Department of Health and Social Care) mindset that ME/CFS was psychological (PACE etc.) then funding/favouring certain approaches would fit.
Caroline Struthers
Senior Member (Voting Rights)
ah. ok. I didn't know that.
Caroline Struthers
Senior Member (Voting Rights)
Exactly! You don't get money for doing reviews, you get membership of Cochrane and publication in a "high impact" journal (The Cochrane Library) which will never retract your review however bad or misleading it is - fantastic! If Cochrane team up with NICE, they can sell this to their authors as an added extra incentive. If NICE use your review, it is now truly untouchable as a piece of great science
Caroline Struthers
Senior Member (Voting Rights)
I don't think they will ever admit that they are being used and abused - either by BPS enthusiasts nor by academic clinicians with vested interests in particular drugs or devices.
Caroline Struthers
Senior Member (Voting Rights)
I have a friend who used to work for NICE and then worked for Cochrane as a contracted systematic reviewer on a suite of Cochrane reviews. She left Cochrane absolutely appalled by the vested interests, the waste, and she was bullied by her boss and the lead statistician on the project to use methods she wasn't comfortable with. She told a story of a Cochrane lead author insisting on including a randomized trial with seven participants. She had many many stories.
I can check this, but I thought she told me that NICE technical review teams did not take views of the committee into account when presenting the evidence. They would work to the protocol set out by the committee, but that's not the same thing surely. I *think* she also said that at NICE they used GRADE much more strictly than at Cochrane. So if a study is downgraded in one of the five domains - risk of bias, inconsistency, indirectness, imprecision and publication bias - then it could not be graded higher in another domain. Makes sense.
[edited for clarity]
Last edited:
From this it appears that NICE is doing what it should be i.e. evaluating the evidence. It's difficult to see how NICE could maintain those standards and use Cochrane; particularly since Cochrane seems no closer to changing it's systems to align with the [NICE] protocols you outline.
Interesting insight into Cochrane - far removed from the "world" described by Paul Garner and other supporters of Cochrane. Sounds like an unpleasant experience.
Jonathan Edwards
Senior Member (Voting Rights)
I am not in a position to give an account first hand but my understanding is that they may not be quite such different things as might appear on the surface.