Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

I dare say, if you got really bored, you could write an entertaining spoof trial.

 

I have plans to do something a bit more substantive! There might be publication bias because I know the result. But interestingly it is not the result I expected.

 

A study in effing and blinding?

 

From the perspective of someone who worked in an entirely different field -

In my line of work I was often asked, indeed pressured, to come up with documentation that would enable less qualified personnel to carry out at least some of the work that those more qualified and with more experience did. The idea being to free up the really experienced & qualified staff to do more of the complex and expensive stuff.

The inevitable snag was always that if you needed to be guided by a process to make decisions you weren't qualified to make then the chances are, sooner or later, you would fail to assess a situation properly and make a wrong decision anyway. Then not have the skill set to either realize or recover it. Costing many more £££££.

What works is having suitably qualified people making the decisions and assessing the information. Transparency in that they can then explain why they made a certain decision. Accountability in that if you muck it up, it's your career on the line.

It seems to me that systems like GRADE remove accountability & transparency. The bottom line is they are more about saving money by not employing enough of the right staff to make the appropriate decisions. When something does go wrong they make it much more difficult to get to the bottom of what went wrong.

 

I thought the same thing; the Cochrane reviews on the MMR vaccine might be one to look at (I tried but my brains just not functioning right now); the 2012 one didn't use GRADE but the most recent one 2020 did. This is the link to the history
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD004407.pub4/information#history

but it might not work (several things I bookmarked, I can't get back to because I deleted their cookies).

 

That is probably the norm. I think other forces are also at play at NICE though.

A huge amount of work already done by Cochrane was repeated involving sorting results of studies into subgroups and all sorts of things relating to GRADE that were completely unneeded. I suspect NICE is very keen to have its own staff (who are supposed to be free of bias) doing the bulk of assessment. The staff may over time get good at ensuring that all aspects are considered. But they probably move on to other jobs pretty quickly.

 

Was it unnecessary though?

Should NICE take Cochrane's work at face value? I don't think so. NICE have a responsibility to the British people that Cochrane do not. We also know that Cochrane have their own issues.

No. As far as I'm concerned if you are going to take responsibility for making important decisions and be held accountable for those (as NICE should be) then you don't take anyone else's say so for granted.

The exception would be when you have no choice but to hire in expertise you don't have but then that expert is held fully accountable for their input.

In my experience, anything else results in loss of accountability and all the issues that follow on from that.

 

I am certainly not talking about taking over Cochrane's conclusions. I am simply referring to all the searching and sorting into subgroups that must have gone on at both - prior to any evaluation. Here at S4ME as a community we already knew enough about the scope of the data before NICE began. Why trawl through it all again and more importantly why sort it into all these mindless subgroups? We knew from looking at the abstracts that the studies were incapable of providing usable information on efficacy.

 

Fair enough. Still, I don't disagree with them doing their own groundwork which their future decisions will be made on.

The boring groundwork and detail gathering is where it is easy to miss something that could affect decisions taken later on and lead to claims of bowing to pressure etc.

Sure we already knew the scope of the thing because we've been looking at it for a helluva long time. The NICE people probably haven't. They shouldn't take what anyone says at face value. Even us.

I say hats off to them for doing it. This way no one can accuse them of simply taking anyone's word for for the scope of what should and shouldn't be included. Take responsibility from start to finish, I say.

As for the subgrouping....presumably they have their methodology for going through the stuff. I sympathise having had to spend many a miserable week doing scoping exercises and documents with project managers before I could roll my sleeves up and crack on. Then have to keep an eye and review as we went. However tedious it did mean that things weren't overlooked and everyone knew exactly what was expected to be in place & when. It often seemed a bit OTT but....it stopped a lot of arguments further down the line.

If these are the standard procedures NICE use then it's important they do that in our case. Otherwise they'll be seen as making an exception for us and that ultimately wouldn't to us much good.

 

All evidence I have seen so far suggests GRADE is potentially useful but fully manipulable. In a nutshell: it is not itself a reliable process, it offers no guarantee that the outcome even respects such things as reality and the linear passage of time. Put it in the right hands and it may be useful. Put it in the wrong hands and it will be harmful because it will only serve whatever purpose those hands have planned to do.

The danger is in trusting that the process works simply because it gave results stakeholders wanted, or calling it out as RIGGED! if it doesn't with open calls to dismiss the entire process. This makes it a fully arbitrary process. The flaws seem to be in the very system of EBM, so not particular to GRADE. All it does is provide a few bumpers to decisions that are themselves entirely down to the preference consensus among those present, unless someone is present to actually make the process transparent, which rarely happens.

So I don't think that GRADE is necessarily unjustified, because what's actually unjustified is the very method of EBM, at least how it's commonly used. It is used as an alternative to the scientific method, rather than as a complement. So GRADE is flawed, but EBM is even more flawed. Any process that can produce 100% unreliable and misleading evidence, such as the Cochrane reviews, is ultimately useless in itself. Frankly "evidence-based" has come to mostly mean "stuff that more people want to be true than there are people who don't", a popularity contest with arbitrary rules and even less reliable enforcement of the rules.

 

But my point is that much of these 'standard procedures' are simply the nonsensical antics of GRADE grading - pages and pages of them - which in the end is a bad idea because we do not want grading we just want to know if evidence justifies recommendation. In reality the result is decided by the committee readjusting what the NICE staff have presented to them.

I am not saying that NICE should not have staff to make sure the scope is comprehensive. But my point is that making the staff go through the antics of meaningless grading exercise is probably not to save money - it is to do with politics of appearances. If it really meant being unbiased OK but GRADE seems to bring in more problems than it solves. In the end the committee made sensible adjustments but if they had competing interests we can be very sure they wouldn't have made those adjustments - Dr Busse has made that very clear.

 

But what is its use? I am unclear, except as a list of things not to forget to consider. Why do we need 'grades'. As Kavanagh also points out we do not do that in real life when making decisions, so why do it for important decisions about health?

Yes, but if those bumpers actually prevent assessors giving the right assessment - such as uninterpretable or strongly indicative against - then they are not just bumpers; they are routes to harm surely?

 

Ah ok. I understand. It's not simply the process as it relates to us, it's the process they use full stop.

 

I guess that an argument in favour of using GRADE at NICE (but not Cochrane) is that it means that you have two sets of assessments. One is done by a committee of people who know the diseases and treatments involved well, but might be biased. The other is done by people who know nothing of the diseases and treatments so should not be based. If both can come up with the same answer, or at least both can come an agreement on the same answer that would seem to have some merit. The only question is whether or not the 'unbiased robot' assessment is actually worth doing or even soundly based. Would it be better for the biased experts to come to a decision and then to have unbiased adjudicators grill them as in cross examination in court?

The situation at NICE seems pretty complex because in addition to the 'techie' assessors there are supervisors employed by NICE who presumably have some sort of guiding function and who may harbour hidden preferences or competing interests. The selection of the committee seemed to be a rather mysterious process, with rules that looked pretty irrational.

Id Dr Busse had been appointed as chairman presumably there is a high chance that the result would have been different. That makes the system fragile. On the other hand somebody somewhere set things up for ME so that some very sensible people were in key positions.

This thread is really intended to be about GrADE in general but in the context of the ME committee my feeling is that NICE turned out to be able to work effectively, but despite GRADE rather than helped by it.

 

From a different field, my experience is that both "techies" and non "techie" should be involved in the one process rather than independently and the decision of both teams weighed up at the end.

Techies can go into flights of fancy and can easily get so caught upnin the technicalities they lose sight of the primary objectives. I've been in many a meeting when the least technical person in the room has actually been the one who sometimes makes the most valuable contribution.

Similarly though the non techies may not grasp some of the complexity of what makes something that sounds really simple not practical in the real world.

Put both together and you may get a bit of a bun fight until they settle down but they can keep each other in check.

 

That figures.

 

It would also be interesting to see how much variability of gradings there can be between different teams working on the same data.

 

As I've pointed out before I haven't followed this carefully.

Let's say @Jonathan Edwards has done a study in rheumatoid arthritis using rituximab. Patients were selected using objective measures (e.g. antibody levels and scans), their level of disability was assessed using an objective measure. Half were then given placebo and half were got treatment (rituximab). Study was unblinded, and data collated, and treatment showed improvement re antibody levels, scans and reduced disability.

Why do you need an algorithm to interpret the results? OK the two groups could be too small to be significant but presumably the maths, re what's explainable by chance (noise in placebo results), deals with that. That of course is Jonathan's point---what's the purpose of the grading system [GRADE]?

I can see a point in NICE highlighting the issue of PACE being flawed, since it didn't have objective outcomes (they were dropped!), and that, as others have pointed out, is spooking the [GRADE/PACE---] horses ---- this logically has a read across to research which does not have objective outcomes!

It's [GRADE/PACE---] a pile of poo so why defend it --- a meal ticket lovingly polished for decades has been shown to be a cheap copy [Jonathan]

I hope NICE have the Cahoonas to highlight the read across to other guidance ---- it looks to be glaringly obvious even if it's a huge issue.

And no I don't accept that PACE couldn't have had objective measures --- they were dropped.

 

I used to drink wine and relied on XYZ--- to guide my (cheap) choices

NICE should be able to evaluate evidence and, as @Jonathan Edwards points out, the reasons why X was adopted should be set out --- transparent.

NICE shouldn't need Cochrane particularly since it's a pile of poo - and NICE cannot be blind to the issues which have cropped up in science recently. Unless of course there's someone in NICE whose close to Cochrane or likes the policy results the [GRADE] black box turns out!