Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

Jonathan Edwards said:
That is probably the norm. I think other forces are also at play at NICE though.

A huge amount of work already done by Cochrane was repeated involving sorting results of studies into subgroups and all sorts of things relating to GRADE that were completely unneeded. I suspect NICE is very keen to have its own staff (who are supposed to be free of bias) doing the bulk of assessment. The staff may over time get good at ensuring that all aspects are considered. But they probably move on to other jobs pretty quickly.
Click to expand...

Well to be fair, if you were in NICE, and you had to answer for the mess, then it would be reasonable to ask for the money to do the job properly.
 
Jonathan Edwards said:
I am certainly not talking about taking over Cochrane's conclusions. I am simply referring to all the searching and sorting into subgroups that must have gone on at both - prior to any evaluation. Here at S4ME as a community we already knew enough about the scope of the data before NICE began. Why trawl through it all again and more importantly why sort it into all these mindless subgroups? We knew from looking at the abstracts that the studies were incapable of providing usable information on efficacy.
Click to expand...

They might have wanted to show they had looked at all of the crap - even if you're correct i.e. they should have said it's crap and here's why--- would have raised a few hackles --- disgruntled X---
 
Jonathan Edwards said:
I guess that an argument in favour of using GRADE at NICE (but not Cochrane) is that it means that you have two sets of assessments. One is done by a committee of people who know the diseases and treatments involved well, but might be biased. The other is done by people who know nothing of the diseases and treatments so should not be based. If both can come up with the same answer, or at least both can come an agreement on the same answer that would seem to have some merit. The only question is whether or not the 'unbiased robot' assessment is actually worth doing or even soundly based. Would it be better for the biased experts to come to a decision and then to have unbiased adjudicators grill them as in cross examination in court?

The situation at NICE seems pretty complex because in addition to the 'techie' assessors there are supervisors employed by NICE who presumably have some sort of guiding function and who may harbour hidden preferences or competing interests. The selection of the committee seemed to be a rather mysterious process, with rules that looked pretty irrational.

Id Dr Busse had been appointed as chairman presumably there is a high chance that the result would have been different. That makes the system fragile. On the other hand somebody somewhere set things up for ME so that some very sensible people were in key positions.

This thread is really intended to be about GrADE in general but in the context of the ME committee my feeling is that NICE turned out to be able to work effectively, but despite GRADE rather than helped by it.
Click to expand...

I think the problem with GRADE is that it's not that easy to define the dependent variables i.e. measure X, Y, Z --- and if the study is flawed then the number is above "n" --- bit like the meaning of life the universe and everything 42! Theoretically you should be able to produce the model but the fact that GRADE rates things that are useless as +ve means that it has no real world use.
The way these things are supposed to work is that you decide on a course of action and then monitor the outcome --- if it works then OK if not you review/revise. So the fact that a lot of people have had CBT/GET and the likelihood that they are able to participate in work, study --- is no different from those who haven't had CBT/GET, means that objectively it doesn't work. Hint - you don't measure the outcome of your intervention that way you don't have to own up that your pet idea didn't work! Bear in mind that you guys are invested in a positive outcome, the folks who benefit from this are too [CBT/GET] - it's just that they benefit from the failed intervention!

EDIT - It's expensive to build in "monitoring" re an intervention; also, if politically your sold on an intervention then your servants might "lose" the monitoring. I'm with Jonathan re it's generally cock up not conspiracy. Just don't think those in power care!
 
Last edited:
I started to read the GRADE manual, but must confess I lost the will to live soon into it.

I'm not averse to trying to bringing order out of chaos, far from it, but all too often such attempts just replace one kind of chaos with another.

The notion of "weighing the evidence" is long established, and well founded. But it is useless if you impose artificial limits on how much weighting you can apply to the various components; the process itself then imposes artificial bias. Especially if something, or some combination of somethings, means their weight should eclipse all the other things out of the equation. Which is where logical operations come into play. There is no sense giving something a weight of infinity to artificially achieve that logic, you just need ... logic.

If you are assessing confidence in the marketability of a car - upholstery comfort, seating position, controls ergonomics, heated seats ... but the heated seats are a bit suspect, then:
Code: 
if (high confidence seats can electrocute driver)
{
    // Dead in the water. Weights of everything else totally irrelevant, so ignore.
    Confidence car is marketable = 0;
}
else
{
     // blah ...
}
 
Jonathan Edwards said:
I am simply referring to all the searching and sorting into subgroups that must have gone on at both - prior to any evaluation. Here at S4ME as a community we already knew enough about the scope of the data before NICE began. Why trawl through it all again and more importantly why sort it into all these mindless subgroups? We knew from looking at the abstracts that the studies were incapable of providing usable information on efficacy.
Click to expand...
I would not have relied on Cochranes searches one bit given the way they were done for their exercise reviews and the poor upkeep of the Cochrane Central db by the Cochrane Common Mental Disorders Group in relation to ME/CFS research.

But for me the issue is being approached from the wrong end; ie they should be coming up with a better, more standardised, more transparent way of presenting/writing research papers which has for so long been a rather vague set of guidelines largely open to interpretation.

eg: certain information should have to be included in the abstract in a concise, structured way not tucked away somewhere in the main body of the paper (eg limitations of the research).
 
Sly Saint said:
I would not have relied on Cochranes searches one bit given the way they were done for their exercise reviews and the poor upkeep of the Cochrane Central db by the Cochrane Common Mental Disorders Group in relation to ME/CFS research.
Click to expand...

Yes, well I suppose another way of looking at it is that if a GRADE assessment is so unreliable that you have to do it again yourself it is not achieving its aim of reproducibility too well!
 
Jonathan Edwards said:
Yes, well I suppose another way of looking at it is that if a GRADE assessment is so unreliable that you have to do it again yourself it is not achieving its aim of reproducibility too well!
Click to expand...
Yes. They actively don't claim GRADE achieves reproducibility of gradings (a rather tragic irony), just reproducibility of how things are graded. But what is the point of that? It is reproducibility of gradings (within sane variance) that matters, together with those gradings being valid.

ETA: It basically means that each organisation, NICE in this case, has to home in on their own interpretation of what GRADE means to them, and to comply with that. If the GRADE designers don't like that, then they should have come up with something better.
 
Last edited:
Barry said:
Yes. They actively don't claim GRADE achieves reproducibility of gradings (a rather tragic irony), just reproducibility of how things are graded.
Click to expand...

I would agree with @Sly Saint that they don't even achieve reproducibility of how, since Cochrane failed to even select the right sorts of trials. In the Larun review the specification was for controlled trials but none of the trials were controlled in any real sense - of having adequate controls.

It all seems to come down to Eric Morecambe: they are playing all the right notes but not necessarily in the right order.
 
I think the first thing any grading system should do is have a list of basic requirements clinical trials have to meet to be included in any review. If a trial does not meet any one of the requirements they would be automatically excluded. Only when these basic requirements are met should the trial be considered for inclusion in grading systems, systematic reviews and clinical guidelines.

These requirements might include:
- No unblinded trials with purely subjective primary outcomes.
- No outcome switching after approval that makes it easier to show 'success'.
- No rolling preliminary trials into main trials with switching of primary outcomes.
- No trials of psychological therapies where the 'control' group is not a genuine control where the only difference in patient experience is the active therapy being tested and is presented to patients in an equally or neutral way.
- No trials in which the active arm has been promoted in recruitment as effective and where the patients can tell whether they are on the active treatment.
- No data is included on monitoring of short and long term worsening and harms, which should include following up of patients who drop out of the trial.

And treatment should not be allowed to be claimed to be effective if, for example:
- The primary outcome measure is meaningless to the patient population, or where the claimed benefit is statistically significant but not clinically significant.
- The 'benefit' is transient and not maintained at followup at least a year after the trial finished.


etc etc.
 
Trish said:
I think the first thing any grading system should do is have a list of basic requirements clinical trials have to meet to be included in any review.
Click to expand...

That looks a good selection to go in a list. I think we need to be careful about exclusion categories though. There are probably exceptions to every rule that are easy to see the justification for. For instance, for an unblinded trial with subjective outcomes that studied six doses of treatment and found a classical sigmoidal dose-response curve with a four fold increase in successful outcome between dose 2 and 3 and between 3 and 4 but not much difference between 1 and 2 or 4, 5 and 6 then if the treatment was safe and met cost conditions it probably should be recommended. But the conditions are very narrow because once this trial has reported expectation bias relating to dose would colour further studies. Also in practice it is reasonable for the vast majority of conditions to wait until an initial dose ranging study has been replicated blind but there are exceptions like lupus or melanoma where recommendation can be fast-tracked.

All in all I am against any rules because if you understand the reasons for evaluating a trial you do not need any rules.

The overarching problem is competing interest. Cochrane seems to have sunk without trace on that one. NICE has had its problems but has passed muster this time - more or less!
 
Jonathan Edwards said:
All in all I am against any rules because if you understand the reasons for evaluating a trial you do not need any rules.
Click to expand...


I think rules are often necessary for those who don't understand - and unfortunately its those who really don't understand who feel the rules shouldn't apply to them and they should be allowed to use their intuition. Often guidance can be good but where if people deviate they have to explain why they are ignoring the rules.
 
Rules are also being applied by people who don't understand and used to justify their poor judgments as scientific. That is as much, if not more of a problem than people who don't understand not applying the rules. Even worse when the rules are invented by people who don't understand either.
 
I keep forgetting what I read and don't have sufficient knowledge of statistics, so will leave the discussion mainly to others.

From skimming many points seem to me important and plausible. Especially, I share the criticism about GRADE that it allows downgrading only to very low certainty of evidence and not to zero certainty.

I was wondering though if one needed to specify the category of outcomes that is of interest first. Depending on the category there might be different requirements to compile relevant evidence ?

Don't know how those points are addressed in the GRADE handbook.

Not able to word my thoughts better at the moment, so just leave here my notes:

Does research on the following outcomes all need the same methodology to get results of relevant evidence?

1) evidence for benefit of a treatment regarding improvement of the medical condition itself: regain qualitative and quantitative functionality (being able to resume specific physical and cognitive activities per se; increase time being able to do specific activities; ideally: recovery, return to school / work)

2) evidence for helping not making a progressive condition worse or slow down progression / stabilizing fluctuating conditions

3) secondary effects of the medical condition -- slow down / prevent deconditioning / counteract negative effects of unavoidable deconditioning

4) improve quality of life

5) maintain mental well-being

6) adverse side-effects/ harm

7) costs / cost-effectiveness

For all those outcomes: consider both short term and long term effects/ consequences.

Maybe rather off-topic but I was thinking that there still are many chronic illnesses for which no or no satisfactory treatment exists. People too ill to work, to go to school or to resume any productive activity deserve evidence-based health care nevertheless.
 
MSEsperanza said:
Does research on the following outcomes all need the same methodology to get results of relevant evidence?
Click to expand...

I think what this emphasises is that all sorts of different methods are relevant indifferent situations. There are some commonalities but also an infinite number of variations. We have to be able to make a useful judgment on the reliability of every new method variation. Most of the time that is not too hard you have seen what actually goes on during trials - the people side of things, how they behave, how easily they are tempted to cheat and so on.
 
Related to my post above:

If a study is designed in a way that it can't provide sufficient evidence for a specific outcome that it aimed to investigate, can it still provide sufficient evidence for another effect (or non-effect)?

That also would speak against a standardized GRADE-style approach to rate evidence as well as against a general exclusion of studies that don't meet certain requirements.

One needed to look at each study individually and ask for which outcome it could provide relevant evidence (and for which it doesn't). Also perhaps the more 'hidden' outcomes?
 
Last edited:
MSEsperanza said:
If a study is designed in a way that it can't provide sufficient evidence for a specific outcome that it aimed to investigate, can it still provide sufficient evidence for another effect (or non-effect)?
Click to expand...
Although I can empathise with wanting to do this, would this not count as post-hoc analysis, given it was not in the original trial protocol. Something we have criticised other trials for. Would it mean that a further trial might be needed to provide valid evidence?
 
Barry said:
Although I can empathise with wanting to do this, would this not count as post-hoc analysis, given it was not in the original trial protocol. Something we have criticised other trials for. Would it mean that a further trial might be needed to provide valid evidence?
Click to expand...

Yes we had a discussion about this with Hilda Bastion who argued that you should not throw away any trials just because they do not come up to standard on one particular point. I think she is right in principle but in practice if there is a fatalflaw in the method for extracting a primary outcome everything else is pretty much damage limitation or useful negative information. The fact that PACE shows that the original hypothesis makes no sense however generous you are with the results would be an example.
 
MSEsperanza said:
If a study is designed in a way that it can't provide sufficient evidence for a specific outcome that it aimed to investigate, can it still provide sufficient evidence for another effect (or non-effect)?
Click to expand...
GRADE rates each outcome separately. I think there is even an example of an unblinded trial where the subjective outcome was downgraded but the objective one (mortality) isn't.
 
You must log in or register to reply here.
Back
Top Bottom