Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

[MSEsperanza]

GRADE rates each outcome separately. I think there is even an example of an unblinded trial where the subjective outcome was downgraded but the objective one (mortality) isn't.

So according to GRADE, if you rate one outcome of a study you always have to rate the primary outcome as evidence too, at least as very low evidence. The subjective outcome in that example still counts as evidence for a positive or negative effect.

I thought it must be possible to say the results for the primary outcome of that study is unusable but other results from the same study are useful.

If I understand properly this could sometimes apply.

Not sure but is it also true that you don't neceassarily need the same methodological requirements to find out about evidence for a positive effect on an outcome as you need to find about a negative effect on the same outcome?

In any case, I think the issues discussed all corroberate the argument that was made on another thread:

One of the issues I am beginning to see with GRADE is the idea of a graded recommendation to be dispensed to GPs.

That seems inconsistent with the policy that doctors and patients should make shared informed decisions.

Basically, GRADE should aim to identify the arguments in favour of using a treatment and the arguments against. Those should be crystallised in such a way that GPs can present the arguments to the patient and they can decide the strength of the evidence.

If the patient wants the GP to advise r'recommend' then the GP should be in a position to explain why they do so strongly or weakly, not just say 'oh well people using GRADE come out with a medium recommendation, although goodness knows why'.

That seems to require though that those evaluating the evidence for guidelines need better education with regard to assessing evidence and doctors need better training in coping with uncertainty.

(Edited for clarity. Edited again -- prepositions.)

 

[MSEsperanza]

Friends of our friends are working on a solution already!

They developed additional tools...

https://www.informedhealthchoices.org/claim-evaluation-tools/

https://thatsaclaim.org/health/

https://www.informedhealthchoices.org/our-solution/

Meet a supporter of investigating the 'LP' with inappropriate trial design, a coordinator of the authors team of the Cochrane review on exercise for CFS, and at least one more defender of exercise therapy for ME/CFS:

https://www.informedhealthchoices.org/about/the-ihc-team/

Edited to add: I only had a very quick look at those tools, so not able to assess them. They might make mostly reasonable points.

 

[FMMM1]

I think what this emphasises is that all sorts of different methods are relevant indifferent situations. There are some commonalities but also an infinite number of variations. We have to be able to make a useful judgment on the reliability of every new method variation. Most of the time that is not too hard you have seen what actually goes on during trials - the people side of things, how they behave, how easily they are tempted to cheat and so on.

It seems like the best you could do is arrive at some sort of basic rules re what is acceptable/unacceptable e.g. @Simon M had a basic (but sound) rule "subjective outcomes only acceptable in blinded trials" "objective trials acceptable in unblinded trials".

However, I'm not sure you could develop a black box (GRADE) in practice since you can't distil out a set of relationships between X (methodology -- blinded/unblinded etc.), Y (measurement of outcome - subjective/objective ---) and Z (acceptable/valid study). Oh and there's the real world "tempted to cheat and so on".

So maybe NICE should have some guidance re the types of outcomes they expect to see objective (where possible) and subjective (blinding required); confirmatory studies, possibly even using different methods i.e. to reduce the risk that something is an artefact. These would be published on the NICE website - maybe they are there! Then each study would be critiqued and the reasons why it's been included/not included set out.

I still can't quite grasp why NICE would use GRADE - if there was one organisation who should know whether GRADE was sound/unsound it's NICE --- in fact if a GRADE type evaluation system was useful/deliverable then there's an argument that NICE is well placed to develop/commission it. I think GRADEs unsound/dangerous by the way --- which makes me wonder where it came from and why --- what "need" did it fulfil?

I went to a talk on the use of statistics in policy development given by Andrew Dilnot -- he discussed the measurement of cost/benefit for something like a broken hip [waiting X days versus y days ---] --- he summarised his view by using his 5 year old daughters thoughts --- a pile of poo. It looks like GRADE is a pile of poo too; but it also appears that it's made some people a lot of money!

 

[MSEsperanza]

Apologies for my cynical post above (There is a tool for everyone...)

I think the tool designed for those doing meta analysis and for health care professionals who want to make good decisions is rather to be found there:

https://www.decide-collaboration.eu/evidence-decision-etd-framework

https://www.bmj.com/content/353/bmj...e10283bb1e600b1e419cb6a9&keytype2=tf_ipsecsha

Just leave the links here in case they are of relevance for the discussion on how to make recommendations transparent.

 

[ME/CFS Skeptic]

Thought this is a good example of why what Jason Busse propose is problematic. How Covid-sceptics were duped by the “wonder drug” ivermectin (newstatesman.com)

COVID-skeptics promote the drug ivermectin and claim that all studies have reported positive effects. The author of the article (Stuart Richie) explains that this doesn't necessarily mean much because there might have been publication bias and the trials are all of deplorable quality (very small and with lack of blinding). He argues that "at the moment the only scientific response is to withhold judgement."

According to the interpretation of GRADE Busse proposes guideline panels should recommend ivermectin even though the quality of evidence is very low.

 

[cassava7]

Thought this is a good example of why what Jason Busse propose is problematic. How Covid-sceptics were duped by the “wonder drug” ivermectin (newstatesman.com)

Exactly. The article mentions the "garbage-in, garbage-out" issue with meta-analyses:

A meta-analysis is only as good as the studies that it’s meta-analysing; the “garbage-in, garbage-out” principle applies. For ivermectin and Covid, at the moment we mostly have garbage – and it wouldn’t be the first time that entire lines of low-quality medical research all pointed in the wrong direction. Now that at least one large, high-quality study of ivermectin and Covid is planned, we’ll soon have some better data. But at the moment the only scientific response is to withhold judgement.

Why Busse, Guyatt, Schünemann et al. are so adamant on making recommendations from low-quality evidence is unclear to me.

Within and beyond the field of medicine, the relevance of any sort of data analysis depends on the input data. The modern example of machine learning illustrates this issue best and must be a tale of caution: the biases that are prevalent in many datasets used to train ML algorithms result in serious, adverse societal consequences (including healthcare decisions). The GRADE working group should read Cathy O'Neil's book Weapons of Math Destruction, which I highly recommend to anyone wanting to gain an understanding of this problem.

 

[alex3619]

The proper process is for people with enough experience and skill in logic to view the evidence available and decide what its implications for recommended management are in one integrated decision step. Any intermediate steps of forcing information into grading levels and using arbitrary rules for moving up and down grading levels is logically invalid and bound to interfere with, rather than assist, a decision.

I have been in agreement with this line of thinking for maybe nine years. I was working on background reading on this kind of thing when my cognition went into a downward spiral. A very similar argument applies to evidence based medicine rankings, and formulaic meta analysis. This is where I was first looking at it. Rather than evidence based medicine, this is covered under evidence based practice. Don't give doctors formulas and dogmatic advice, teach them to figure it out if they need to.

Scientists in research and medical professionals in research and clinical practice can be taught the necessary skills. Instead we replaced one dogmatic system with another. Published medical reviews are useful, but I am not convinced that formulaic approaches work effectively in many cases.

I was in particular looking at intuitive-heuristic medicine versus rational medicine. Both have their uses.

The only question I have is about one integrated decision step. At any point, yes, but asking questions, looking for answers, and getting better answers, should be part of the process. Medicine evolves as our understanding and technologies evolve.

 

[alex3619]

Exactly. The article mentions the "garbage-in, garbage-out" issue with meta-analyses:

Psychobabble loves GIGO as a methodology. This is why I proposed BIBO, but it never took off. Babble in, babble out. Of course the issue in much of medicine is not babble, so GIGO still applies to a proportion of it.

 

[alex3619]

It all seems to come down to Eric Morecambe: they are playing all the right notes but not necessarily in the right order.

Yes, tick-box mentality. There is a danger in substituting bureaucracy for science and reason.

 

[Sly Saint]

just linking to this thread as might be of interest here:
https://www.s4me.info/threads/assessing-randomised-controlled-trials.5420/

 

[Jonathan Edwards]

I still can't quite grasp why NICE would use GRADE - if there was one organisation who should know whether GRADE was sound/unsound it's NICE --- in fact if a GRADE type evaluation system was useful/deliverable then there's an argument that NICE is well placed to develop/commission it.

The central irony of GRADE is that it is designed to show 'how the people who know how to do it do it, so that you can do it too'. But NICE are supposed to make use of the people who know how to do it, surely.

 

[Kitty]

But NICE are supposed to make use of the people who know how to do it, surely.

What I find hard to grasp is how you persuade distinguished specialists to agree in the first place that they need to set aside all the expertise and experience they've accumulated over decades, and make decisions by going through a half-inch thick printout of an algorithm instead. A sort of grown-up version of a Year 5 workbook.

It just sounds as it it deserves some kind of chutzpah award. I'm sure distinguished specialists are still capable of being remarkably dim, but I doubt they see it that way themselves.

 

[FMMM1]

The central irony of GRADE is that it is designed to show 'how the people who know how to do it do it, so that you can do it too'. But NICE are supposed to make use of the people who know how to do it, surely.

Exactly i.e. "NICE are supposed to make use of the people who know how to do it, surely". OK if it's difficult to get hold of the experts, understandable as they are treating patients, then an idea would be to use a screening tool - but not when it doesn't work [GRADE]! I have to say that you couldn't make it up - the body charged with deciding on treatment paths [NICE] uses a black box which is a piece of crap [GRADE]!

 

[Invisible Woman]

OK if it's difficult to get hold of the experts, understandable as they are treating patients,

Not necessarily all of the time, though.

In other fields specialists have appraisals and career development paths. Some of it technical and some might be more personal development.

There are training days allocated to meeting up with specialists with similar technical backgrounds where you discuss how you are going to handle developments that are coming down the track in the next year or so.

Other professional fields have to handle a lot more of this kind of information all the time & without the benefit of scientific trials. Often you might have the manufacturing blurb which sounds great until you try to implement the thing in a real world environment.

They do it by employing the right level of expertise to assess the new tech in the first place and then using those same experts to mentor and train up more junior colleagues.

It seems to me the difference here is empire building and removing responsibility & power in order to build that empire.

 

[FMMM1]

They do it by employing the right level of expertise to assess the new tech in the first place ---
It seems to me the difference here is empire building and removing responsibility & power in order to build that empire.

Think that's the problem "They do it by employing the right level of expertise to assess the new tech in the first place" ---- or in the case of NICE that doesn't appear to have happened re the original ME guidance.

My point was that Doctors like @Jonathan Edwards when they are dealing with patients --- progressing research --- may be the people you want [NICE] but not easy to get hold of --- understandably.

I think it's likely cock-up not conspiracy but your post makes me think that yes there will be people who have the critical thinking skills, who are available, and willing to participate [NICE].

 

[Jonathan Edwards]

My point was that Doctors like @Jonathan Edwards when they are dealing with patients --- progressing research --- may be the people you want [NICE] but not easy to get hold of --- understandably.

In fact we were not difficult to get hold of at all. Expert people are more or less always ready to put aside the time and have the time. When rituximab for RA went to NICE I was not asked to advise, despite the fact that I had far more experience than anyone internationally. In particular I had more knowledge of the adverse effects.

There is paranoia about using people who might have competing interests. The irony is that the paranoia tends to lead to silly rules that allow exactly those people to dominate the situation. Somebody must have worked hard in NICE to turn the tables for ME this last time. Somebody made a very clever decision to refuse me a place on the committee but offer me the option to present evidence, which was much more useful.

 

[FMMM1]

Interesting --- at least there's a positive outcome for the ME community.

 

[FMMM1]

I am certainly not talking about taking over Cochrane's conclusions. I am simply referring to all the searching and sorting into subgroups that must have gone on at both - prior to any evaluation. Here at S4ME as a community we already knew enough about the scope of the data before NICE began. Why trawl through it all again and more importantly why sort it into all these mindless subgroups? We knew from looking at the abstracts that the studies were incapable of providing usable information on efficacy.

If you're wondering about NICE dividing studies into mindless sub-groups then it might be that those doing the sifting don't have much insight.

I did wonder if one reason was to refer to every possible study. The easy grounds to judicially review something are procedural grounds e.g. that they did not consider relevant studies [the court won't really consider whether they're relevant or not!]. It's easy to refute that challenge since the court will merely ask --- did you refer to said study/s and "consider" them - in fact the court might decide that it's enough to have "mentioned X study" as evidence that it was considered. So some of this might be NICE avoiding providing grounds to take a judicial review.
I don't know that much about it - it may be obvious!

 

[Caroline Struthers]

Meet a supporter of investigating the 'LP' with inappropriate trial design, a coordinator of the authors team of the Cochrane review on exercise for CFS, and at least one more defender of exercise therapy for ME/CFS:

https://www.informedhealthchoices.org/about/the-ihc-team/

Edited to add: I only had a very quick look at those tools, so not able to assess them. They might make mostly reasonable points.

This explains a lot. This team also includes Sir Iain Chalmers, founder of Cochrane. Paul Glaziou, adviser on exercise review, another founder Cochrane person. Atle Fretheim, Larun's boss . Andy Oxman - another Cochranite and GRADE author who I think was approached to adjudicate over the withdrawal/amendment of the exercise review before it was then given to Guyatt? It's all coordinated by the pro-LP, exercise, CBT etc Norwegians at the Institute of Public Health in Norway. Many of whom seem to be friends with Hilda Bastian which somewhat undermines her independence overseeing the new exercise review. But I guess this will be disclosed in her declaration of interests if and when the new review ever happens.

 

[Jonathan Edwards]

The irony this organisation is quite striking. In their blurb they say;

People’s mistaken trust in unreliable claims has led to billions of dollars being wasted every year, untold suffering and millions of unnecessary deaths. Conversely, failure to believe and act on reliable claims also leads to inefficient use of resources and unnecessary suffering. This creates a disproportionate burden for economically disadvantaged people who can least afford to waste resources.

This looks like evangelism based in emotion rather than - er what they are claiming to be base in - sound reasoning.