Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

i didnt have the energy to write anything, or even, tbh, to really read/absord whats being said about grade, my brain just wont take it in. but then i read Wonko's post & he has said precisely what i want to say about the whole thing.

 

Measurement is literally the starting point of all science, it's the point at which it can become rigorous. The idea that it can be replaced by assigning pseudo-numbers to vague concepts in a fully interpreted process is delusional.

 

The issue is really that it's a default explanation, it requires no evidence for it, it's what's left after a process has exhausted, whether that process was rigorous or not. Which means a dead chicken leading that process would have the exact same conclusion. As would a rock, or The Rock.

The idea of a default explanation in any science is absurd, even more so in medicine. But it is a cornerstone of medicine going back decades. So we are essentially stuck because of a tradition defaulting to a delusional belief. And it's completely blocking the process of actually applying medical science to the issue, the only way out of this rut.

 

I sooooo want to be an offensive research officer!

 

Thank you for this. I agree surely NICE could employ scientist graduates to do this "I think it would be fair to ask technical staff to search for studies and document a list of features ----." E.g. is the study blinded, does it use object outcome indicators [FitBit or whatever] ---?

Hope @Caroline Struthers doesn't mind the mention, but there are bound to be well qualified, capable, people to carry out assessments of published studies --- it looks like most of the current psychological studies could be disregarded using the list of features.

 

And very especially disregarded when egregiously misapplied to physiological conditions.

I truly believe there should, within the medical system, be some kind of enforceable sanity check applied to psychological trials if there is a non-trivial chance the condition under test is potentially physiological in origin, and potential for harm by being so misrepresented.

There seems to be a hinterland between the two worlds, except the psych brigade seek to hoover up all who get lost within there.

 

I think it should be applied to everything.

I understand there are many conditions where we just don't have a clue. Where we either do nothing or we try to alleviate some suffering.

Where we are trying to simply alleviate suffering until we know more then it is just as important as ever to be alert to assumptions & hypothesis with nothing to underpin them and any potential harms or stigma caused.

Any system that doesn't allow for ranking of zero evidence or even that the treatment is harmful & so has a negative score simply doesn't reflect reality and is a danger to patients. Especially when used by people not competent to understand it's shortcomings and those are the people GRADE has been partly developed for, if I understand the issue (& I mightn't).

Psychiatry has a barbaric history. I had a relative who was a psychiatric nurse 50+ years ago and they had some horror stories to tell. Those with poor mental health are more vulnerable than most and most likely to have complaints of harms dismissed. When it comes it psychiatry or psychology patient safety should be held to a higher safety standard IMO, because it's so much harder for the patient to be believed.

 

"Id Dr Busse had been appointed as chairman presumably there is a high chance that the result would have been different. That makes the system fragile."
As a (very) general rule the proposed position/outcome is consulted upon and the hope is that the consultation should reduce the risk that the final position/outcome is crap. E.g. the 2007 (draft) guidelines didn't consider the quality of the evidence produced by the studies; if the process worked properly then this might have been picked up in the responses to the consultation. I'm assuming that GRADE rated the studies as "moderate" while in reality they were "low or very low" quality. Then there's the issue of the power imbalance which can occur i.e. the folks from University of York’s Centre for Reviews and Dissemination, who participated in the production of the (2007) guidelines, knew the studies were flawed* but that wasn't reflected in the final guidelines. So the system is indeed "fragile" and the consultation on the guidelines doesn't always pick up the issue of the underlying analysis being crap --- as for GRADE ---. This also illustrates the value of the folks who participated in the current review and indeed in trying to resolve the current impasse/"pause" --- @Jonathan Edwards @Brian Hughes +++

*https://thesciencebit.net/2021/08/1...the-new-nice-guideline-ask-about-the-old-one/
& https://jamanetwork.com/journals/jama/article-abstract/194209

 

It's a bit confusing, but there's essentially two different scales going on. There's the different domains for determining the level of the risk of bias of studies (randomization, blinding, attrition, selective outcome reporting), and then that level of the risk of bias of the various studies in an outcome determines how much that outcome in GRADE should be downgraded for the overall "risk of bias" domain, which is one of the GRADE domains (indirectness, imprecision, inconsistency etc.).

So in terms of the question of whether blinding with subjective outcomes alone can cause enough risk of bias for the GRADE outcome to be downgraded by two increments, that doesn't seem to be the case for me. I've tried reading though some of the guidance on the risk of bias tool, which can be found here: https://sites.google.com/site/risko...-2-0-tool/current-version-of-rob-2?authuser=0. Which isn't perfect, because the tool is updated, and the one used in the cochrane review would have been a previous version. Blinding falls under the "Risk of bias due to deviations from the intended interventions" domain in that, and the blinding part of the domain seems to simply look at whether the participants and investigators were blinded, there doesn't seem to be a basis, within that single domain, for judging the risk of bias to be even higher because of the kinds of issues with PACE and other trials.

Those issues could fall under other domains of risk of bias though, like bias in the measurement of the outcome, though I'm not entirely sure. That risk of bias domain didn't seem to be used at the time of the cochrane review though.

It does seem to be possible for an outcome to be downgraded two increments if the issues with a single risk of bias domain are serious enough, but I don't get the impression it would have been possible to do that under the "blinding domain" in the cochrane review, I think they would have had to rate it at high risk of bias for another domain.

 

Anyone know what prompted this? (Guyatt and Flottorp are signatories). Wondered if I was missing some background to it:

https://www.sciencedirect.com/science/article/abs/pii/S0895435622002426?via=ihub

Strong and high quality evidence synthesis needs Cochrane: A statement of support by the GRADE Guidance Group

 

I think maybe Cochrane is getting a bit desperate?? I have heard anecdotally that Gordon Guyatt is not a fan of Cochrane any more. But he intervened as an "independent arbitrator" to get them out of trouble with the Exercise review by approving the authors' dodgy use of GRADE to rate the evidence on one outcome as moderate rather than low or very low (as they were asked to do by the outgoing Editor in Chief)

 

I see that Schünemann is listed as

1. Department of Health Research Methods, Evidence, and Impact
2. Michael G. DeGroote, Cochrane Canada & McMaster GRADE Centres, McMaster University, Hamilton, ON, Canada
3. Institute for Evidence in Medicine, Medical Center and Faculty of Medicine, University of Freiburg, Germany
4. Department of Medicine, McMaster University, Hamilton, ON, Canada

Which could potentially explain why COMET initiative: Core Outcome Set for Chronic Fatigue Syndrome/Myalgic Encephalomyelitis is happening at McMaster as well.

 

Maybe they should re-name it McEvidence University?

 

Perhaps we can get fries with our evidence as well, or maybe even super-size it?!?!

 

Perhaps it's just that Cochrane funding sources have threatened to pull the plug on McMaster and the author?

 

Maybe the McMaster (lacks evidence) School --- they'll probably think "loads of money"!

 

Wait, it's the GRADE guidance group basically writing flowers about themselves? Or in support of Cochrane? Or of Cochrane using GRADE?

This mutual admiration society sure is very admiring of everything about itself.

 

Sorry I'm getting confused on what that last bit means for sure - probably because I'm not familiar with what normal process would be and what circumstances an independent arbitrator would be required. Did both the editor and the author want it as moderate, or disagree?

 

The author wanted to keep it moderate, but the outgoing editor in May 2019 (David Tovey) wanted it downgraded to low. But he left before it was decided. The new editor from June 2019 (Karla Soares Weiser) decided not to argue for what David wanted and so threw it over to Gordon Guyatt to arbitrate. The whole correspondence is here .

Gordon Guyatt was suggested to Karla by Andy Oxman, who works for Cochrane Norway https://www.cochrane.no/contact-us. Cochrane Norway is hosted by the review authors' institution, the Norwegian Institute of Public Health https://www.fhi.no/en/cristin-projects/ongoing/cochrane-norway/. Andy Oxman therefore has an interest in not upsetting the institution that funds his work. So Gordon Guyatt was not an independent choice.