Anomalies in the review process and interpretation of the evidence in the NICE guideline for (CFS & ME), 2023, White et al

Mark Vink said:
Why is GRADE garbage?
Click to expand...

Because it reduces reasoning based on detailed arguments to arbitrary 'numerical' grades that have no arithmetical validity as numbers and then adds them up, which in reasoning terms is garbage.

If a type of trial is known to be totally unreliable because it is open to very easy abuse through expectation bias then the evidence from it is prima facie valueless, not just one pip or two pips down from some non-existent number signifying validity. Moreover, there may still be useful information if you interrogate in detail, so an overall score is not good reasoning. You might be able to see that the trial makes a positive result not only uncertain but very unlikely indeed to be real - i.e. the trial is reliably negative, like PACE. A positive difference against control might be so huge (compared to similar trials) that a real result is actually likely after all. GRADE gets nowhere near this sort of subtlety, which, despite being subtle, is what we expect to be part and parcel of everyday decision-making. Human decision making depends on vastly complex and mostly unconscious comparison processes going on in our brains which if we are lucky we can trace back through and lay out in text in clear reasoning. It has nothing to do with giving scores of minus one or minus two to reliability.

This has been pointed out in the literature. It is not my idea. GRADE comes from the people at McMaster who are typical of the 'quality police' types who so often are actually the least insightful on such matters.
 
Jonathan Edwards said:
Because it reduces reasoning based on detailed arguments to arbitrary 'numerical' grades that have no arithmetical validity as numbers and then adds them up, which in reasoning terms is garbage.

If a type of trial is known to be totally unreliable because it is open to very easy abuse through expectation bias then the evidence from it is prima facie valueless, not just one pip or two pips down from some non-existent number signifying validity. Moreover, there may still be useful information if you interrogate in detail, so an overall score is not good reasoning. You might be able to see that the trial makes a positive result not only uncertain but very unlikely indeed to be real - i.e. the trial is reliably negative, like PACE. A positive difference against control might be so huge (compared to similar trials) that a real result is actually likely after all. GRADE gets nowhere near this sort of subtlety, which, despite being subtle, is what we expect to be part and parcel of everyday decision-making. Human decision making depends on vastly complex and mostly unconscious comparison processes going on in our brains which if we are lucky we can trace back through and lay out in text in clear reasoning. It has nothing to do with giving scores of minus one or minus two to reliability.

This has been pointed out in the literature. It is not my idea. GRADE comes from the people at McMaster who are typical of the 'quality police' types who so often are actually the least insightful on such matters.
Click to expand...

Thank you. I had difficulty finding that sort of literature which shows the problems of GRADE. Can you recommend a good article about that?
 
Mark Vink said:
Can you recommend a good article about that?
Click to expand...

There is one in particular that points out the bogus nature of the numerical scale. It is hidden in a thread here somewhere. I will see if I can find it. The first author has I think a short Asian name but I may be dreaming.

Edit:

Have a look at the thread :
Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

At least one critical paper comes up there - I think probably the one I am thinking of.
 
Last edited by a moderator:
Jonathan Edwards said:
Because it reduces reasoning based on detailed arguments to arbitrary 'numerical' grades that have no arithmetical validity as numbers and then adds them up, which in reasoning terms is garbage.
Click to expand...
Which seems much the same flaw as with some of the psych questionnaires. As if allocating fairly arbitrary numbers to points on what may well be very non-linear and different scales in reality, and then totting them up and aggregating them as if all the scales are the same and linear ... it's no wonder they can be massaged to "prove" whatever you want to prove.
 
Jonathan Edwards said:
There is one in particular that points out the bogus nature of the numerical scale. It is hidden in a thread here somewhere. I will see if I can find it. The first author has I think a short Asian name but I may be dreaming.

Edit:

Have a look at the thread :
Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

At least one critical paper comes up there - I think probably the one I am thinking of.
Click to expand...
Thank you!
 
Last edited by a moderator:
Jonathan Edwards said:
Have a look at the thread :
Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

At least one critical paper comes up there - I think probably the one I am thinking of.
Click to expand...
Just had a look back at this, and reminded that the flaws in GRADE mean it cannot adequately process factors that should have significantly different weightings. So if just one thing is sufficiently flawed to render a trial totally worthless, then GRADE cannot reflect that.

Like if a trial was run to assess a recipe for making a tasty omelette, but the trial was run with everything good ... except the cooker did not work. My understanding is that GRADE would be incapable of rating that trial as totally worthless, even though they ended up with nothing to taste.

If something falls over in an extreme case like this, it often indicates it may fail in more subtle ways away from the edge cases; ways that might get misinterpreted to be of differing quality from the reality.
 
Last edited by a moderator:
Barry said:
Which seems much the same flaw as with some of the psych questionnaires. As if allocating fairly arbitrary numbers to points on what may well be very non-linear and different scales in reality, and then totting them up and aggregating them as if all the scales are the same and linear ... it's no wonder they can be massaged to "prove" whatever you want to prove.
Click to expand...
About exactly what was going through my head reading the comment above. They are using basic arithmetic on numbers that aren't even natural. Where they do the equivalent of 1+1+1+1+1=5, but each number representing 1 doesn't have a fixed value of 1 in a valid number system. This way of doing things is mathematically invalid. Every single number has its own internal metric / exchange rate with multiple factors influencing them, from the way questions are framed, to which options are available and adding up individual experience and interpretation.

It isn't even an apples to oranges kind of thing, it's apples to the color mauve if it was a spirit animal rendered as an interpretive dance using lasers.
 
I suspect many researchers using these endless questionnaires so much have forgotten this basic of psychometrics that such numerical values do not represent actual numbers but are effectively a metaphor, that is if they ever knew it.

[edited to remove a confusing preposition]
 
Last edited:
rvallee said:
It isn't even an apples to oranges kind of thing, it's apples to the color mauve if it was a spirit animal rendered as an interpretive dance using lasers.
Click to expand...
Peter Trewhitt said:
I suspect many researchers using these endless questionnaires so much have forgotten this basic of psychometrics that such numerical values do not represent actual numbers but are effectively a metaphor, that is if they ever knew it.
Click to expand...
Yes, akin to saying: I want to aggregate somehow, goats, cars, wheelbarrows, and the wind in the willows ... how can I possibly do that? Ah, I know, I'll assign numerical identities to them, then it's simple ... just add them up :).
 
Barry said:
Yes, akin to saying: I want to aggregate somehow, goats, cars, wheelbarrows, and the wind in the willows ... how can I possibly do that? Ah, I know, I'll assign numerical identities to them, then it's simple ... just add them up :).
Click to expand...
Kind of reminds me of the budget jokes:

Folks help me out is my monthly budget OK?

Food: $300
Rent: $800
Utilities: $250
Loan on aircraft carrier: $2.6M
Internet + mobile: $100
Misc, repairs, etc: $150

I think I need to cut back on Internet, but maybe I could eat less food, too? What do you think?​
 
You must log in or register to reply here.
Back
Top Bottom