Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

rvallee said:
I don't know if it will happen soon, but it will be possible within 1-2 years. AI systems like this have to be trained, so someone would have to spend resources on it. This is mostly money and labor.

But being to copy (well, do a lot better and faster anyway) the process of systematic reviews is on the low-end of what AI will be capable of. Real systematic reviews, not cherry-picking that arbitrarily filters out 99% and pretends this isn't a reflection of the evidence.

Only question is who would build such a system. It would essentially make Cochrane mostly obsolete, so that could be awkward. It will happen for sure, but the uncertainty is political more than technical. Lots of jobs doing nothing but this GIGO process. And removing people from the loop removes a lot of the bias that keep many of those jobs going.

We all know how this review would look like if the Cochrane handbook were applied, I don't think a single study actually qualifies. For sure it would not have come to the conclusion that this is worth a damn.
Click to expand...

Programs like this will not only make Cochrane absolete but an enormous percentage of white collar jobs. My brother used to train helpdesk-employees, he'd have to build training programs so that people could actually give solid advice within a couple courses. The jobs on either side of that equation will be obsolete not too long from now as it wouldn't be too hard to hook up ChatGPT to a bunch of user manuals, have it extrapolate useful information and then answer the phone for you. Or just answer online.

My work pre-ME was as a mid-level accountant I think you could say(3 years of study). I've worked for a company that did the books for self-employed people and small businesses. At an internship some woman was still writing stuff down manually whereas we were thought in excel, which was much quicker at the time. Excel got replaced by an accounting-program which got replaced by a better one and so on and so fort, tremendously increasing output of a single accountant. I'd imagine the coming evolutions would eliminate the job almost completely.

We're already able to scan just about anything with our phones, we can pay with our phones too. It's a very small step from that to having your phone immediately log it in your accounts. Government being disastrous at IT will make the final hurdle a bit harder to climb, but once you can log everything into your accounts at once it should also be very doable to get a summary at the end of each year which you just have to gloss over before sending it to the inland revenue.

With machine-learning the easy to spot errors can already be found in research papers, once this thing blows up further it can likely review thousands if not millions of papers at once and spot every tiny thing wrong with them.

But as you said, the algorithms need training. The thing is it gets easier to train with each new iteration and we likely won't be dependent on Cochrane for setting something up. Some angry scientist who has a buddy that specializes in automation could likely set it up somewhere in the future, from my limited understanding.
 
Solstice said:
But as you said, the algorithms need training. The thing is it gets easier to train with each new iteration and we likely won't be dependent on Cochrane for setting something up. Some angry scientist who has a buddy that specializes in automation could likely set it up somewhere in the future, from my limited understanding.
Click to expand...

I need to see a movie or mini-series about this happening, please.
 
Solstice said:
Excel got replaced by an accounting-program which got replaced by a better one and so on and so fort, tremendously increasing output of a single accountant. I'd imagine the coming evolutions would eliminate the job almost completely.
Click to expand...

Given what happened to postmasters and postmistresses in the UK because of accountancy software with bugs in it I think the prospect is terrifying.

https://en.wikipedia.org/wiki/British_Post_Office_scandal
 
Caroline Struthers said:
A spokesperson for Cochrane told The BMJ that its editors have to consider whether a review “would add value to the existing evidence base,” highlighting the work of the UK’s National Institute for Health and Care Excellence, which looked at puberty blockers and hormones for adolescents in 2021. “That review found the evidence to be inconclusive, and there have been no significant primary studies published since.”
Click to expand...
Thanks Caroline.
My initial reaction is that it's terribly convenient for Cochrane - they're politically clever enough to realise this [puberty blockers and hormones for adolescents] is risky territory. OK to misrepresent those with ME/CFS of course --- less controversial target.

Great idea to use this against Cochrane!
 
Caroline Struthers said:
“if scientists come to rely on AI programs to prepare literature reviews or summarise their findings, the proper context of the work and the deep scrutiny that results deserve could be lost”.
Cochrane is so good at that "deep scrutiny" isn't it?! https://www.theguardian.com/science...ban-listing-of-chatgpt-as-co-author-on-papers
Click to expand...
The weird thing about this is it doesn't seem to understand that AI-generated content can, and should, be checked by humans.

This is basically the same as not trusting mechanical calculators because human computers, it used to be a job title, are much better at it.

It's not as if the content has to be used as is, without even checking. It's a multiplier, it doesn't remove human intelligence from the loop. But it does have the potential to remove bias, and that is basically disastrous for psychology and everything that relies on their methods, like EBM, where they need bias, otherwise they have nothing.

But on the part of Cochrane, I don't really see their product, the reviews, as their main thing. It's a club, it's about people connecting with other like-minded people and using that influence. The reviewers are the real product, not the reviews. If all that's left is the reviews, the entire organization is superfluous, as they can be produced far better, faster and cheaper without the bloat of their private influencers club.
 
Adrian said:
As should content that is generated by humans
Click to expand...
Speaking of which, I saw a few people experimenting with ChatGPT to review some of their old papers.

Lots of work ahead but it seemed to do better at the most basic stuff than the obscure, which is pretty much the opposite we see with humans, who can argue forcefully about semantic trivia while being perfectly OK with overlapping too-disabled-to-work and recovered-and-fully-healthy thresholds.

Or somehow being OK with open label trials with subjective outcomes from biased researchers, even when they're not even allowed. Bots won't accept the "pseudoscience is Ok for pseudo-illness" nonsense unless it's explicitly written in their instructions, which I wouldn't put past organizations like Cochrane.

Not sure what an editor would think of a bot review that says "didn't bother reading past the abstract", though.

Only a matter of time before it does better than humans, although of course better than humans is not desirable to some, and basically a death sentence for the BPS ideology.

For further discussion go to this thread:
OpenAi's new ChatGPT
 
Last edited by a moderator:
Back on topic.

Trial By Error: What the Hell is Going on with Cochrane’s Long-Delayed Re-Review of “Exercise Therapies”?

"What is going on with Cochrane’s much-delayed re-review of exercise therapy for the illness the organization is these days calling ME/CFS? It is more than three years since Cochrane announced plans to develop this completely new review, but no one outside the organization seems to have much idea of what’s happening with that project.

Maybe no one inside really does, either.

Patients are frustrated and angry—and rightly so. The existing review, as flawed as it is, still gets cited as evidence that exercise therapies have been shown to be effective treatments. Recently, for example, the authors of a propaganda piece presented in the guise of a “research agenda for post-COVID-19 fatigue” referenced the contested Cochrane review to assert this exact point and suggest the same interventions would be helpful for the new global cohort of long Covid patients. However, in a telling omission that highlights the weakness of their argument, the authors failed to mention the current ME/CFS guidelines from the UK’s National Institute for Health and Care Excellence (NICE). These were published in 2021 and reversed NICE’s prior exercise therapy recommendation; the agency assessed the quality of the evidence in favor of these interventions as uniformly “low” or “very low.”"

More at https://www.virology.ws/2023/03/21/...long-delayed-re-review-of-exercise-therapies/

Click to expand...
 
Caroline Struthers said:
Yes, interesting what you say about the protocol and NICE. I was reading this BMJ article today https://www.bmj.com/content/380/bmj.p382?utm_source=substack&utm_medium=email.

You don't need to read the whole thing, but towards the end it says

"Cochrane, an international organisation that has built its reputation on delivering independent evidence reviews, has yet to publish a systematic review of gender treatments in minors. But The BMJ has learnt that in 2020 Cochrane accepted a proposal to review puberty blockers and that it worked with a team of researchers through 2021 in developing a protocol, but it ultimately rejected it after peer review. A spokesperson for Cochrane told The BMJ that its editors have to consider whether a review “would add value to the existing evidence base,” highlighting the work of the UK’s National Institute for Health and Care Excellence, which looked at puberty blockers and hormones for adolescents in 2021. “That review found the evidence to be inconclusive, and there have been no significant primary studies published since.”
Click to expand...

dave30th said:
this is an interesting point.
Click to expand...

Agree it is interesting. But is it a good example?

I think the situation is hardly comparable -- only able to skim-read but if i understood correctly, the NICE commissioned an external reviewer to evaluate the evidence who found the evidence base not strong enough to issue an actual position:

"The evidence reviews were published in March 2021. Unfortunately, the available evidence was not deemed strong enough to form the basis of a policy position."

https://cass.independent-review.uk/nice-evidence-reviews/

https://www.england.nhs.uk/commissi...mme/implementing-advice-from-the-cass-review/

From skimming the bmj article, the references and also the responses, I don't have the impression the mentioned bmj article gives a fair account, neither of some details regarding the evidence nor the involved parties' disagreement on interpreting the evidence.

The evidence on exercise for ME/CFS I think is pretty conclusive as is the current NICE guidance on ME/CFS -- exercise is not an effective treatment for ME/CFS.

Yet, for decades, proponents of exercise for ME/CFS both have done bad trials and have misrepresented the evidence, and they continue to do so.

And the bmj has played its part in misrepresenting the evidence and also how NICE arrived at its conclusions in the new ME/CFS guideline.

Anticipated apologies if I misunderstood or misrepresented anything, but I think we need to be very cautious with referring to material and decisions that seem to suit our case without taking a closer look.

Also, I'm reminded of the NICE guidance on chronic pain (discussed here) -- would we want Cochrane to adopt NICE' evidence review of this guidance, too?


(Edit to replace centuries with decades :facepalm: )
Edit 2 to remove redundant part.
Edit 3 to remove another redundant part and to clarify the point about the bmj's role in the tangled story of ME/CFS
 
Last edited:
MSEsperanza said:
I think the situation is hardly comparable -- only able to skim-read but if i understood correctly, the NICE commissioned an external reviewer to evaluate the evidence who found the evidence base not strong enough to issue an actual position:

"The evidence reviews were published in March 2021. Unfortunately, the available evidence was not deemed strong enough to form the basis of a policy position."
Click to expand...

For me what is relevant is not whether the ME exercise situation or the use of puberty blockers have any specific parallels, but rather that it is a precedent of Cochrane using NICE reviews to inform their own action. Obviously Cochrane should not be bound to slavishly adopt any NICE position, but I think Cochrane now needs a clear justification for not withdrawing their flawed exercise review completely and why they need to commence a new review, in the light of the seemingly clear conclusions from the NICE evidence review and the consequent new NICE ME/CFS guidelines.
 
Peter Trewhitt said:
For me what is relevant is not whether the ME exercise situation or the use of puberty blockers have any specific parallels, but rather that it is a precedent of Cochrane using NICE reviews to inform their own action.
Click to expand...
I realized that bit but the reasoning behind this officially seems to be there is no sufficiently strong evidence base to do a review. That is a step before doing the actual assessment of the evidence.

And again, just to follow NICE's decisions per se seems not a good point -- see the old NICE guideline (based mainly on the same evidence base but different criteria in the assessment process) and also the current chronic pain guideline.

Not defending Cochrane in any way just don't think this is a good point on our behalf.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom