David Tuller: Trial by Error: My exchange with Professor Bishop

http://www.virology.ws/2018/06/25/trial-by-error-my-exchange-with-professor-bishop/

 

So much fail
So if i did a trial on homeopathy, played with the trial protocol and reported its effective thats now good science?

If i use a magic 8 ball enough times it will eventually say "All signs point to yes"

 

What is going on with Bishop?

She keeps seeming to misunderstand the details of CFS research in sloppy ways, and it's always in ways that favour the researchers doing poor work.

 

I would think she believes CFS is psychosomatic hence any evidence run by her is all about confirmation bias and reinforcing the predetermined conclusion

 

Prof Bishop does come across as sincere. And she seems to genuinely believe that Crawley was just testing LP out of some sort of commitment to the scientific process, in spite of having doubts about it (we know that story is false). I think she probably genuinely believes that these researchers are ethical and its just a bunch of crazy patients making a great big fuss over nothing.

But then aren't these people the real problem? These people who claim to be standing up for a better, more open science? The people who ought to look more carefully, whose opinion carries weight, but they just gloss over stuff, believe the lines the authors spin, and remain complacent? I find myself getting quite angry about it.

Bishop's quite outspoken about the horrors of psychodynamic approaches to autism (the 'refrigerator mothers' stuff). But similar harms are being done to PwMEs, and she dismisses us as no better than climate change deniers.

I'm not sure what to make of Prof Bishop's intellectual abilities. She seems to do fairly solid work and lots of it. But razor sharp? Not sure. She asked a question on twitter once which kind of shocked me. She'd done this study that involved comparing two groups that were selected to be different on some measure (I think it might have been scores on a reading or language test). In her study, she compared how these two groups performed on some experimental task. But then she combined the two groups together and ran some correlations.

This is a super no-no. If you choose two groups on the basis of their distinctness from one another, you can't then mix them all up and perform correlations. The reason is that you have taken out a lot of the natural variance by selecting people that were maximally different for your groups in the first place. This will have the effect of inflating any correlations that you find.

Anyway, she was criticised by a reviewer for doing this combined correlation, and she asked on twitter whether the reviewer was right or not. She had never heard of that before. I was a little shocked by that! I mean, kudos to her for actually having the courage to ask I suppose. But still a a bit scary.

 

She's a reactionary skeptic. She once pounced on a tweet of mine in which I was suggesting a book for a popular science prize, and she derided it as being too pop sciency(!) It was clear she had neither read the book in question or understood the research behind it, even though it should have been her area (developmental neuroscience). [Although, maybe it was because it was her area and she disapproved.] I steered well clear of her after that.

 

As you point out @Woolie, it is not something you need to have 'heard of before'. It is barn door common sense. Anyone who does something like this simply does not understand how the world works. Academia is full of them, as you know. Again, it seems that the quality police are so often people of not particular quality.

 

Can't help noticing that this has a familiar ring to it.

 

If PACE taught me anything, besides unblinded trials relying on questionnaire being unreliable, it's that science is hard and even the pros make a lot of mistakes. Many of these mistakes are probably due to a lack of time. Having a good understanding of a problem is very time consuming.

 

That said it is very disappointing that Prof. Bishop doesn't seem to have recognized LP as exploitative pseudoscience. Kids being instructed by a guru figure to stand in "magical circles" and telling their illness to stop, at the cost of several hundred pounds. Or perhaps she really has bought fully into the delusional somatization disorder narrative which seems to make this kind of exploitation suddenly OK in the eyes of some.

 

Maybe I am being hopelessly naive for asking this.

But is there a chance that some of the CBT proponents who support LP don't even know how the LP actually functions?

It honestly wasn't until the Buzzfeed article last year that I found out just how crazy the LP actually was. I thought it was ''just'' an intense form of CBT, and not as it turned out, an inbred form of CBT on acid.

 

Yea. My take on the Bishop reply is that she is kind of uncomfortable with the fringey-ness of LP, but can see absolutely nothing wrong with the way the trial was conducted.

This enables her maintain a position that Schrodinger would be proud of: able to support her follow SMC member, while at the same time preserving her reputation as a "proper" scientist.

 

I agree that these are the main issues specific to LP. The technical flaws in the methodology of the study are a secondary issue and speak more generally to the quality of work being done by specific individuals. I think if you want to really put someone on the spot just ask them why they think the salutary findings of a study investigating the effects of brainwashing are reliable. The 'sterile' technical flaws - however egregious they may be - provide a bit of an escape inasmuch as they are fertile ground for distracting haggling and prevarication. Even if every facet of the study external to the specific machinations of the lightning process were done as well as possible, it would still be neither science, nor ethical.

 

I have serious concerns about the ethics involved with LP that I haven't seen mentioned. Parents are desperate for their children to get better so it is not like making a decision about a new car, say. Someone saying "this will cure your child and give them their life back" - parents will go to moneylenders to pay for it.

On another forum we had a mother who was upset when we didn't back her taking her daughter to it. This was someone living off benefits.

A peer reviewed "evidence based" trial finding LP gets better results than standard care only feeds into this desperation. I do not think the extra pressure parents feel is taken into account.

Saw a TV drama where unscrupulous people were selling quack cures to parents of children with terminal cancer; these so called doctors are doing the same thing.

 

But the name: LIGHTNING process. It sounds religious-esoterical to me. Alone this makes LP very uninteresting for me, and I didn't even care to check about it were it not for clinical trials about it.

Sometimes it feels like they think "crap science for a crap disease".

 

Absolutely.

And I didn't mean all those people who care about applying real research. Those people obviously respect people with ME.

 

This is completely true. However, I'm a pragmatist and believe in pressing where you can get the most traction. The methodological flaws are egregious and should doom the study before even having to think about the ethics of the LP itself. Pointing out these flaws also makes it clear that the investigators manipulated their data and engaged in what could be considered academic or scientific fraud. The trial registration violation alone should lead to immediate retraction. The outcome swapping, and the fact that all these shenanigans were not disclosed in the paper, makes is much worse. Now, we don't yet know what will happen. But BMJ has been unable to answer the questions for almost six months now, so clearly editors there know they've been caught and are trying to figure out what to do.