Bruce Levin to speak at Columbia: "How NOT To Conduct A Randomized Clinical Trial"

Unfortunately I just saw this, and as it's happening in 12 hours I won't be able to get myself there. If this was posted, I didn't see it...here or anywhere else, until just now. I'm guessing this won't be recorded, but who knows. Wish I could go to point out the webpage from Oxford, among other things, but...

https://visit.columbia.edu/events/how-not-conduct-randomized-clinical-trial

 

Well, as it turns out I'm going to head up there for this. Don't know if they permit recording, but if they do I will do my best.

 

I emailed them and just got a reply:

The presentation will be recorded and will be posted on our website (slides and video):
http://irvinginstitute.columbia.edu/resources/biostat_educational_initiatives.html

eta: did have a little giggle as the reply called me Dr. xxx

 

@Three Chord Monty - thought I'd tag you in case it might save you some trouble. If you still go, send Levin our thanks!

 

That's great, I look forward to seeing the video. Thanks for asking them about it, Dr xxx!

 

Good talk. Covered what for him are the most egregious issues, as far as the statistics, and quite a few other points as well. Light turnout, they had had 75 RSVPs but I think there were maybe 25 people there. Don't think there were any other ME peeps, but not completely sure. Students mostly. He just tore it up, without getting too much into any weeds, great for lay audiences I think. I suspect the PACE team won't like it.

 

A PDF from the talk is up now: http://irvinginstitute.columbia.edu/pdf/How_Not_to_Conduct_in_Randomized_Clinical_Trial_Levin.pdf . Good stuff!

 

"PECULIARITIES"

Only skimming through - up to page 40, it's mostly summerizing the issues well known to most patients advocates. Recovery criteria, switching out-come measures etc.

Page 41 starts with: "BIAS ISSUES" - if one has a low-reading ability day, it might be a good starting point for Levines view of the conduction of the PACE-trial

It's really something about it - seeing professionals make the same points patient activist have been hammering on about for years

 

As is my way, I've only pulled out the slight problems with Levin's slides. For a lot of it, I was impressed that he seemed to have a good grasp of the details, and it seemed that he had dug into it and was mentioning problems that don't get a lot of attention. Thanks a lot to Levin for all his work.

PACE was actually worse than it sounds here:

The Statistical Analysis Plan did not include the PACE trial's new recovery criteria. We now have the TSG & TMG minutes available to us, and there is no sign of approval for the revised recovery criteria: https://www.s4me.info/threads/pace-trial-tsc-and-tmg-minutes-released.3150/

The PACE researchers have been deeply evasive on all this, classing a request for clarification of the timing of protocol changes from Matthees as 'vexatious', but it really looks to me like the PACE recovery criteria was devised after they had already been analysing trial data: https://www.whatdotheyknow.com/request/timing_of_changes_to_pace_trial

In their 2013 paper they say: "We changed three of the thresholds for measuring recovery from our original protocol (White et al. 2007) before the analysis, as explained below." It is perfectly reasonable for people to assume that this means before they had begun analysing trial data, but that's not what they say. They say before the analysis, and so, potentially after other analyses.

After the 2011 PACE paper was published a number of patient groups made an FOI request for outcomes laid out in the trial protocol: https://www.meassociation.org.uk/2011/05/6171/

QMUL responded by saying that some of these outcomes, including the protocol specified recovery outcome, were exempt from the FOIA as they were due to be published in an academic journal. This would put the date for changing the recovery criteria as being after the data for the 2011 paper had been analysed, and this paper included the 'normal range' outcomes for the SF36-PF and Chalder Fatigue Questionnaire that were a core part of their later recovery criteria: http://www.meassociation.org.uk/wp-content/uploads/2011/06/FOI+from+Queen+Mary.pdf

edit: I think Levin may have gone a bit wrong there as he just assumed that PACE had done the basics of what competent, honest researchers do. The evidence we currently have available to us does not support that assumption. We can't yet be certain of when the PACE recovery criteria was changed, but it seems that it was after the final TSC & TMG meetings.

PACE dropped actometers as a potentially objective outcome, but this was before the protocol was published. The protocol primary outcomes and recovery outcome both relied only on subjective self-report outcomes.

I can't think of an example of them rejecting those specific requests as 'vexatious'... I doubt PACE would want to challenge this point though, as it brings up cases like the Matthees one above, where they had unfairly classed his request as vexatious.

I really like how this talk took the time to present some of the PACE team's responses, and then pick them apart. I can see why some people don't bother, as their responses are so rubbish, but I think it's worth doing.

Going through that was deeply cheering - PACE really is a piece of shit, isn't it?

 

I'm afraid I don't understand that reasoning?

 

After the Lancet paper had come out that FOI response said that the results for the recovery criteria laid out in the protocol was exempt from the FOIA as they were due to be released in an academic journal (the FOIA has a specific exemption for information due to be released in an academic journal). If that claim from QMUL was true, then it must be that the decision to drop that recovery criteria came after the date of that letter.

 

Okay thanks. I suppose that would be literally true. But the PACE Trial investigators may have used some licence to avoid having to release the data and simply meant that recovery criteria data would be published.

 

Yeah, if what QMUL said was not strictly accurate, then we can't say much from it.

That there does not seem to be approval for their revised recovery criteria in the TSC or TMG minutes, and that it was not mentioned in their published statistical analysis plan, also makes it seem more likely that it was decided upon after analysis for other outcomes has begun. They described the 'normal range' analyses in the Lancet paper as post-hoc (apparently included at the suggestion of a peer reviewer) and these 'normal ranges' went on to be the basis for their revised recovery criteria. They've refused to provide info on when the recovery criteria was decided upon (eg: https://www.whatdotheyknow.com/request/timing_of_changes_to_pace_trial#comment-59096).

That all still doesn't mean we can be certain that the recovery criteria were decided upon after they had begun analysing trial data, but to me that now looks the most likely explanation for the evidence we do have on this. It's pretty annoying that they've been able to get away with not clarifying this issue for so long.

 

They called the entire class of requests "vexatious" in one of the statements to the IOC. So I think it's fair to apply that to all the requests. On a couple of the other nuances, you're right. They never claimed to have gotten permission to change the recovery definition--they just did it on their own. That's the assumption one can make from the fact that no such approval is referenced in the recovery paper itself. It was the change in the primary outcome that they claimed was made before looking at any data. The redefining of physical function and fatigue thresholds was identified as post-hoc in The Lancet so those thresholds are clearly post-hoc for "recovery" as well.

 

Did they? I'd forgotten that. I need to keep better notes on those ICO stuff. There's so much damning info in there.

I guess they could claim that the recovery criteria was pre-specified, but that the idea of using just the SF36-PF and CFQ cut-offs in the Lancet paper was post-hoc? Would sound pretty dodgy though. It's hard to know how far to bend trying to find a way to give them the benefit of the doubt.

 

I don't think they can claim any of it was pre-specified. they carefully have never said that, as far as I can tell, about the "recovery" definition--only about the change in the primary outcome to the comparison of means, which they did get oversight approval for. They have fudged tremendously to make it as confusing as possible to figure out.

 

I think you're right, but we just can't yet be 100% certain on that.

I think I'd encourage people who didn't really know all the details to be a bit cautious about what they claimed here.

 

@Graham Wasn't your request for data labelled as vexatious?