1. Guest, the 'News in Brief' for the week beginning 10th May 2021 is here.
    Dismiss Notice
  2. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Bruce Levin to speak at Columbia: "How NOT To Conduct A Randomized Clinical Trial"

Discussion in 'PsychoSocial ME/CFS News' started by Three Chord Monty, Sep 12, 2018.

  1. Three Chord Monty

    Three Chord Monty Senior Member (Voting Rights)

    Messages:
    133
    Likes Received:
    1,022
    Unfortunately I just saw this, and as it's happening in 12 hours I won't be able to get myself there. If this was posted, I didn't see it...here or anywhere else, until just now. I'm guessing this won't be recorded, but who knows. Wish I could go to point out the webpage from Oxford, among other things, but...

    https://visit.columbia.edu/events/how-not-conduct-randomized-clinical-trial
     
    Joel, Woolie, TiredSam and 39 others like this.
  2. Hutan

    Hutan Moderator Staff Member

    Messages:
    13,454
    Likes Received:
    55,445
    Location:
    New Zealand
    Fantastic :).

    (Bruce Levin is Professor of Biostatistics at Columbia University, was an early signatory to the Tuller letter calling for the retraction of the 2011 PACE trial paper and a co-author of the Wilshire et al paper "Rethinking the treatment of chronic fatigue syndrome—A reanalysis and evaluation of findings from a recent major trial of graded exercise and CBT".
    https://www.me-pedia.org/wiki/Bruce_Levin.)

    :):):)
     
    Joel, Woolie, Indigophoton and 34 others like this.
  3. Three Chord Monty

    Three Chord Monty Senior Member (Voting Rights)

    Messages:
    133
    Likes Received:
    1,022
    Well, as it turns out I'm going to head up there for this. Don't know if they permit recording, but if they do I will do my best.
     
    Joel, Milo, Woolie and 18 others like this.
  4. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    6,058
    Likes Received:
    49,751
    Location:
    UK
    Joel, Milo, Woolie and 34 others like this.
  5. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,069
    Likes Received:
    33,002
  6. EzzieD

    EzzieD Senior Member (Voting Rights)

    Messages:
    352
    Likes Received:
    3,784
    Location:
    UK
  7. Three Chord Monty

    Three Chord Monty Senior Member (Voting Rights)

    Messages:
    133
    Likes Received:
    1,022
    Good talk. Covered what for him are the most egregious issues, as far as the statistics, and quite a few other points as well. Light turnout, they had had 75 RSVPs but I think there were maybe 25 people there. Don't think there were any other ME peeps, but not completely sure. Students mostly. He just tore it up, without getting too much into any weeds, great for lay audiences I think. I suspect the PACE team won't like it.
     
  8. EzzieD

    EzzieD Senior Member (Voting Rights)

    Messages:
    352
    Likes Received:
    3,784
    Location:
    UK
  9. inox

    inox Senior Member (Voting Rights)

    Messages:
    508
    Likes Received:
    4,551
    Location:
    Norway
    "PECULIARITIES" :D

    Only skimming through - up to page 40, it's mostly summerizing the issues well known to most patients advocates. Recovery criteria, switching out-come measures etc.

    Page 41 starts with: "BIAS ISSUES" - if one has a low-reading ability day, it might be a good starting point for Levines view of the conduction of the PACE-trial :)

    It's really something about it - seeing professionals make the same points patient activist have been hammering on about for years :thumbup: :D
     
    Woolie, rvallee, MEMarge and 14 others like this.
  10. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,069
    Likes Received:
    33,002
    As is my way, I've only pulled out the slight problems with Levin's slides. For a lot of it, I was impressed that he seemed to have a good grasp of the details, and it seemed that he had dug into it and was mentioning problems that don't get a lot of attention. Thanks a lot to Levin for all his work.

    PACE was actually worse than it sounds here:

    The Statistical Analysis Plan did not include the PACE trial's new recovery criteria. We now have the TSG & TMG minutes available to us, and there is no sign of approval for the revised recovery criteria: https://www.s4me.info/threads/pace-trial-tsc-and-tmg-minutes-released.3150/

    The PACE researchers have been deeply evasive on all this, classing a request for clarification of the timing of protocol changes from Matthees as 'vexatious', but it really looks to me like the PACE recovery criteria was devised after they had already been analysing trial data: https://www.whatdotheyknow.com/request/timing_of_changes_to_pace_trial

    In their 2013 paper they say: "We changed three of the thresholds for measuring recovery from our original protocol (White et al. 2007) before the analysis, as explained below." It is perfectly reasonable for people to assume that this means before they had begun analysing trial data, but that's not what they say. They say before the analysis, and so, potentially after other analyses.

    After the 2011 PACE paper was published a number of patient groups made an FOI request for outcomes laid out in the trial protocol: https://www.meassociation.org.uk/2011/05/6171/

    QMUL responded by saying that some of these outcomes, including the protocol specified recovery outcome, were exempt from the FOIA as they were due to be published in an academic journal. This would put the date for changing the recovery criteria as being after the data for the 2011 paper had been analysed, and this paper included the 'normal range' outcomes for the SF36-PF and Chalder Fatigue Questionnaire that were a core part of their later recovery criteria: http://www.meassociation.org.uk/wp-content/uploads/2011/06/FOI+from+Queen+Mary.pdf

    edit: I think Levin may have gone a bit wrong there as he just assumed that PACE had done the basics of what competent, honest researchers do. The evidence we currently have available to us does not support that assumption. We can't yet be certain of when the PACE recovery criteria was changed, but it seems that it was after the final TSC & TMG meetings.

    PACE dropped actometers as a potentially objective outcome, but this was before the protocol was published. The protocol primary outcomes and recovery outcome both relied only on subjective self-report outcomes.

    I can't think of an example of them rejecting those specific requests as 'vexatious'... I doubt PACE would want to challenge this point though, as it brings up cases like the Matthees one above, where they had unfairly classed his request as vexatious.

    I really like how this talk took the time to present some of the PACE team's responses, and then pick them apart. I can see why some people don't bother, as their responses are so rubbish, but I think it's worth doing.

    Going through that was deeply cheering - PACE really is a piece of shit, isn't it?
     
    Last edited: Sep 15, 2018
    Woolie, rvallee, MEMarge and 15 others like this.
  11. Dolphin

    Dolphin Senior Member (Voting Rights)

    Messages:
    2,956
    Likes Received:
    16,434
    I'm afraid I don't understand that reasoning?
     
    Robert 1973 likes this.
  12. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,069
    Likes Received:
    33,002
    After the Lancet paper had come out that FOI response said that the results for the recovery criteria laid out in the protocol was exempt from the FOIA as they were due to be released in an academic journal (the FOIA has a specific exemption for information due to be released in an academic journal). If that claim from QMUL was true, then it must be that the decision to drop that recovery criteria came after the date of that letter.
     
  13. Dolphin

    Dolphin Senior Member (Voting Rights)

    Messages:
    2,956
    Likes Received:
    16,434
    Okay thanks. I suppose that would be literally true. But the PACE Trial investigators may have used some licence to avoid having to release the data and simply meant that recovery criteria data would be published.
     
  14. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,069
    Likes Received:
    33,002
    Yeah, if what QMUL said was not strictly accurate, then we can't say much from it.

    That there does not seem to be approval for their revised recovery criteria in the TSC or TMG minutes, and that it was not mentioned in their published statistical analysis plan, also makes it seem more likely that it was decided upon after analysis for other outcomes has begun. They described the 'normal range' analyses in the Lancet paper as post-hoc (apparently included at the suggestion of a peer reviewer) and these 'normal ranges' went on to be the basis for their revised recovery criteria. They've refused to provide info on when the recovery criteria was decided upon (eg: https://www.whatdotheyknow.com/request/timing_of_changes_to_pace_trial#comment-59096).

    That all still doesn't mean we can be certain that the recovery criteria were decided upon after they had begun analysing trial data, but to me that now looks the most likely explanation for the evidence we do have on this. It's pretty annoying that they've been able to get away with not clarifying this issue for so long.
     
    Last edited: Sep 15, 2018
  15. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    1,303
    Likes Received:
    18,721
    They called the entire class of requests "vexatious" in one of the statements to the IOC. So I think it's fair to apply that to all the requests. On a couple of the other nuances, you're right. They never claimed to have gotten permission to change the recovery definition--they just did it on their own. That's the assumption one can make from the fact that no such approval is referenced in the recovery paper itself. It was the change in the primary outcome that they claimed was made before looking at any data. The redefining of physical function and fatigue thresholds was identified as post-hoc in The Lancet so those thresholds are clearly post-hoc for "recovery" as well.
     
  16. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,069
    Likes Received:
    33,002
    Did they? I'd forgotten that. I need to keep better notes on those ICO stuff. There's so much damning info in there.

    I guess they could claim that the recovery criteria was pre-specified, but that the idea of using just the SF36-PF and CFQ cut-offs in the Lancet paper was post-hoc? Would sound pretty dodgy though. It's hard to know how far to bend trying to find a way to give them the benefit of the doubt.
     
    MEMarge and Trish like this.
  17. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    1,303
    Likes Received:
    18,721
    I don't think they can claim any of it was pre-specified. they carefully have never said that, as far as I can tell, about the "recovery" definition--only about the change in the primary outcome to the comparison of means, which they did get oversight approval for. They have fudged tremendously to make it as confusing as possible to figure out.
     
  18. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,069
    Likes Received:
    33,002
    I think you're right, but we just can't yet be 100% certain on that.

    I think I'd encourage people who didn't really know all the details to be a bit cautious about what they claimed here.
     
  19. Adrian

    Adrian Administrator Staff Member

    Messages:
    5,177
    Likes Received:
    26,801
    Location:
    UK
    I think the decision to drop recovery as a secondary outcome was made when the SAP was written. But they had not dropped to idea of a paper on recovery! But that is not clear since the SAP is for the main analysis (i.e. the main paper) and shouldn't override the protocol.
     
  20. MEMarge

    MEMarge Senior Member (Voting Rights)

    Messages:
    1,988
    Likes Received:
    12,090
    @Graham Wasn't your request for data labelled as vexatious?
     
    andypants and Graham like this.

Share This Page