1. Sign our petition calling on Cochrane to withdraw their review of Exercise Therapy for CFS here.
    Dismiss Notice
  2. Guest, the 'News in Brief' for the week beginning 18th March 2024 is here.
    Dismiss Notice
  3. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews (2017) Sterne [SMILE, etc on lack of blinding prob]

Discussion in 'Research methodology news and research' started by Esther12, Nov 26, 2017.

Tags:
  1. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,393
    Looks open access: https://academic.oup.com/aje/advance-article/doi/10.1093/aje/kwx344/4604571


    Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: the ROBES Meta-Epidemiologic Study.
    Savovic J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, Sterne JAC.
    Abstract
    Flaws in trial design may bias intervention effect estimates and increase between-trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the ROBES study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding and incomplete data) from a large collection of meta-analyses published in the Cochrane Library, issue 4, 2011. We categorized outcome measures as mortality, other objective or subjective, and estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 trials in 228 meta-analyses, intervention effect estimates were on average exaggerated in trials with high or unclear risk-of-bias judgements (versus low) for sequence generation (ratio of odds ratio = 0.91 (95% credible interval 0.86, 0.98)), allocation concealment (0.92 (0.86, 0.98)) and blinding (0.87 (0.80, 0.93)). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-trial heterogeneity associated with lack of blinding in meta-analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgments applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in trials.

    There are lots of papers pointing out these sorts of problems, and I've yet to read this one, but it stood out to me because of Sterne's involvement, so thought I'd post it now. He's a frequent collaborator with Esther Crawley (and George Davey Smith), including on the non-blinded SMILE trial.

    Also looks like they didn't find the previously reported difference between subjective and objective outcomes, and the details of that may be of interest.
     
    MSEsperanza, Inara, Webdog and 8 others like this.
  2. Amw66

    Amw66 Senior Member (Voting Rights)

    Messages:
    6,261
  3. Adrian

    Adrian Administrator Staff Member

    Messages:
    6,478
    Location:
    UK
    He was an author on the paper where they didn't seem to have ethical approval but claimed it was a service evaluation.
     
    MSEsperanza, Inara, Barry and 4 others like this.
  4. Adrian

    Adrian Administrator Staff Member

    Messages:
    6,478
    Location:
    UK
    They talk about subjective vs mortality.

    I would have thought trials like PACE and Smile were particularly bad for bias on subjective measures because interventions try to change perception of the illness. But if that is not included as a variable in any model then such an effect would be swamped out by open label drug trials.

    Smile of course didn't have much in terms of objective results!

    The question for me is about whether there are trials with differences between subjective and objective outcomes. Assuming there are then it is an effect that needs to be controlled for or questioned when interpreting the data. Even if the effect is not common when it occurs and hence should be thought about in a trial design.
     
    MSEsperanza, Inara, Trish and 4 others like this.
  5. Esther12

    Esther12 Senior Member (Voting Rights)

    Messages:
    4,393
    Although if participants receiving an experimental drug are given reasons for positive expectations of it's effect, I'd still expect that to lead to greater biasing of subjective outcomes.

    Yes, that seems an important thing to examine when looking at these sorts of problems.

    Did you see anything much that would be particularly useful with regards to SMILE?
     
    MSEsperanza, Inara and Adrian like this.
  6. Barry

    Barry Senior Member (Voting Rights)

    Messages:
    8,385
    Just to be sure I'm understanding this right. I presume they are saying that mortality is an objective measure ... sort of feels like it might be :). Though if a questionnaire to PwME asked "Are you dead?", they might get some indication of how subjective measures might be at odds with objective ones :rolleyes:.
     
    MSEsperanza, Inara, Trish and 4 others like this.
  7. TiredSam

    TiredSam Committee Member

    Messages:
    10,482
    Location:
    Germany
    Apparently EC has put in a grant application to assess mortality in parrots by doing an owners' plumage assessment by telephone.
     
  8. Barry

    Barry Senior Member (Voting Rights)

    Messages:
    8,385
    Probably includes a section in the questionnaire for the parrot to respond to as well.
     
    MSEsperanza, Trish and TiredSam like this.

Share This Page