1. Sign our petition calling on Cochrane to withdraw their review of Exercise Therapy for CFS here.
    Dismiss Notice
  2. Guest, the 'News in Brief' for the week beginning 25th September 2023 is here.
    Dismiss Notice
  3. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Homeopathy can offer empirical insights on treatment effects in a null field, 2023, Sigurdson et al

Discussion in 'Research methodology news and research' started by CRG, Feb 5, 2023.

Tags:
  1. CRG

    CRG Senior Member (Voting Rights)

    Messages:
    1,750
    Location:
    UK
    Journal of Clinical Epidemiology

    Homeopathy can offer empirical insights on treatment effects in a null field

    Matthew K Sigurdson, Kristin L Sainani, John P A Ioannidis

    Abstract

    Objectives: A "null field" is a scientific field where there is nothing to discover and where observed associations are thus expected to simply reflect the magnitude of bias. We aimed to characterize a null field using a known example, homeopathy (a pseudoscientific medical approach based on using highly diluted substances), as a prototype.

    Study design: We identified 50 randomized placebo-controlled trials of homeopathy interventions from highly-cited meta-analyses. The primary outcome variable was the observed effect size in the studies. Variables related to study quality or impact were also extracted.

    Results: The mean effect size for homeopathy was 0.36 standard deviations (Hedges' g; 95% CI: 0.21, 0.51) better than placebo, which corresponds to an odds ratio of 1.94 (95% CI: 1.69, 2.23) in favor of homeopathy. 80% of studies had positive effect sizes (favoring homeopathy). Effect size was significantly correlated with citation counts from journals in the Directory of Open Access Journals and CiteWatch. We identified common statistical errors in 25 studies.

    Conclusion: A null field like homeopathy can exhibit large effect sizes, high rates of favorable results, and high citation impact in the published scientific literature. Null fields may represent a useful negative control for the scientific process.

    https://pubmed.ncbi.nlm.nih.gov/36736709/ not open access
     
    CRG, Feb 5, 2023
    #1
    Chezboo, Sean, Lilas and 7 others like this.
  2. Shadrach Loom

    Shadrach Loom Senior Member (Voting Rights)

    Messages:
    1,052
    Location:
    London, UK
    What exactly is a placebo in a study of homeopathy - are the substances insufficiently diluted, or do they use active ones?
     
    Shadrach Loom, Feb 5, 2023
    #2
    hellytheelephant, alktipping, Lilas and 8 others like this.
  3. RedFox

    RedFox Senior Member (Voting Rights)

    Messages:
    1,167
    Location:
    Pennsylvania
    RedFox, Feb 5, 2023
    #3
    Sean, Lilas, Hutan and 6 others like this.
  4. cassava7

    cassava7 Senior Member (Voting Rights)

    Messages:
    924
    Perhaps this should prompt some reflection on trials of interventions in other fields that are riddled with biases but achieve the same standard deviation of around 0.3 to 0.4 in favour of the intervention.

    From the 2008 Cochrane review of CBT for CFS:

     
    cassava7, Feb 5, 2023
    #4
    Sean, alktipping, Lilas and 6 others like this.
  5. rvallee

    rvallee Senior Member (Voting Rights)

    Messages:
    11,571
    Location:
    Canada
    Oh, the irony of saying that about homeopathy when it applies far more to everything biopsychosocial, which is basically intellectual homeopathy: ideas diluted out of any meaning that no longer have any. Applying the same evaluation to everything BPS would find far greater biases, the field is built entirely out of it and unlike homeopathy, it has the added weight of being officially recognized, which made it coercive, but without any actual oversight or accountability, the very worst features of both.
     
    rvallee, Feb 5, 2023
    #5
    Sean, alktipping, Hutan and 3 others like this.
  6. Hutan

    Hutan Moderator Staff Member

    Messages:
    24,783
    Location:
    Aotearoa New Zealand
    Thanks @CRG and @cassava7, together those findings suggest CBT isn't worth much at all.

    SMD = standardised mean difference
    [​IMG]


    And, it's worth keeping in mind that the homeopathy trials were presumably nominally blinded, whereas the CBT trials were not. And the CBT trials frequently included other positive biases, like extended contact with a therapist who is a proponent of the treatment, and the treatment itself urging participants to downplay their symptoms.

    I think it indicates that many of the current approaches to research of medical treatments don't work very well. Where researchers have a big incentive to find a positive result, they will probably find a way, even if they aren't conscious of it. There needs to be replication by researchers who don't have 'skin in the game', who have the required level of equipoise.

    Until then, I shall assume that findings of relatively small levels of improvement as a result of a treatment probably aren't real.
     
    Hutan, Feb 5, 2023
    #6
    hellytheelephant, RedFox, CRG and 6 others like this.
  7. Hutan

    Hutan Moderator Staff Member

    Messages:
    24,783
    Location:
    Aotearoa New Zealand
    Just further on the Standard Mean Difference,
    Cochrane says:
    So, that would be something to check for, when looking at SMD's. 'What was the standard deviation that was used for the denominator in the calculation? Was it the standard deviation of the treatment and control arms combined, or was it just the standard deviation of the treatment arm? It might even be a standard deviation from another trial that is taken to be more applicable. The choice of the standard deviation used could swing a standard mean difference in the direction that works best for the researcher.
     
    Hutan, Feb 5, 2023
    #7
    hellytheelephant, CRG, hibiscuswahine and 4 others like this.
  8. Hoopoe

    Hoopoe Senior Member (Voting Rights)

    Messages:
    5,075
    I'm surprised homeopathy is better than a placebo according to many studies. I thought it was no better.
     
    Hoopoe, Feb 5, 2023
    #8
    hellytheelephant, hibiscuswahine, alktipping and 2 others like this.
  9. Shadrach Loom

    Shadrach Loom Senior Member (Voting Rights)

    Messages:
    1,052
    Location:
    London, UK
    Blinding failure. Homeopathy is placebo.
     
    Shadrach Loom, Feb 5, 2023
    #9
    CRG, hibiscuswahine, shak8 and 4 others like this.
  10. Sean

    Sean Moderator Staff Member

    Messages:
    6,225
    Location:
    Australia
    Bullshit is acceptable when it is officially approved.
     
    Sean, Feb 5, 2023
    #10
    hellytheelephant, Lilas, hibiscuswahine and 3 others like this.
  11. Hutan

    Hutan Moderator Staff Member

    Messages:
    24,783
    Location:
    Aotearoa New Zealand
    :D a failure of blinding, and a blinding failure

    (For those not familiar with all the meanings and nuances of the word 'blinding', as well as the scientific meaning where participants (and sometimes some of the researchers) aren't aware of which treatment a participant is receiving;
    blinding can mean an obstruction of vision or depriving of understanding, reason and sense
    "he was blinded by his faith"
    "they try to blind you with science"

    It can also mean 'very intense', or 'remarkably skilful')


    @Medfeb, @Hilda Bastian, in the increasingly forlorn hope that something useful will come out of the Cochrane review of exercise therapy for ME/CFS, I think this homeopathy example is well worth considering. Can the reviewers really be sure that any apparent small benefit reported in exercise therapy trials for ME/CFS (or indeed chronic fatigue) is not just a product of the bias created by subjective outcomes in unblinded trials, and the bias created by the enthusiasm of the researchers for the treatment?
     
    Hutan, Feb 5, 2023
    #11
    hellytheelephant, Lilas, CRG and 6 others like this.
  12. rvallee

    rvallee Senior Member (Voting Rights)

    Messages:
    11,571
    Location:
    Canada
    There's always bias in trials. Always. Hell, biases. They're not rigorous precisely to keep the churn of fake positives going. There's just no courage to end the Big lie.

    And frankly homeopathy studies show this with high certainty. You know there can't be any effect from a sip of water. And yet there are, because the "effects" are all simple artifacts of poor methodology.

    Homeopathy is the ultimate placebo, the null comparator. You can easily do a double-blind experiment. And you still get false positives. That's the impact of biases that exist throughout medicine, it has nothing to do with some effect on the patient, the "placebo" is entirely the sum of small errors made by poor methodology, it's just that they're always in favor of something working.
     
    rvallee, Feb 5, 2023
    #12
    hellytheelephant, Shadrach Loom, Sean and 3 others like this.
  13. hibiscuswahine

    hibiscuswahine Senior Member (Voting Rights)

    Messages:
    460
    hibiscuswahine, Feb 6, 2023
    #13
    Mithriel, Shadrach Loom, shak8 and 4 others like this.

Share This Page