Reanalysis of STAR*D trial (PACE-like outcome switching scandal)

Sid

Sid

Senior Member (Voting Rights)
The largest and most impactful trial of depression of all time has now been reanalysed using individual participant data obtained from the NIMH. The STAR*D trial was originally published in the American Journal of Psychiatry in 2006 and has amassed over 5,000 citations. It cost the US taxpayers 35 million USD.

The study purported to show that following four sequential (and increasingly aggressive) treatment steps, the cumulative remission rate is 67%. These findings have become accepted wisdom in psychiatry and pretty much every paper on depression cites this trial.

A new reanalysis of protocol-specified outcome found that the investigators switched the outcome from the blinded observer-rated Hamilton Depression Rating Scale to the unblinded self-reported QIDS, a novel (at the time) scale that the investigators made up. This resulted in near doubling of the published remission rate (protocol-stipulated rate 35% vs 67% in the outcome-switched published paper).

Another bit that made me chuckle and reminded me of PACE is that, as a result of outcome switching, they ended up including in the analysis patients who already met Hamilton remission criteria before the treatment.

Cherry on top:

Published on: 26 July 2023
Note from the Editor team at BMJ Open
  • Amy Branch-Hollis, Research Editor BMJ Open
We invited the authors of the STAR*D study to provide a response to this article, but they declined.
Click to expand...

At least these authors are keeping their mouths shut instead of making a spectacle of themselves like White, Chalder et al with their incessant wailing and flailing.

Abstract
Objective Reanalyse the patient-level data set of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study with fidelity to the original research protocol and related publications.

Design The study was open label and semirandomised examining the effectiveness of up to four optimised and increasingly aggressive, antidepressant therapies in depressed adults. Patients who failed to gain adequate relief from their level 1 trial on the SSRI citalopram could receive up to three additional treatment trials in levels 2–4.

Setting 41 North American psychiatry and primary care treatment centres.

Participants 4041 adults screened positive for major depressive disorder. In contrast to most clinical trials, STAR*D enrolled patients seeking care (vs recruited) and included patients with a wide range of common comorbid medical and psychiatric conditions to enhance the generalisability of findings to real-world clinical practice.

Interventions STAR*D evaluated the relative effectiveness of 13 antidepressants therapies in treatment levels 2–4 for depressed patients who failed to gain adequate benefit from their level 1 medication trial.

Main outcome measures According to the STAR*D protocol, the primary outcome was remission, defined as a score <8 on the blinded Hamilton Rating Scale for Depression (HRSD). Response was a secondary outcome defined as ≥50% reduction in HRSD scores. STAR*D’s protocol specifically excluded all non-blinded clinic-administered assessments from use as research outcome measures.

Results STAR*D investigators did not use the protocol-stipulated HRSD to report cumulative remission and response rates in their summary article and instead used a non-blinded clinic-administered assessment. This inflated their report of outcomes, as did their inclusion of 99 patients who scored as remitted on the HRSD at study outset as well as 125 who scored as remitted when initiating their next-level treatment. These patients should have been excluded from data analysis. In contrast to the STAR*D-reported 67% cumulative remission rate after up to four antidepressant treatment trials, the rate was 35.0% when using the protocol-stipulated HRSD and inclusion in data analysis criteria.

Conclusion STAR*D’s cumulative remission rate was approximately half of that reported.
Click to expand...

https://bmjopen.bmj.com/content/13/7/e063095
 
At this point, it's probably guaranteed that this is accepted standard practice. And explains the mass indifference to doing the same thing with PACE. If they don't cheat, they have nothing. So they always do. The psychiatric discipline produces almost no effective outcomes outside of sedating people, or otherwise blunting their senses. Or simply lying about producing effective outcomes, and somehow getting away with it.

If the star trial of the profession features this much cheating, and we've seen it everywhere for years, it really is all they've got, and they simply can't accept this much failure. So they just keep covering up the same way, and think nothing of doing it again, it's how they always do it anyway.

I'd even suggest that the reanalyzed rates here are still highly inflated, simply because of all the bias inherent in how they rate outcomes, then how they evaluate them, then how they review and summarize them in guidelines. They're able to take a tiny 1% statistical significance of dubious origin and inflate it into a 100% surefire cure. It's a hype profession for the most part. Not entirely, but for the most part. They have an important mission to address, but have simply always been massively out of their depth.

I'm pretty sure that similar reanalysis of most drug trials, especially antidepressants, would yield similarly distorted outcomes. And it's not surprising, medicine works best when they understand the biology and target it directly. The further they are from understanding both the pathology and the drug to target it, the worse it is. And in depression, which has no test and no known pathology, just as with ME, they understand neither and mostly make stuff up instead.

It's really absurd how this stuff, especially psychosomatics, is invading all the nooks and crannies of medicine with their BS about MUS, or FND or BDD, or whatever. They literally can't tell when their expertise applies, and somehow even though they deliver almost nothing of value, they always want more. It's so damn weird, and clearly acts as a release valve for the massive inadequacies of medicine. Especially for liability and legal obligations. Just declare the patient insane, liability waived entirely. Problem buried alive.

More than anything, though, what they have done is block progress. And that is immoral. Fully and entirely immoral, and corrupt to a level that is simply impossible to believe for anyone who isn't familiar with it. It's not as if it's well-known that the brain is the most complicated organ in the body. And still they have the hubris, the utter shamelessness, to pretend that they can claim to whenever convenient. It's simply too big a scandal to blow up, it threatens the very credibility of all medicine, which is entirely complicit in allowing this.

Worth looking into, @dave30th.
 
Method
Restoring Invisible and Abandoned Trials initiative
The Restoring Invisible and Abandoned Trials (RIAT) initiative started in 2013 calling on funders and investigators of abandoned (unpublished) or misreported studies to publish undisclosed outcomes or correct misleading publications.19 If investigators failed to correct a study identified as misreported, independent investigators were encouraged to correct the record by reanalysing the study’s patient-level data set consistent with the research protocol and analytic plan.
Click to expand...

What is RIAT?
RIAT is an international effort to tackle bias in the way research is reported with the goal of providing more accurate information to patients and other healthcare decision makers.

Randomized controlled trials are known as medicine’s “gold standard” for reliable evidence. However, they are falling short of that standard, in large part due to two fundamental problems:

  1. MISREPORTING: many trials that are published are inaccurately or incompletely reported (misreported trials)
  2. INVISIBILITY: not all trials conducted are published (unpublished trials)
When the original investigators or sponsors do not correct misreporting, or even leave the entire trial unpublished, they can be considered to have abandoned their trial. And the downstream effects can be substantial, drawing to false conclusions about the effectiveness and safety of medical interventions.

The RIAT initiative aims to address these problems by offering a methodology that allows other people to responsibly correct the record. Anyone who can access trial data and document trial abandonment can use RIAT to:

  1. CORRECT MISREPORTING by republishing a trial (or just those sections of a trial where misreporting has occurred)
  2. CORRECT INVISIBILITY by publishing an unpublished trial
Some RIAT projects will be large, while others will be far more focused. It all depends on the nature of the trial abandonment. To get a sense of the effort involved, see How to RIAT.
Click to expand...
The RIAT Support Center
“RIATing” a trial (i.e., fixing misreporting/non-publication) is a large and complex undertaking. The RIAT Support Center aims to provide support to researchers working towards the complete and accurate dissemination of misreported or unpublished clinical trials. We aim to do this by:

  • Raising awareness of misreporting or invisibility of clinical trials, which are experiments on human beings
  • Providing instruments for the identification of misreported or invisible trials
  • Making sponsors and authors take responsability for the misreporting or invisibility of their trials
  • Publishing a restoration of misreported or invisible trials if sponsors refuse to take responsability (abandonment)
  • Supporting the restoration of the written record with advice and resources
Click to expand...
https://restoringtrials.org/whatisriat/

eta:
now if only someone could get hold of the full data...
eta2: for the PACE trial.
 
Last edited:
ME/CFS Skeptic said:
Interesting, thanks for sharing.

The original trial had severe limitations though. This paper writes: "To mimic clinical practice, STAR*D used an open-label research design with no control group during any phase of the study."
Click to expand...

Raters were blinded and were administering the Hamilton Depression Rating Scale. These blinded ratings were supposed to be used as the primary outcome. However, the investigators switched to an unblinded self-rated scale in their trial report.

There was no placebo control group and the patients were not blinded. In step 2 of the trial, CBT was one of the arms so blinding wouldn’t have been possible anyhow.
 
rvallee said:
I'd even suggest that the reanalyzed rates here are still highly inflated
Click to expand...

Yes, the 35% remission rate includes passage of time, regression toward the mean, “placebo effect” and various biases. As there was no placebo control group, we have no idea how the 35% remission rate compares to doing nothing or howling at the moon or taking a sugar pill.
 
Oof. It looks exactly as bad as PACE. Seriously, this is standard operating practice and it's why we're getting nowhere denouncing it for us. They need to cheat to produce fake outcomes:
Only 7% of subjects in remission remained stable and stayed in the study until the end. This represents only 3% of subjects according to the original inclusion and exclusion criteria (108 out of 3,671). This has not been specified.[8]
Click to expand...
I have a feeling that anyone looking closely enough would find even more issues with it. And likely the response from institutions and MDs would be to band together in defense. There is clearly a huge bunker mentality applying here that views criticism of this kind of work as invalid, simply because it's indefensible and the only response is to attack critics and DARVO all the way.

It's going to be extremely revealing once AIs can automate this kind of work and reveal just how research in medical psychology is. We already know how excessively bad it is, and it's even worse. The whole discipline of psychiatry is in for a very rough few years, and they will deserve every bit of it.
 
Sid said:
Ah I see. To be honest, the limited dataset that’s out there already following FOIA victory is enough to establish that the treatments don’t work.
Click to expand...
yes, but the ongoing saga of the 'full data now being available' continues and yet (afaik) no one who has applied appears to have been deemed 'eligible' to retrieve it even
though the cat is out of the bag thanks to the FOI data.

(Who knows what else might be lurking in the data that we don't know about.)
 
Sid said:
I wonder if anyone in the depression community is going to campaign for retraction or at least a corrigendum. This is as shocking as PACE but it affects many more people because depression is vastly more common than ME/CFS.
Click to expand...
Most of the data for SSRIs come from large drug trials. I doubt this trial actually has that much influence, frankly. Maybe in the psychiatric profession it's held up as some powerful evidence for what they do, but it's very small compared to the many large drug trials and systematic reviews. Hundreds of millions have taken SSRIs already. 4K people is nothing compared to this. And most MDs feel that they work, and that's even more influential.

I also don't think there is such a thing as the depression community. Lots of people who may or may not have something labeled as such, but little in the way of organized advocacy. This isn't something patients should be doing, however, and probably wouldn't succeed at. We debunked PACE and it made no real difference.

Depression in general terms (which includes the reckless application to us) is so much larger than us, it's hundreds of billions in drug revenues per year, lots and lots of MDs, way more than with us, who have been dispensing them for decades and genuinely think it would be disastrous to acknowledge basic facts like this. That it would scare people away from them. The strong belief is that it's worth giving them to 100x as many people as need it on the off chance that a few are helped. And it makes sense to them since they deny all the side effects, reattribute them to some psychological reaction, or whatever.

Really, I don't see how any of this is resolved until AIs automate the work of reviewing studies, and rate studies like this as worthless, or worse. It has to come from authority, and patients have basically negative authority here. Once AIs become a trusted authority, it will come quickly. It will be a sight to behold, and for sure it will happen much faster than any reaction to this reanalysis, because no one actually has to do anything here, there's no authority forcing anyone to.
 
The STAR*D Scandal: Scientific Misconduct on a Grand Scale
https://www.madinamerica.com/2023/09/the-stard-scandal-scientific-misconduct-on-a-grand-scale/


That happened to be the same year that psychologist Ed Pigott and colleagues published their deconstruction of the STAR*D trial. Pigott had filed a Freedom of Information Act request to obtain the STAR*D protocol and other key documents, and once he and his collaborators had the protocol, they were able to identify the various ways the NIMH investigators had deviated from the protocol to inflate the remission rate. They published patient data that showed if the protocol had been followed, the cumulative remission would have been 38%. The STAR*D investigators had also failed to report the stay-well rate at the end of one year, but Pigott and colleagues found that outcome hidden in a confusing graphic that the STAR*D investigators had published. Only 3% of the 4041 patients who entered the trial had remitted and then stayed well and in the trial to its end.
...
The protocol violations and publication of a fabricated “principal outcome”—the 67% cumulative remission rate—are evidence of scientific misconduct that rises to the level of fraud. Yet, as Pigott and colleagues have published their papers deconstructing the study, the NIMH investigators have never uttered a peep in protest. They have remained silent, and this was the case when Pigott and colleagues, in August of this year, published their latest paper in BMJ Open. In it, they analyzed patient-level data from the trial and detailed, once again, the protocol violations used to inflate the results. As BMJ Open wrote in the Rapid Responses section of the online article, “we invited the authors of the STAR*D study to provide a response to this article, but they declined.”
...
Thus, given the documented record of scientific misconduct, in the largest and most important trial of antidepressants ever conducted, there is only one conclusion to draw: In American psychiatry, scientific misconduct is an accepted practice.​


IMO none of this is especially egregious compared to what we've seen in the last few years in psychosomatic medicine. For sure PACE was far more fraudulent than this, but those are clearly standard practices and not just in psychiatry, but about anything that can be blamed on mental illness. I commonly see worse than this, the main difference is the scale. The cost and hype around STAR*D are similar to why PACE is so harshly defended.

Small trials used to justify decisions for years are less embarrassing when 97%+ of them simply get filtered out as being too awful to even consider. Even though by cost and numbers of patients they add up to far more, also true in years wasted. But they can be endlessly funded because the whole system is broken and can't stop doing harm, since admitting to doing harm is the worse thing that can happen. Harm is fine, admitting it is forbidden.

The fraud seen in STAR*D is terrible, of course, but this is indeed standard practice, including in anything that falls under clinical psychology, and even worse in non-pharmaceutical interventions. It's easy to see why pieces like PACE can't fall, the whole structure of mental health care is tied together with fraud and deceit on a mass scale.

Including the common framing of equating antidepressant prescriptions with cases of mental illness, even though they are commonly prescribed for any physical illness that can't be easily/cheaply figured out, and how more often than not, most "antidepressant" drugs are not ceased, even if by then natural remission could have occurred, since withdrawal symptoms are not only awful, but also denied on a mass scale.

Obviously the pharmaceutical industry is a major player in this, they made trillions out of this fraud, but the medical profession as a whole deserves very strong condemnation for doing the equivalent of parents getting their toddlers drunk just so they can have some peace and quiet. At its core, the very notion of "do no harm" is just as fraudulent, the premise is simply a giant lie.
 
You must log in or register to reply here.
Back
Top Bottom