PACE trial TSC and TMG minutes released

[Stewart]

No, I agree with Trish that it is very odd for there to be an Analysis Strategy Group in existence after the trial has formally commenced. By 2000 it was very clear to people running trials that your analysis needed to be decided before you started. It rather looks as if the MRC fell well below basic standards on this in its 'CTU'.

From the Trial Steering Committee meeting of 29 June 2005 (ie. several months *after* recruitment of patients had started - so the trial was underway - but still a full 16 months before the text of the protocol was submitted to BMC Neurology):

"The analysis plan will be written once it is felt no further amendments to the protocol are likely" (page 7)​

I'm not sure how I'm supposed to interpret this statement other than as an open admission that the protocol - supposedly agreed before the trial began - was being routinely tweaked after the trial had started and that the TSC fully expected this state of affairs to continue for some time.

From what I can tell from the TMG minutes, the Analysis Strategy Group didn't even *start* work on the strategy until October 2006 (the same month the protocol was finally submitted to BMC - what a coincidence!) by which time the trial had been underway for over 18 months. And yet incredibly the MRC don't seem to have had a problem with any aspect of this.

 

[Adrian]

I'm not sure how I'm supposed to interpret this statement other than as an open admission that the protocol - supposedly agreed before the trial began - was being routinely tweaked after the trial had started and that the TSC fully expected this state of affairs to continue for some time.

I would read it as a statement that the analysis plan wasn't expected to override and change the protocol which appears to be what happened. As far as I can tell the TSC never approved the final protocol changes just an analysis plan that failed to point them out, When the results were presented to the TSC and DMEC this was recorded in the minutes

8. Presentation of statistical analysis for main paper presented an overview of the statistical analysis strategy
for the trial. The changes made to the analysis since the original protocol was drafted were highlighted and it was noted that the analysis plan was agreed by the TSC and signed off before analysis commenced.
ACTION 1:
to ensure that the review and sign off of the analysis strategy by the TSC is well documented

It sounds a bit like some may have been surprised at the protocol changes at the point where results were presented and got a 'but you approved the analysis strategy comment'. But there was no discussion of protocol changes in the previous meeting just discussion of the tables.

There is a whole paragraph of redacted comments when the TSC comment on the results.

 

[Esther12]

It sounds a bit like some may have been surprised at the protocol changes at the point where results were presented and got a 'but you approved the analysis strategy comment'. But there was no discussion of protocol changes in the previous meeting just discussion of the tables.

I hadn't thought of that. Another interpretation is that they wanted it absolutely clear on the record that the TSC had approved of the changes laid out in the statistical analysis plan (which failed to include the deviations to the recovery criteria) before analysis commenced.

There is a whole paragraph of redacted comments when the TSC comment on the results.

Annoying. I wonder why that would be. It could be that in involved Action for ME?

 

[Adrian]

I hadn't thought of that. Another interpretation is that they wanted it absolutely clear on the record that the TSC had approved of the changes laid out in the statistical analysis plan (which failed to include the deviations to the recovery criteria) before analysis commenced.

The stats plan dropped recovery as a secondary outcome.

I think their claim is that the stats analysis plan replaces the protocol.

There is nothing earlier in the minutes suggesting any awareness of the protocol changes. Quite a lot of detailed discussion around the table formats but nothing about the protocol changes. My view is if it is not recorded in the minutes it was not raised as an issue.

 

[Sean]

There is nothing earlier in the minutes suggesting any awareness of the protocol changes. Quite a lot of detailed discussion around the table formats but nothing about the protocol changes. My view is if it is not recorded in the minutes it was not raised as an issue.

Generally the legal and bureaucratic view too.

No formal record? Didn't happen.

Oops.

 

[Stewart]

This tickled me. From the minutes for Trial Management Group Meeting #30, 11 March 2009. Page 8

14. Final PACE team day 2009
This is on Wednesday 17th June 2009. Topics for presentation during the academic morning were discussed...
...Two suggestions were made for the social afternoon:

a) Treasure hunt around London.
b) Boat trip to Hampton Court.​

I can just imagine the discussion...

"So, any ideas for the social afternoon?"
- "How about a treasure hunt around London?"
"Great idea! But what would we hunt for?"
- "Errr... statistically significant results?"
{tumbleweed}
"No-one here finds you funny. I hope you know that."

 

[Adrian]

14. Final PACE team day 2009
This is on Wednesday 17th June 2009. Topics for presentation during the academic morning were discussed...
...Two suggestions were made for the social afternoon:
a) Treasure hunt around London.
b) Boat trip to Hampton Court.

I thought the PACE day where they get all the therapists together was interesting. Is this where they get to hear what is going on in all the trial centers? It was intended to be a moral booster but its also ensures information exchange about progress (in an open label trial).

 

[JohnTheJack]

I hadn't thought of that. Another interpretation is that they wanted it absolutely clear on the record that the TSC had approved of the changes laid out in the statistical analysis plan (which failed to include the deviations to the recovery criteria) before analysis commenced.



Annoying. I wonder why that would be. It could be that in involved Action for ME?

That was my interpretation on both counts.


There was a specific mention by the ICO of redacting comments by AfME rep, and really that is the only chunk that is redacted (apart from what is clearly the discussion about membership). Looks to me like the AfME response to the results. Which actually would be interesting to see.

 

[JohnTheJack]

I thought the PACE day where they get all the therapists together was interesting. Is this where they get to hear what is going on in all the trial centers? It was intended to be a moral booster but its also ensures information exchange about progress (in an open label trial).

I think that's a good point. The suspicion is that the PIs were must have been getting informal briefings on effectiveness.

 

[Joel]

Step test - a participant at Edinburgh reported an AE as a result of the step test. Reported physical set back from doing the test and emotional set back at not feeling they had done very well. In conclusion the participant acknowledged that they had pushed themselves too hard (possibly in response to wanting to do well for the RN).

Discussion held regarding instructions to the step test to try to avoid a repeat of this in the future.

As data from the step test was never published, in combination with the above fact, this leads me to wonder: was adverse event data associated with this part of the trial (the step test) reported still? If not then was the step test data not reported as a means of hiding associated adverse events in the trial?

 

[Robert 1973]

Not only an unblinded trial which primed participants on the effectiveness of CBT and relied on subjective outcomes measures, but also one in which patients volunteered to join because they were already convinced of the potential effectiveness of CBT and GET and became depresssed if they did not get these treatments. This really is a textbook example of how not to conduct a clinical trial – or how to ensure a positive result for an ineffective treatment.

 

[Jonathan Edwards]

I think that's a good point. The suspicion is that the PIs were must have been getting informal briefings on effectiveness.

There would have be no end of gossip about how the patients did if it bore any relation to the way clinical staff normally behave under such circumstances.

 

[Jonathan Edwards]

Not only an unblinded trial which primed participants on the effectiveness of CBT and relied on subjective outcomes measures, but also one in which patients volunteered to join because they were already convinced of the potential effectiveness of CBT and GET and became depresssed if they did not get these treatments. This really is a textbook example of how not to conduct a clinical trial – or how to ensure a positive result for an ineffective treatment.

I think this is a good illustration of how one has to realise that clinical trials are episodes in real life and that being part of a trial is a good way to play merry hell with all sorts of beliefs and emotions. It also highlights the fact that nobody was under any illusion that specialist medical care was anything other than - zilch.

 

[Adrian]

I hadn't thought of that. Another interpretation is that they wanted it absolutely clear on the record that the TSC had approved of the changes laid out in the statistical analysis plan (which failed to include the deviations to the recovery criteria) before analysis commenced.

I thought this was interesting in the TMG minutes from the 4th of november 2009

7. Update from recent meetings
a) TSC/ASG

fed back that the analysis strategy was close to completion. There had been a number of final issues to resolve. It was highlighted that an important change has been made to the reporting of the primary outcome measures. Previously it had been decided that the results would be presented categorically using thresholds derived from the binary scoring of the Chalder questionnaire and the continuous SF36 scale. It has since been decided that the original question posed by the study would be better answered by comparing the continuous scores on both the Chalder and SF36 scales. The originally planned categorical scores will also be reported in the main paper, as a secondary analysis, reflecting clinically important differences.

A few concerns were raised that making a change at this stage may invite criticism. It was highlighted that the change will increase the sensitivity of the study and that the changes have been made before the reviewing of any data, and that the change will be reported in the paper. It was agreed by the TMG that these changes should go ahead.

But the minutes from the previous TSG meeting are dated 29th April 2009 with the following meeting being 10th Sept 2010

In the previous minutes of the TMG on the 23rd of June 2009 they seem to report on the 29th April 2009 TSG meeting (below) which makes me wonder if there is a set of minutes missing or whether the above referred to email (or just inaccurate). They still seem to be referring to primary outcome changes with no mention of dropping recovery as a secondary outcome.

c) Analysis Strategy Group

The ASG had met prior to the TMG and had approved the final Analysis strategy. Discussions focused on multiplicity and analysis of the safety data. The main analysis will compare combined APT/GET/CBT with SSMC, APT with GET and APT with CBT, without adjustment for multiplicity. Further exploratory analyses will compare each individual therapy versus SSMC and CBT versus GET. These exploratory analyses will be adjusted for multiplicity using Bonferoni’s correction. The outcomes of fatigue and disability will be looked at separately and therefore will not require correction. The TMG agreed with this strategy and asked that any final comments should be sent to as soon as possible so that can proceed.

Serious deterioration rates will be compared across treatment arms. Adverse event data will be presented descriptively and events with a twofold increase across treatment arms would be considered of interest.

ACTION 6: to send round the final analysis strategy for official sign off by the TMG and TSC.

xxx was thanked for all hard work on the trial as will be leaving shortly.

It is hoped that baseline lock should take place shortly in order for writing groups to be able to commence with the baseline papers. There is however a local centre issue concerning eligibility which needs to be resolved before this lock can occur and this will be returned to under item 12a. The TMG agreed that the section of the baseline papers which explains the derivation of the sample should be the same for all papers. The group was keen for the baseline data lock to occur only once.

It was noted that there are a small number of outstanding red and black book queries which it is hoped with be rectified shortly.

[Adding]

On the 13th Feb 2008 there seemed to be a presentation of the analysis plan to the TMG. The end of this quote they just refer to a morning meeting (I assume of the ASG) approving the primary and secondary outcomes but no record of discussions, concerns or reasons for change. There is more debate over referring to therapies as 'active'.

Analysis strategy for PACE

xxx gave a presentation about the Analysis Strategy that has been developed over the last 18 months. The slides and Analysis Strategy document were sent out with the documentation for this meeting.

The TMG thanked and for the enormous amounts of hard work put into the creation of this document and to congratulate them on their achievement in producing it.

There was discussion regarding the semantics of the terms ‘active’ (CBT and GET) and ‘non-active’ treatments (APT) and the controversy these terms may cause with patient groups and negative impact of morale it might have on APT therapists. It is proposed that from now on APT will be referred to as a ‘new’ treatment and the others as ‘established’.

ACTION 11: to alter this in the next version of the trial protocol and other associated documentation.

Health Economics
ACTION 12: The statisticians to supply data in due course to to allow to build models for the health economic analysis.

ACTION 13: The statisticians will provide data in due course with the agreement of the TMG for the production of other papers.
Therapist effects

xxx explained that in terms of therapist effect, there will be two analyses: treatment outcomes on the basis of the expected therapist versus actual therapists (and combinations) of therapists seen by each participant.

It was noted that ongoing supervision and training should take account of any issues of competence and treatment differentiation. Analysis is not expected to identify any great outliers in treatment competence between therapists.
The TMG approved the analysis strategy.

ACTION 14: All to send any comments on the Analysis Strategy to by 7th March.

ACTION 15: will send finalised (subject to comments) the Analysis Strategy to DMEC and the TSC.
8. Update from Analysis Strategy group

The morning meeting was concerned with approval of the analysis strategy for the primary and secondary trial outcomes.

 

[Esther12]

which makes me wonder if there is a set of minutes missing or whether the above referred to email (or just inaccurate).

Could be. Seems odd that no-ones found more detailed discussion about the protocol deviations. There also seemed to be some COI statements missing from that release.

Thanks for putting that stuff together.

General point: My instinct is to be cautious with making criticisms of the basis on these minutes until we have a good idea of exactly what happened. If we're trying to push the UK academic community to go against their self-interest, and take action on the problems with PACE, it's probably worth being aware that many of them will feel they have skeletons in their closets, and feel some sympathy for PACE having TSC minutes released like this. There are already so many problems identified with PACE that it could be counter-productive to risk making further criticisms that are not rock-solid.

It looks like the minutes further support the view that they failed do get TSC approval for the Recovery criteria used in the psych med paper. Maybe that's the point to focus on now? That paper was always the key candidate for retraction, and Robin Murray's response to concerns was scandalous.

 

[Stewart]

In the previous minutes of the TMG on the 23rd of June 2009 they seem to report on the 29th April 2009 TSG meeting (below) which makes me wonder if there is a set of minutes missing or whether the above referred to email (or just inaccurate). They still seem to be referring to primary outcome changes with no mention of dropping recovery as a secondary outcome.

I think the explanation is that we don't have minutes for the meetings of the Analysis Strategy Group and it seems likely (to me at least) that these meetings are where all the protocol changes were proposed/discussed/agreed. We know that the TSC attended the meeting of the ASG that took place on 4 September 2009 and that deviations from the protocol were discussed at this meeting - this is almost certainly the meeting that the TMG minutes for 4 November 2009 are referring to in the section you highlighted.

As we don't have the minutes for that meeting we don't know which specific changes were discussed, what justifications were given or whether the TMG voted on that date to approve any of the ASG's proposals. And as we don't have the rest of the ASG minutes we don't know whether the TSC's attendance at this meeting was a one off or if they made a habit of attending the ASG's subsequent meetings. Perhaps it was at one of the meetings of the ASG that the TSC approved the analysis strategy - they certainly don't seem to have given it approval at any of their own meetings...

Without access to the ASG minutes it's going to be impossible to work out how the changes happened. To be honest it probably wouldn't be much easier with the minutes... But I think the minutes that have been released so far show that the official trial records do not support the PACE team's claim that the changes they made from the protocol were correctly approved by the TSC.

 

[Adrian]

We know that the TSC attended the meeting of the ASG that took place on 4 September 2009 and that deviations from the protocol were discussed at this meeting - this is almost certainly the meeting that the TMG minutes for 4 November 2009 are referring to in the section you highlighted.

Did we know that?

It looks like the minutes further support the view that they failed do get TSC approval for the Recovery criteria used in the psych med paper. Maybe that's the point to focus on now? That paper was always the key candidate for retraction, and Robin Murray's response to concerns was scandalous.

My feeling for a while is that they slipped in the changes to the analysis plan rather than being explicit about them. These minutes seem to me to confirm that but I think it is up to them to provide evidence of approval. I think the other important question is around the secondary outcome of the step test data that we have only seen in a graph.

I think the minutes are showing a lack of governance from both the TSC and the MRC.

One thing that seems strange it that the TMG minutes identify action items but there is no review of them in the next meeting minutes so some appear to be forgotten or delayed.

 

[Dolphin]

Apologies if this has been posted already:

 

[Stewart]

Did we know that?

We did. In response to an FOI request the MRC made this statement in a letter dated 12 October 2016:

"The Trial Steering Committee joined a meeting of the Analysis Strategy Group held on 04 September 2009. It was at this meeting that the change in the analysis of the Chalder Fatigue Questionnaire was discussed and agreed i.e. the planned secondary outcome measure of the scale using the Likert (0,1,2,3) scale would become part of the primary outcomes instead of the bimodal (0,0,1,1) scale."
​

Presumably the details of this discussion - and the fact that this change in analysis was agreed by the TSC - appear in the ASC minutes, because there's certainly no mention of the TSC taking this decision in their own minutes. So did the TSC agree any other changes at this meeting, or at any subsequent ASG meetings? Without the ASC minutes there's no way of knowing.

How many other decisions did the TSC make that aren't recorded in their own minutes in any way?

 

[Jonathan Edwards]

I have to say that I am so impressed by the mastery of the facts some of you have here. I cannot cope. I am hoping for a simple precis once the dust has settled. Keep at it.

I work a lot with solicitors on medicolegal cases and you remind me of the very best team I have worked with - like very smart indeed. And they would pay me a grand for any one of the nuggets in the last few posts - not because they had not already thought of it themselves but the rule was that I had to come to that conclusion myself, even if with a bit of help in terms of making sure I actually read all the words in the documents in front of me. Helped by some nice biscuits with the coffee, I usually did. And with good reason. You do not build a case worth hundreds of millions on dodgy arguments. Barristers are not that smart, they are just good at verbal sparring. It is the nice quiet friendly solicitors who know how to unravel a stitched up argument.