PACE trial TSC and TMG minutes released

Horton described the PACE paper as 'eagerly awaited'.
Was he short of good copy?
Considering that the great majority of physicians have still to this day never heard of the PACE trial - all about some physio or counselling for chronic fatigue or something? - why on earth would he think it newsworthy?

 

Sounds dodgy to me. Sounds like Kings had issues.

 

Hi @JohnTheJack can you please explain what TSC and TMG stand for?

 

I seem to remember that the stats analysis plan doesn't raise the issue of protocol change but just introduces them. So this raises the issue of whether the protocol changes were explicitly approved.

 

Trial steering committee (TSG) which I believe is meant to be independent
Trial management committee (TMG)

 

Did they select the people who wrote the editorial in the Lancet that made recovery claims based on their 'normal range'

 

Any mention of changes to the recovery criteria anywhere? That's always been the one where it was not clear when/if approval for changes was given.

Thanks for all your work John.

 

An interesting question.

 

My interpretation is they dropped it as a secondary outcome in the analysis plan and then later produced an adhoc recovery definition created after the initial paper was published.

They seem more concerned about PR strategies than the analysis plan and I've yet to come across protocol changes being mentioned.

 

An interesting comment on data sharing given their refusal to release data and the MRC backing up this decision in the information tribunal.

 

I'm not going to have time to look through these, but just opened up the last two minutes available to see how late they went, and they were both held prior to the release of the first Lancet paper, which is disappointing. What happened with trial oversight after that?

My notes on these two files, although I probably excerpted too much.

A lot of blacked out info here, and that is just removed when copy/pasted, so some of the excerpts I post might seem odd.

20100910 Joint meeting TSC and DMEC.pdf

Friday 10th September 2010

Early involvement of SMC:

"SC #8 ACTION 16: to speak to regarding the Science Media Centre."

Pleased to see a robust outcome?

"Discussion of preliminary main results by TSC and DMEC
The TSC and DMEC praised the trial team for the low dropout and high follow up rates which are indicative of a very well conducted trial. Therigour with which the trial has been conducted gives confidence in the results. on behalf of the MRC commented on the excellent oversight of the trial by the TSC and DMEC and was very pleased to see a robust outcome."

They were discussing PACE results in relation to FINE at this point, maybe adding further justification to view that FINE's null result was known of when decision to alter PACE's primary outcomes was made:

"Discussion of preliminary Health Economic results by TSC and DMEC
was thanked for hard work on the complex health economics analysis. The TSC felt that there was more work to do and specific actions will be recorded separately.
It was suggested that a comparison with the FINE trial would be very interesting although this data has not yet been analysed.
ACTION 3: was asked to liaise with colleagues to complete health economics analysis for the FINE trial as soon as possible and contact to assist with a comparative analysis"

Some interesting stuff here. The participant newsletter repeated their spin on "back to normal":

"16.Publication strategy:
a.Preliminary plans for public dissemination of main results
The intended publication strategy was discussed. The Lancet is the firstchoice for publishing the paper and they have already agreed to fast trackthe submission. The aim is to submit by the end of October. All the relevant press offices (sponsor, funders, and MRC) should be notified at the point of submission (or when the likely submission date is known). It will be important to work with the press offices so that an explanatory press statement is ready ahead of time. The sponsor’s press office should take the lead in drafting the press release and accompanying frequently asked questions. The MRC’s press office, The Lancet (if accepted) and will work closely with the team at Queen Mary University of London and it will be important for all press offices to present the same message, working from a single set of FAQs.
will need to brief his colleagues at regarding the results and will require permission from as to do so.

A participant newsletter should be drafted ready to distribute at the time of publication."

I wonder what the WAPOC thought of all the problems in the PACE papers? I wonder who was on this, and what real oversight they provided?

"ACTION 5: Writing and publication oversight committee (WAPOC) to plan a timetable of when the results will be disclosed and to whom (with confidentiality agreements in place)"

LOL at this:

"Policy for third party access to data

The intention to publically release PACE data to legitimate researchers inline with the MRC policy on data sharing was discussed. It was agreed thefirst priority is to release the results of the trial into the public domain. Thetrial team could then consider releasing part or all of the dataset to external third parties however it was noted that coming to a dataset cold with no access to the trial team for clarifications would be difficult. There could be more potential for harm than good if the data was analysed incorrectly or misinterpreted. There would also be a cost associated with data extraction and manipulation to create a format suitable for distribution.

emphasised the aim of the MRC to maximise the use of public money whilst maintaining high quality outputs. It was suggested that in response to enquiries about data sharing, the team should emphasise that the trial has been registered with ISRCTN, the protocol has been published, the treatment manuals will be publically available on the trial website and the results are due to be published in a peer reviewed journal. Access to raw data can be available by request to the PIs but only after receipt of a robust, fully funded and ethically approved proposal written by bona fide scientists. The TSC were happy with the draft statement circulated."

It should be noted that there has been no release of a health economics analysis using LTFU PACE data:

"The long-term follow-up study also appeared to be proceeding well with current return rates of 75%. The importance of this data for the health economic outcomes was noted."

20100210 TMG #33.pdf
Wednesday 10th February 2010

"TMG #32 ACTION 8: The analysis strategy for the 2.5 year follow up study will be discussed at the next ASG meeting and should be decided before the main results are available."

ASG?

Looks like there were other groups having meetings on key issues:

"Update from recent meetings
a) PACT, 1st December
Further to discussions at TMG# 32 the PACE Analysis Coordinating Team (PACT) has been set up to oversee the stage by stage completion of the main analysis. The first meeting has already taken place and the group is due to meet again on 19th February.
b) Analysis Strategy Group #15, 6th January
The Analysis Strategy group met in January to review the analysis strategy. The best strategy for the main comparison is still not clear, due to the complexities of four arms, 2 outcome measures and the issues of multiplicity and therapist clustering. Further discussion is also required on the policy for defining a clinical response to treatment. Another meeting is planned for 18th February, when it is hoped the strategy can be finalised and this will then require sign off by the TMG.
POST MEETING COMMENT: Since the TMG now only exists as a virtual committee it is suggested that the Analysis Strategy Group and TSC sign off the analysis strategy and it is circulated to the TMG for information only [looks like their approach to analysis had still not been finalised by the time of this final meeting]
c) WAPOC #7, 10th February
The TMG were updated on various trial timelines. It is anticipated the baseline data will be released to writing groups mid March. The main analysis is estimated to be complete by the end of August and a TSC/DMEC meeting will be planned for early September. The decision to publish the statistical analysis plan has been deferred, instead an audit trail will be kept of any changes to the document, so it is clear that decisions were made before the results were known.[Aside from the fact that this is all part of a nonblinded trial?]
ACTION 3: to ask if the TMG may attend the TSC/DMEC meeting as observers"

This might relate to claims that PACE data is now inaccessible:

"Centre Leaders provided feedback regarding their local archiving arrangements as follows:
Oxford: Records may be kept locally until the end of 2010 after which there will be a fee to store with R&D (who outsource this)
Royal Free: has been working hard on trying to finalise arrangements with R&D.
Kings: Records will be archived locally for full archiving period
Edinburgh: Space locally to store records until main analysis complete and further access unlikely to be required. R&D (who outsource) will then store at a charge
Barts: Two options including a local archive which is free, but as the trial sponsor, the QMUL/Bart’s R&D department have confirmed records must go to their central store. No cost has been identified for this but is in further discussions with R&D to clarify arrangements
Bristol: TBC"

 

Apologies in advance for the length of this post, and for any difficulties that the formatting causes people. I've struggled to work out how best to lay it all out to make it easy to follow, but I accept I've probably failed miserably.

I've flicked back and forth through these documents trying to follow the development of the the analysis strategy, which seems to be where the protocol changes came from. Here are the agenda points that struck me as significant.



Trial Steering Committee meeting #7 - 9 April 2008 (page 5)

7. Analysis Strategy for PACE

[REDACTED] spoke to this document. [REDACTED] would like comments and feedback from the TSC and permission to use a complex analysis process on the results.

ACTION 7: [REDACTED] to meet with [REDACTED] to discuss the analysis strategy in detail. This may take the form of a wider meeting with any other interested members on the morning of the next TSC meeting.

Discussions were held as to whether the proposed analysis methods could be applied to pre-existing datasets of other trials to evaluate the accuracy and effectiveness of the methodology.

ACTION 8: [REDACTED] to speak with [REDACTED] for analysis strategies for safety data as this aspect of the strategy plan needs further development.

My comments: This seems to be the start of a protracted discussion about the analysis strategy, which was being developed by the Analysis Strategy Group (composed of the PACE PIs, the trial statisticians and a couple of others). This 2008 discussion isn't that interesting in itself but it sets the scene for a fuller discussion of the strategy at the next TSC meeting, which would take place the following year.



Trial Steering Committee meeting #8 - 29 April 2009 (page 7)

8. Feedback from morning presentation of Analysis Strategy

[REDACTED] thanked [REDACTED] and [REDACTED] for an informative presentation which lead to a good discussion.

Health Economic Analyses

[REDACTED] commented on the high quality of the health economics aspect of PACE, which includes both cost effectiveness and societal costs.
There may be issued of multiplicity to return to, but there were no issues that required the TSC’s input.

The importance of the 2.5 year follow up study for looking at economic differences including patient’s return to walk was discussed. [REDACTED] agreed that this data would be important but explained that it is unlikely the DWP would be able to offer any financial contribution to this as the DWP generally fund research where return to work is the final outcome. Although this is relevant to PACE, the main outcome is clinical.

Main Analysis

ACTION 11: Actions for the statisticians regarding the dummy data presented are summarised below

A footer stating that tables and figures are composed from dummy data should be listed on each page of the “Presentation of the PACE analysis strategy” and future versions of mock presentations in addition to stating the data is not the actual PACE data on the first page.

Table 1: Responders of Disability (SF36-PF) and Fatigue (CFQ) by treatment group and time, was considered too “busy”. To improve the table only percentages will be shown rather then displaying the patient count and percentage in each cell. The total number of participants at each treatment group and time point will be displayed so that the reader can calculate the data we no longer will include in the table.

Figures 1 to 4 and 6 will not be included in the primary paper. Figure 5: Percentage of responders to Fatigue and Disability by treatment and time was deemed to be the best way to display the outcome for the primary paper. The final figure will also include confidence intervals.

Figure 7: Comparing response to Fatigue and Disability in the treatments CBT and GET, displayed a scenario where the TSC considered whether they should combine CBT or GET. It was noted that the profiles did not match that of figure 5. It was decided that in order to combine CBT and GET the difference between the response in CBT and GET must be no greater than 10% at each time point (12, 24 and 52 weeks). When analysing the real data a line plot of the proportion of difference in response between CBT and GET will be displayed with 95% confidence intervals.

Figure 8 and table 2 will be included in the study report only.

The TSC was happy with the way the analysis was displayed in table 3, 4, 5 and 6.

Figures of odds ratios and 95% Confidence intervals: It would be preferred if unadjusted differences and 95% confidence intervals were displayed rather than odds ratios to ease interpretation for the reader. It was planned for the primary paper that 2 figures would be displayed side by side. The first figure would display results of Fatigue and the second displaying Disability. It was also planned that the sensitivity analyses could be displayed within each figure although this idea might be dropped if the figures look overcrowded.

The TSC approves the PACE analysis strategy principles but will give extra time to TSC members to approve the PACE analysis strategy text.

Action: [REDACTED] to circulate an email to TSC members asking for comments by June. If no comments are received by that date it will be assumed that the relevant TSC member agrees with the analysis strategy.

The analysis strategy will be presented to the Mental Health Research Network (MHRN) Methadology group on the 7th July.

After taking the MHRN’s comments on board a TSC teleconference will be held to finalise the analysis strategy. During the teleconference it will be decided who will be responsible for signing of the analysis strategy. The data of the TSC teleconference will be decided by email.

My comments: I know, I know - on the face of it this discussion doesn't look that interesting either. But there are two points that I think are worth recording here. First, at this meeting it appears that the text of the analysis strategy is *almost* finalised, and it doesn't appear that any deviations from the trial protocol have been proposed at this point. Second, note the reference to the teleconference in the final paragraph. Did this happen? If so, why does no note of the discussion appear to have been kept?



Trial Management Group Meeting #31 - 23 June 2009 (Page 3)

7. Updates from recent meetings

c) Analysis Strategy Group

The ASG had met prior to the TMG and had approved the final Analysis strategy. Discussions focused on multiplicity and analysis of the safety data.

The main analysis will compare combined APT/GET/CBT with SSMC, APT with GET and APT with CBT, without adjustment for multiplicity. Further exploratory analyses will compare each individual therapy versus SSMC and CBT versus GET. These exploratory analyses will be adjusted for multiplicity using Bonferoni’s correction. The outcomes of fatigue and disability will be looked at separately and therefore will not require correction. The TMG agreed with this strategy and [REDACTED] asked that any final comments should be sent to [REDACTED] as soon as possible so that [REDACTED] can proceed.

Serious deterioration rates will be compared across treatment arms. Adverse event data will be presented descriptively and events with a twofold increase across treatment arms would be considered of interest.

ACTION 6: [REDACTED] to send round the final analysis strategy for official sign off by the TMG and TSC.

My comments: This summary from a meeting of the TMG seems to corroborate what was said at the TSC two months earlier - the Analysis Strategy Group has drawn up their final plan, and all that needs to happen now is that the TMG and TSC sign it off. Again there was no suggestion of deviating from the trial protocol at this point in time...



Trial Management Group Meeting #32 - 4 November 2009 (Page 2)

7. Update from recent meetings

a) TSC/ASG

[REDACTED] fed back that the analysis strategy was close to completion. There had been a number of final issues to resolve. It was highlighted that an important change has been made to the reporting of the primary outcome measures. Previously it had been decided that the results would be presented categorically using thresholds derived from the binary scoring of the Chalder questionnaire and the continuous SF36 scale. It has since been decided that the original question posed by the study would be better answered by comparing the continuous scores on both the Chalder and SF36 scales. The originally planned categorical scores will also be reported in the main paper, as a secondary analysis, reflecting clinically important differences.

A few concerns were raised that making a change at this stage may invite criticism. It was highlighted that the change will increase the sensitivity of the study and that the changes have been made before the reviewing of any data, and that the change will be reported in the paper. It was agreed by the TMG that these changes should go ahead.

My comments: We're now 6 months on from the last meeting of the TSC at which the analysis strategy was supposedly all but agreed - but it now becomes apparent that some pretty massive changes have been proposed to the strategy in the meantime and the trial protocol is being deviated from. Why? What happened to necessitate this? These minutes suggest that this news is an update from a recent meeting of the TSC and the Analysis Strategy Group - so when did this take place? Where are the minutes of this discussion? Was it a teleconference, as suggested at the April TSC meeting? Or was there a joint meeting of the TSC and the ASG on 4 September 2009, as the MRC stated in their reply to this FOI request? Why have those minutes not been released?



Trial Management Group Meeting #33 - 10 February 2010 (Page 3)

7. Updates from recent meetings

b) Analysis Strategy Group #15, 6th January

The Analysis Strategy group met in January to review the analysis strategy. The best strategy for the main comparison is still not clear, due to the complexities of four arms, 2 outcome measures and the issues of multiplicity and therapist clustering. Further discussion is also required on the policy for defining a clinical response to treatment. Another meeting is planned for 18th February, when it is hoped the strategy can be finalised and this will then require sign off by the TMG.

POST MEETING COMMENT: Since the TMG now only exists as a virtual committee it is suggested that the Analysis Strategy Group and TSC sign off the analysis strategy and it is circulated to the TMG for information only

My comment: This is important because it clearly states that a) in February 2010 the analysis strategy was *still* not finalised and consequently b) it hadn't been signed off by the TSC at this point. It also suggests that the Analysis Strategy Group were *very* busy boys and girls around this time. Do minutes exist for all these meetings too?



Joint meeting of the Trial Steering Committee and Data Monitoring and ethics committee - 10 September 2010 (Page 3)

8. Presentation of statistical analysis for main paper

[REDACTED] presented an overview of the statistical analysis strategy for the trial. The changes made to the analysis since the original protocol was drafted were highlighted and it was noted that the analysis plan was agreed by the TSC and signed off before analysis commenced.

ACTION 1: [REDACTED] to ensure that the review and sign off of the analysis strategy by the TSC is well documented.

My comment: The PACE team like to point out that the changes to the primary outcomes were approved by the TSC and the Data Monitoring and Ethics Committees. I assume that when they make this claim they are referring to this particular agenda point during this particular joint meeting. The TSC papers that have been released suggest that this is certainly the first time that protocol changes were discussed at one of their official minuted meetings.

But this is a problematic claim, to put it lightly. The minutes don't record what the exact changes were - or what discussion took place (if any) concerning the justification for these changes. But most importantly the minutes don't state that either committee actually approved the changes - they only state that they 'noted' that the TSC had agreed the analysis plan and this happened before analysis commenced.

That's past tense - they're noting that the TSC had *already* agreed the plan before this meeting took place. The analysis of the results was already well underway. The very next item on the agenda of this September 2010 joint meeting was titled "Presentation of preliminary main results" so they'd obviously been crunching the numbers for some time. So when did the TSC take this decision (presumably some point after February 2010) and why do there not appear to be any records of the meeting where this decision was made?

 

Slightly OT, but I was just reminded of Wessely's comment under Rehmeyer's piece about the release of the reanalysis of PACE's results using the protocol criteria for recovery:

https://www.statnews.com/2016/09/21/chronic-fatigue-syndrome-pace-trial/comment-page-6/#comments

Currently no sign of TSC approval for their redefinition of 'recovery'.

 

From 20020607 Draft minutes of PACE team meeting.pdf

Some stuff on pacing here that further supports peoples concerns about 'pacing' being turned into something more like CBT/GET in order to be a part of the PACE trial, rather than the minimal patient-led approach patients use. Their concerns about problems with APT being too different from CBT/GET leading to problems with bias are legitimate, but that's because of the central problem with the trial being nonblinded and relying on subjective self-report questionnaires! They can't get around that by just tinkering with how pacing is defined.

 

I hadn't seen that before. It makes it look like the MRC really don't know what was going on. I doubt they know when the recovery criteria was changed.

 

It seems clear that despite all the bluster they were very apprehensive about the fragility of the narrative to be put out. Robust data do not have more potential for harm than good from being freely available. If somebody misinterprets them that can be pointed out. The idea that coming to the dataset cold without talking to the trial team would be a problem is very odd. They clearly thought that the data would look odd in some way - which they do.

 

So they talk of the Analysis Strategy group.

From the PACE paper, my bold:

http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext

PACE trial group

Trial Steering Committee (independent members)—Janet Darbyshire (Chair), Jenny Butler, Patrick Doherty, Stella Law, M Llewelyn, and Tom Sensky. Observers—Sir Mansel Aylward (Department for Work and Pensions, London, UK), Sir Peter Spencer and Chris Clark (Action for ME, Bristol, UK), Stephen Stansfeld (Queen Mary University of London, London, UK), Alison Wearden (Fatigue Intervention by Nurses Evaluation trial), and members of analysis strategy group and writing and publication oversight committee. Data Monitoring and Ethics Committee—Paul Dieppe (initial Chair), Astrid Fletcher (final Chair), and Charlotte Feinmann. Independent assessors of the trial safety data—Hiroko Akagi, Alastair Miller, and Gavin Spickett. Independent assessors of therapy—Barbara Bowman and Deborah Fleetwood.

So no names are given in the PACE paper of who was involved with these groups, just a mention of their existence.

Do we have any idea from an other sources who were in the Analysis and Strategy Group or the Writing and Publication Oversight Committees?

Jonathan, is this normal practice to have such committees for a research trial?

 

Keith Geraghty is tweeting about this

 

I am not sure whether or not it would be usual to have such a large number of 'independent' people. For a multi centre trial I suspect it would be usual to have some sort of oversight committee. I think this would be standard if MRC and/or NIHR were involved and probably all funding bodies would be represented.

I suspect various people may have moved on by the tie the trial started. I think Paul Dieppe had moved on and Aylward had left DWP.