Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

[ME/CFS Skeptic]

Any update on the letter regarding effort preference and EEfRT? Has a summary of the issues identified here already been sent to the NIH researchers?

@andrewkq @EndME

 

[andrewkq]

Any update on the letter regarding effort preference and EEfRT? Has a summary of the issues identified here already been sent to the NIH researchers?

@andrewkq @EndME

It's still in the works. I crashed and had to take a break from working on it for a bit. The current plan is for me to have the next draft of revisions ready for the co-authors to review in mid to late June. So hopefully looking at sending it to the NIH researchers in July and submission a few weeks after that.

 

[ME/CFS Skeptic]

It's still in the works. I crashed and had to take a break from working on it for a bit. The current plan is for me to have the next draft of revisions ready for the co-authors to review in mid to late June. So hopefully looking at sending it to the NIH researchers in July and submission a few weeks after that.

Great to hear it is ongoing. Thanks for working on this.

 

[Medfeb]

Posted on twitter by Jeanette Burmeister

Part 1 of my new 4-part article about NIH’s Effort Preference claim for ME. Read about how NIH has misrepresented their own data. Start w/ understanding that NIH built on Wessely’s work: ME as a disorder of the perception of effort.

Her first of four blog posts is here - https://thoughtsaboutme.com/2024/06/10/1/

Part 1 goes through a lot of historical background. She said she will share in part 2 her analysis of the EEfRT data and of NIH’s misrepresentation of the data that was required to make their Effort Preference claim.

In the meantime, not sure if this was already discussed here but in case not, her blog references comments about effort preference NIH made in their response to reviewer comments (page 12 line 244)

"The approach selected with GEE was necessary to determine the primary objective of our study, the existence of EffRT performance difference between the PI-ME/CFS and HV groups. The Cooper 2019 approach is not designed to determine group differences in performance. Rather, it is designed to dissect out how participants are making their decisions (i.e. which aspects of the task are being weighed in making decisions about hard/easy task selection). Use of the Cooper 2019 approach would help determine the contribution of individual aspects of the task to the performance outcome, such as how subjects integrate reward, effort, and probability to guide decision-making.

As our data did not show differences in reward sensitivity and probability sensitivity by group, this approach seems unlikely to provide information regarding the primary outcome. We have added this sentence to the Supplement, page 10, lines 419-420: As Models 2 and 3 did not show differences in reward sensitivity and probability sensitivity by group, further analysis was not performed. We have also justified the reporting of trial timeouts with the following sentence in the Supplement, page 10,lines 421-422: No difference in decision timeliness was observed as measured by task decision timeouts (0.3% versus 0.6%, p = 0.19)."​

 

[Dakota15]

Sharing this message I received today from an individual with HHS, if helpful to know for anyone still working on this here:

"Our office did refer this complaint to the Office of Research Integrity ( https://ori.hhs.gov/ ) which does deal with issues like research misconduct and misrepresentation of research results in publications. Because of this, you may be contacted by that office in the future."

 

[bobbler]

I think this could strengthen your argument if positioned in such a way.

If PHTC shows effort preference, which (ahem) is “avoiding feelings of fatigue.” That should be strongly correlated with reported disability. Rather, percent complete is more closely tied to disability. The fact that ability to complete wasn’t wholly determinate of PHTC is also descriptive. (They are definitely related - high p- value with small sample and 0.3 is meaningful enough to break the assumptions you’re going after. But also illustrates that many chose hard tasks anyway.

Does it in some way avoid the ‘correlation issue’ of what is causing what given disability existed before the task —> correlation with completion can’t be ‘ in reverse’

and I’m not sure it makes logical sense to think someone would complete a smaller % because they chose the hard task less rather than vice versa.

that you can’t ignore those and just say disability level correlated to % hard tasks might help avoid obfuscation?

although whilst I try my hardest to predict the mind of…

 

[Binkie4]

Part 2 of Jeanette Burmeister's article on the NIH intramural study is posted on her blog. It is long and requires concentration to read but she shreds the study including Madian's response to an advance question.

https://thoughtsaboutme.com/2024/06...e-study-lies-damn-lies-and-statistics-part-2/


The blog is complex and I have done no more than skim it. Even this might be an exaggeration.

Jeanette's description of it is as follows.

"This post is the longest in the series and requires a fair amount of stick-to-it-iveness both in terms of length and complexity of the issues and details discussed. I realize that this will, unfortunately, be beyond the limits of many ME patients, but I decided not to divide it into smaller parts due to the connectedness of the issues and in order to allow for easy sharing with and reporting to the appropriate authorities and other interested parties of the main reasons for why this study should be urgently investigated and retracted."

 

[Colleen Steckel]

Part 2 of Jeanette Burmeister's article on the NIH intramural study is posted on her blog. It is long and requires concentration to read but she shreds the study including Madian's response to an advance question.

https://thoughtsaboutme.com/2024/06...e-study-lies-damn-lies-and-statistics-part-2/


The blog is complex and I have done no more than skim it. Even this might be an exaggeration."

While this is a very complex article covering the data, this quote sums it up nicely:

"It is easy to see why the authors chose not to generate a visual of what actually happened during the EEfRT and instead resorted to manipulating the data with statistical tools until they arrived at a figure that fit their desired outcome (Figure 3a and Supplemental Figure S5e). The latter allowed them to make it look as though patients chose significantly fewer hard tasks for every single trial throughout the EEfRT while the former shows clearly that their Effort Preference claim has no legs."

 

[forestglip]

While this is a very complex article covering the data, this quote sums it up nicely:

"It is easy to see why the authors chose not to generate a visual of what actually happened during the EEfRT and instead resorted to manipulating the data with statistical tools until they arrived at a figure that fit their desired outcome (Figure 3a and Supplemental Figure S5e). The latter allowed them to make it look as though patients chose significantly fewer hard tasks for every single trial throughout the EEfRT while the former shows clearly that their Effort Preference claim has no legs."

The comparison is astounding:

NIH chart:


Chart created from raw data (from blog):

And even if counting the little difference in the middle of the real chart as meaningful, the task was not to do as many hard tasks as possible, it was to win as much money as possible, which required strategically choosing between easy and hard tasks.

The assertion that the proportion of hard-task choices is the primary measure of the EEfRT is demonstrably false. Based on the EEfRT instructions, it is improper to use the EEfRT as a measure of motivation—or an alleged false perception of effort as NIH has done. In the structure of the EEfRT, always choosing the hard task over the easy task (even choosing the hard task a large majority of the time) is not optimal if one is trying to receive the maximum reward. The use of rewards is designed to be the motivating factor, and winning as much money as possible is the goal of the test. For example, one EEfRT paper clearly and simply states the following (other EEfRT papers contain similar language):

“The goal of the EEfRT is to win as much money as possible by completing easy or hard tasks.”

Hence, merely looking at the relative proportion of hard versus easy tasks is not the correct way to assess results of the EEfRT because that approach would most definitely not lead to a maximization of rewards. If the instructions had been to choose as many hard tasks as possible, then the proportion of hard tasks chosen would be the primary outcome measure, but that is not the case in EEfRT studies and was not the case in the NIH study.

Oh hey, the pwME won more money with their strategy (though just barely):


chart from blog

 

[Sean]

That is stark indeed, especially the final chart that clearly shows that, if anything, patients were playing a better strategy than the healthy controls. Which hardly supports the patients are misinterpreting/incompetent/imagining/delusional/whatever view.

It is also dangerously close to straight fraud, on the part of the authors. Or at least one of them.

I think there is a whole story behind this that has yet to come out, and it will not be flattering to Wallit. I very strongly suspect he pulled rank to bulldoze his bogus spin into the paper over the objections of others.

I am extremely angry about this. Every single concern expressed about Wallit from the moment he was put in charge of this critical project nearly a decade back, and which were sneeringly dismissed by the head honchos at the NIH, has proven completely justified.

He has to go. Now. No possible good can come of his continued involvement, at any level.

 

[bobbler]

While this is a very complex article covering the data, this quote sums it up nicely:

"It is easy to see why the authors chose not to generate a visual of what actually happened during the EEfRT and instead resorted to manipulating the data with statistical tools until they arrived at a figure that fit their desired outcome (Figure 3a and Supplemental Figure S5e). The latter allowed them to make it look as though patients chose significantly fewer hard tasks for every single trial throughout the EEfRT while the former shows clearly that their Effort Preference claim has no legs."

Yes! I'm glad that she has pulled up the usage of the following charts. ANd how they seemed to be presented as if they showed the data, when they are instead statistical legerments.

It shouldn't take people more than a few seconds to eg look at the end point and see that it doesn't map to the number of rounds completed even, except most people won't have downloaded the raw data, and they would never think that someone would have done this so would naturally assume a picture of this kind would be showing the data?

 

[bobbler]

That is stark indeed, especially the final chart that clearly shows that, if anything, patients were playing a better strategy than the healthy controls. Which hardly supports the patients are misinterpreting/incompetent/imagining/delusional/whatever view.

It is also dangerously close to straight fraud, on the part of the authors. Or at least one of them.

I think there is a whole story behind this that has yet to come out, and it will not be flattering to Wallit. I very strongly suspect he pulled rank to bulldoze his bogus spin into the paper over the objections of others.

I am extremely angry about this. Every single concern expressed about Wallit from the moment he was put in charge of this critical project nearly a decade back, and which were sneeringly dismissed by the head honchos at the NIH, has proven completely justified.

He has to go. Now. No possible good can come of his continued involvement, at any level.

I agree, when you roll in the lack of calibrating for disability (which is in Treadway's instructions, despite his reply), and the news articles and discussion noting that participants who had ME and not healthy volunteers were warned that they should be thinking about how these decisions affect their health. Using parametric tests when the data was obviously not 'normally distributed'. ..

Is there other stuff I've missed?

And then combine it with it being 4 individual choices across approx 40 trials numbers with not even even numbers of participants in each group (15 vs 17, and then there is the HV that got excluded).

I don't know what 'would seem to be manufactured finding' counts under regarding all the definitions of any sort of wrongdoing or being misleading etc?

But [to eventually end up with] 4 decisions (out of how many hundreds?) and it turns out that the ME group won more money, across only 15 participants, ALL of these events/errors/warnings/non-calibration etc had to take place, AND they had to fish around with what they inputted into their GEE (and was it here they used the wrong tests statistically, so they didn't apply to the data shape?), to even get some warped statistical output?


So to then present with diagrams (representing that statistical creation instead of the data itself) that suggest to most readers visually there is a consistent, or large difference, without clear labels flagging this is actually displaying some legerment that is particularly unusual...

That combination of things ceases to be in the realms of error or mistake, as it isn't even the same error carried through or anything?


What I cannot get past is the gall to then take a non-finding - as the paper confirms, a legerment - and try and use it to underpin some comparison with a handgrip and then treat this like some truism throughout the whole paper to re-frame and ergo 'pollute' other potential biological findings that were done at least with methods appropriate to the sample size and aren't such statistical artifacts...

 

[MSEsperanza]

Thank you to everyone working on dissecting the study.

In skimming-mode only and most is way beyond me anyway but stumbled across a quote from Jeannette's blog:

In the meantime, not sure if this was already discussed here but in case not, her blog references comments about effort preference NIH made in their response to reviewer comments (page 12 line 244)

"The approach selected with GEE was necessary to determine the primary objective of our study, the existence of EffRT performance difference between the PI-ME/CFS and HV groups.

Too unwell to check myself -- so not clear to me: Do they mean the primary objective of the EffRT performance study as part of the whole intramural study besides other incorporated studied or do they mean the effort preference stuff was the primary objective of their whole study?

So only a primary outcome or the primary outcome of the whole project?

Anyway the "deep phenotyping" article seems to be the study report of the whole study.

I wonder how and when the effort preference stuff came up for the whole study? Was it in the protocol right from the beginning? If yes, what weight had the investigators given to that particular part of the study in relation to the other parts of the study -- all the biomedical investigations?

Anyone had a look at the protocol(s)?

And what did they say in the protocols about the required number of participants to have statistical significance?

If that has already been discussed -- could someone point me to the according posts?

 

[Robert 1973]

This is Jeanette Burmeister’s summary of part 2 of her blog: https://threadreaderapp.com/thread/1800987624619782625.html

@dave30th Are you intending to blog about this too? (Apologies if I’ve missed it.)

 

[Trish]

I've been reading Jeanette Burmeister's analysis, not having managed to find the time to do any analysis myself. Huge thanks to Jeanette for all her hard work.

It's absolutely shocking just how badly the NIH team misrepresented practically every aspect of the EEfRT task results. No wonder the person who ran and analysed the task and presented it at the symposium seemed so awkward and embarrassed reading out his expanations. He and his puppet masters must surely have known by then just how badly they misrepresented their data, and that the whole weight of the effort preference hypothesis that Walitt chose to give central prominence to in the published paper relied on false information.

If they don't retract the paper, it will confirm for sure that Walitt, enabled by Nath, is a huge barrier to understanding of ME/CFS.

This is as big a scandal as the PACE trial, and potentially even more powerful in lasting influence if it isn't retracted.

And they dare to berate us and say they are hurt by criticisms when all they want to do is help and they have done such a wonderful study.

Bah humbug.

 

[Jonathan Edwards]

I may be have been naïve, but to begin with I gave everyone the benefit of the doubt. But every time I go through these three stages now:

1. I can see that there are so many confounding factors and potential biases that results reported are not going to mean much anyway. I tend not to analyse further.
2. When others struggle through the data (or when I see a truncated Y axis when copying a graph to us at NICE) it becomes clear that there wasn't even a meaningful result.
3. Tracking the history it becomes clear that the authors must have known 1 and 2. In fact they had worked very hard to conceal both.

 

[FMMM1]

Oh hey, the pwME won more money with their strategy (though just barely):

chart from blog

When I looked at these graphs I thought --- basically there's no difference between the two groups -- just the sort of "noise" you'd expect (measurement error). Then I looked again &, as per your comment, they're the wrong way around i.e. contradicts the claim!

He has to go. Now. No possible good can come of his continued involvement, at any level.

We can't demand that he goes but I do think he's clearly making unfounded claims and that surely is in conflict with NIH's mission - perhaps I'm wrong, but the controversy around this will be noted by the higher echelons in NIH!

 

[Binkie4]

No wonder it took Walitt and Nath so long to write the paper. They had to find a way to present the data so it showed what they needed it to show, rather than what it did demonstrate.

Ugh. This was predictable and predicted once Walitt was given the role he was.


edit: duplicated word omitted

 

[Eddie]

The sample size was so small that by hunting around in the data you could have found a way to support any number of hypothesis. Especially with the number of tests run, the authors could easily pick and chose what they wanted to highlight. From what I remember there were results in the appendix that were interesting, and may have been close to statistical significance, but weren't mentioned in the paper.

 

[sneyz]

Oh hey, the pwME won more money with their strategy (though just barely):

Just to underline the absurdity..
Pair this with the repeated grip strength testing, and imagine the title: “ME/CFS is associated with better risk/reward navigation.”