Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

[ME/CFS Skeptic]

@ME/CFS Skeptic Apologies if this is stated elsewhere, but do you mind if I ask if there are plans to send to the authors or to Nature? Fighting through difficult cognitive dysfunction today and trying to decipher

I don't have any plans but believe that @andrewkq and @EndME are working on a letter.

FWIW - I’m severe, but have masters in statistics and spent 21 years working with scientists and business teams. My background isn’t medical or research. And I’m rusty on stat details. Happy to collaborate or support as I’m able.

Thanks. It would be great if could someone with stat expertise could look at the GEE modelling as I had some troubling replicating their findings. I don't have any statistical training so it is possible that I've made an error. I've attached my code to this post if that helps - I've used Python and the statsmodels package.

For example they write:

"Given equal levels and probabilities of reward, HVs chose more hard tasks than PI-ME/CFS participants (Odds Ratio (OR) = 1.65 [1.03, 2.65], p = 0.04"
​

I found an odds ratio of 1.61 and p-value of 0.41 0.041 using a GEE with a 'exchangeable' covariance structure. This is reasonably close but they also write:

"HVs were more likely to complete hard tasks (OR = 27.23 [6.33, 117.14], p < 0.0001)"
​

Here I found an odds ratio of 10.68 and p-value of 0.002 which is pretty far off. I think Andrew found similar results in R so it would be interesting if someone could have a further look at this.

 

[Dakota15]

Thanks @ME/CFS Skeptic. If you have any updates @andrewkq and @EndME just let me know.

 

[JoClaire]

It would be great if could someone with stat expertise could look at the GEE modelling as I had some troubling replicating their findings.

I’d love to.
Reminder that:

My background isn’t medical or research. And I’m rusty on stat details.

Rusty and severe and new terrain. lol!

But very motivated. Ironically.

 

[Dakota15]

2. Because this research is funded by the US federal government directly it should be possible to get our hands on information. Freedom of Information Act covers NIH and if they won't give info nicely someone can pursue them for it. An interesting request might be All correspondence and minutes of meetings relating to the use of EEfRT and susbequent analysis of said data - see if Nath and Wallitt had any disputes about how far to push it.
https://www.nih.gov/institutes-nih/...public-liaison/freedom-information-act-office. Probably a job for a US citizen to do the actual filing!

FYI I filed a FOIA on this, @Murph

 

[JoClaire]

"Given equal levels and probabilities of reward, HVs chose more hard tasks than PI-ME/CFS participants (Odds Ratio (OR) = 1.65 [1.03, 2.65], p = 0.04"

@andrewkq
(Edit to add my enthusiasm and fatigue conspired against me. May not be as bad a liar as I thought - at least for this particular statistic.)

From the notes of Figure 3a -
"The Odds Ratio for the probability of choosing the hard task at the start of the task is 1.65 [1.03, 2.65], p = 0.04 using Fisher’s Exact test."




(Edit to add my enthusiasm and fatigue conspired against me. May not be as bad a liar as I thought - at least for this particular statistic.)

What a liar! "Liar, liar, pants on fire!"

He's only comparing the data from the first trial. (Which he doesn't confess in the results section. Rather, he buries it in the figure notes.)

Note well: Looks like healthy volunteers lost preference for effort as the test progressed. (And started to be much closer to the PwME.)

Great find, again, y'all. We wouldn't have looked this closely. This is evidence that his main finding is based on a statistic that would never have been planned in the design. Why would they collect all 50 points, if the plan was to compare the first observation/trial only.

Data ommission -
(b) Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.

Sorry rambling shaky again! must go rest now.

Will check other later. lmk if I'm not clear!

 

[Dakota15]

Worth noting how "Matters Arising" works in Nature Communications.

You write your piece, then you submit it to the authors of the paper. You resolve what you can with them, and then, if you still think it necessary, you submit it to Nature Comms as "Matters Arising".


So it gets sent to the authors twice, once by you, informally, once by Nature Comms, formally.

Then it's either accepted or not. If it's not accepted, you write it as a comment on the website.

https://www.nature.com/ncomms/submit/matters-arising

Edit: added a sentence to clarify.

@Evergreen Did this get sent to authors (Nath/Walitt) then or still fluid?

 

[Murph]

FYI I filed a FOIA on this, @Murph

Excellent diligence, Lone wolf! I respect your grind a lot.

 

[Dakota15]

Appreciate your kind words, @Murph

 

[JoClaire]

Edit to add my enthusiasm and fatigue conspired against me. May not be as bad a liar as I thought - at least for this particular statistic.

@ME/CFS Skeptic - I looked at your results today and rusty tired brain didn’t see anything to help you out yet. Sorry. Will look more with fresher brain whenever that happens.

Lol - Re:My earlier comment I hadn’t looked at your results as closely yesterday and thought NIH hadn’t gotten a significant result looking at all data points/trials. (Your comment said 0.41 vs 0.041 p-value typo.)

I still think it’s odd - suspicious -that they would only include that statistic - only first trial?

Makes me wonder if there’d be different results if all trial runs were included. And whether they told the participants the first ones were throwaway.

 

[JoClaire]

Now come the plots of the trial rounds:

Do we know if NIH declared how many trials were to happen before the official "start" of trial 0?

This graph shows more PwME selecting hard trials than HV's. With higher failure rate. (

See my comments above about them only using the first trial in the statistic they shared in the paper. (notes from Fig. 3a)
Not sure what other factors were above, but still very suspicious to me.

"The Odds Ratio for the probability of choosing the hard task at the start of the task is 1.65 [1.03, 2.65], p = 0.04 using Fisher’s Exact test."

 

[JoClaire]

For example they write:

"Given equal levels and probabilities of reward, HVs chose more hard tasks than PI-ME/CFS participants (Odds Ratio (OR) = 1.65 [1.03, 2.65], p = 0.04

I found an odds ratio of 1.61 and p-value of 0.41 using a GEE with a 'exchangeable' covariance structure. This is reasonably close "

1. As I shared earlier, see notes under Figure 3a -
"The Odds Ratio for the probability of choosing the hard task at the start of the task is 1.65 [1.03, 2.65], p = 0.04 using Fisher’s Exact test."

(As I've shared in other comments, this is very odd. And is the only statistic they share for this variable. They also do not mention Fisher's Exact in the statistical methods details.)

"HVs were more likely to complete hard tasks (OR = 27.23 [6.33, 117.14], p < 0.0001)"
Here I found an odds ratio of 10.68 and p-value of 0.002 which is pretty far off. I think Andrew found similar results in R so it would be interesting if someone could have a further look at this.

2. I found a potential clue on p. 23 Statistical analysis of effort expenditure for rewards task:
"The model included reward probability, reward magnitude, expected value, trial number, participant diagnosis, and participant sex, as well as a new term indexing the difficulty of the task chosen (easy or hard). The three-way interaction of participant diagnosis, trial number, and task difficulty was evaluated in order to determine whether participants’ abilities to complete the easy and hard tasks differed between diagnostic group"

It’s a bit ambiguous to me whether the interaction effect was a separate analysis.

Perhaps adding the interaction variable would make a difference?

3. I'm curious what these models would look like, keeping the first 4 weeks.
(It's suspicious to use just the start trial as basis for "effort preference". I'll look to see if the other EEfRT papers did omitted the beginning trials. (Unless someone knows? ) The PwME in chose hard more than HV in the first four trials.)

(See previous comment.)

4. I checked both of your scripts for GEE. And double checked that binomial distribution defaults to logit links and binary distribution appropriately (Python vs SAS9). I didn't see any issues.

5. Apologies I am not able to run any of these myself. I did data modeling early in my career, but moved to customer facing and leadership roles. Then, there were other team members who did that work. I did some modeling work at my last job (start-up - lots of hats to wear). I mostly collaborated with business and science teams to ensure we were using right tools to address their problems. I also worked to ensure the science was understandable for the client.

Now I feel like I struggle to write a sentence and also even use a filter in Excel!

 

[Dakota15]

FYI I filed a FOIA on this, @Murph

I’ve already connected with the agent processing my FOIA claim on this. If I get back an abundance of material to sort through, would any here be willing to possibly help me go through? Just asking in advance.

(For reference on FOIA filed, from @Murph)

"2. Because this research is funded by the US federal government directly it should be possible to get our hands on information. Freedom of Information Act covers NIH and if they won't give info nicely someone can pursue them for it. An interesting request might be All correspondence and minutes of meetings relating to the use of EEfRT and susbequent analysis of said data - see if Nath and Wallitt had any disputes about how far to push it.
https://www.nih.gov/institutes-nih/...public-liaison/freedom-information-act-office. Probably a job for a US citizen to do the actual filing!"

 

[ME/CFS Skeptic]

Yes, I could try to help a bit (depending on how much info they send and how feasible it is to go through it).

 

[JoClaire]

2. I found a potential clue on p. 23 Statistical analysis of effort expenditure for rewards task:
"The model included reward probability, reward magnitude, expected value, trial number, participant diagnosis, and participant sex, as well as a new term indexing the difficulty of the task chosen (easy or hard). The three-way interaction of participant diagnosis, trial number, and task difficulty was evaluated in order to determine whether participants’ abilities to complete the easy and hard tasks differed between diagnostic group"

It’s a bit ambiguous to me whether the interaction effect was a separate analysis.

Perhaps adding the interaction variable would make a difference?

2b - Just noticed that they included easy versus hard as a variable. Your model filters to include only hard tasks selected.

Note: It seems strange to me to add in a three way interaction effect variable. They did not reveal the selection process for variables as they did for the choosing harder tasks model.

@EndME @andrewkq @ME/CFS Skeptic - tagging you all bc I believe you each did some modeling.

 

[ME/CFS Skeptic]

Apologies I am not able to run any of these myself. I did data modeling early in my career, but moved to customer facing and leadership roles.

No problem, I think it is very useful to have more people like you with a statistical background on the forum. I hope you will stick around!

 

[Murph]

If I get back an abundance of material to sort through, would any here be willing to possibly help me go through? Just asking in advance.

Getting organised to submit an FOI request is very much not my skillset but trawling through 1000 pages of correspondence looking for a smoking gun is my happy place. (Executive function vs hyperfocus; I never had adhd symptoms pre mecfs but i have them now!)

 

[Evergreen]

@Evergreen Did this get sent to authors (Nath/Walitt) then or still fluid?

I don't know if anyone has sent anything to the authors. I have not. My understanding is that others on this thread are working on a response/responses.

 

[Dakota15]

Getting organised to submit an FOI request is very much not my skillset but trawling through 1000 pages of correspondence looking for a smoking gun is my happy place. (Executive function vs hyperfocus; I never had adhd symptoms pre mecfs but i have them now!)

Thank you @Murph & @ME/CFS Skeptic. Will keep you on standby

 

[JoClaire]

Thank you @Murph & @ME/CFS Skeptic. Will keep you on standby

Will you tag me as well? I'm not able to commit, but I'd love to support as I'm able.

 

[andrewkq]

I am almost done with my draft of the letter to the editor. The bulk of it is from @ME/CFS Skeptic's draft, so it's really more of a revision of that.

@EndME and @Jonathan Edwards, would you both still like to be co-authors on the letter? If so, I was thinking I'd share a private google doc version of my draft for you to edit and provide feedback on. I'm thinking that once we have all authors from here assembled and a polished draft, our last step could be to reach out to Treadway with the letter to see if he is willing to join, but I'd be open to asking him earlier on if people think that'd be better.

@Jonathan Edwards would you be open to asking Brian Hughes if he'd be willing to join as a co-author? You'd mentioned a while back that he might be interested.

If anyone else on here is interested in joining the letter as an official co-author, you would definitely be welcome. Let me know by Wed. April 3rd if you would like to join.

I'd ask that co-authors agree to upholding the 4 ICMJE Authorship Criteria, which just helps ensure everyone is on the same page with what authorship entails. Here are the 4 criteria:

The ICMJE recommends that authorship be based on the following 4 criteria:

• Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND
• Drafting the work or reviewing it critically for important intellectual content; AND
• Final approval of the version to be published; AND
• Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

In my opinion, any contribution to this thread checks the box for #1, so it's mostly agreeing to do a critical review of the letter, approve the final version, and be accountable for the work in the future (e.g., responding to rebuttals from Wallit et al. and any requests from the Nature Communications editor).