Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

Trish said:
I don't get how this passed peer review.
Click to expand...
This is a gross systems failure. There is no way the small group of hardcore psychosomatic fanatics at the core of this could have got away with all of it, for decades, without serious sustained support and protection from the broader governance structures, including outside of medicine itself.

And it is clearly still happening.
 
I noticed that 2 weeks ago Walitt et al. have added new files on EEfRT to the code Github repository for the study that were not there before.
https://github.com/docwalitt/Nation...igue-Syndrome-Code-Repository/tree/main/EEfRT

The Readme was also updated and now reads:
EEfRT Data and Supporting Materials
The "EEfRT" folder contains the data used to make the statistical models and generate the graphs for the paper. It is included as an excel file and as a tab delimited text file.

Additionally, the SAS scripts used to model the dataset and generate our graphs are included. The Hard Task Choice script runs the analysis and creates the graph used for Figure 3a, whereas the Button Press Rate script runs the analysis and creates the graph used for Figure 3b. These scripts are attached as SAS files and as text files. If one has a copy of SAS and the attached dataset, replicating our results should be as simple as importing the dataset into SAS and then running the included scripts.

All elements of the scripts and models were constructed closely following the official SAS user’s guide for the PROC GEE procedure (see the attached gee.pdf file) and the UCLA Statistical Methods and Data Analytics seminar on analyzing and visualizing interactions in SAS (which can be found here: https://stats.oarc.ucla.edu/sas/seminars/analyzing-and-visualizing-interactions/).

The first half of both scripts takes the variables in the dataset and retitles them to make their names shorter and more manageable and to make the outputs easier to read and understand. The “Trial” variable is also adjusted to begin at 0 instead of at 1, which by my understanding is best practice, and a copy of the variable is also generated, which is necessary for the GEE procedure.

The second half of both scripts controls the construction and testing of the model itself and the creation of the graphs. The only element of these which I believe requires significant extra explanation is the “where” statement, which removes some data from the analysis. This statement does three things:

1). Data is only included if the participant is adjudicated AND if their data is valid

2). Data from the practice trials (indicated by a negative value for Trial) are dropped

3). Trials in which the participant failed to make a choice about trial difficulty within the allotted 5 seconds (thus requiring the computer to make a random choice for the participant) were also excluded from the analysis, since the participant did not actually make their own difficulty choice that trial

For the graphs, default settings were used so that they were drawn for the condition in which the model covariates (e.g., reward value, reward probability) are set to their average values. The only place where this was adjusted was for the Sex covariate, since the default behavior for the graph function when working with binary covariates is to create the graphs for only one of the two conditions (i.e., only men or only women). It was specified that the graphs should instead be made for the “average” participant (which is an approximately 6:4 sex ratio).
Click to expand...
 
Replicating this experiment with pwME would be impossible now, as anyone asked to take it can google it, find this and similar discussions, and decide deliberately to, for example, choose the hard task every time, regardless of whether they are too fatigued to complete them.
 
Their Sas code looks like this:
https://github.com/docwalitt/Nation...ain/EEfRT/EEfRT Hard Task Choice Proc GEE.sas
data work.Import;
set work.Import;
by ID;
Trial_Number = Trial -1;
CTrial_Number = Trial_Number;
Status = Status_MECFS_is_1;
Sex = Sex_Male_is_1;
Difficulty_Choice = Trial_Difficulty_Hard_is_1;
Successful_Completion = Successful_Completion_Yes_is_1;
Reward_Value = Value_of_Reward;
Reward_Probability = Probability_of_Reward;
Expected_Value = Reward_Value*Reward_Probability;
run;
proc gee data=work.Import descending;
where Adjudicated_is_1 AND Valid_is_1 = 1 AND Trial_Number >= 0 AND Choice_Time ^= 5;
class ID CTrial_Number Status Sex Difficulty_Choice Successful_Completion;
model Difficulty_Choice = Status Trial_Number Reward_Value Reward_Probability Expected_Value Sex /dist = binary link = logit;
repeated subject=ID /type = exch covb corrw within=CTrial_Number;
lsmeans Status /oddsratio CL diff;
ods graphics on / width=1000px height=750px;
effectplot slicefit (x = Trial_Number sliceby = Status) /clm AT(Sex(CODED)=0.6 0.4);
effectplot slicefit (x = Reward_Value sliceby = Status) /clm AT(Sex(CODED)=0.6 0.4);
effectplot slicefit (x = Reward_Probability sliceby = Status) /clm AT(Sex(CODED)=0.6 0.4);
store gee;
run;
Click to expand...
 
Nitro802 said:
My mapMECFS.org account is pending. Can you share the rewards chart here?
Click to expand...

My account request was denied. "Thank you for your interest in mapMECFS. Unfortunately, the scope of our tool is designed to allow researchers access to preliminary results from studies, with the understanding that these results may not yet be replicated elsewhere. As such, our site is not appropriate for your stated goal, and your registration request was not approved. However, you can see a list of datasets on site without logging in and access the original publication. If you have a specific study of interest, we can try to point you to the relevant publication."

Trish said:
Replicating this experiment with pwME would be impossible now, as anyone asked to take it can google it, find this and similar discussions, and decide deliberately to, for example, choose the hard task every time, regardless of whether they are too fatigued to complete them.
Click to expand...

The long covid study includes this test. Its the updated version as it lists the max you can earn as $8.60 now. I think they will have a higher sample size than the first test. So we are going to see this again.
 
Just want to make sure everyone saw Betsy Keller's response in her recent 2 day CPET paper. She addresses ‘effort preference’ and the "not central or peripheral fatigue" claims from an exercise science perspective:

"A recent study of post-infectious ME/CFS (N = 17) and healthy controls (N = 21) assessed a comprehensive panel of physiological, physical, cognitive, biochemical, microbiological, and immunological variables [84]. Of these measures, only 8 ME/CFS and 9 controls completed a single CPET with an average VO2peak about 40% higher in the control group.

Based on this small sample size and inappropriately matched control group, authors suggested that impaired ANS function in ME/CFS, evidenced by diminished HRV, abnormal tilt-related symptoms, and other abnormal orthostatic responses, led to lower metabolic energy production and work output, and may be contributed to by a reduced ‘effort preference’. Effort preference was assessed in this study using the Effort-Expenditure for Rewards Task [85], which utilized a small motor task to assess for anhedonia typically associated with major depressive disorder.

The Effort-Expenditure for Rewards Task is not highly associated with measures of whole-body oxygen consumption or power output compared to conventional indices of effort (%peak HR, RER, RPE), none of which were reported in the study. Whereas the link between ANS dysfunction and impaired energy metabolism is not inconsistent with the systemic CPET data reported herein, their reasoning is misguided.

It has long been known that the magnitude of cardiovascular responses to exertion is predominantly influenced by the relative level of muscle activation (number and intensity of activated muscle fibers) via feedback loop from peripheral interoceptors (e.g., Golgi tendon organs, muscle spindles, etc.) to the motor cortex then to the brainstem [85]. Disruption of this feedback loop at any level, for example, due to infection of the vagus nerve postulated by VanElzakker [86] to emanate from the gut of ME/CFS, would negatively impact this tightly controlled process and downregulate central nervous system signaling of cardiovascular support peripherally for energy production. Consequently, during incremental exercise (i.e., CPET) accumulation of local muscle metabolites from insufficient blood flow coupled with dysregulated central signaling at the brainstem, will directly inhibit the relative level of muscle activation and thereby reduce effort.

Given that both ME/CFS and well-matched CTL in the present study achieved similarly high metrics at peak effort during CPET, we saw no evidence of reduced peak effort in ME/CFS."

https://translational-medicine.biomedcentral.com/articles/10.1186/s12967-024-05410-5
 
Last edited by a moderator:
Jeanette Burmeister has today posted a series of 4 tweets

1) NIH’s Agency Intramural Research Integrity Officer (AIRIO) has begun assessing research-conduct complaints made by ME advocates against certain NIH investigators involved in intramural ME study. This is the 1st in a 4-step process. 1/n

2)Due to the large number of issues raised with the intramural study and the fact that we are several weeks into this process already, the assessment stage will likely take more than the usual 30 days. 2/

3)
2nd step: inquiry
3rd step: investigation
4th step: determination by the Deciding Official whether NIH accepts investigation report’s findings and recommended actions
3/n

4) I feel optimistic that the requirements of the Public Health Service Policies on Research Misconduct (42 CFR Part 93) & NIH Policy will be satisfied for stages 1-3; that investigation report will, consequently, find research misconduct; and that Deciding Official will concur. 4/4

I have not been around for a little while so have not kept up with the hard work of those here who have been examining the study.

Hope so very much that Jeanette's optimism that the Deciding Official will find research misconduct comes to pass.

edit: I am not on twitter but occasionally receive tweets via email.?? I glimpsed a follow up conversation to these 4 points but now cannot access it. Someone on twitter might be able to.
 
Last edited:
@Binkie4 Curious (and hoping) if it will be a different conclusion for Jeannette (or hopefully this previous submission only helps..?)

Pasting from 5/1/24

Dakota15 said:
I anticipated this response from AIRIO (Agency Intramural Research Integrity Officer) at NIH after I filed a complaint on this test & Walitt, but sharing for visibility.

"I am writing in response to your email of March 19, 2023 on the Nature Communications article, “Deep phenotyping of post-infectious myalgic encephalomyelitis/chronic fatigue syndrome.” My office has assessed the allegations, and determined that they do not fall within the definition of research misconduct, in that you are not alleging falsification, fabrication, or plagiarism in this paper. Although there is a scientific difference of opinion about whether the Effort-Expenditure for Reward Task (EEfRT) test was appropriately used and appropriately discussed, this assertion does not meet the definition of research misconduct, as research misconduct does not include honest error or differences of opinion. Therefore, my office has closed this matter with no further action. As you may be aware, NINDS will be hosting a symposium about this topic on May 2, 2024 [https://mregs.nih.gov/ninds/f1p5-f2l3320], which is one of the remedies I believe you had proposed. We wish you the best."

Sidebar: Will see how tomorrow goes now.
Click to expand...
 
Unless I hear about about the effort to send a letter to the journal and can contribute there, I am now going to start working on a powerpoint presentation in hopes of doing a webinar or even making a youtube video. I plan to use the charts and graphs in this thread and hope that's OK.

A wall of text is tough but I think a video / powerpoint is the perfect medium to get the concepts across.
 
Hi all--I'm getting around to writing a post on this after spending too much time recently thinking about FND. In resposne to @Nitro802's question, has a letter been sent at this point? And does anyone know what exactly promoted the NIH's review? Was it Jeannette's piece, or a direct appeal to NIH that forced them to at least make a show of investigating? Was there an NIH announcement, or do we just know this from Jeannette's tweets? Thanks!
 
dave30th said:
Hi all--I'm getting around to writing a post on this after spending too much time recently thinking about FND. In resposne to @Nitro802's question, has a letter been sent at this point? And does anyone know what exactly promoted the NIH's review? Was it Jeannette's piece, or a direct appeal to NIH that forced them to at least make a show of investigating? Was there an NIH announcement, or do we just know this from Jeannette's tweets? Thanks!
Click to expand...

1) To my knowledge, no letter has been sent to the journal. I would like to help get this over the hump and submitted and have some spoons for this
2) Jeannette had to send a complaint directly to AIRIO at NIH. Dakota already had his dismissed. I think there is a very high bar for the misconduct here. We only know because Jeannette annouced on Twitter, though.
3) Jeannette's call to action at the bottom of part 4 of her series has likely led to more AIRIO and other complaints. Unfortunately I think we have shown there are issues with points 3 and 5. Pasted below:

Call to Action
I urge readers to file complaints by sharing my four-part analysis with the following authorities:

  1. your U.S. Senators and Representatives
  2. the NIH Director, Dr. Monica Bertagnolli:
    • monica.bertagnolli@nih.gov
  3. the Director of Research Integrity and the Agency Intramural Research Integrity Officer (AIRIO), Dr. Kathy Partin (https://oir.nih.gov/sourcebook/ethical-conduct/research-misconduct):
  4. the HHS Office of the Inspector General (OIG). OIG complaints can be filed in a variety of ways as follows:
    • online: https://oig.hhs.gov/fraud/report-fraud/
    • by U.S. mail:
      • U.S. Department of Health and Human Services
        Office of Inspector General
        ATTN: OIG HOTLINE OPERATIONS
        P.O. Box 23489
        Washington, DC 20026
    • by fax: (800) 223-8164
with requests to:

  1. investigate the study
    1. with respect to potential gross misconduct and potential research misconduct, including, but not limited to, falsification of data, i.e., manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record
    2. constituting a significant departure from accepted practices of the relevant research community and committed intentionally, knowingly, or recklessly,
  2. discipline any individuals involved in the foregoing improprieties, if any, and
  3. retract the NIH intramural study.
As we cannot know all the individuals who were involved with the EEfRT testing, the complaints should be made against:

  1. Dr. Avindra Nath as principal investigator of the study and
  2. any other individuals at NIH involved in any the EEfRT testing in any way (including, but not limited to, the design, administration, and interpretation of the EEfRT testing), including, but not limited to, Dr. Brian Walitt and Dr. Nicholas Madian.
Please feel free to use the following summary of some of the potential misconduct, the inclusion of which in your correspondence may expedite the authorities’ determination to investigate the matter. Please make it clear that this is a non-exhaustive list of potential misconduct:

  1. The inclusion in the paper of a graph, Figure 3a, in order to support the Effort Preference claim, that was the result of statistical manipulation resulting in a completely misleading presentation of the data, which, when graphed, present a picture that contradicts Figure 3a
  2. The claim that ME patients chose fewer hard tasks than controls “at the start of and through the [EEfRT]” with Figure 3a claiming that patients chose fewer hard tasks in every single trial
    1. During the May 2, 2024 NIH Symposium on the study, Dr. Madian stated, “We did again see a difference at baseline, which persisted throughout the task, indicating differences in effort discounting.”
    2. Out of the first four trials, ME patients and controls chose the exact same number of hard tasks per participant. For the very first trial, arguably “the start” of the EEfRT, patients chose twice as many hard trials as controls, even though the patient cohort consisted of one fewer individual than the control cohort.
    3. For 34% of the trials, ME patients chose hard tasks at a higher rate than controls. For another 2% of trials, both groups chose the same percentage of hard tasks. During an additional 14% of tasks, both groups’ hard-task choices were nearly identical, and the difference was, therefore, not statistically significant.
  3. The inclusion of randomly assigned tasks (hard versus easy) in the analysis of hard-task choices despite no choice having been made in those cases, which occurred substantially more often in the case of patients than controls and with a substantially higher percentage of the randomly assigned tasks being easy tasks in the case of patients compared to controls
  4. The omission of an analysis of the obvious impact of patients using a game optimization strategy and the conclusory claim (without discussion) that there was no resulting group difference in probability sensitivity despite the fact that there was a significant difference between groups for 12% and 50% probability trials but not for 88% probability trials, negating any basis for the Effort Preference claim
  5. The use of an improper metric, number/ratio/probability of hard-task-trial choices, in support of the Effort Preference claim, as opposed to the correct metric, the average rewards earned by both groups, for which there was no significant difference between the two groups (less than 1%), negating any basis for the Effort Preference claim
  6. The failure to address or even acknowledge significant confounding factors and to attempt to control for them or at least minimize their impact, contrary to other EEfRT studies, for example:
    1. the failure to exclude from the EEfRT patients taking benzodiazepines
    2. the failure to control for patients’ motoric or other physical impairment to complete hard tasks by calibrating the maximum required button-press rates to individual physical ability despite numerous prior EEfRT studies emphasizing the need to do so
    3. the failure, contrary to what prior EEfRT studies have done, to exclude five patients who were physically unable to complete hard tasks at a reasonable rate or at all (the combined hard-task completion for those five patients was less than 16%) leading to a significant group difference in the ability to complete hard tasks (96.43% for controls versus 67.07% for patients), invalidating the EEfRT data and analysis
      • During the May 2, 2024 NIH Symposium of the study, Dr. Madian stated, “What the [original EEfRT] paper describes is that the EEfRT was designed so that the sample of patients used within that original study could consistently complete the task. This does not mean that everyone who takes the task must be able to complete the task without issue for the administration or data to be valid or interpretable. It seems that the creators wanted to ensure that in general as many people as possible would be able to complete the task but without compromising the task’s ability to challenge participants. Furthermore, I think, it bears mentioning that although our ME participants did not complete the task at the same 96-100% rate as the participants in the original study or at the same rate as our healthy controls, they still completed the task a large majority of the time. To wrap things up, to answer the question, consistently completing the task is not a requirement for a valid EEfRT test administration, and by all accounts we believe our data is valid and is, thus, interpretable as a measure of impaired effort discounting.” This is a misrepresentation of the what the original EEfRT study found (required task completion by “all subjects”) and what subsequent EEfRT studies have stressed. Furthermore, it is untrue that patients “completed the task a large majority of the time.”
  7. The inappropriate use of a test (the EEfRT), that was designed for and has been exclusively used for mental-health issues (or in healthy individuals), in order to support a novel and newly introduced term and concept, Effort Preference, in a physical disease
  8. The failure to discuss the validity of the use of the EEfRT in an unprecented way, i.e., to measure alleged disrupted effort discounting as opposed to the established use of EEfRT results as an assessment of effort-based, reward-based motivation
  9. The failure to identify any limitations of the EEfRT testing contrary to what other EEfRT studies have invariably done
  10. The failure to exclude the data of four “spontaneously recovered” ME patients (about a quarter of the patient cohort), a recovery rate well above of what has been found by credible researchers, indicating that at least some of those patients were misdiagnosed
  11. The over-generalization of the unsupported Effort Preference claim beyond the expending of effort for small gambling rewards, i.e., for any effort exertion by ME patients
  12. The over-generalization of the unsupported Effort Preference claim to millions of ME patients worldwide based on the one-time EEfRT performance of 15 ME patients, some of whom seem to have been misdiagnosed
  13. The inclusion of data from healthy controls with diseases that have substantial symptom overlap with ME (orthostatic issues in high numbers, Chronic Lyme Disease, and Psoriasis) as well as the inclusions of two blood relatives (siblings) of ME patients in the study despite the fact that there seems to be at least an infectious component to ME
  14. The choice of a new and exceedingly prejudicial label for a patient community that has suffered grave harm from decades of misrepresentation of the disease nature and from sustained and relentless defamation, including by NIH
  15. The use of three vastly different criteria (two of which are overly broad) for patient selection, including one set of criteria, the IOM definition, that is not a research definition, which likely resulted in including individuals in the patient group who were not ME patients
  16. The claim to have established the ME phenotype based on an exploratory, hypothesis-generating study of a cohort of only 17 patients, with many tests run only on even smaller sub-subsets of patients
  17. The misrepresentation of the nature of ME by reducing it to mere fatigue, exercise intolerance, malaise, and cognitive complaints, which is a non-specific description that does not capture ME, a multi-system disease with a variety of other disabling symptoms
  18. The assigning of a researcher to design the study and run its day-to-day activities, Dr. Brian Walitt, who is on record with his unscientific views of ME, for example, that it is merely a normal way of experiencing life and not a medical entity
  19. The hostility, derision, and unprofessional conduct by the principal investigator Dr. Avindra Nath, a high-level civil servant, toward the ME community:
    1. his persistent demands of unqualified praise from the ME community,
    2. his veiled threats as well as his overt intimidation with respect to future ME research in an attempt to silence criticism by advocates of the intramural study and NIH’s research and conduct, causing pain and suffering to ME patients,
    3. his presenting himself as a victim of ME advocates, gravely sick patients, causing reputational harm to the ME and ME advocacy community, and
    4. his relentless stressing that he and the other researchers in the study have allegedly been forced to donate their time and work without compensation for ME patients
 
dave30th said:
Ha, multiple 3s and 5s in that list. Which points specifically are you identifying?
Click to expand...

5. NIH used a column called "Reward is granted" It actually did not mean that, it just meant whether the run of probability was positive. Patients did not earn more "total rewards" than the controls. At no point were players told to maximize their total rewards or even maximize the number of hard trials so personally I don't see this as mattering.

ME/CFS Skeptic said:
I was only able to skim it briefly but it looks like a good blog that identified many of the same issues discussed here on the S4ME thread.

One issue that has the title 'False Recording of EEfRT Data' was new to me. Burmeister argues that there are errors in the raw data because the value 'Reward_Granted_Yes_is_1' is often 1 when 'Successful_Completion_Yes_is_1' is 0. So that would suggest that the reward is given even when the participant did not successfully complete the trial.

However, this seem to have happened more than 60 times which is quite a lot. Another possible explanation is that Reward_Granted_Yes_is_1 is simply the end result of the probability that the reward would be given (yes or no). A bit like you toss a coin and write up the end result: head or tails. If that is the case, then the rewards given are those that have a value of 1 for 'Reward_Granted_Yes_is_1' and for 'Successful_Completion_Yes_is_1' (instead of only 1 for 'Reward_Granted_Yes_is_1' which Burmeister calculated).

If I do that, I get slightly different results for the mean reward per group.

55.54 dollar for controls
48.42 dollar for patients​

So patients had a lower amount, but not because of hard task avoidance but because of lower successful completion rates.
Click to expand...

3. The forced choices were not included in NIH's analysis

ME/CFS Skeptic said:
So they did exclude those with a choice time of 5 seconds. @Nitro802 @andrewkq

I reran the Python GEE analysis I did before and now got the exact same result as theirs: (OR) = 1.652 [1.029, 2.653], p = 0.038. So the minor difference (I previously got an OR of 1.60) was due to those with a choice time of 5 seconds.
Click to expand...
 
Trish said:
Replicating this experiment with pwME would be impossible now, as anyone asked to take it can google it, find this and similar discussions, and decide deliberately to, for example, choose the hard task every time, regardless of whether they are too fatigued to complete them.
Click to expand...
If a person chooses the hard task every time, or the easy task every time, their data is deemed invalid! So for the deliberate strategy you describe, you'd have to choose the hard task every time it makes any sense to, in terms of the size of the potential reward and the probability displayed on the screen, regardless of whether you think you can do it or not.

Any future studies using the EEfRT or other effort tasks will hopefully take account of participants' different abilities so that the hard task would be hard, but doable, for each individual.
 
Evergreen said:
Any future studies using the EEfRT or other effort tasks will hopefully take account of participants' different abilities so that the hard task would be hard, but doable, for each individual.
Click to expand...
It seems a pretty daft task for pwME anyway. It hasn't been validated for ME/CFS and is inappropriate. I hope it's never used again. It was designed for people with depression, not people with physical diseases.

The idea that we would waste our limited energy repeatedly pressing buttons to gain a paltry couple of dollars when our ability to expend energy is so limited is insulting. The pwME who participated in the NIH study made a major effort to do all the tasks required in the study. That effort preference to take on all the travel, sensory and exertion challenges shows just how willing pwME are to push ourselves to and above our limits when it's something worthwhile or necessary for survival makes a mockery of this silly little task being interpreted as pwME having defective effort preferences.
Sorry, rant over. This stuff sickens me.
 
You must log in or register to reply here.
Back
Top Bottom