Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

[bobbler]

Yes, in the paper's discussion they make it pretty clear that being unable to do tasks and having an effort preference not to choose to do them are two mutually exclusive alternatives. That is to say that their concept of effort preference does assume that the person, like HVF, can do the task.

I am still unclear how they explain ME patients' failure to complete the hard tasks. If it was 'deliberate' it is like HVF. If it was not then the problem isn't effort preference, it is not being able to do the task - at least on a binary hypothesis. It seems much more likely that inability to do at least repeated tasks and a strategy of avoiding hard tasks would be part and parcel of the same scenario but with the cause being the inability.

Would it be ridiculous to suggest that really getting cognitive psychologists involved on this sort of thing seems sensible?

I say this because they are big on heuristics, touch on decision-making and tasks involving capability but also are interested in the task itself. I'm thinking eg air-traffic control and the need to take breaks and so on.

Ironically they might not be interested because of the small sample size meaning we are almost looking at the anecdote level of groups of 5 etc. But I see development of these types of tasks/tools as relevant to their territory as it is to neuro people or whatever the background is here?

And PS it might have been just via a good reminder to me as I struggled with magic-eye excel tables, but I think there is also potentially a perceptual/perception element here when we are talking psychometrics.

It's genuinely quite an interesting thing when we consider other things going on (including whether psychometrics would be useful for certain aspects of ME-CFS beyond them being interpreted in the psychosomatic but more things like reaction times), and I've never fully understood why given how many in the population it could affect those 'schools' haven't been involved?

 

[Murph]

Ooh, you can play EEfRT on your own computer.

https://www.millisecond.com/downloa...rewardtask/effortexpenditureforrewardtask.web

requires download and I am not guaranteeing it is virus free but I downloaded it and my computer still seems to work!

I failed the first two hard tasks I played! required a technique adjustment to actually achieve it and it still wasn't easy. I am in no rush to do more hard tasks after doing only three.

Gives us some useful context but of course the specifics in the NIH intramural study may have been different in minor but material ways.

 

[Murph]

Some screenshots from the game. Note that there's no clock ticking when you're filling up the bar so you don't know if you're going to make it.



Here's how the bar looks before you fill it up.


This is it about halfway full.

 

[bobbler]

I think having a contrast between the practice and real rounds is good. I still need to look at the graph on low brightness to be able for it, to make everything paler/more muted. If the rose were paler and the background cells were paler, it would be easier. Thank you so much for all the experimentation. Do stop when it makes sense to.

ok I've made paler and greyer both the blue background and the pinks - let me know. I'm saving the hex numbers

 

[SNT Gatchaman]

Ooh, you can play EEfRT on your own computer.

EDIT: superseded, see later post.

I found EEfRT-9-9-16 on GitHub — "The Effort Expenditure for Rewards Task (EEfRT) (Treadway et al., 2009) created and redesigned using Psychopy (Peirce, 2009)"

PsychoPy is a Python-based environment — "Free software for creating psychology/behavioural/neuroscience/psycholinguistics/economics experiments."

I'll see if I can get it running later. Looks reasonably straightforward to get running. Here's the magic incantation for "wizards of the Mac" —

git clone https://github.com/bizerkmaverick/EEfRT-9-9-16.git
brew install psychopy

Find and launch the newly installed PsychoPy.app in your /Applications folder.
The user interface is a bit "funky"
Open the new-eefrt.psyexp file.

EDIT: Spoke too soon. PsychoPy looks to be broken on recent macOS versions. Will require some messing around but it might be easier to try on Windows next.

 

[bobbler]

OK I'm trying to figure out if there is anything in button-press rate. As a sketchy by eye estimate (looking at the people who failed a bit then comparing their button-press for fails vs not) I reckon that

- for hard the button-press rate to complete is around 4.7
- for easy it is around 4.3

Given that I'm just curious at this stage who/how many were 'well above this' (wiggle-room) and if their rate dropped, by how much - and was it an even spread genuinely within groups even if Walitt notes that there is a similar gradient across groups - I'm trying to think how I might begin to tackle that in the easiest way.

 

[Hutan]

I thought it might be useful at this point to remind ourselves what is said about the EFFrT experiment in the paper:

The Effort-Expenditure for Rewards Task (EEfRT)15 was used to assess effort, task-related fatigue, and reward sensitivity (Supplementary Fig. S5A–D). Given equal levels and probabilities of reward, HVs chose more hard tasks than PI-ME/CFS participants (Odds Ratio (OR) = 1.65 [1.03, 2.65], p = 0.04; Fig. 3a).

Two-way interactions showed no group differences in responses to task-related fatigue (Relative Odds Ratio (ROR) = 1.01 [0.99, 1.03], p = 0.53; Fig. 3a), reward value (ROR = 1.57 [0.97, 2.53], p = 0.07), reward probability (ROR = 0.50 [0.09, 2.77], p = 0.43), and expected value (ROR = 0.98 [0.54, 1.79], p = 0.95).

Effort preference, the decision to avoid the harder task when decision-making is unsupervised and reward values and probabilities of receiving a reward are standardized, was estimated using the Proportion of Hard-Task Choices (PHTC) metric. This metric summarizes performance across the entire task and reflects the significantly lower rate of hard-trial selection in PI-ME/CFS participants (Fig. 3a).


There was no group difference in the probability of completing easy tasks but there was a decline in button-pressing speed over time noted for the PI-ME/CFS participants (Slope = −0.008, SE = 0.002, p = 0.003; Fig. 3b). This pattern suggests the PI-ME/CFS participants were pacing to limit exertion and associated feelings of discomfort16. HVs were more likely to complete hard tasks (OR = 27.23 [6.33, 117.14], p < 0.0001) but there was no difference in the decline in button-press rate over time for either group for hard tasks (Fig. 3b).

One point I'd make from this is that Fig 3e shows the relationship between time to failure in a grip test and proportion of hard task choices. This could be an alternative to SF-36 data, with maintenance of grip force perhaps even more relevant to the ability to perform the tapping than the SF-36. There does seem to be a relationship between ability to maintain grip force and the proportion of hard tasks chosen in the ME/CFS group.

 

[bobbler]

I thought it might be useful at this point to remind ourselves what is said about the EFFrT experiment in the paper:


View attachment 21350


One point I'd make from this is that Fig 3e shows the relationship between time to failure in a grip test and proportion of hard task choices. This could be an alternative to SF-36 data, with maintenance of grip force perhaps even more relevant to the ability to perform the tapping than the SF-36. There does seem to be a relationship between ability to maintain grip force and the proportion of hard tasks chosen in the ME/CFS group.

Gosh with those quotes you've picked out I have just realised how divisive the choice of phrasing is 'avoiding the hard task' for example.

Anyway, on the following quote:

There was no group difference in the probability of completing easy tasks but there was a decline in button-pressing speed over time noted for the PI-ME/CFS participants (Slope = −0.008, SE = 0.002, p = 0.003; Fig. 3b). This pattern suggests the PI-ME/CFS participants were pacing to limit exertion and associated feelings of discomfort16. HVs were more likely to complete hard tasks (OR = 27.23 [6.33, 117.14], p < 0.0001) but there was no difference in the decline in button-press rate over time for either group for hard tasks (Fig. 3b)

Isn't that non-sequitur ?

If the ME-CFS group were able to manage the easy task why note the decline in button-pressing speed as 'pacing'? It could be but they have no proof. It just happens to be that there is more headroom in ability for the easy task vs capability of ME-CFS cohort that they can get slower and still complete it.

Whereas on the hard task we already know that many of the ME-CFS already couldn't complete it. 'there was no difference in the decline in button-press rate over time for either group for hard tasks' is not the same as 'there was no decline' and if your rate is close to the 4.7 needed to complete the task the 'same decline' is a very different impact to if you have a rate above 5?

I'm also a bit confused by his chart of button-press rate for hard. It probably is correct and he has chosen carefully what 'stat/graph' to use, but the average button-press I have for ME-CFS H is 3.87. And I have an ave hard button-press rate for ME-CFS C of 6.11, indeed the fastest rate of all participants. I know mine are averages rather than over time, but I'm struggling to see eg how that average can map over time into being within the pink curve?

 

[bobbler]

I thought it might be useful at this point to remind ourselves what is said about the EFFrT experiment in the paper:


View attachment 21350


One point I'd make from this is that Fig 3e shows the relationship between time to failure in a grip test and proportion of hard task choices. This could be an alternative to SF-36 data, with maintenance of grip force perhaps even more relevant to the ability to perform the tapping than the SF-36. There does seem to be a relationship between ability to maintain grip force and the proportion of hard tasks chosen in the ME/CFS group.

I don't know precisely what is in the probability of choosing hard task graph but:

after task 40 I'm not sure that curve is correct. By task 50 very few participants are still playing.

At 40-44 it is the full lot of ME-CFS but only 7 HV. and you have the repeat pattern for those 4 trials of 1/7 HVs picking hard vs 2/16 ME-CFS (so technically slightly higher % if you don't count its fewer people) and then 0/7 HVs vs 1/16 ME-CFS picking hard. In later trials there are points where there are obviously more ME-CFS picking hard vs HV too - % or not.

COuld someone else have another check? I just did it by filter to have a quick scratch around so far.

 

[bobbler]

I thought it might be useful at this point to remind ourselves what is said about the EFFrT experiment in the paper:


View attachment 21350


One point I'd make from this is that Fig 3e shows the relationship between time to failure in a grip test and proportion of hard task choices. This could be an alternative to SF-36 data, with maintenance of grip force perhaps even more relevant to the ability to perform the tapping than the SF-36. There does seem to be a relationship between ability to maintain grip force and the proportion of hard tasks chosen in the ME/CFS group.

the easy task graph doesn't ring true either after task 40 (I haven't checked the earlier ones). By task 45 ME-CFS range is 4.69-6.5 and HV is 4.0-6.6, and the trials 41-44 have consistently had a few HVs with button-presses in the 4s.

at trial 44 HV O had the slowest button-press with 3. something.

 

[bobbler]

I thought it might be useful at this point to remind ourselves what is said about the EFFrT experiment in the paper:


View attachment 21350


One point I'd make from this is that Fig 3e shows the relationship between time to failure in a grip test and proportion of hard task choices. This could be an alternative to SF-36 data, with maintenance of grip force perhaps even more relevant to the ability to perform the tapping than the SF-36. There does seem to be a relationship between ability to maintain grip force and the proportion of hard tasks chosen in the ME/CFS group.

and I'm aware there is likely to be more rolled into the 'metric' proportion of hard task choices than it seems, but ME-CFS % hard task choices range from 16.98-52.5%, whereas HVs range from 25.49 - 77.14 % hard task choices. SO I don't know where the straight line has come from for the HVs on the PHTC axis.

 

[bobbler]

I thought it might be useful at this point to remind ourselves what is said about the EFFrT experiment in the paper:


View attachment 21350


One point I'd make from this is that Fig 3e shows the relationship between time to failure in a grip test and proportion of hard task choices. This could be an alternative to SF-36 data, with maintenance of grip force perhaps even more relevant to the ability to perform the tapping than the SF-36. There does seem to be a relationship between ability to maintain grip force and the proportion of hard tasks chosen in the ME/CFS group.

and another thing with him using the graph he has for the proportion of hard task choices vs time/trial. Well of course it would look like that. Because the way the game works means that the ones who do pick 70% hard choices only get to trial 39 for example (HVH and HVP), so necessarily by that point there will have to be an increasing proportion of participants who have chosen fewer hard tasks, simply because they wouldn't have got to that trial time-wise otherwise.

So why does the band of blue on the chart stay the same width when you include the overlap? BY trial 50 there are only 3HVs and 4 ME-CFS still getting trials. All of them go for easy. At trial 46 and 47 there is a higher proportion (60%), but the rest of them are around the 1/7 %

49 it is 0/5HV (excl F who selects easy) and 1/5 ME-CFS

48 it is 1/7 HV (excl F who selects easy) and 1/5 ME-CFS

47 5/9 HV and 4/8 ME-CFS select hard

46 6/11 HV and 6/10 ME-CFS select hard

45 1/11 HV (excl F who selects easy) and 1/12 ME-CFS selects hard

44 1/13 HV (excl F who selects easy) and 2/13 ME-CFS selects hard

43 0/14 HV (excl F who selects easy) and 1/14 ME-CFS selects hard

42 2/14 HV (excl F who selects easy) and 1/14 ME-CFS selects hard

41 1/14 HV (excl F who selects easy) and 0/14 ME-CFS selects hard

40 2/14 HV (excl F who selects easy) and 1/15 ME-CFS selects hard

 

[Jonathan Edwards]

Would it be ridiculous to suggest that really getting cognitive psychologists involved on this sort of thing seems sensible?

We have some good psychologists in the membership. They just don't seem to be chipping in much at present!

 

[Jonathan Edwards]

All fascinating analysis. I think there must be something misleading in the decline in button press rate, as bobbler suggests.

I am coming round to the simple take home message being:

They wanted to see if people failed because they couldn't or because they chose not to try. The most obvious finding is that PWME failed despite choosing to try.

So choosing not to is not the cause of failing.

 

[Jonathan Edwards]

Another thought recurring:

The problem the authors are faced with is finding a task that PWME cannot actually do because of central signals. The Catch22 is that a task that patients cannot do in any clear cut fashion will be a task they wouldn't volunteer for in the first place. So you end up studying people who can just about do the study tasks. Some of those may still choose not to do the harder tasks, not unreasonably, but the actual data are hanging by the thread of an excluded outlier.

With flu, both of two situations apply - symptoms that strongly discourage you from doing and involuntary inhibition of doing. Joint pain is another good example of this. Knee pain can strongly discourage you from standing up but it can also produce an involuntary inhibition of quadriceps that means that however much you ignore what it feels like you cannot stand - you fall over, even without that much sensed pain.

The authors do not seem to understand that there is no clear cut distinction between able to do and not able to do that they can fit their data around and that they should not have expected to.

 

[EndME]

Yes, this invalidation of HV F seems to me a key point. Hopefully I've summarised correctly below without erroneous assumptions.

HV F is said to be a 21yo healthy male. He deploys an optimal strategy per the rules of the challenge. Motivation? Nothing more needed than "I like to win". (Possibly he had developed greater insight into the challenge than the researchers that were setting the rules. Similar sorts of theoretical tests are very common when interviewing for programming jobs in Silicon Valley.)

HV F was fully capable of completing the hard task at will, at every attempt. No fatiguing element.

By definition therefore, HV F was also fully capable of completing the easy tasks.

He chose not to complete easy tasks as part of the optimal strategy. His data was excluded due to failure to complete easy tasks.

Vs

How do we know that there weren't ME patients also attempting the same optimal strategy?

But with them the key difference is they couldn't complete the hard task at will at every attempt. If they had, their data would have been invalid also.

---
I don't think you can exclude HV F and not exclude any relevant (all?) ME.

It's entirely possible to exclude HV F and not exclude anybody else. Whether this is what should have been done is something we cannot currently judge since we don't have access to this data.

The EEfRT is designed to study a set of behaviours X in a group of people. The set of behaviours is strictly smaller than the set of all behaviours.

The EEfRT can only study these behaviours if the group of people use a strategy out of the set of strategies Y, which is a proper subset of the set of all strategies.

Now it maybe the case that the strategy of HV F is not in Y whilst the strategy of all other participants is. Whether or not that is the case is something we cannot judge as we don't know what Y is. We have to know what Y is and when it was decided what Y was. @Murph has mentioned that it's possible for us to gain access to this data. It's quite plausible that all of this was handled correctly.

Note also: Optimal strategy is a rigourously defined mathematical term and we know for certain that HV F is not following an optimal strategy, in fact he's one of the easiest people to tell that he is not following an optimal strategy since he has perfect abilities, for pwME that don't have these abilities that is actually harder to tell, which is why your abilities are intrinsically part of your strategy, I had discussed this here and here.

 

[EndME]

The IDs are different in the publicly available data and the mapMECFS data. So I don't think you will be able to do this, unless I'm missing something.

Are there not unique participants ID's in the mapMECFD data and that ID appears both in the SF36 data as well as in the EEfRT data? Or are neither datasets complete and there's a mismatch when you try to connect them?

 

[bobbler]

and I'm aware there is likely to be more rolled into the 'metric' proportion of hard task choices than it seems, but ME-CFS % hard task choices range from 16.98-52.5%, whereas HVs range from 25.49 - 77.14 % hard task choices. SO I don't know where the straight line has come from for the HVs on the PHTC axis.

OK so I now realise that it isn't a correlation at all on the hand-grip vs PHTC for HVs , it actually says it on the graph that p is way off being significant and I've now looked more closely to see the dots in blue are all over the place.

So the EEfRT test was unvalidated in ME/CFS (as Treadway says: 'as an attempt at validation') ,

and there is no correlation between the two for HVs

For ME-CFS the p is significant but n = 14

- it isn't terribly unfeasible that just level of physical disability as per SF-36 or something else to do with the hands for example could be underlying both that isn't to do with 'effort'.

and there are a large number of hard tasks that were non-complete for ME-CFS based on participants being unable to do that many clicks in the time.

does that non-complete rate for hards actually fit better with the hand-grip test or worse?

 

[Eddie]

It's entirely possible to exclude HV F and not exclude anybody else. Whether this is what should have been done is something we cannot currently judge since we don't have access to this data.

The EEfRT is designed to study a set of behaviours X in a group of people. The set of behaviours is strictly smaller than the set of all behaviours.

The EEfRT can only study these behaviours if the group of people use a strategy out of the set of strategies Y, which is a proper subset of the set of all strategies.

Now it maybe the case that the strategy of HV F is not in Y whilst the strategy of all other participants is. Whether or not that is the case is something we cannot judge as we don't know what Y is. We have to know what Y is and when it was decided what Y was. @Murph has mentioned that it's possible for us to gain access to this data. It's quite plausible that all of this was handled correctly.

Note also: Optimal strategy is a rigourously defined mathematical term and we know for certain that HV F is not following an optimal strategy, in fact he's one of the easiest people to tell that he is not following an optimal strategy since he has perfect abilities, for pwME that don't have these abilities that is actually harder to tell, which is why your abilities are intrinsically part of your strategy, I had discussed this here and here.

Not quite sure what your point is here. It is clear that HV F's strategy is better from the point of view of achieving more money. It may not be optimal but just look at the graph of the expected prize money.

Why should the study authors chose which strategies to exclude unless participants have been explicitly told beforehand? Why not conclude that HFV just had much lower effort preference? The EEfRT should ensure that effort and rewards are not in conflict as they can be in this test. If you have a crap test you are going to get crap data even if the data is handled correctly (which I'm not convinced is the case).

 

[EndME]

Not quite sure what your point is here. It is clear that HV F's strategy is better from the point of view of achieving more money. It may not be optimal but just look at the graph of the expected prize money.

Why should the study authors chose which strategies to exclude unless participants have been explicitly told beforehand? Why not conclude that HFV just had much lower effort preference? The EEfRT should ensure that effort and rewards are not in conflict as they can be in this test. If you have a crap test you are going to get crap data even if the data is handled correctly (which I'm not convinced is the case).

It's not that easy to just say his strategy is "better" than some of those strategies of people with ME/CFS by doing a post-hoc analysis of his rewards, because participants all have entirely different capabilities and the assessment of your capabilites is part of your strategy (he'll most likely end up having a better strategy than almost anybody else, especially the ones that are healthy, but I wouldn't be suprised either if someone else has a "better" strategy than him, because it's abundantly clear that he makes multiple "wrong choices"). It's a bit like saying Christiano Ronaldo is the smartest tactician when in reality he might just be the player with the most physical abilities. He might be the best tactian as well, but it's a bit harder to tell when his physical capabilities are enough to outshine the rest.

At the end of the day it's in any case not about how good your strategy is, it's about whether your strategy is part of those strategies that are eligible to be examined in the EEfRT (the set of strategies I called Y above).

"Why should the study authors chose which strategies to exclude unless participants have been explicitly told beforehand?" Because they can argue that the EEfRT is only designed to study what I above called X, which requires participants to use Y. Many other EEfRT studies have done exactly the same. That has nothing to do with whether the test is actually good or accurate.