JoClaire
Established Member (Voting Rights)
Thx for sharing.
I'm still in PEM and digesting. But curious how you were able to connect the grip data to the PHTC data.
-
New Forum Software Installed (Still a few issues but reopening the forum) More details.
Grip test results and brain imaging in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al
- Thread starter Andy
- Start date
Thx for sharing.
I'm still in PEM and digesting. But curious how you were able to connect the grip data to the PHTC data.
The EEfRT data and grip data are both shared on the MAP ME/CFS website. Grip data is in the "Post-Infectious MECFS at the NIH: Body Composition, Exercise, and Bioenergetics" data set and EEfRT is in the "Post-Infectious MECFS at the NIH: Neurophysiology" data set. The EEfRT data there is only summary variables, not trial by trial, and you can't connect the MAP ME/CFS data to the data shared with the nature article because they have different participant IDs, so some analyses aren't possible but I think the ones you are interested in are.
Just adding this paper as a link: Endurance time of grip-force as a function of grip-span, posture and anthropometric variables - ScienceDirect
"The results indicate that the endurance time decreases significantly as the grip span deviates from the optimal in both directions. On the other hand, the considered shoulder postures did not have a significant effect on the endurance time. "
Janna Moen PhD
Established Member
Hi everyone, I am working on my writeup of this study for my blog and I've come across what I believe might be some major issues with the statistics in this analysis, specifically in regards to the analysis of the EMG data (slope of the Dimitrov Index) and the MEP amplitude data. These are their measures of "peripheral" vs "central" fatigue, which I'll try to explain briefly:
These are the images from the manuscript of the figures in question:
The claim is that the ME/CFS group could not have muscle fatigue because the Dimitrov Index decreased following fatigue onset, whereas they also could not have central fatigue because their MEP amplitudes increased over the course of fatigue development. I won't get into the fact that cortical hyperexcitability is actually well described phenomenon in other neurological disorders.... Regardless, they use this to argue that ME/CFS patients do not experience muscle fatigue despite being prone to grip failure and easily fatigued vs HVs.
Now I started to get a little bit curious because I noticed there are actually no statistical annotation on these graphs. Considering this data is an ordered time series, one would assume they would use something like a repeated measures 2-way ANOVA to characterize the interactions between both groups over time, or use some kind of mixed model. Not so. Apparently they used t-tests?
Right off the bat this analysis seems inappropriate considering t-tests are only useful for differentiating the relationship between two variables, but the data they present are repeated measures over a time series. ANOVA was developed specifically to avoid performing repeated t-tests between groups in this kind of experimental design because it is very prone to false positive errors, but here we don't even know which blocks are being compared. It's also unclear to me where these means they quote come from (0.2±0.5 for ME/CFS, -0.43±0.3 for HVs). In the graph above for the Dimitrov Index, the mean values all appear to be between 0.005 and 0.02 with no negative values. Then I thought maybe they were comparing the change of the slope over time, so subtracting the fatigue block from the first block, but that math doesn't add up either and doesn't explain the decision to use a t-test instead of something like 2-way ANOVA.
For the MEP data, again from the supplement:
Once again, we have a t-test used inappropriately on an experimental design with two dependent variables (time and patient group). These numbers at least seem a little bit more plausible if they were subtracting fatigued blocks from initial block, but again don't seem to add up when the differences between b1 and f1 seems much larger than 0.13 MEP units. It also seems problematic that we do not know how the data was handled for this analysis and which specific datapoints it draws from, and I would be very interested to hear the justification for using t-tests here instead of a more informative analysis - it's very odd because the effect size seems fairly large in both cases, but the small sample size might have tipped the ANOVA into a non-significant interaction.
I guess I am looking for a bit of a sanity check here - am I missing something, or is this really as egregious as it looks? If so, I feel like I need to re-analyze their data using the correct stats and depending on what that tells me, write a letter to the editors at Nat Comm. Would love for someone with more statistical prowess to chime in before I get ahead of myself.
- Peripheral fatigue is muscle fatigue that occurs due to changes at the neuromuscular junction or within the muscle itself, i.e. recruitment of muscle fibers, accumulation of lactate. During the repetitive grip task, they recorded the electrical activity of the muscles in the arm (via EMG) and looked at the way the activity changed over the course of the 30 second grip block. A steep/larger change means that muscle activation dropped off quickly due to developing fatigue.
- Central fatigue is a "break" on motor activity from somewhere along the corticospinal tract, typically in response to afferent sensory input from the muscles. We can measure this using transcranial magnetic stimulation (TMS) directed at the motor cortex - before the grip strength session, they would find the location within the motor cortex they could stimulate to induce activation of the same arm muscles. They can then repeat the stimulation in between grip sessions to see how the motor-evoked potential (MEP) is changing due to fatigue onset. When central fatigue starts to develop, MEP amplitude will decrease due to the recruitment of inhibitory signals within the corticospinal tract.
These are the images from the manuscript of the figures in question:
The claim is that the ME/CFS group could not have muscle fatigue because the Dimitrov Index decreased following fatigue onset, whereas they also could not have central fatigue because their MEP amplitudes increased over the course of fatigue development. I won't get into the fact that cortical hyperexcitability is actually well described phenomenon in other neurological disorders.... Regardless, they use this to argue that ME/CFS patients do not experience muscle fatigue despite being prone to grip failure and easily fatigued vs HVs.
Now I started to get a little bit curious because I noticed there are actually no statistical annotation on these graphs. Considering this data is an ordered time series, one would assume they would use something like a repeated measures 2-way ANOVA to characterize the interactions between both groups over time, or use some kind of mixed model. Not so. Apparently they used t-tests?
Right off the bat this analysis seems inappropriate considering t-tests are only useful for differentiating the relationship between two variables, but the data they present are repeated measures over a time series. ANOVA was developed specifically to avoid performing repeated t-tests between groups in this kind of experimental design because it is very prone to false positive errors, but here we don't even know which blocks are being compared. It's also unclear to me where these means they quote come from (0.2±0.5 for ME/CFS, -0.43±0.3 for HVs). In the graph above for the Dimitrov Index, the mean values all appear to be between 0.005 and 0.02 with no negative values. Then I thought maybe they were comparing the change of the slope over time, so subtracting the fatigue block from the first block, but that math doesn't add up either and doesn't explain the decision to use a t-test instead of something like 2-way ANOVA.
For the MEP data, again from the supplement:
Once again, we have a t-test used inappropriately on an experimental design with two dependent variables (time and patient group). These numbers at least seem a little bit more plausible if they were subtracting fatigued blocks from initial block, but again don't seem to add up when the differences between b1 and f1 seems much larger than 0.13 MEP units. It also seems problematic that we do not know how the data was handled for this analysis and which specific datapoints it draws from, and I would be very interested to hear the justification for using t-tests here instead of a more informative analysis - it's very odd because the effect size seems fairly large in both cases, but the small sample size might have tipped the ANOVA into a non-significant interaction.
I guess I am looking for a bit of a sanity check here - am I missing something, or is this really as egregious as it looks? If so, I feel like I need to re-analyze their data using the correct stats and depending on what that tells me, write a letter to the editors at Nat Comm. Would love for someone with more statistical prowess to chime in before I get ahead of myself.
Well done @Janna Moen PhD, it does look odd.
I thought the numbers might have been the slope of the lines (ie a regression) over the 5 timepoints. That would sort of make sense for a t test, the mean slope for the ME/CFS group versus the mean slope for the controls - are they different, considering the number of observations of the slopes ((6+8)-2df=12) and the variability? (I mean, my stats is way too rusty for me to know if it acceptable or not, but I can see how someone might think it was.)
In terms of the signs and comparative magnitudes of the figures given, that would sort of make sense. i.e. for the Slope of Dimitrov (0.2 versus -4.3), the slope of the series is a bit positive for the controls and substantially more negative for the ME/CFS group. For the motor evoked potential (-0.13 versus +0.13), the trendlines in Fig 3c are symmetrical but in opposite directions i.e. a bit negative for the controls and an equal bit positive for the ME/CFS group.
But, I can't make the slopes actually equal the those numbers. So yeah, I don't know what they did and I haven't looked at the data. Whatever they did, I don't think they have explained it sufficiently well, so I reckon it's very reasonable to ask the NIH about it.
I thought the numbers might have been the slope of the lines (ie a regression) over the 5 timepoints. That would sort of make sense for a t test, the mean slope for the ME/CFS group versus the mean slope for the controls - are they different, considering the number of observations of the slopes ((6+8)-2df=12) and the variability? (I mean, my stats is way too rusty for me to know if it acceptable or not, but I can see how someone might think it was.)
In terms of the signs and comparative magnitudes of the figures given, that would sort of make sense. i.e. for the Slope of Dimitrov (0.2 versus -4.3), the slope of the series is a bit positive for the controls and substantially more negative for the ME/CFS group. For the motor evoked potential (-0.13 versus +0.13), the trendlines in Fig 3c are symmetrical but in opposite directions i.e. a bit negative for the controls and an equal bit positive for the ME/CFS group.
But, I can't make the slopes actually equal the those numbers. So yeah, I don't know what they did and I haven't looked at the data. Whatever they did, I don't think they have explained it sufficiently well, so I reckon it's very reasonable to ask the NIH about it.
Would it be possible to ask for the statistical analysis plan in detail, including which tests were used on which data points and why? And to ask for another statistician to reanalyse the data using what they consider appropriate tests? I'm sorry my stats is too rusty to be useful.
I have completely forgotten what this was all about and can't wrap my head around it, but from what I can remember @Janna Moen PhD I see this similar as @Hutan. The graph is the graph of the slope of the Dimitrov index. I don't think that they argue that "ME/CFS group could not have muscle fatigue because the Dimitrov Index decreased following fatigue onset", but rather that they argue that there was a "relative decrease in the slope of the Dimitrov index" which I think would be a stronger statement (I'm not cognitively well enough, but I think the first statement were to only roughly tell you that the Dimitrov index was concave whilst the second one implies it is strictly concave). The slope decreasing essentially means negative second derivative (with the caviat that the slope isn't always decreasing) and the relative means comparing those values.
That is to say I think they are doing exactly what your second thought was.
"In the graph above for the Dimitrov Index, the mean values all appear to be between 0.005 and 0.02 with no negative values."
So the 0.2 vs -0.43 sort of makes sense to me (a slight increase for HV and a much larger decrease for ME/CFS), I think the numbers for HV's come before those of ME/CFS.
I can't comment about appropriate testing, but wouldn't be suprised if what they did is inappropriate as it seems to also have been the case elsewhere. I also wouldn't be surprised if they went with something simply because it shows what they want to show, rather than being the more valid thing to do.
I will have another look once my brain clears up a bit.
That is to say I think they are doing exactly what your second thought was.
"In the graph above for the Dimitrov Index, the mean values all appear to be between 0.005 and 0.02 with no negative values."
So the 0.2 vs -0.43 sort of makes sense to me (a slight increase for HV and a much larger decrease for ME/CFS), I think the numbers for HV's come before those of ME/CFS.
I can't comment about appropriate testing, but wouldn't be suprised if what they did is inappropriate as it seems to also have been the case elsewhere. I also wouldn't be surprised if they went with something simply because it shows what they want to show, rather than being the more valid thing to do.
I will have another look once my brain clears up a bit.
Last edited:
Janna Moen PhD
Established Member
Thanks all - yes I think it's a good point that they might be doing some kind of regression analysis here and then using a t-test to compare slopes. That would probably be the most straightforward answer, although again it's disappointing that we do not have any information regarding how they derived these slopes. If a regression analysis was part of this I would also expect to see the regression lines and stats overlaid on the graphs, no? I'm going to run this past a few other folks in my lab and reach out to the NIH team to see if they can provide any clarity.
Janna Moen PhD
Established Member
This is definitely what I meant but didn't explain it well, thanks for clarifying!!
ME/CFS Skeptic
Senior Member (Voting Rights)
If you click on 'Source Data' in the Walitt et al. paper you find the data behind the figures. For figure 4B you get the data for each of the participants for 5 blocks (the first , the last before fatigue and the first three after fatigue).
In the graph they seem to have plotted the means per group for these 5 blocks. I quickly calculated the regression line for these means per group in Excel and got -0.0042 and 0.0002 as slopes. I assume they calculated the slopes for each participant and then did a t-test for the between-group difference.
ME/CFS Skeptic
Senior Member (Voting Rights)
If I try to do this using statsmodels in python I get:
t = 3.1995698023875945, p-value = 0.007638634835067758
There is one NaN (for HV-5 blok 5) and in my calculation I simply ignored this value.
Janna Moen PhD
Established Member
Ah, finally some answers to the numbers problem! I mean, it's still off by several orders of magnitude, but at least it makes sense as an attempt to explain the analysis.
It still strikes me as an odd choice of analysis and IMO it's misleading to not plot this with the figures to make it clear what kind of statistics they are running. Sounds like I need to reactivate my GraphPad Prism subscription or figure out how to do ANOVA in python.
Do you know whether the Dimitrov index has been validated. The thread on the original study suggests it wasn't validated at the time and I'm wandering whether that has changed?
The fact that one number is off an additional order of magnitude than the other one suggest that either something else is going on which cannot be corrected by taking the relative rate (i.e. doing something like the taking difference of these values diving by initial value and multiplying by 100) or more likely that there is simply a typo and 0.2 is supposed to be 0.02.
What do you think @ME/CFS Skeptic ?
Janna Moen PhD
Established Member
It looks like it had been published on previously, although muscle physiology is way outside of my wheelhouse. This is the initial publication on the Dimitrov index:
https://journals.lww.com/acsm-msse/...e_fatigue_during_dynamic_contractions.14.aspx
Not sure what the critique was in the original thread, but I don't feel like I know enough about this field of study to really comment on how valid the slope of the DI is for determining the presence of peripheral fatigue. Muscle EMG recordings seem fairly objective and the slope of the DI is basically telling you how much the firing frequency changes over the course of the grip task, which sounds like it should help control for things like electrode placement as the change over time should stay constant. But again, this is much closer to physics than biology at this point and I'm just a lowly neuropharmacologist, so grain of salt and all that jazz.
Sid
Senior Member (Voting Rights)
Maybe they ran a more conventional analysis like linear mixed effect model and didn’t get the desired result.
ME/CFS Skeptic
Senior Member (Voting Rights)
Yes I might be a typo because when trying to do a t-test of all participants' slope, I got the same results they report namely t =32 and p = 0.008.
I think the most straightforward approach would be to use a linear mixed effect model that tests for the interaction of patient_group and blocks while using participant ID as a grouping, random effect. When I try to do this (Caveat: I have little experience with this) I got a p-value of 0.013.
As @EndME says, do check out that thread on the Dimitrov Index, especially Simon's comment.
Muscle Fatigue during Dynamic Contractions Assessed by New Spectral Indices, 2006, Dimitrov et al.
It would be worthwhile getting an opinion on it from someone who knows about this stuff.
Here, they are comparing the linear changes at 5 rather arbitrary points for data series that have very different lengths. (ie the ME/CFS people typically fatigued more quickly and didn't do so many reps.). Some of the healthy people didn't stop doing reps until they got to the maximum allowable number of reps. If you look at the chart replicated above in post 71, it looks as though the healthy group's line would have kept going down, perhaps following a similar trajectory as the ME/CFS group if everyone had been allowed to continue on with the reps until they were fatigued.
I would like to see the charts for each individual at all of the measured timepoints. I think possibly they have measured different things in the ME/CFS and healthy groups, because not all of the healthy people exercised to exhaustion.
