Questionnaires - design, validation and use in ME/CFS research - discussion thread

@rvallee posted this in another thread but this BMJ opinion piece is also relevant here.

Using condition specific patient reported outcome measures for long covid
https://www.bmj.com/content/376/bmj.o257
Currently long covid services are using PROMs developed for other conditions such respiratory conditions (Medical Research Council Dyspnea Scale), anxiety disorder (Generalised Anxiety Disorder Assessment) and depression (Patient Health Questionnaire)
[...]
However, this approach has several limitations. Such measures, in our experience, are cognitively burdensome to long covid patients, do not comprehensively capture the spectrum of symptoms, cannot directly engage with the underlying biological mechanisms, and are reported not to be meaningful by patients, families, and clinicians.
[...]
There is the added danger of misleading management, for example individuals scoring highly on anxiety scores may get diverted to psychological services when their anxiety is being driven by underlying dysautonomia (increased heart rate) which needs medical optimisation.
[...]
Patient groups—including under-served populations—need to be involved in the development, selection, and co-design of systems to implement long covid specific PROMs in care pathways.8 There needs to be early engagement with other key stakeholders—clinicians, health informatics, governance, researchers, and service commissioners.
 
This blog was linked elsewhere but the quoted bit is relevant here. It's unclear to me if the part I bolded below means they're only going to analyse existing questionnaires or develop new ones. Either way, are any of the Canadians here involved in this? If not, can they muscle in?
Working group five led by Dr. Nacul and Kathleen Kerr from the Ontario Environmental Clinic in Toronto are leading the development of a consistent strategy and criteria for research – how to define which subjects have ME/CFS so that research results are meaningful and which questionnaires are valid and will provide useful information.
https://www.eleanorsteinmd.ca/blog/ICanCME
 
I think researchers usually calculate test-retest reliability and internal consistency (Cronbach alpha).

Then, they look how strong the correlation is with other questionnaires that have been used a lot and should measure the same or a similar thing.

Sometimes they do a study with test subjects or patients to see if they manage to comprehend the questions, if there are any problems with interpretation.

And that's about it, I believe.

There's little testing if a questionnaire actually measures what its name suggest. Because that would require analysis and judgement and that is often seen as subjective. I think that's why we have scales for central sensitization, catastrophizing, kinesiophobia etc.

I could create the "diabetes scale" with questions as: are you often thirsty? are you often fatigued? do you feel like your blood sugar levels are low? etc. I could then measure test in ME/CFS patients and conclude that diabetes is an overlooked aspect of ME/CFS, this might explain patients' symptoms etc.

Apologies, not able to read through the thread, but stumbled across this study on fatigue in Lupus. Not sure if it aimed to be a validation study, but they used actimetry/ accelerometers to examine the correlation between subjective and objective measures and also tested the internal consistency of the subjective measures (patient reported outcomes).

Fatigue, patient reported outcomes, and objective measurement of physical activity in systemic lupus erythematosus. Lupus. 2016 Oct ;25(11):1190-9. https://pubmed.ncbi.nlm.nih.gov/26869353/

Forum thread here.

Haven't read beyond the the abstract so don't know how the idea that physical activation could decrease fatigue impacts the authors' overall reasoning.



I think both the conclusion and how they present the results doesn't suggest the study showed more than a strong correlation between fatigue and physical activity?

That means the subjective measures were consistent with the objective measures.

Don't know however if this is what they wanted to find out.

What interests me in this regard is:

Is the risk of bias in a validation study comparable to the risk of bias in a clinical trial?

If a validation study shows a strong consistency between subjective and objective measures, does that mean there would be a similar consistency when using the same measures in a clinical trial?

Edit:
I don't think there is a comparable risk of bias in both sorts of studies so don't think that in an unblinded trial even the best validated subjective measure could make an additional objective measure redundant.
 
Last edited:
This looks relevant to this thread.

The heading is:
Assessment of body-focussed Beck Anxiety and Beck Depression Inventory items in chronic illness: A preliminary analysis of emotional versus physical causes. by Jill Albertson et al.
 
Last edited:
This looks relevant to this thread.

The heading is:
Assessment of body-focussed Beck Anxiety and Beck Depression Inventory items in chronic illness: A preliminary analysis of emotional versus physical causes. by Jill Albertson et al.

I wonder if there is a link to something other than an unreadable photograph? That looks very useful.

Also, whats with the cucumber hanging over that woman's head? :D
 
I've just completed round 2 of a Delphi Consensus for long COVID evaluation. See https://www.pc-cos.org. I only just came across this on Twitter last night and hadn't seen it discussed previously. I rushed and just managed to register in as a "health professional with Long Covid" before close today and the process took a couple of hours.

Many of the questionnaires discussed here were being assessed. I hope to have rated appropriately as discussed above and elsewhere. Apart from the ridiculous overload of anxiety/depression/psych instruments, they only offered Fukuda for ME type evaluation. I zero-rated and suggested ICC.

Cognitive instruments also focused on dementia/AD or depression. Physical symptoms seemed overly focused on "lung damage" rather than eg dysautonomia/metabolic failure. I made some suggestions there too, but mucked up at the end when I intended to add the NASA lean test. There was no ability to return and review from the penultimate page, grrr.
 
It's a shame I didn't catch this for the initial intake — and others here could have been involved too, or perhaps S4ME as an organisation. Suggests it can't have been placed on the radars of the large ME groups?

Highly surprising (and also not) that Fukuda was the only one on offer. I presume CCC/ICC weren't offered up by those initially involved. If that is so, I am doubly glad I came across it. There will be two more surveys that I have to complete.

In the second and third surveys, you will rate each outcome measure again after viewing results (from the first and second survey) of the different groups taking part (e.g. people with long COVID, health professionals, etc.).

I'll see and report back how things progress, but yes it might require further direct correspondence.
 
I've just completed round 2 of a Delphi Consensus for long COVID evaluation. See https://www.pc-cos.org.

Are they including patient experts in this round, or is this a case of the blind leading the blind? (so to speak, no offence to blind people) There are no "long-COVID experts" for a Delphi Consensus process to be valid...
 
Last edited:
As I recall, the options I could choose definitely included "patient with Long Covid" or words to that effect; as well as healthcare professionals who are or are not involved in caring for pwLC, or who themselves have LC; and researchers into LC. I think there were other categories also (eg perhaps patient support group rep). I had to hustle to make the deadline, so I didn't spend too much time looking at these preliminaries.

Hopefully, I'll get a sense of the contributors from the responses when they come through, but I guess there will be varying degrees of relative expertise (and the inevitable self-assumed expertise). As you point out, no-one can be an expert in a condition that has yet to be explained. Let's hope the process can make something reasonable, although I expect it will inevitably be well short of ideal.
 
I'll see and report back how things progress

I've just completed round 2 of the Delphi process. This is the part where you can change your responses depending on your review of the spread of others' responses. They divided into: patients with LC, medics/researchers with LC, and medics/researchers without LC. Groups 2 and 3 seemed broadly similar as I was going through.

Perhaps unsurprisingly, I was at significant variance with the majority of my responses. Those I was happy with tended to also be favoured by others, but where I disliked the instrument, most favoured. Eg various* fatigue scales were included, which I had independently rated at 2, or occasionally 1. Many respondees were clustered at 7-9. Now that I review commentary here and in other threads, I think I probably managed to represent our thoughts, but suspect it'll end up as a minority opinion.

* You know.
 
They divided into: patients with LC, medics/researchers with LC, and medics/researchers without LC. Groups 2 and 3 seemed broadly similar as I was going through.

Just to expand on that. The medics/researchers with or without LC looked to have a similar mode +/- mean, but the deviation was more in those with LC. As I was going through I wondered if this meant that only some of those with lived experience were overcoming their professional biases and seeing the deficiencies of the existing instruments (and paradigm). I may be trying to read too much into it.
 
I was doing a bit of work on something else, and came across this paper:
I won't make a separate thread for it because it's pain, not ME/CFS related, but it's a terrific example of how the problems of poorly conceived measures are then compounded by data analysis techniques that take the data out of context. And then there is no one with common sense and decency in the report drafting, or editing or peer review stages to stop really appalling conclusions causing harm.

Ethnic- and sex-related differences in pain characteristics, psychological distress and pain-related disability in patients attending a New Zealand teaching hospital pain service, 2018, Burri et al )

This study took the data of 798 people who attended a hospital pain service over a few years. Each person was required to fill out a battery of surveys to assess the usual, depression, anxiety and so on, and also 'pain catastophising'. The results of these surveys were used to help work out what sort of help the person needed (which I think was probably mostly thought to be helping them to learn to live with the pain - but that's another story). As part of an initiative to demonstrate the utility of analysis of patient records, some researchers came along and analysed these surveys.

One of the conclusions was “Pacific Islanders had the highest tendencies to catastrophise about their pain followed by Maori and Asians”. Which is just staggering. As background, many Pacific Islanders in Auckland are newly arrived from the islands. These immigrants don't have much capital and they end up working in the low paid physical jobs - cleaning, bus-driving, labouring. There maybe some English language issues and certainly a hesitancy about interacting with institutions. People pay to see the GP, so, if money is tight, health care gets put off.

So, Strike 1 is that the scale is called the Pain Catastrophising Scale, when what it is actually is a Pain-related worry scale. Put in those terms, it's probably useful for a clinician to get an idea of how much concern pain is causing someone. For example, a cleaner who may not be able to keep doing their job, and who then probably could not keep covering the costs of rent and food for their family is likely to be worrying a lot more than someone who is finding it difficult to keep playing tennis on the weekend. I guess if someone seems to be worrying a lot about their pain, out of proportion to the difficulties they face, maybe reassurance that what they have isn't cancer, just the wear and tear of age, something like that, could be part of the clinical response. But the survey doesn't say anything about the difficulties someone is facing, so it can't tell you how reasonable the worry is.

Strike 2 is the data analysis, that takes that data and uses it out of context, reporting the results still with that 'catastrophising' label. It's worse than that though. The paper says that Pacific Islanders constituted 10% of the hospital catchment population, but only 2.6% of the people attending the pain clinic were Pacific Islanders. To be clear, that is 21 people out of the sample of 798 people that they are basing this abstract-worthy conclusion on. It's extremely likely that the average pain level of the Pacific Islanders making it to the pain service was higher than most, even if it isn't showing up in the results. And 21 people is not a big enough sample to make conclusions about a whole ethnicity (in fact, 'Pacific Islanders' covers multiple ethnicities). There were only 30 Māori people and 44 Asian people in the sample also.

I don't think the p-values in the chart showing the results make sense. It's not at all clear to me what comparisons the p-values actually relate to.

Screen Shot 2023-01-23 at 11.19.09 am.png


Strike 3 is that no one at any point in getting the paper into the New Zealand Medical journal thought, hang on, that sounds pretty offensive. It might cause harm saying that, are we really really sure? It went though peer review. A fair proportion of the doctors in New Zealand probably read that abstract and went, 'yeah, those Pacific Islanders, they catastrophise, they make a big deal of their pain, not like me'. I can't see any record of anyone writing in to the journal saying 'I don't think this is right'.
 
I can't see any record of anyone writing in to the journal saying 'I don't think this is right'.
Maybe all our doctors are so overworked and burnt out they don't bother to read the New Zealand Medical Journal? Or maybe they quickly worked out that this sort of nonsense is not worth wasting reading time on? It's not like this article provides any actionable information when faced with an individual patient. Even if the data was rock solid - which almost certainly it is not (though I've only read the abstract) - it's hardly relevant if men as a group on average have "less pain acceptance an willingness" or whether as a group on group average "Middle Eastern/Latin/African patients showed highest levels of kinesiophobia".

Doesn't explain though why anyone thought this was a good way to study pain experiences, why anyone thought it was a good study to fund, or why anyone thought it was worth publishing. Sigh
 
As part of the 2021 NICE ME/CFS guideline there are some recommendations for research. They include this one:
Individual research recommendation details
Recommendation ID
NG206/02
Question

A core outcome set: What core set of relevant health outcome measures should be used for trials of treatments for ME/CFS and managing symptoms of ME/CFS?

Any explanatory notes
(if applicable)
For a short explanation of why the committee made this recommendation, see the rationale section on managing ME/CFS. Full details of the evidence and the committee's discussion are in evidence review G: nonpharmacological management of ME/CFS.

Source guidance details
Comes from guidance
Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management
Number
NG206
Date issued
October 2021
Other details
Is this a recommendation for the use of a technology only in the context of research? No
Is it a recommendation that suggests collection of data or the establishment of a register? No
______________________

There is a new research project under way that is discussed on this thread:
UK:ME Association funds research for a new clinical assessment toolkit in NHS ME/CFS specialist services, 2023
 
One of the conclusions was “Pacific Islanders had the highest tendencies to catastrophise about their pain followed by Maori and Asians”. Which is just staggering.

It reads like a variant of patient-blaming, only with racist overtones.

Even if "catastrophising" actually exists (which is not a given, despite the researchers' peculiar lack of curiosity about it), so what? The patients would still be in pain even if they stopped.

Perhaps unsurprisingly, I was at significant variance with the majority of my responses. Those I was happy with tended to also be favoured by others, but where I disliked the instrument, most favoured. Eg various* fatigue scales were included, which I had independently rated at 2, or occasionally 1. Many respondees were clustered at 7-9.

I wonder if some respondents haven't gained insight into some of the issues yet? For patients at least, it can take a long time to develop an appreciation of the differences between fatigue, fatiguability, and weakness, and even then it can be hard to make clear distinctions. But an awareness of the differences does at least highlight the problems with some fatigue scales, which don't offer opportunities to rate them separately or even acknowledge they exist.
 
Back
Top Bottom