Questionnaires - design, validation and use in ME/CFS research - discussion thread

[Trish]

I am raising this topic for discussion because I think there are fundamental problems with the design and use of questionnaires in ME/CFS and other medical research.

Questions for consideration:

1. Should questionnaires ever be used to draw conclusions about psychological factors affecting people with physical diseases?
a) what are the ethical considerations involved?
b) Do ethics committees have a duty to look beneath the surface at what is actually being asked in questionnaires, or is it sufficient for them to see the list of named questionnaires?
c) Should comparitor groups always be other equally disabling chronic physical diseases?

2. How are questionnaires validity and reliability tested? Do these methods result in useful questionnaires from ME/CFS patients' perspective?
Here's a paper and a chart found from a quick search:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5463570/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5463570/figure/F1/

3. Why are so many quesionnaires used in ME/CFS psychosocial research so inappropriate and badly designed?

4. Why do funding bodies keep supporting PhD studentships and academics' research based entirely on questionniares? Should such studies require expert patient involvement from the start?

4. What can we do about this?

This thread is open for discussion in whatever direction it goes. I hope some sort of action might come out of it.

In my next post I'll give a couple of examples of particularly bad use of questionnaires in recent and current research discussed on this forum.

 

[Trish]

Example 1:
40-item Cognitive and Behavioural Responses Questionnaire (CBRQ)
The questionnaire is on pages 33 - 36
Thread here
Used in this study
Generalised worry in patients with [CFS] following Cognitive Behavioural Therapy - a prospective cohort study in secondary care, 2022, Chalder et al
This questionniare is subdivided under headings: fear avoidance, catastrophising, damage, embarrassment avoidance, symptom focusing, all or nothing behaviour, and avoidance/resting behaviour.

This particular study only used the parts on fear avoidance, and avoidance behaviour and claimed the study showed
Avoidance/resting behaviour at baseline completely mediated the association between baseline worry and work and social adjustment scores following CBT (Figure 5).

Comment: It is clear from the use by the Chalder team that this is used to judge negatively factors most of us would describe as appropriate pacing behaviour and accurate interpretation of symptoms as physical in origin. The fear avoidance CBT model used by these researchers would therefore be judged successful on the basis of the researchers' misinterpretation of the significance of the results for the patients' health.

Example 2:
The Experiences in Close Relationships-Revised Questionnaire
This was used in this ongoing PhD project:
Irritable bowel syndrome (IBS), chronic fatigue syndrome (CFS) and fibromyalgia in those aged 45 and older – the impact of emotions
The questionnaire consists of 36 statements about emotional aspects of the person's intimate relationships.
As I commented in my letter to the researchers, I felt this was completely inapproprate in the context of an online ME research study, and in any case would give no useful information about the relationship between emotions and physical symptoms.
Why not just ask the question straightforwardly: Has your physical ill health affected your close relationships on a scale from badly negatively affected through unaffected to positively affected. And a second question about whether personal relationships have affected your physical health.

Example 3:
The Brief Illness Perception Questionnaire
This was also used in the study in Example 2 above

Comment: I understand from this paper that it was developed for use in other chronic conditions where there are known causes and treatments such as asthma, type 2 diabetes and heart disease, so questions on the person's understanding of their disease and how much control they feel they have over it make more sense and may be useful in assessing whether patients need more information and help with self administered treatment.
For ME/CFS this questionnaire seems not to provide any useful information, let alone a valid assessment of the severity level which seems to be the intention in this particular study.

I suggest if you want to discuss these questionnaires in the context of the studies linked, it would be best to do so on the relevant threads. My aim on this thread is to look at the wider issues raised by them.

 

[Hoopoe]

1. Should questionnaires ever be used to draw conclusions about psychological factors affecting people with physical diseases?

I've asked myself what the concept of illness is. I think a key element is "abnormal functioning", meaning that something is no longer working right. It's normal to feel sleepy, but there is also such a thing as excessive sleepiness. A fireman that has worked in an emergency for 20 hours will be extremely tired, and despite the extreme level of tiredness what he is experiencing is normal. The context is essential. For mental illness it is the same. There is a big difference between very low mood in someone who has no reason for this, and in someone who is facing ongoing unresolvable adversity like homelessness due to untreatable chronic illness. I don't think it's an illness to feel despair in the face of unmanagable adversity in life. But if the mood issues consistently occur in an inappropriate manner then it's reasonable to think that something in the body is not working right.

Questionnaires usually exclude the context in which distress occurs entirely. Vital information is lost through this.

The recent paper by Chalder on worrying in CFS seems to reveal a view of mental illness as something that can be diagnosed by patients scoring sufficiently high on a questionnaire. No attempt is made to understand whether these patients might be worried over having a progressive illness, possible homelessness or poverty or other very good reasons to be worried.

To solve a problem you need to know the cause.

 

[Mithriel]

First thoughts are that use of a questionnaire is valid. It mimics the questions a doctor would ask to reach a diagnosis and is a valuable concept.

Having said that, very few are fit for purpose, especially the ones used for ME and for mental health in general. A doctor will see you in context and take that into account when diagnosis. A question which asks how well did you sleep last night is one thing but if you have turned up at the GP surgery with baby triplets in tow the question does not need asked!

It is impossible for me to understand how anyone can think the Chalder Scale is of use. What information was it designed to get from patients and why did they think this form was the best way to get it?

Mental health ones should obviously not be used by patients with physical disease as they stand. Someone involved in developing them should ask how many of the questions would give the wrong answer if there was physical disease. If doctors genuinely want to know if there is a mental health problem you would think they would want the questions to be focussed just as much as we do.

It leads to the suspicion that things like the depression questionnaires were used in ME because they confirmed Wessely's views. In CFS, FND, MUS all the studies done by the BPS are looking for confirmation and no one checks to see if there is an alternative explanation because it might show their theories are wrong!

The answer to the Chalder Scale may be it was maybe made so opaque so people would need to be steered toward how to answer to make all the CBT and GET treatments look successful.

I could write a book, if I could write a book!

 

[Peter T]

The recent paper by Chalder on worrying in CFS seems to reveal a view of mental illness as something that can be diagnosed by patients scoring sufficiently high on a questionnaire. No attempt is made to understand whether these patients might be worried over having a progressive illness, possible homelessness or poverty or other very good reasons to be worried.

I wonder if part of the root of this, is the idea that if validated the actual questions in a psychometric tool are irrelevant. The idea that once validated it ceases to be a questionnaire and rather an objective measuring tool.

As an undergraduate, some forty years ago, I was taught that once validated, the individual questions become irrelevant as long as you have the evidence when developing that tool that everyone with the specific trait being measured answers these questions in more or less the same way. However for most psychological traits there are no objective ways to independently validate a questionnaire, as the only measures for these traits are asking questions. So validating a questionnaire is like trying to lift yourself by your own boot straps.

Given independent verification is unachievable, the individual questions remain vital. You need to be able to distinguish possible different reasons for particular answer patterns, for example not socialising may be the result of social avoidance or it could be the result of being house bound. Unfortunately too many people do not realise their questionnaires are not an objective psychometric tool but a list of questions the answers to which may be context dependent rather than necessarily indicative of a particular psychological trait.

 

[duncan]

The patient voice frequently is best heard from a questionnaire, but it can and is distorted all the time. Questionnaires have for many of us become synonymous with sleaze and bias and corrupt intent. A patient-selected patient advocacy needs to work with an independent questionnaire company to formulate one that is representative and will elicit statistically meaningful and salient patient data.

Who prioritizes that data also must be patient led. I've seen researchers skew results in such a way as those symptoms that debilitate most or many of us are relegated to "minor" import.

 

[Peter T]

The patient voice frequently is best heard from a questionnaire, but it can and is distorted all the time. Questionnaires have for many of us become synonymous with sleaze and bias and corrupt intent. A patient-selected patient advocacy needs to work with an independent questionnaire company to formulate one that is representative and will elicit statistically meaningful and salient patient data.

Who prioritizes that data also must be patient led. I've seen researchers skew results in such a way as those symptoms that debilitate most or many of us are relegated to "minor" import.

It is important to distinguish between a questionnaire as a psychometric tool designed to measure specific psychological traits and a questionnaire designed to elicit information about an individual or a group of people. I suspect in reality such as the Calder scale is the later, misused as the former.

 

[Hoopoe]

I wonder if part of the root of this, is the idea that if validated the actual questions in a psychometric tool are irrelevant. The idea that once validated it ceases to be a questionnaire and rather an objective measuring tool.

I believe the BPS people have said repeatedly that you don't need to understand the cause of distress to treat it. I think they're just fooling themselves, and it's easy to see how they're fooling themselves: they see that CBT always reduces scores on questionnaires, and interpret this as evidence that the causes of distress don't matter. This then feeds the belief that happiness is independent of circumstances and simply a choice.

They ignore the issue that the typical clinical trial of CBT doesn't adequately control for response bias and would therefore be expected to nearly always produce at least modest improvement on questionnaires.

 

[ME/CFS Skeptic]

Good questions.

I think researchers usually calculate test-retest reliability and internal consistency (Cronbach alpha).

Then, they look how strong the correlation is with other questionnaires that have been used a lot and should measure the same or a similar thing.

Sometimes they do a study with test subjects or patients to see if they manage to comprehend the questions, if there are any problems with interpretation.

And that's about it, I believe.

There's little testing if a questionnaire actually measures what its name suggest. Because that would require analysis and judgement and that is often seen as subjective. I think that's why we have scales for central sensitization, catastrophizing, kinesiophobia etc.

I could create the "diabetes scale" with questions as: are you often thirsty? are you often fatigued? do you feel like your blood sugar levels are low? etc. I could then measure test in ME/CFS patients and conclude that diabetes is an overlooked aspect of ME/CFS, this might explain patients' symptoms etc.

 

[Kitty]

1. Should questionnaires ever be used to draw conclusions about psychological factors affecting people with physical diseases?

My first response would be: only if those psychological factors are what is being studied. Psychological factors aren't directly relevant to the diagnosis or management of physical diseases.

If living with a physical disease results in psychological, emotional or social problems, then of course patients should be referred on so that the issues can be treated in their own right by someone with the expertise to do so. Just as they would be if someone with psychological problems harmed themselves physically.

 

[Kitty]

3. Why are so many quesionnaires used in ME/CFS psychosocial research so inappropriate and badly designed?

Because they're pretending to research ME. They're not; they're researching psychosocial issues, which are not specific to ME and frequently not even relevant to it.

Obviously, there might be psychosocial issues that are relevant to certain illnesses or groups of illnesses, for instance those resulting from the way sufferers are treated by GPs, specialists, families, partners, employers, benefits agencies, insurance agencies, and so on. But those questions are not the ones being asked.

 

[Kitty]

4. Why do funding bodies keep supporting PhD studentships and academics' research based entirely on questionniares?

Because questionnaires are really, really cheap to compile and distribute? You don't even need to pay for envelopes and stamps these days.

And they're really, really easy for supervisors to manage. Nothing messy or uncontrolled, like having to meet people or test them for something.

Should such studies require expert patient involvement from the start?

Of course it should, but universities seem to be about graduating as many of their students as possible for as little expense, with as little inconvenience, and using as few staff as possible, whilst raking in as much profit as possible from rackets like accommodation. Dealing with actual patients is bristling with all kinds of expensive complications.

Obviously that's all very cynical, but there's probably some truth in it. I suspect it's genuinely difficult at times for tutors and supervisors to obtain the resources they'd need to enable a broader range of investigations. I also wonder if students are being trained—consciously or not—to expect that sizeable components of both learning and research are done at a distance, over the internet, and they propose these approaches because that's what they expect to be doing. (Given the direction in which some types of healthcare are moving, they're probably not entirely wrong.)

 

[Fainbrog]

If one could trust that a questionnaire/answers would not be twisted/misconstrued by those assessing an ME patient, perhaps some of the questionnaires available could have a place. However, as they can easily be used to suit a desired narrative, I'd say, they have no place.

Having been required to have a number of assessments for various work related benefits and ongoing 'rehab' (don't even go there; I'm still very bloody ill, how can I begin rehab?), using The Chalder Fatigue Scale, GAD7 and PHQ9, Mini Mental State Examination (MMSE), that I'm aware of, I am now extremely wary of how I answer them.

Having seen scores be used to suggest that I am severely depressed (which couldn't be further from the truth), where a set of questions were posed, without any context or explanation of how the answers would be used.

PHQ9 Questionnaires ask amongst other things:

Over the last two weeks, how often have you been bothered by any of the following problems?

- Trouble falling or staying asleep, or sleeping too much?
- Feeling tired or having little energy?
- Trouble concentrating on things, such as reading the newspaper or watching television?
- Moving or speaking so slowly that other people could have noticed?

Which are questions that we would probably all answer Nearly every day to if posed in the course of a general questionnaire as we are answering based on our physical/cognitive impairment.

I was lucky that I had a chance to have the final report changed on one occasion, when I took the time to see what some of the reported values actually meant. Many may not have had the energy or inclination to understand what was being said about them.

In another case, I only saw the report after the conclusions had been reported and the data conclusions have been quoted back to me suggesting lesser impact of my illness than is actually the case.

Many of us will have similar experiences, I'm sure - albeit each with an individual flavour.

So, returning to the start of my post, I think it's about trust..and I aint got any left!

 

[NelliePledge]

Good questions.


I could create the "diabetes scale" with questions as: are you often thirsty? are you often fatigued? do you feel like your blood sugar levels are low? etc. I could then measure test in ME/CFS patients and conclude that diabetes is an overlooked aspect of ME/CFS, this might explain patients' symptoms etc.

It would almost be worth doing this just to prove the point, sadly I doubt that those in most need of understanding the point would actually take it in.

 

[duncan]

How do you qualify what patients experience, in a statistically compelling fashion, if not through a questionnaire?

Just because so many have been crafted poorly and even, arguably evilly, it doesn't mean a good and useful one cannot, or should not, be built.

 

[Kitty]

How do you qualify what patients experience, in a statistically compelling fashion, if not by through a questionnaire?

The problem historically is that many of them have had a psychosocial basis, which is pointless. When it comes to other types of questionnaire, I guess it's possible to compile meaningful statistics about length of illness, work/education status, age at onset, etc, and to pick up some broad patterns from this.

It depends on what you're going to do with the data, and whether that actually gets us anywhere. How much do clinicians and researchers really learn about ME from questions where the answers are unequivocal enough to use as statistics? (Are there any such questions?)

Patients have often pointed out that we lack a good clinical description of PEM. I'm not convinced there's any realistic chance of capturing that from questionnaires, though, because all the nuance is lost. Perceptive researchers can pick up important detail from individuals with good insight or descriptive skills, especially if they're free to ask any supplementary question they think useful. None of that happens if they just send them a list of questions with tick-box answers.

My position would be that people communicate experience most accurately and effectively through story, something we've probably understood since the Mesolithic. A lot can be learned from people telling in their own words, but we'd need someone to listen.

 

[duncan]

My position would be that people communicate experience most accurately and effectively through story, something we've probably understood since the Mesolithic. A lot can be learned from people telling in their own words, but we'd need someone to listen.

Yes. I couldn't agree more. But medically speaking I fear such stories would fall under anecdote or, if lucky, a case study.

I used to work for a research firm. We did loads of questionnaires and focus groups and the like. We had a discerning group of clientele, too, so they had to pass muster. But for the life of me I cannot remember how.

 

[Kitty]

But medically speaking I fear such stories would fall under anecdote or, if lucky, a case study.

I suppose it depends how you set it up, really.

If a clinician wants to understand how something is experienced, they listen to patients outlining their symptoms, explaining how they manage them, and describing what they do to avoid provoking them. This is a normal part of diagnosis and management, but it could also be part of a panel led by researchers trying to characterise PEM.

They'd learn so much more than by feeding rigidly standardised data into a computer—which might tell them what percentage of patients have X symptom to several decimal points of accuracy, but nothing actually about X symptom. Nor would it reveal that some of the patients unknowingly have X symptom, it's just that they haven't yet identified it as a particular thing, or been able to coin a name for it.

 

[Ravn]

1. Should questionnaires ever be used to draw conclusions about psychological factors affecting people with physical diseases?
a) what are the ethical considerations involved?
b) Do ethics committees have a duty to look beneath the surface at what is actually being asked in questionnaires, or is it sufficient for them to see the list of named questionnaires?
c) Should comparitor groups always be other equally disabling chronic physical diseases?

1. Context is everything. It's fine to use an appropriate questionnaire to assess a psychological comorbidity or the psychological effect of living with a disease. It's incompetent and biased to ask questions about symptoms of the physical disease and then use those answers to make psychological claims.
a) The latter approach is unethical, too, especially if not telling patients that their answers will be reinterpreted in this way.
b) They do have a duty to look beneath the surface where there is reason to believe there might be issues. And when you're assessing research into a contested disease, there is strong reason to assume there will be issues.
c) Yes. Although not using any inappropriate questionnaires at all would be even better.

2. How are questionnaires validity and reliability tested? Do these methods result in useful questionnaires from ME/CFS patients' perspective?
Here's a paper and a chart found from a quick search:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5463570/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5463570/figure/F1/

I strongly doubt content and construct validity have ever been properly tested in any of the questionnaires foisted upon us. Hence their utter awfulness. There may have been some cursory motivated reasoning going on, as well as the circular thinking involved in comparing a new questionnaire to older "validated" ones. Trouble is if the latter are poor to start with...

The graphic on developing a questionnaire doesn't mention patient involvement. Theoretically patients could be part of the expert committee though I doubt this is common practice.

3. Why are so many quesionnaires used in ME/CFS psychosocial research so inappropriate and badly designed?

Some are very well designed, always give the desired result (sarcasm). Other than that, repurposing questionnaires designed for other diseases is cheap. Add ignorance about ME, i.e. simply assuming it's similar to fatigue in other diseases or believing ME is psychological/psychosomatic.

4. Why do funding bodies keep supporting PhD studentships and academics' research based entirely on questionnaires? Should such studies require expert patient involvement from the start?

See point 3, it's cheap and the people involved are ignorant about ME.

Yes, expert patients should be involved at all stages, including on the ethics committees and as patient peer reviewers. If only we had the health to take all this on!

4. What can we do about this?

Involving expert patients should help but no guarantees. I spent a lot of energy trying to make Leonard Jason aware of the shortcomings of his PEM questionnaire but I don't think a single thing changed from his draft as a result of my feedback. From what I saw in the FB groups where he sought patient feedback at the time, most of it was in the 'we're so grateful anyone's listening to us' category without offering much in the way of analysis, positive or negative. The sort of thing that's nice to hear but doesn't constitute useful feedback.

What's needed instead is a bunch of experienced, constructive, critical grumps, as found on this forum. But even if we sat down and helped write a review on the unsuitability of commonly used questionnaires in ME, plus created some less bad ones ourselves, that would have little impact unless we can get it all published, disseminated and widely accepted. For which we need strong allies in the right places. And I've no idea where to find those.

Also, I think there are some organisations starting to push for expert patient involvement in research (in general, not ME specific)? What strategies do they use?

 

[Lilas]

1. Should questionnaires ever be used to draw conclusions about psychological factors affecting people with physical diseases?

No.

First, and as mentioned earlier, because "psychological" questionnaires are designed to assess psychological aspects only, without considering the context of a physical illness, which we know makes them extremely inadequate.

Secondly, the “Psychs” who use them for research purposes are unfortunately mostly of strong BPS allegiance. Their belief that any disease (even physical) has a psychological component automatically results in them not really caring about the context, have a biased view that "psychologizes everything" and observe only from the psychological and/or social point of view, convinced that if a treatment improves the psychological dimension of the person then it will necessarily have a positive impact on his physical health. The impact of the biological is thus minimized or even eluded.

Thirdly, apart from constructing a new questionnaire, with the help of a pwme expert group as proposed above, to study the impact of ME in the person's life, or to question on their experiences with health personnal, for example, I don't see the point.

In short, it seems to me that the accent should rather be placed on rigorous criticism of the inadequate, absurd use of this kind of questionnaires for the study of pwme. Biomedical research is what we need, that's all.