Playing devils advocate here - Would it be acceptable for a study of an asthma intervention to use any old criteria, including those that specifically do not require people to actually have asthma? Especially if that intervention to be tested was known to exacerbate the core symptom of asthma?
IMO, this is a key methodological issue right up there with subjective findings in unblinded trials, outcome switching, etc.
There are different sorts of asthma that you ideally might want to stratify out and treat differently, although they have often been lumped together in one study e.g. asthma caused by an allergy, asthma caused by exercise, and, for all I know, it may be hard to differentiate these, maybe they overlap, maybe we now know other sorts of asthma, maybe we will discover that there other sorts of asthma that we don't know now. Even with exercise-induced asthma, it could be a reasonable study question to ask - does careful exposure to exercise help?
So, it could be reasonable to do a study combining different sorts of asthmatics and see if exercise helps. Maybe it helps nobody, and then you know something. Maybe it only helps some, and so you have questions for another study. The biggest problem is if you use the faulty study design that gives you a positive result, regardless of treatment.
Your choice of the asthma example is interesting because there is that fantastic study where they told people that they were being given an effective asthma treatment and gave some people a placebo. The people with the treatment and the placebo both reported that their asthma was considerably better. But only the people with the treatment had improved respiratory function. That is the demonstration of the problem we are concerned about. I'll see if I can find a link.
If you do a mixed asthma study (allergic asthma, exercise-induced asthma, maybe some people have both) with only subjective outcomes and no placebo, everyone might seem better. Was it because they hoped they were better, or were they really better? You don't know. You have not learned anything.
*****
Honestly, how good are the ME/CFS diagnoses? Even experienced clinicians are struggling. We've seen that in the rate of misdiagnoses identified. Even if you think you have a "pure" ME/CFS cohort it will be mixed.
Take a group of people diagnosed under the Fukuda criteria, use a good objective outcome such as tracking their activity levels over 6 months, have two treatments that the participants think are equally likely to work, and you might be able to say that the treatment truly helped some people. Or it didn't. You have learned something. Until we have a biomarker, if you truly find exercise does help some people with chronic fatigue, it could be a reasonable thing to say 'look, we know that exercise truly does help some people with chronic fatigue, but our data also shows that it truly can set some people back'. So, we need to investigate the people who improved versus the ones who didn't because we aren't very good at knowing in advance. And, we need to be very alert to whether you might have PEM if you do decide to exercise.
Alternatively, if you find that exercise truly helps no one with Fukuda chronic fatigue, things are simple, don't recommend it to anyone.
Or, take a group of people carefully diagnosed under CCC or IOM. Give them a programme of a treatment you say is great, even exercise, then ask them how they feel. Some will drop out, but you don't count them. The ones who make it to the end of the six weekly sessions and the celebratory cake will report on your subjective survey, 'I feel better'. And that goes into a review with other similar studies, and there you have it, scientifically proven!, Cochrane gold-standard endorsed! Impossible to question. Exercise helps people with CCC criteria ME/CFS.
