Is the International Consensus Criteria (ICC) "valid and reliable"?

In the thread about translating the ICC into Hindi, there was this claim:

"International consensus criteria (ICC) originally published in English (ICC-E) is a valid and reliable tool for identifying cases of ME."

How are they validating that claim? There is no clinical test for ME to verify that someone has ME. So, is the claim just a circular argument? "100% of people who fit our ICC fit our ICC!" Of course that's valid and reliable, but also meaningless in terms of whether someone actually has ME. The criteria are vague enough that there will be people who have ME who don't quite fit the criteria, and people who fit the criteria who don't actually have ME.

The ICC is better than nothing, but claiming that it's "valid and reliable" seems misleading to me. If the goal is to provide victims with official diagnoses of impairment for insurance, work, or other needs, then a set of criteria for those impairments would be useful, whether the person has ME or long-covid or fibromyalgia or whatever.

I think a more useful task for ME would be to come up with sets of criteria for study cohorts: clear definitions of mild, medium and serious ME, with POTs or without, with digestive symptoms or without, etc. Let the studies be about apples, kiwis, bananas and oranges, rather than comparing studies about "fruit". A similar set of criteria for controls is important too.

 

That other thread about ICC is about what should be included. I started this thread, because I thought about the Hindi translation thread, and the part about "valid and reliable" suddenly struck me as inappropriate. It's like an advertisement based on claiming that their product is "The best in the world!", without any evidence supporting it, and just hoping that people will keep accepting and word-of-mouth passing on the on the "best in the world" claim without question.

We still don't know what ME is, so we can't prove that anyone has it. At present, ME is defined by personal judgments of constellations of symptoms. My judgement of which of my symptoms qualifies for the categories will be different from someone else's. In addition, I'm not sure that some of my qualifiers are actually due to ME. People have sleep problems without having ME, so why can't PWME have those same problems for the same non-ME reasons, which would technically disqualify them from ICC?

I agree with the purpose of the ICC: to ensure that studies on ME are done on subjects who actually have it. I'm not okay with ignoring its limitation by boldly claiming that it's "valid and reliable". I don't have any suggestions on how to improve it, but I think the research community should at least admit that it's not 100% reliable.

FWIW, my qualifier for "Energy production/transportation impairments: At least one symptom" is "recurrent feelings of feverishness with or without low grade fever". I reasonably often feel slightly feverish, despite my oral temperature at those times usually being below normal for me. In the first few years, my temperature would reliably rise a few tenths along with my other symptoms, so that could be described as "Loss of thermostatic stability". Personally, I think that's not what the committee on ICC had in mind, and I'd be a terrible choice for a study on "normal ME". I expect that a fair percentage of study subjects will be stretching their judgments of symptoms similarly (and perhaps subconsciously), to qualify for ICC, possibly including symptoms of non_ME causes that kinda-sorta fit the criteria.

If my "feverishness" doesn't qualify me, I did have maybe half a dozen incidents when I felt like I was going to black out (actually did black out once). I haven't experienced that symptom for 10+ years. The ICC doesn't say anything about whether the qualifiers have to be present every day, or whether having them at one time (maybe decades before) still qualifies as valid. So, this is another flaw in comparing study cohorts via ICC: one research group insists on subjects meeting the criteria consistently on a daily basis, while another one accepts subjects who had qualifying symptoms individually at various times over a few decades, never fully qualifying on any given day. Then there's the problem of different sets of qualifiers: if one cohort qualifies based on "sensitivity to light", and another cohort qualifies on "muscle weakness", what are the studies comparing?

I think that the ICC is better described as "least horribly flawed" than "valid and reliable".

 

This issue raises another question: How do you verify the validity of diagnostic criteria when there is no gold standard? Reliability means that something is robust with respect to random errors, and validity means that it is robust with respect to systematic errors.

The reliability of criteria can be verified e.g. by repeating the screening of patients with an interval long enough for people to forget what they said during the last evaluation but short enough so there are no major changes in the illness trajectory. If the diagnoses of the participants remain the same, the criteria are reliable.

But how do you test validity? Another way of phrasing the meaning of validity is that something should show what you really intended it to show. But we don’t know what ME really is or what the underlying pathology is. We have ideas and clues, but no reliable answers. When testing the validity of Alzheimer’s disease, there is a gold standard. You can perform an autopsy of patients after they have passed away. Pathological changes in the brain may verify or disprove the diagnosis, and we can then check whether the criteria used at an early stage were valid.

The only thing I can come up with is that you should check patients with diagnosed illnesses that may easily be confused with #MECFS and see if the criteria exclude these patients.

@Jonathan Edwards, do you have any thoughts on this matter?

 

Not really any greater thoughts than you have admirable expressed @Obermann.

The additional problem with ICC is that it brings in all sorts of hypothetical aspects of ME/CFS and asks for symptoms to be evidence of those rather than just looking for stable clusters of symptoms. To my mind it is very much the least good of the options.

 

I think the ICC is unnecessarily strict. The most important symptom of ME is, by far, PEM. Requiring as many other symptoms as the ICC needlessly weeds people out, probably biasing samples towards more severe or complex presentations.

 

I did have one idea on how to improve consistency of studies based on the ICC: create cohorts based on each specific option, such as "muscle weakness" or "light sensitivity" and test them for as many factors as possible, most importantly the things that are presently being studies the most, such as viral RNA, mitochondrial function markers, blood vessel properties, etc. I expect that some of the factors will correlate with the options, and others won't. Then if someone wants to study blood vessel dysfunction in ME, they could narrow their selection to the appropriate subset of options. Limiting cohorts according to severity of symptoms would help too.

Ideally, this would be done for all combinations of the options (and severity), but probably impractical. I still think that having data from sets of cohorts (ME and long-covid and appropriate controls) being available for any researchers would be useful for consistency of research and save money. I assume there would be resistance, because they can't claim credit for the selection and sampling, the way it has been done in the past.

 

Complex presentations are more likely when comorbidities are present.

Using the ICC in research could be similar to only recruiting people with ME that also have other health problems. It would unnecessarily adds noise and complexity to the data.

The stated reason the ICC is being promoted by some patients is that strict criteria are necessary for progress. But an unstated reason seems to be gaining an advantage in health insurance disputes. It's a part of a competition for resources.