Developing a blood cell-based diagnostic test for ME/CFS using peripheral blood mononuclear cells, 2023, Xu, Morten et al

[Jonathan Edwards]

Why didn't Morten look for a single measure to differentiate PWME from healthy people?

I presume that no single measure comes anywhere near that. So this technique is not going to be a step towards that - there isn't one.

But my real point is that a test that differentiates is not actually what we are looking for. You can differentiate PWME from healthy people by asking them about their symptoms. The point of doing tests is to establish some objective evidence of a process that gives us enough insight into how the illness might come about to convince us it tells us something useful.

Specificities and sensitive are not important. If a metabolic screen showed that 30% of PWME had a level of A above the 95th centile and a level of B below the 50th centile (such that this combination occurred in no more than a few per cent of healthy people) where A is normally regulated by or along with B in some known way in which they normally tended to go up and down together then we would be motoring. We would have good evidence that at least 30% of PWME seemed to have something wrong with a pathway involving A and B.

In simpler terms I think it is probably fair to say that one is unlikely to get a useful result from machine learning for a test of diagnostic level for individual patients unless at least one measure shows a robust statistical difference across a population. For instance, at minimum we would expect 70% of levels of A for PWME to be above the 50th centile. It is just about conceivable that machine learning would pick up some subtle multiple imbalance without this but in general terms diseases are not as obscure as that.

 

[DMissa]

But my real point is that a test that differentiates is not actually what we are looking for.

Clinical criteria change with the years, with borders, and with egos (to say nothing of the challenges now apparently introduced by pandemics). Having something reproducible and objective as a bulwark against these aspects, and as a clear signpost of legitimacy as a silver bullet to skeptics is incredibly important imo.

Specificities and sensitive are not important.

They may be, especially if this corresponds to differing clinical presentations within the particular ME/CFS population (eg: symptoms or onset patterns). What if we had two tests that produced outcomes residing within useful percentiles of each population as you suggest with the one metabolite example, but one of them has a much greater overlap with people without ME/CFS while possessing a greater fidelity for distinguishing people with ME/CFS of X or Y clinical pattern? A combined or staged protocol using both measurements with the specific utilities of each in mind has obvious value. One is better for filtering out people without ME/CFS and one is better for confirming a precise ME/CFS diagnosis. It may be more economical or more practical to proceed with only one test and then the other when more detail is needed. These statistics inform that process.

in general terms diseases are not as obscure as that

Who is to say that we are dealing with a disease that is best described in general terms?


I understand the general (not specifically this paper) criticisms of machine learning or taking too many concurrent and individually meaningless measurements and applying them to diagnostic applications. Especially given the randomly generated dataset shown earlier in the thread (useful and impressive demonstration by @chillier btw, love it). I actually agree with many of these critical ideas in a broad sense. This applies to my own past work; I am not coming from a place of defensiveness whatsoever. I think the problem is in the language. Blanket language is dangerous. Generalised criticisms, delivered from a position of expert authority, are dangerous. These threads are very public. There are varying degrees to the issues raised here (and earlier in the thread) and they interact with different studies variably. Yes, this stuff can be overused or sometimes spurious. But equally not all studies using machine learning will be producing throwaway fluff from thousands of otherwise meaningless measurements. Not every inclusion of sens/spec is a cheap way to put a "score" on a throwaway biomarker paper for clout. I think that the criticism of these approaches needs to be very carefully communicated so as not to herd less knowledgeable community members into a habit of rejecting things upon recognition of superficial red flags in an abstract. The fine details of each case at hand must always, always, always be of chief importance.

 

[Jonathan Edwards]

I think that the criticism of these approaches needs to be very carefully communicated so as not to herd less knowledgeable community members into a habit of rejecting things upon recognition of superficial red flags in an abstract. The fine details of each case at hand must always, always, always be of chief importance.

I can see how you feel but I have been carefully communicating with detailed debate on these issues on this forum for six years now. I am not herding anybody into anything, just maybe pointing out that the reality is that most publications on disease mechanisms and diagnostic tests these days are crap - 95%. Back in the 1970s that was not the case but now almost everything is 'me too'. It is not difficult to produce a good quality abstract if you have something valid to say. If you read a journal volume from 1969 you will find most of the articles are saying something that is still considered valid fifty years later and you can see that in the abstract.

The point I was making was a basic theoretical one that scientific progress in these matters is not about labelling or 'legitimacy'. It is about having an understanding of what is going on. Patients are entitled to want something that gives legitimacy but in reality true legitimacy only comes from something that increases our understanding. The rheumatoid factor test is neither particularly specific nor sensitive but it led us to an understanding of rheumatoid disease that allowed us to control it. That was because it gave us an understanding that we could work on and finally see what it was we needed to do.

If I am criticising from a position of expertise I'm afraid I cannot help that. I spent fifty years working in this area and came to see all the pitfalls. Devising effective treatments for RA had nothing to do with having a 'diagnostic test' - there wasn't one - it had everything to do with tests that told us what was going on.

 

[Hoopoe]

If we had an easily accessible diagnostic test for PEM, we could tell schools to use it on children that are increasingly absent from school. That would be enormously helpful for diagnosing PEM early before too much damage is done by the school system that is generally designed to put pressure on children to attend as much as they can (and with PEM, they'll often feel like they CAN, even if they will relapse afterwards).

That would be useful even if it didn't tell us what going wrong in the body.

I suppose one could criticize me for proposing a technological solution (diagnostic test) for a social problem (not believing patients, etc.). That's valid, but in practice it's still useful to have a test to avoid ambiguities.

A lot depends on how accessible test is. In an ideal fantasy it would cost a few cents, require a spit sample, and not require any training. If it was expensive and less practical than say looking up an information page on PEM and/or going to the GP for an opinion, then it would not be as useful.

 

[duncan]

I can think of a half dozen or more diseases - off the top of my head - where I could not care less about learning more about the disease process, politically. All I would want is a reliable yes/no diagnostic, at least in the near term. It might not move science forward, but it immediately could help validate patients, and help keep the wolves at bay while science and understanding play catch up.

ME/CFS is one of those.

 

[FMMM1]

I can think of a half dozen or more diseases - off the top of my head - where I could not care less about learning more about the disease process, politically. All I would want is a reliable yes/no diagnostic, at least in the near term. It might not move science forward, but it immediately could help validate patients, and help keep the wolves at bay while science and understanding play catch up.

ME/CFS is one of those.

Yes validity is a problem e.g. the UK Government keeps funding crap (psychological) research assessed subjectively (questionnaires) - I'm thinking of the latest one by Crawley and others. That "feeds" the validity problem.

If Government/public attitudes changed (validity) then there would hopefully be more support. However, "validity" and the identification of a biomarker, i.e. to identify the disease mechanism (thereby what needs to be treated), are not mutually exclusive goals - we need both. If there were effective treatments (following the identification of a biomarker) then I for one wouldn't give a shit about the views of those who disputed the existence of ME/CFS.

GWAS is a potential way to identify mechanism - biomarker - and thereby treatments.

 

[duncan]

If Government/public attitudes changed (validity) then there would hopefully be more support. However, "validity" and the identification of a biomarker, i.e. to identify the disease mechanism (thereby what needs to be treated), are not mutually exclusive goals - we need both

We do, yes. But I doubt things will be so fortunate as that. Nice to consider, though.

If there were effective treatments (following the identification of a biomarker) then I for one wouldn't give a shit about the views of those who disputed the existence of ME/CFS.

I would, in particular family, family first and foremost, but yes, all those rat schmucks who mischaracterized and lied about us at our lives' expense. Give me an unequivocal yes/no diagnostic. I've already come to an uneasy acceptance I might not see a cure, but there's a different mending and accounting that only a vetted diagnostic can offer.

 

[Jonathan Edwards]

If we had an easily accessible diagnostic test for PEM, we could tell schools to use it on children that are increasingly absent from school. That would be enormously helpful for diagnosing PEM early

But you see, this points to the reason why we do not want a test with high specificity and sensitivity. We want a test that shows something even before the diagnosis can be made clinically. We want a test that is positive when the clinical diagnosis is not made yet.

The real value of a test is in indicating what we can expect to happen later. That need not involve an established mechanism but it has to be something that we can credibly expect to link to a mechanism - that determines what happens next.

 

[duncan]

The real value of a test is in indicating what we can expect to happen later. That need not involve an established mechanism but it has to be something that we can credibly expect to link to a mechanism - that determines what happens next.

I wouldn't even presume to ask for so much. Just a simple gold standard positive or negative for something definitive. That's my porridge. Something that cannot be disputed or dismissed or even minimized.

 

[Hubris]

In the UK and US there is a good number of doctors who, I suppose, are at least somewhat decent at diagnosing ME/CFS, after we exclude the BPS fanatics. But most countries aren't like that.

Where I live, 99+% of doctors are completely incapable of diagnosing this illness. I have confidently been told I cannot have CFS because "CFS is when you have only fatigue and no other symptoms, you have others so it can't be that". Furthermore, most of the time, you cannot get disability without having another illness like Chron's disease, because there is this widespread belief that if you don't have an identifiable cause for your CFS (like an inflammatory illness), then you don't actually have CFS. CFS is basically the symptom "fatigue", and most doctors will not diagnose it because they think it's redundant (of course, they are unaware of what CFS actually is, so even if they do diagnose it, it's only harming our cause).

Having a diagnostic test would be tremendously useful as it would force these people to start learning about the illness instead of living in complete ignorance, and the patients would actually get diagnosed properly. It would allow us to recruit them and start running proper studies. And it would help against gaslighting of the more severe cases. I was forced to push myself at school and my life was ruined from it. I couldn't care less that the test doesn't have predictive powers.

 

[Jonathan Edwards]

I wouldn't even presume to ask for so much. Just a simple gold standard positive or negative for something definitive.

So like a definitive test for Thursdays?
That proves they really are Thursdays?
Reality doesn't work like that!

 

[Jonathan Edwards]

I couldn't care less that the test doesn't have predictive powers.

So it doesn't need to predict that very likely you will still be ill tomorrow?
Because if not nobody is going to fork out disability payments for tomorrow.
That is what I mean by predict. That is the way the whole of medicine works even if it is presented terms of 'diagnoses'.

I remember listening to a rather unimpressive colleague teaching students on their first ward round about why we take a history with all those peculiar questions about dizzy spells and how many times you have your bowels open. I thought he was going to say 'to make a diagnosis'. But my estimation of him went up. He said 'to work out what is wrong and what is likely to happen next'.

 

[Hubris]

So it doesn't need to predict that very likely you will still be ill tomorrow?

It probably would, given that only a few ME patients recover? So if you were positive for the test, chances are it would keep being positive and doctors would assume that also as they start seeing more and more patients. If the clinical diagnosis is predictive, I don't see why the biomarker wouldn't be.

By predictive power I meant that I don't care that the test isn't positive before you can make the diagnosis clinically - which is what you described. Being able to know you will get CFS soon would be nice but it's not a requirement. Most of the damage is done after the diagnosis. I have never heard anyone complain that they were damaged by exercising before the onset of symptoms, or that doctors should diagnose before symptoms appear. They don't even diagnose you if you are bedbound and ill for 20 years. Let's not get ahead of ourselves. Incremental progress.

 

[duncan]

So like a definitive test for Thursdays?
That proves they really are Thursdays?
Reality doesn't work like that!

Reality? The reality is our's is a contested disease, and we need a diagnostic that cannot be contested. Moreover, we're not alone in that regard.

So it doesn't need to predict that very likely you will still be ill tomorrow?

That is not required of many diagnostics. It seems to me that really should fall under disease characteristics.

 

[Trish]

Perhaps it's a bit like if someone is fatigued because of anaemia, then there is a simple blood test to show that their fatigue is caused by anaemia.

As a result of having a simple blood test 3 things can happen:

1 the doctor prescribes iron supplements
2 if they don't work, the doctor investigates further to see what's causing the anaemia and hence the fatigue not to go away as expected
3. The doctor believes there is something physical causing the fatigue, and doesn't dismiss the patient as malingering or send them to a psychologist to overcome their 'perceived' fatigue.

I think the reason some people with ME say they want a test that shows their symptoms are 'real' even if it doesn't lead to better understanding of the cause or to treatment is in order to stop our doctor disbelieving us and sending us to psychiatrists to be disabused of our false beliefs. In other words, it's for number 3 on my list.

Of course ideally we want numbers 1 and 2 as well, but number 3 would make a huge difference even on its own.

 

[Robert 1973]

Jo, I think I sort of understand your point but I’m not sure I agree.

First, I’m sure I was not alone in struggling to accept the accuracy of my ME/CFS diagnosis. For years I desperately sought an alternative diagnosis to explain my symptoms, and I would still be delighted if I was given an different (non-BPS) diagnosis – even if there was no treatment and the prognosis was bad. If there was a diagnostic test with high specificity and sensitivity I would have more confidence that my diagnosis was correct and some other explanation for my symptoms has not been missed.

Second, we don’t know if ME/CFS is one illness with different presentations or different illnesses with similar presentations. If diagnostic tests were able to differentiate between sub-groups that would be useful (particularly if they were subsequently found to have different outcomes).

Third, I can’t think of why a reliable diagnostic test with high specificity and sensitivity would not be useful for research – and therefore lead to more accurate prognoses (ie predictability) based on that test. If the test also accurately reflected the severity of the illness that would surely be useful for research too – potentially much easier and more reliable than using self-reports.

Fourth, If the test had high specificity and sensitivity, I can’t see how it could not also tell us something useful about the mechanism of the illness.

Fifth, it would help to reduced prejudice and stigma associated with the diagnosis. That may not be rational or scientific but nor are most people’s attitudes towards people with ME.

I’m sure I could think of more points, and perhaps modify the ones I’ve made, but this is just what’s come to mind as I write.

 

[FMMM1]

Jo, I think I sort of understand your point but I’m not sure I agree.

First, I’m sure I was not alone in struggling to accept the accuracy of my ME/CFS diagnosis. For years I desperately sought an alternative diagnosis to explain my symptoms, and I would still be delighted if I was given an different (non-BPS) diagnosis – even if there was no treatment and the prognosis was bad. If there was a diagnostic test with high specificity and sensitivity I would have more confidence that my diagnosis was correct and some other explanation for my symptoms has not been missed.

Second, we don’t know if ME/CFS is one illness with different presentations or different illnesses with similar presentations. If diagnostic tests were able to differentiate between sub-groups that would be useful (particularly if they were subsequently found to have different outcomes).

Third, I can’t think of why a reliable diagnostic test with high specificity and sensitivity would not be useful for research – and therefore lead to more accurate prognoses (ie predictability) based on that test. If the test also accurately reflected the severity of the illness that would surely be useful for research too – potentially much easier and more reliable than using self-reports.

Fourth, If the test had high specificity and sensitivity, I can’t see how it could not also tell us something useful about the mechanism of the illness.

Fifth, it would help to reduced prejudice and stigma associated with the diagnosis. That may not be rational or scientific but nor are most people’s attitudes towards people with ME.

I’m sure I could think of more points, and perhaps modify the ones I’ve made, but this is just what’s come to mind as I write.

I think a biomarker would be a huge step forward e.g. if GWAS identified a gene and from that a disrupted pathway was identified ("X" is elevated --- "Y" lowered--). At that stage you'd start to have a clue what's wrong and what your aim is --- e.g. get "X", "Y" --- into normal range.
The biomarker doesn't have to be highly specific e.g. if it's 3X lower/higher in some people with ME --- but also elevated in diseases with a different (known) pathology.
So yea, really progress seems to be based on finding that biomarker.
I'm in the UK & I support Duncan's etc. views re accessing benefits etc. However, to get back to a normal life we need that biomarker/understanding of pathology.

 

[Jonathan Edwards]

If the clinical diagnosis is predictive, I don't see why the biomarker wouldn't be.

But this is the whole Catch22 of the argument. If you are sure of the clinical diagnosis you do not need the test. Tests are only of value when they tell you something about what is likely to happen that you cannot reliably judge clinically. That means they should not correlate perfectly with what you judge clinically because if they did that would be proof that your clinical judgement was all you needed.

 

[Jonathan Edwards]

The reality is our's is a contested disease, and we need a diagnostic that cannot be contested.

Exactly, and it will only be incontestable if it more reliably predicts what is going to happen than clinical judgment. Moreover, to be taken seriously by the professionals who make such decisions it willed to make some sort of sense in terms of being related to a mechanism. Metabolic studies aim to do that but the practical question here is whether metabolic findings relate to the disease itself or epiphenomena. Fo professionals to have any confidence in tests they either need to see some explanatory power or some reliably documented predictive power or preferably both.
And if you get those you have an incontestable test and it doesn't matter if it is rather nonspecific and insensitive. Rheumatoid is a clear example of that.

 

[Jonathan Edwards]

I think the reason some people with ME say they want a test that shows their symptoms are 'real' even if it doesn't lead to better understanding of the cause or to treatment is in order to stop our doctor disbelieving us and sending us to psychiatrists to be disabused of our false beliefs. In other words, it's for number 3 on my list.

Indeed, but doctors are only going to believe the test indicates something real if there is reliably demonstrated predictive power over and above clinical judgment or it makes some mechanistic sense. Getting better predictive power than clinical judgment may be quite easy if clinical judgment is poor but then the test isn't going to correlate well with that judgment because it will often be wrong.

There are further layers to this -many. But I trying to persuade PWME that they do not in fact want a '100% test for ME' whatever and that researchers should not fall into that trap either. In RA there has never been such a test and never will be but we have gone from knowing almost nothing to being able to control most disease - using tests intelligently as clues to much more complex situation.