Analyzing ordinal data with metric models: What could possibly go wrong?, Liddell et al. 2018

Abstract
We surveyed all articles in the Journal of Personality and Social Psychology (JPSP), Psychological Science (PS), and the Journal of Experimental Psychology: General (JEP:G) that mentioned the term “Likert,” and found that 100% of the articles that analyzed ordinal data did so using a metric model. We present novel evidence that analyzing ordinal data as if they were metric can systematically lead to errors. We demonstrate false alarms (i.e., detecting an effect where none exists, Type I errors) and failures to detect effects (i.e., loss of power, Type II errors). We demonstrate systematic inversions of effects, for which treating ordinal data as metric indicates the opposite ordering of means than the true ordering of means. We show the same problems — false alarms, misses, and inversions — for interactions in factorial designs and for trend analyses in regression. We demonstrate that averaging across multiple ordinal measurements does not solve or even ameliorate these problems. A central contribution is a graphical explanation of how and when the misrepresentations occur. Moreover, we point out that there is no sure-fire way to detect these problems by treating the ordinal values as metric, and instead we advocate use of ordered-probit models (or similar) because they will better describe the data. Finally, although frequentist approaches to some ordered-probit models are available, we use Bayesian methods because of their flexibility in specifying models and their richness and accuracy in providing parameter estimates. An R script is provided for running an analysis that compares ordered-probit and metric models.

Full article: https://www.sciencedirect.com/science/article/abs/pii/S0022103117307746

 

A bit of background: This is mostly about statistics but it may be relevant to many ME/CFS studies.

In many psychological studies, including many studies on ME/CFS researchers use a Likert-type questionnaire.

A Likert item typically refers to a question for which the response is indicated on a discrete ordered scale ranging from one qualitative endpoint to another qualitative endpoint. For example, patients are given a treatment and then have to indicate how their health changed with 5 possible options: "much deteriorated", "deteriorated", "stayed the same", "improved", "much improved".

These data are usually analyzed by giving each outcome a number from 1 to 5 and then it is treated as if this is a measurement like a count or measurement of height. The problem is that this assumes that the distance between each of the responses is exactly the same while this isn't necessarily the case. There is clearly an order in that much deteriorated" is worse than "deteriorated" etc. but the distances between the responses can vary.

This paper argues that ignoring this problem can lead to misleading results (in both directions: falsely suggesting there is a difference when there is none or falsely suggesting there is none where there is one). There is a solution to this, namely modeling the responses as a cumulative distribution. Unfortunately, this is rarely done. The authors looked at 68 articles in psychological journals and found that, "every one treated the ordinal data as metric and used a metric model."

 

@Lucibee as previously highlighted this issue in regard to the Chalder Fatigue Scale but I think it applies to many other scales of this format as well.

 

This does not surprise me in the slightest. I've thought since first encountering this notion, that it is dangerously naive to assume it valid to simplistically map arbitrarily scaled ordinal data onto an analogue scale. Especially when there is no telling what the real intervals are, let alone assuming those intervals to be the same for everyone. As the paper's title says, "What could possibly go wrong?".

 

It would be interesting if we could reanalyze the data with an appropriate ordinal regression model and see if the results differ.

With the Chalder Fatigue Scale there is the additional difficulty that it combines multiple of these Likert scales. I think that to model these in an appropriate way we would need the responses of each participant to each individual question. And the Chalder Fatigue Scale data that is currently available is to my knowledge, just the total score summarized.

 

It is good to see a detailed analysis of this.
It is good example of the old adage 'lies, damned lies and statistics'. Statistics so easily creates make believe data.

 

Perhaps it should be called the Simon fatigue scale since Trudie Chalder has disowned it and said it was really Simon's.

 

I did not know this. Why did she feel the need to do this?

 

I cannot remember where I read this but apparently she was suggesting that this was really Simon's idea and that she just made up a bit of a scale for him or something. So maybe disowned is a bit strong. Maybe she was just too modest to acknowledge her mastery of method

 

I think I have read somewhere, or maybe it was in an interview of his, how describes coming up with it in an afternoon, or something very akin to that. It seemed a very ad hoc and - dare I say it - subjective process.

 

An after lunch fatigue scale then?

 

This is the only reference I can find to that, but thought there was a bit more somewhere:

https://meassociation.org.uk/2012/0...-fatigue-syndrome-journal-article-march-2012/

SW definitely claims ownership of it though.

 

I thought it was a back ofa beer mat exercise with Powell for the 1989 paper Fatigue syndromes http://jnnp.bmj.com/content/jnnp/52/8/940.full.pdf

It clearly underwent a little development after that.

 

I can only access the abstract at that link, but can get to full article here:

https://sci-hub.wf/10.1016/j.jesp.2018.08.009

 

Yes that as it.

And I think the reason it has been used so much is because its very short, it’s easy to administer, it includes items related to physical and mental fatigue, and its got eleven items

Really rigorous stuff then.

 

Exactly. All the right ingredients a scientific measure could ever possibly want.

 

Was that why Brian Hughes said something about scoring 11 for nostalgia?

Seems it was.

I should go back to sleep.