The stats gap, 2023, Ellen Pasternack

The Works in Progress Newsletter

The stats gap

Ellen Pasternack

Extract:
What you’ve been taught is something anthropologist Richard McElreath calls a ‘golem’. Golems, most famously the Golem of Prague, are powerful clay giants (or so the legend goes) created to defend local Jewish populations from persecution, but which, having no intelligence of their own, will cause disaster if not carefully directed.

Statistical algorithms – whether a simple t-test or something more complex – are like golems. Whether calculated by hand or by computer, you put the data in, and it gives some numbers as output.

Sometimes the output is ‘statistically significant’, which might be shown in statistical software by a little asterisk. But seeing a little asterisk is not a substitute for actually understanding what is going on.

What ‘statistically significant’ means in the context of an algorithm like this is: given the data you’ve just fed me, I have performed some calculations, and the output of those calculations is a number which is lower than 0.05. The algorithm (aka the golem) can’t make real-world inferences for you, and it can’t tell you whether it was the correct algorithm to use in this instance. If the data is of the wrong sort, it will still blindly attempt to carry out its instructions. If the answer it gets is wacky for some reason, it won’t necessarily notice or care.

Armed with these golems, you’re let loose on real research as a graduate student. Let’s say you’re trying to apply a golem to your experimental data, and your statistical software throws back an error: the command you typed hasn’t worked.

You don’t understand why it hasn’t worked, or for that matter what the jargon-filled error message even means, so you Google it, and see that someone has posted on StackExchange a few years ago describing what sounds like a similar problem. ‘You should be applying a Tischbein-Fischbein correction’, suggests one of the replies. ‘Actually, this analysis is probably worthless without it.’

Full article: https://worksinprogress.substack.com/p/the-stats-gap

 

When I read a paper and it claims that whatever their theory is, the statistical evidence is somewhere around the 'accepted' level (.05 or whatever), I assume that they massaged the data and/or reworked the experiment to make it reach that level ... and then ignore their conclusions/claims.