Adam Mastroianni - blogs on research methodology

[Michelle]

You may remember Adam Mastroianni from such threads as this (about why peer review is crap) or this (about how people believe things could be better). But he recently posted an appeal to join his secret science society (that's not really secret):

I hereby invite every curious human to do science and post it on the internet.

Ask questions, collect data, write stuff, and make it available to everyone. You should feel as free to do and share research as you would feel uploading a video to YouTube or a song to Spotify.

You don’t actually need my or anyone else’s permission to do this, but sometimes people need a little encouragement, so: come on in!

Actually, let me make that a little more urgent: Please come in, we need you.

See, scientific progress has slowed. We fund more research than ever and get way less bang for our buck. We spend 15,000 years of collective effort every year on a peer review system that doesn’t do its job. Fraudsters can publish dozens of papers before they get caught, if they get caught at all.

This is bad. Our world is full of problems, and science is the main way we solve them. We’ve got climate change, an obesity epidemic, and a lot of sad people. There are folks dying of poverty and preventable disease. Heck, we still mainly make electricity by burning dinosaur bones. This can’t be as good as it gets.

It's a great follow-up piece to the peer review piece mentioned above. Among his points is one that @Jonathan Edwards has made that good science does not necessarily require a lot of money.

In fact, if you're willing to use simple methods, you actually have an advantage over professional scientists. The pros wanna look cool to their colleagues (and win big grant money from the government), so they have to use the fanciest, most advanced techniques, even when simpler stuff would do them better. That's great for you, because it means the professionals will rarely investigate important questions if they don't require giant magnets or ten thousand computer cores or whatever. Cheap ideas are just lying around for you to scoop up. So scoop ‘em, darn it!

He's even set up a Discord server for people to join if you reach out to him explaining the science you're doing. Given that we have had people here off and on talk about setting up studies, this might be one such avenue to explore.

I also like that he despises the term "citizen scientist" as he finds it terribly condescending.

If you call what I’m describing here “citizen science,” I will karate chop you. I despise that phrase. All science is science, regardless of the author’s credentials. Slapping the label “citizen” on science done by people working outside of institutions is just a way of widening the moat around the ivory tower, of reinforcing the false idea that only people with PhDs and academic jobs get to do “real” science.

 

[Hutan]

Declining trust in Zeus is a technology

The latest blog from Adam Mastroianni:

There are some stats suggesting there is declining trust in educational institutions, then

Here’s an example. Lots of people are skeptical of the medical establishment and prefer not to interact with doctors. But very, very few of those skeptics would refuse to go the hospital if they, say, fell down the stairs and snapped their leg in two. That’s because emergency departments are so undeniably good at fixing busted legs that you'd be crazy to prefer poultices and witch-doctoring instead. This is what victory looks like: a case where the experts have vanquished the cranks so thoroughly that everyone can see it.

Those successes should be way more common. Anybody who is tethered to reality by believing in empirical evidence, experimentation, testing theories, all that science stuff—they have a gigantic advantage over the people who form their worldviews by looking at goat entrails or consulting the stars or whatever. It shouldn’t even be close. If patients can’t tell the difference between you and the goat entrail people, you better tug harder on your tether and pull yourself closer to reality, because you’re embarrassing yourself. It’s like spending years practicing your Super Smash Bros. technique and then losing to a 10-year-old who just mashes the buttons.

That’s why I’m not sympathetic to the most common worry about declining trust: if the public stops trusting mainstream experts and abandoning traditional institutions, they’ll start trusting sweaty podcasts charlatans and joining terrorist cells instead. I, too, would like to beat the charlatans and the terrorists, which is why I want to do better than, “Don’t trust those guys—they lack the proper accreditation!” If that’s all you got, people shouldn’t trust you. Instead of arguing from expertise, you should use your expertise to make better arguments.

It reminded me of listening to a doctor on the radio recently who was very concerned that people are spending a lot of money on (often expensive) medical tests that aren't arranged by a doctor, and then spend more money on treatments designed to fix the (often spurious, sometimes completely wrong) findings from the tests. A lot of that concern is valid. But part of the problem is that the people turning to those tests have been to the medical profession with real symptoms and didn't get any answers that genuinely helped (and were probably patronised and gaslighted as well). And I'm reminded of the so-called tests for conversion disorder - medically sanctioned approaches to diagnosis that are just as made-up as the worst of the expensive retail medical tests. The answer to the declining trust is not to shout louder 'Trust only us, we know what we are doing!'.

 

[Arnie Pye]

Interesting article, thank you for posting.

 

[Kitty]

The answer to the declining trust is not to shout louder 'Trust only us, we know what we are doing!'.

"...and no, we're not going to explain why we're doing it. You're not in our special club."

 

[bobbler]

The latest blog from Adam Mastroianni:

There are some stats suggesting there is declining trust in educational institutions, then



It reminded me of listening to a doctor on the radio recently who was very concerned that people are spending a lot of money on (often expensive) medical tests that aren't arranged by a doctor, and then spend more money on treatments designed to fix the (often spurious, sometimes completely wrong) findings from the tests. A lot of that concern is valid. But part of the problem is that the people turning to those tests have been to the medical profession with real symptoms and didn't get any answers that genuinely helped (and were probably patronised and gaslighted as well). And I'm reminded of the so-called tests for conversion disorder - medically sanctioned approaches to diagnosis that are just as made-up as the worst of the expensive retail medical tests. The answer to the declining trust is not to shout louder 'Trust only us, we know what we are doing!'.

Hmm

I'm not a big fan of the term 'better arguments'. Too ambiguous.

I think the real solution lies in them understanding that - like integrity is something that is easily lost because basically means '100% of the time sticking to truth' and that 'doing the talking' it is about stepping back and realising what their values and what they do actually are. Not what they think (or would like to think) they are.

In certain procedures the 'quality' is measured by the rate of accident or averse consequences or occurences of side-effects. In conditions deemed more important than ME it is proper outcomes ie 5yr follow-up that are measured and reported. Not through the lens of the people marking their own homework for their department. Yellow card data and statistics on which treatment works for whom/what type of issue, along with knowledge on side-effects and who certain things might tend to occur with can be incorporated to decisions on treatment or changes to it.

I think that is called 'transparency' and it corresponds to trust and integrity because someone submitted to it/is OK with that. Nothing to do with argument.

It would have been hard for a bad broken leg fixer to claim that something now at an angle visibly and definitely dodgy on an x-ray is 'perfect healing'. Could that be why the system works best there? Plus of course I'm unaware of it being a big earner area for woo, in the UK apparently it is the A&E type stuff that would never be taken on by privates' cherry-picking so I'm guessing it's expensive and harder to business-model.

The usual issue of people who think the starting point is them talking, instead of it being listening and self-effacement. Identifying where medicine itself is just a merely 'aping/feigning/performative' of what it claims to be, in the same way 'other traditions' might have copied as a sales technique.

If there are areas where they are worse or no better than whatever made-up pretend thing they want to call these 'others' then really it isn't about 'arguments' but 'acknowledgement'. Where observation happens properly and research design isn't 'framed' to get results that fit with gut-ideas of what a condition might be.

And if they believe that they/their approach can do better one day in these areas then they need to remember what those values and approaches actually require - rather than looking for something to offer that fills the gaps.

Are all of their departments 'science-based' or whatever 'good things' they believe are rolled under that term they call themselves of 'traditional'? Is it time that within this they identify the departments who are science-based vs belief-based and fill the gaps for certain conditions where no science-based is there - with people who also know and believe in proper science only, and do not rely on trying to argue their 'belief-based' can 'look like science if you do a few random correlations'?

Does this all actually need less 'argument' and more listening and proper assessment to check those saying these things aren't the ones with the point.

 

[Eleanor]

Unserious Science
The first section of this article by Adam Mastroianni made me think of quite a few studies that show up in the psychosomatic subforums...

https://www.experimental-history.com/p/surely-you-can-be-serious

I once saw someone give a talk about a tiny intervention that caused a gigantic effect, something like, “We gave high school seniors a hearty slap on the back and then they scored 500 points higher on the SAT.”1

Everyone in the audience was like, “Hmm, interesting, I wonder if there were any gender effects, etc.”

I wanted to get up and yell: “EITHER THIS IS THE MOST POTENT PSYCHOLOGICAL INTERVENTION EVER, OR THIS STUDY IS TOTAL BULLSHIT.”

If those results are real, we should start a nationwide backslapping campaign immediately. We should be backslapping astronauts and before their rocket launches and Olympians before their floor routines. We should be running followup studies to see just how many SAT points we can get—does a second slap get you another 500? Or just another 250? Can you slap someone raw and turn them into a genius?

Or—much more likely—the results are not real, and we should either be a) helping this person understand where they screwed up in their methods and data analysis, or b) kicking them out for fraud.

[...]


That day, in the lecture hall, none of us were serious. We were all doing a little pantomime on the theme of science. The speaker pretended to show us some results and the we pretended to think about them. As long as we all hit our marks and said our lines, the content didn’t matter—one week you’re doing “Guys and Dolls” and the next week you’re doing “Oklahoma!”, but at the end of the day, it’s all just playacting.

This was the kind of thing that eventually drove me out of academia. I love playing pretend and I love doing science, but not at the same time.

 

[Hutan]

Click on the tag at the top left of the thread - 'Mastroianni' - for more content by Adam Mastroianni e.g.
The Rise and Fall of Peer Review, 2022, Mastroianni
[Preprint] Things could be better, 2022, Mastroianni and Ludwin-Peery

 

[Hutan]

Here's the latest Adam Mastroianni blog - Thank you for being annoying

It isn't about research methodology, it's about what motivates people to do things (which includes being motivated enough to do research and to advocate for things to be better). But, this thread seemed the best place to post about the blog. I always find Adam's blogs thought-provoking.

The blog is less about people being annoying; it's mostly focused on people being annoyed and how that annoyance is what drives human progress.

I think annoyance, like cholesterol, has a good kind and a bad kind. The bad kind makes you want to flee: backed-up traffic, crying babies on planes, colleagues who say they can use Excel when really they mean they’ve heard of Excel. But the good kind of annoyance draws you in rather than driving you away. It’s that feeling you get when there’s something you can and must make right, the way some people feel when they see a picture frame that’s just a bit askew, except a lot more and all the time.

...

Good annoyance is, I think, the only thing that keeps people coming back for more, indefinitely. There is nothing that a human with a normally-functioning brain can do for eight hours a day, every day, for their whole career, that feels “fun” the whole time, or even a large fraction of the time. We’re just too good at adapting to things. And thank God, because if we never got bored, we never would have survived. Our ancestors would have spent their days staring doe-eyed and slack-jawed at, like, a really pretty leaf or something, and they would have gotten eaten by leopards. Fun fades, but irritation is infinite.

The right job for you, then, is the one that puts you in charge of the things that annoy you. And this is where we steer people wrong. We imply that the right occupation for them is the one that lets them float through their days in a kind of dreamy pleasantness, when in fact they should be alternating between vexation and gratification. Or we let them choose proximity over responsibility, prioritizing what they’re working in rather than what they’re working on.

I had a lot of artsy friends in college who did this after graduation—they wanted to play Hamlet, but they instead ended up drafting marketing emails for a summer repertory production of Guys and Dolls. It’s no surprise that they hated this, because being in the presence of your annoyances without being in control of them is a recipe for insanity. That’s like working at the Museum of Slightly Crooked Pictures, where all the frames are wonky but you’re not allowed to straighten them.

"being in the presence of your annoyances without being in control of them is a recipe for insanity"
If not insanity, then, assuming you care enough, at least despair. It's something that is causing me quite a lot of consternation at the moment and goes to the heart of a discussion that we have had in the Emotional Support forum lately. How do we cope when we have relevant knowledge and motivation but can't find a way to get to a role that actually has a good chance of correcting the things that annoy us? Sniping from the sidelines, being the people who are annoying, can frustratingly make us less likely to be seen as the people who can be trusted to make real change.

Some approaches when we feel that we are endlessly consigned to sniping ineffectively on the sidelines are to 1. ignore the problem and stop, or 2. to lower our expectations of personal effectiveness. Someone on another thread mentioned the excellent 'pebbles into a pond' metaphor - the approach of many people each contributing a pebble to the pond that eventually causes it to break its banks.

But, arghh!, life is short, and many of us are impatient for change, for us, for our friends, for our children. To hell with pebbles, I often want a bulldozer.

Another way is to think about how we can create more control for ourselves, so that we do get a chance to make a material difference. The blog is a good reminder - vexation is part of what is needed for gratification. But, we have had more than enough vexation. How can we get substantially more gratification than we currently are experiencing?

 

[Sean]

Fun fades, but irritation is infinite.​

IIRC, Claude Shannon, one of the greats of 20th century science, said he was not motivated by any grand noble principles or ideals, like the search for truth, or wanting to do good for humanity, etc.

No, it was simply being irritated by the flaws and deficiencies in a machine or system or idea, and wanting to fix it.