Saying true things is hard
I think the replication & open science movement is great. I come from an epistemological, first-principles sort of mindset, and so I find this very encouraging:
We replicated 21 social science experiments in Science or Nature. We succeeded with 13. Replication effect sizes were half of originals. All materials, data, code, & reports: https://t.co/Uq1R5SUHNo, preprint https://t.co/aDEctL7yUx, Nature Human Behavior https://t.co/5VSJ86avAC pic.twitter.com/dVrSK922Cb— Brian Nosek (@BrianNosek) August 27, 2018
We should conclude that saying true things is hard. Here’s why.
Let’s note that these experiments were:
- Performed by scientists
- Weeks or months of work
- Peer reviewed
- Accepted by professional journals
…and were still only 60% right. Moreso, the effect sizes were ½ of the original claims.
We might take this as evidence we should apply (say) 30% confidence (60 × ½) to new scientific claims.
That’s rough. Now! Consider that most reporting on science is:
- Written by generalists
- Hours or days of work
- Hopefully edited and fact-checked
- Published in popular outlets
Let’s set aside notions of bias or bad incentives – that debate is speculative. Let’s stipulate that everyone in the signal chain is well-intended, smart and professional.
Even under ideal circumstance, such popular ‘re-broadcast’ should be understood as ‘signal loss’ applied to our low baseline of 30% confidence.
What to do? As readers, the healthy approach is to assume most conclusions, from most reading, should be held very weakly – because saying true things is hard.