Saying true things is hard
I think the replication & open science movement is great. I come from an epistemological, first-principles sort of mindset, and so I find this very encouraging:
We replicated 21 social science experiments in Science or Nature. We succeeded with 13. Replication effect sizes were half of originals. All materials, data, code, & reports: https://t.co/Uq1R5SUHNo, preprint https://t.co/aDEctL7yUx, Nature Human Behavior https://t.co/5VSJ86avAC pic.twitter.com/dVrSK922Cb— Brian Nosek (@BrianNosek) August 27, 2018
We should conclude that saying true things is hard. Here’s why.
In the context of the above, let’s consider that these experiments were:
- Performed by scientists
- Over weeks or months
- Peer reviewed
- Accepted by professional journals
…and were still only 60% right. Moreso, the effect sizes were ½ of the original claims.
We, as readers, then might take this as evidence we should apply (say) 30% confidence (60 × ½) to new scientific claims.
That’s rough. Now! Consider that most reporting on science is:
- Written by non-scientists
- Over hours or days
- (Hopefully) edited and fact-checked
- Published by consumer-oriented outlets
Let’s set aside notions of obvious bias or incentives – that debate is speculative, subjective, and hard to agree on. Let’s stipulate that everyone in the ‘signal chain’ is well-intended, smart and professional.
Even under ideal circumstance, such popular ‘re-broadcast’ should be understood as ‘signal loss’ applied to our baseline of 30% confidence.
What to do? As readers, the healthy approach is to assume most of our conclusions, from most reading, should be held very weakly – because saying true things is hard.