Saying true things is hard

I think the replication & open science movement is great. I come from an epistemological, first-principles sort of mindset, and so I find this very encouraging:

We should conclude that saying true things is hard. Here’s why.

Let’s note that these experiments were:

  • Performed by scientists
  • First-hand
  • Weeks or months of work
  • Peer reviewed
  • Accepted by professional journals

…and were still only 60% right. Moreso, the effect sizes were ½ of the original claims.

We might take this as evidence we should apply (say) 30% confidence (60 × ½) to new scientific claims.

That’s rough. Now! Consider that most reporting on science is:

  • Written by generalists
  • Second-hand
  • Hours or days of work
  • Hopefully edited and fact-checked
  • Published in popular outlets

Let’s set aside notions of bias or bad incentives – that debate is speculative. Let’s stipulate that everyone in the signal chain is well-intended, smart and professional.

Even under ideal circumstance, such popular ‘re-broadcast’ should be understood as ‘signal loss’ applied to our low baseline of 30% confidence.

What to do? As readers, the healthy approach is to assume most conclusions, from most reading, should be held very weakly – because saying true things is hard.

Published August 29, 2018