Saying true things is hard

I think the replication & open science movement is great. I come from an epistemological, first-principles sort of mindset, and so I find this very encouraging:

We should conclude that saying true things is hard. Here’s why.

In the context of the above, let’s consider that these experiments were:

  • Performed by scientists
  • First-hand
  • Over weeks or months
  • Peer reviewed
  • Accepted by professional journals

…and were still only 60% right. Moreso, the effect sizes were ½ of the original claims.

We, as readers, then might take this as evidence we should apply (say) 30% confidence (60 × ½) to new scientific claims.

That’s rough. Now! Consider that most reporting on science is:

  • Written by non-scientists
  • Second-hand
  • Over hours or days
  • (Hopefully) edited and fact-checked
  • Published by consumer-oriented outlets

Let’s set aside notions of obvious bias or incentives – that debate is speculative, subjective, and hard to agree on. Let’s stipulate that everyone in the ‘signal chain’ is well-intended, smart and professional.

Even under ideal circumstance, such popular ‘re-broadcast’ should be understood as ‘signal loss’ applied to our baseline of 30% confidence.

What to do? As readers, the healthy approach is to assume most of our conclusions, from most reading, should be held very weakly – because saying true things is hard.

Published by Matt Sherman on August 29, 2018