🍫🎬 Is Most Published Research Wrong

The full title ends in a question mark, which my file system will not have. Thirteen minutes from a science channel in 2016. I watched it on Sunday 4 May straight after finishing the note on 🔬 Why Most Published Research Findings Are False, to check that I had understood a paper whose arithmetic I had followed with one finger.

I had, mostly. The video does the sum with dots:

• Imagine 1,000 hypotheses, of which 100 are really true.
• Tests with decent power find 80 of the 100.
• Of the 900 false ones, 5 percent pass anyway by chance: 45.
• So 125 results get announced and 45 of them are wrong, more than a third, before anybody has done anything dishonest.

Then he adds the dishonest part, or the human part: journals prefer surprising positives, so the nulls stay in drawers; and a researcher who tries enough outcomes, cut-offs and subgroups will find something under the line. His example is a deliberately silly study showing that chocolate helps people lose weight, run for real, analysed until it gave an answer and reported round the world.

For the course: this is 🎓 Lecture 8 - Hypothesis tests and what p means with pictures. Hale's version used biscuits. Same sum.

For work: we run small experiments all year and celebrate the ones that win. I have never once asked how many of the winners would win twice.

He ends fairly: for all of this, checking each other in public is still the best method anyone has. I agree, and I notice it is also the least comfortable part. Filed under 00. Statistics for Product Decisions I Hub.

Concepts It Touches: 🔤 p-value · 🔤 Statistical power · 🧩 Randomisation · 💎 A significant result is not the same as a true one · 📝 Issue 43 - Significant is not the same as true · 🧪 Report every metric with a range for a month