🧪Why Most Published Research Findings Are False

Whether a published positive result is true depends on three things besides the data: how likely the hypothesis was before the study, how much power the study had to detect a real effect, and how much bias crept in between the measurements and the page. In fields where long shots are tested with small samples and flexible analysis, most results that pass the significance threshold will be false, simply because true effects were rare among the things tried.

The limits, as I read them.

• It is a model, not a measurement. The headline follows from numbers he chooses for prior odds and for bias. Choose kinder ones and most becomes some. Statisticians who later tried to estimate the false share from the published record in medicine got figures well under a half.
• The bias term does much of the work. One parameter stands for every kind of selective reporting, and its value is a judgement.
• It treats a finding as true or false. Real effects are large, small, or negligible. A true effect too small to matter is counted here as a success.
• It is about medicine. I am carrying it across to product experiments by analogy. The analogy is good, since we also test long shots on small samples and look early, but he did not make it and I should not cite him as if he had.
• The title has outrun the paper. People quote it to dismiss research they dislike. The paper is an argument for better studies and not for believing nothing.

Before I call a test a win. The checklist I made from it, used on the triage test:

1. How likely was this idea to work before we tested it, honestly.
2. Was the sample size fixed before we started, or did we stop when it looked good.
3. How many metrics did we look at, and are we reporting the one that moved.
4. Report the interval for the difference, not only whether it crossed the line.

Still open: has anyone tried to repeat it on a second customer.

• Course reading for the hypothesis-testing week of 00. Statistics for Product Decisions I Hub. Read on Saturday 3 May at Lantern Coffee, before slabs with Caleb and seven hours before an anniversary dinner.
• Six pages and a title built to start a fight. It has been starting one for twenty years.
• It is not a survey of studies that failed. It is arithmetic: given how research is done, what share of the claimed discoveries can be true.
• The thing I had backwards until this weekend: a 🔤 p-value under 0.05 does not mean a five percent chance the finding is wrong. How often a significant result is true depends on how many of the ideas being tested were true to begin with.
• Which is the base-rate lecture from February again, in a white coat.