Wasserstein and Lazar on p-values
📚SOURCE
💎PERMANENT NOTES
🧵SYNTHESIS
Paraphrase, not quotation: 🔬 The ASA Statement on p-Values, the statement and its short commentary.
In 2016 the American Statistical Association did something it had not done before: it issued a formal statement on how one specific tool should be used. The tool was the p-value, and the reason was that a generation of researchers had turned a threshold of 0.05 into a machine for deciding what is true.
The statement is six principles. A p-value says how incompatible the data are with a stated model. It is not the probability that the hypothesis is true, or that chance produced the result. Decisions should not rest on whether it crosses a line. Honest inference needs everything that was tried to be reported. It does not measure the size or importance of an effect. And alone, it is weak evidence.
One thing it is, and five warnings about what it is not. Nothing in it is new. That is what makes it damning.
✨KEY IDEAS
A p-value measures how surprising the data would be if a particular model, usually no effect, were correct.
It is not the probability that the hypothesis is true.
A conclusion should not turn on whether p falls on one side of a threshold.
Reporting only the analyses that passed makes the reported p-values uninterpretable.
A tiny p-value can come with a trivial effect, and a large one with an important effect measured badly.
On its own, a p-value near 0.05 is only weak evidence against the null.
📝READING NOTES
Read twice
Once yesterday, once this morning with a pencil. It is short, careful and oddly tense: a committee document in which every sentence has clearly been fought over.
The six, as I would say them at work
- What it is. A measure of how badly the data fit the assumption that nothing is going on.
- Not the chance I am right. It is computed assuming the null. It cannot also be the probability of the null.
- Not a verdict. A line at 0.05 is a convention, and a business decision needs costs, context and prior knowledge as well.
- Not valid if I went fishing. Every comparison I ran, every end date I tried, belongs in the report.
- Not a size. Enough data makes anything significant.
- Not enough. Give the estimate, the interval, and the reasoning.
My own three-line version is at 🔤 p-value.
Where 0.05 came from
🏛️ Ronald Fisher suggested one in twenty in the 1920s as a convenient level at which a result deserved a second look. A rule of thumb for an experimenter deciding whether to repeat a trial. It was never meant as the gate between true and false, and the statement is largely an attempt to return it to that modest job.
What the statement does not do
The six principles are five prohibitions and a definition. The closing section lists alternatives in one paragraph (intervals, Bayesian methods, and others) and endorses none. I understand why a society of statisticians could not agree on a replacement. As a practitioner I am left holding a tool I have been told not to lean on, and a result I still have to report on Thursday.
My working answer, for now: report the estimate and its interval first, mention the p-value if asked, and say out loud how many things I looked at. That is principles 3 to 6 in one habit.
- It is also written for researchers. Business or policy decisions appear in principle 3 and nowhere else. The pressures on a product team (ship or not, this quarter) are different from the pressure to publish, though they bend the same way.
With last week's reading
📖 Ioannidis on false findings gives the mechanism: why a significant result can be probably false. This gives the grammar: what the number means in the first place. Between them I could finally finish 💎 A significant result is not the same as a true one, this afternoon.
Added 6 October: handed in 📌 Statistics II - Reading response - What a p-value is not. Writing 600 words on it five months later was harder than writing this note, which says something about how much I had kept.
Related notes
- 🔤 p-value
- 🏛️ Ronald Fisher
- 📌 Statistics II - Reading response - What a p-value is not
- 📖 Ioannidis on false findings
Reading record
- Times read: twice
🗣️IN SHORT
- In : The ASA Statement on p-Values: Context, Process, and Purpose (2016)
- Wasserstein and Lazar, for the association, argue that : the p-value is a narrow and widely misused measure, and that scientific and practical conclusions should not be reduced to whether it passes a threshold
- by stating that : it measures the incompatibility of data with a model and nothing more: not the truth of a hypothesis, not the size of an effect, and not, alone, good evidence