Ioannidis on false findings
📚SOURCE
💎PERMANENT NOTES
🧵SYNTHESIS
Paraphrase, not quotation: the whole paper, 🔬 Why Most Published Research Findings Are False.
Ioannidis asks a question the significance test does not: given that a study reported a positive result, how likely is it to be true? The answer depends on three things. The prior odds that the kind of relationship being tested is real. The power of the study to detect it. And the threshold for a false alarm.
If a field tests many long shots with small studies, true effects are rare and often missed, while false alarms arrive at a steady five percent of everything tested. The positives are then mostly false alarms. Add bias, which is flexible analysis and selective reporting, and several teams chasing the same result, and it gets worse.
He turns this into rules of thumb: trust a finding less when the studies are small, the effects are small, many relationships were tested, the methods were flexible, money or conviction is involved, or the field is fashionable.
The argument is a model and not a census.
✨KEY IDEAS
The chance that a significant finding is true depends on how plausible it was before the study, and the p-value does not contain that.
With prior odds of one in ten and good power, well under two thirds of positives are true; with poor power, well under a third.
Bias, meaning analysis choices that tilt toward a result, turns some true negatives into reported positives.
The more teams test the same question, the more likely that at least one finds something by chance, and that one publishes.
Small studies, small effects, many hypotheses, flexible designs, vested interests and hot topics each lower the odds that a finding is true.
📝READING NOTES
What the paper is
Eight pages, published in a medical journal in 2005, set for the hypothesis-testing lectures. Read yesterday evening and again this morning, on a day mostly spent with Dad's repair notebooks. The contrast was not lost on me: forty years of date, fault, part, what I would do differently, with no p-values, and every entry true.
The arithmetic, done by hand
Suppose I test 1,000 ideas, and 1 in 11 is truly good (odds of one to ten). That is about 91 real effects and 909 duds.
| Power 80% | Power 20% | |
|---|---|---|
| Real effects detected | 73 | 18 |
| Duds passing at 5% | 45 | 45 |
| Share of positives that are true | about 62% | about 29% |
Same threshold. Same honesty. The only thing that changed is how well the study could see. With underpowered tests, most of what passes is noise, and each pass looks exactly like a discovery.
This is what 🔤 p-value does not tell me. It answers how surprising is this data if nothing is going on. I want to know how likely is it that something is going on, and that needs the prior.
The six rules of thumb, translated
- Small studies. A test that ran for one week on one customer.
- Small effects. We are usually hunting two or three percent.
- Many relationships tested. Twelve metrics on the dashboard; one of them moved.
- Flexible design. We chose the end date after looking.
- Interests. The team that built it analyses it.
- Hot field. Whatever the all-hands was about.
How much of it I believe
Most is derived from assumed values of prior odds and bias, not measured. Other statisticians objected at the time that his assumptions were pessimistic, and later attempts to estimate the false-positive rate in medical journals came out much lower than half. So the mechanism is solid and the headline is an argument. I notice I would have torn into a product claim built this way. The paper is, a little, an instance of corollary six.
Also, it treats true and false as two bins. Most real effects are neither zero nor large. A finding can be true and too small to matter, which the model has no word for.
The companion
🎬 Is Most Published Research Wrong walks through the same sum in twelve minutes with pictures. Watched it after writing the table, and the table matched. I would send a colleague the video and keep the paper for myself.
What it produced
💎 A significant result is not the same as a true one, still in pieces today. I will write it properly after next week's reading.
Added 21 October: Bayes in tonight's lecture. This paper is Bayes' rule applied to a literature, and I did not have the name for it in May.
Questions it left me with
- What are the prior odds for a typical product idea at Halden? Cagan says about half fail outright. I suspect ours are worse and nobody has counted.
- What is the power of our usual test? I do not know, which means it is low.
- If most of our wins are noise, which shipped features would I bet on if I had to rerun them?
Reading record
- Read for: the course
🗣️IN SHORT
- In : Why Most Published Research Findings Are False (2005), the whole paper
- Ioannidis argues that : the probability that a published positive finding is true depends on prior odds, power and bias, and is below one half across much of research
- by stating that : when true relationships are rare and studies are small, false positives at the usual threshold outnumber true ones, and flexible analysis, competing teams and conflicts of interest lower the proportion further