If 94% of published studies say a drug works, is the question settled?

Every published result can be real, and the pile of them can still mislead. Studies that find something get printed; studies that find little often don't, or get written up as if they had.

Antidepressant trials submitted to the FDA, 1987–2004

Positive in journals
94%
of published trials appeared to show the drug worked
Positive in FDA records
51%
of all the same trials, judged by the FDA
Never published
23 of 74
trials, 22 of them negative or questionable

The same 74 trials, two very different pictures

Each square is one trial, in the same spot in both rows

Two grids of the same 74 antidepressant trials. In the FDA's records, 38 were positive and 36 negative or questionable. In published journals, 37 of the positive trials appeared, 11 negative trials were written up as positive, only 3 negative trials were published as negative, and 23 trials, 22 of them negative, were never published. So 48 of 51 published trials, 94%, looked positive. What the FDA saw: all 74 trials 38 positive (51%) · 36 negative or questionable What doctors could read in journals: 51 published papers 48 looked positive (94%) · 3 negative · 23 never published
Positive Negative or questionable Negative, but published as if positive Never published

What gets published isn't a fair sample of what was tested. Almost every trial that worked made it into a journal. Most of the ones that didn't either stayed in a drawer or were written up to sound like successes. A doctor reading every published paper would see a drug that almost always works, when the full record was closer to a coin flip.

The same filter works anywhere only the hits get shown: investing newsletters that advertise their winning picks, diets that only share success stories, a highlight reel of a season. Before trusting a stack of positive results, ask what didn't get shown.

Where do you mostly see the successes and almost never the failures?

What happened to each of the 74 trials
FDA's verdictPublished as positivePublished as negativeNever publishedTotal
Positive370138
Negative or questionable1132236
All trials4832374

The drugs still beat placebo overall. But in the published papers their effect looked about a third bigger (32%) than in the FDA's complete data. Today, U.S. law requires most clinical trials to be registered before they start and to report results, which makes missing trials easier to spot.

Related exhibits

Sources: E. H. Turner et al., "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy," New England Journal of Medicine, January 17, 2008 (74 FDA-registered trials of 12 antidepressants).