This Week’s Most Discussed PLOS Neuroscience Article: Publication Bias in Psychology

By Pierre Mégevand

An article published recently in PLOS ONE, Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size, by Kühberger, Fritz and Scherndl, generated healthy discussion on social media.

Since its publication on Sept 5, 2014, this paper has received 14,000 views, 1252 shares, 523 Tweets and 723 Facebook shares!

The Paper:  Kühberger and colleagues, from the University of Salzburg in Austria, discuss the relationship between effect size and sample size. These two quantities, they argue, should normally be unrelated. However, the authors found a clear correlation between the two, mostly driven by studies with sample sizes below 100. They also looked at the distribution of p values and found that barely-significant values were much more frequently reported than values that just failed to reach the conventional threshold for statistical significance. Both findings, the authors argue, reflect publication bias in psychology.

The Authors’ Post Publication Comment :

“There seems to be agreement that small sample studies are at the center of the problems around publication bias. If so, neuroscience (especially traditional imaging studies with their sample sizes of about 20) could be especially affected”, Kühberger said.

 Blog Coverage:

In an informative blog post, Daniël Lakens (from the Eindhoven Institute of Technology, The Netherlands) takes the discussion further, especially with respect to p curves (plots that illustrate the distribution of p values across studies). There is a risk of misinterpreting these p curves as a sign of scientific malconduct (“p-hacking”); what they reflect instead is precisely publication bias, Lakens indicates.