By Amy Ross, Senior Editor, PLOS Medicine
Replicating the work of others is not as exciting as doing the latest breakthrough research and it’s unlikely to win you a Nobel prize, however awareness of the “reproducibility issue” is increasing and it has become a more important consideration for grant funding and publication of research.
Sunday morning at SfN 2014, I attended a session titled “Enhancing Reproducibility of Neuroscience Studies”, led by Story Landis, recently retired Director of the National Institute of Neurological Disorders and Stroke (NINDS) and Thomas Insel, Director of the National Institute of Mental Health (NIMH). The main goals of the session were to raise awareness about issues with reproducibility in neuroscience research and to outline some of the steps that the NIH, journals, and researchers can take to improve reproducibility.
Despite the lack of exciting breakthrough research being presented, the session was well attended. No doubt helped by the first presentation coming from the Director of the NIH, Francis Collins. In his talk, Francis Collins discussed the Proposed Principles and Guidelines for Reporting Preclinical Research published on the NIH website. He distinguished between hypothesis generating research and the more rigorous hypothesis testing research that needs to be done before preclinical research can be translated into clinical research and highlighted his own Director’s Blog on p-hacking, the practice of trying several statistical analysis methods to identify the p-value that looks best.
The second speaker was Véronique Kiermer from Nature publishing, who talked about the role of journals in enhancing reproducibility, with a focus on improving transparency of research reporting. She discussed identified four key ways that Nature has changed their policies to improve reporting, including the use of checklists for reporting standards, eliminating the length restrictions on methods, increasing the scrutiny of statistics, and re-emphasizing data sharing.
The last two speakers, Huda Zoghbi, from the Baylor College of Medicine, and John Morrison, from Mount Sinai Hospital, provided the important perspective from scientists actually doing the research. In addition to having good research practices and being ok with failed research, Huda Zoghbi highlighted the importance of understanding the disease being studied (at least by collaborating with a physician with an intimate knowledge of the disease) and understanding the value and limitations of the animal models.
John Morrison provided some practical information on resources for best practices for preclinical research in neuroscience from a new paper published in Neuron, which could be used as required reading for not just graduate students and early career researchers, but also faculty members.
Overall, a stimulating discussion and many questions/comments from the floor, including what lies in store for data sharing, thoughts on making peer review open, and how to change the culture of research so that high quality and replicating research is rewarded in addition to novel findings.
All speakers stressed that the session was NOT about research misconduct and acknowledged that the culture of research needs to change in order for reproducibility to improve. The session was focused on how to move forward rather than just identified the problem and I was reminded of a recent paper we published in PLOS Medicine by John Ioannidis on “How to Make More Published Research True”, a follow-up to the widely read “Why Most Published Research Findings Are False” from the same author in 2005.