There was a very nice News Feature in Nature last week on the problems of reproducibility in scientific papers. This isn’t the topic of ‘reproducible documents’ on which I have posted before but rather the question of repeatability of scientific research. The feature isn’t freely available I’m afraid but I can do some summarising here.
Nature’s report Jim Giles (he of the ‘Wikipedia only a little bit less accurate than Britannica‘ story and recent winner of the Association of British Science Writers award for best feature in 2005) did a bit of investigation of a four year old issue of Nature. He looked at the research articles and saw how they had been cited and how they had been followed up.
“impact of the papers has also been within the normal range: a handful have been referenced hundreds of times each, but most have notched up only a few tens of citations.”
Jim went a bit deeper than citation though, going back to the authors of the papers, and others in the fields to ask whether the results hade been reproduced or independently replicated. Some had been repeated, some had been corroborated (you can’t repeat the description of a fossil!) but some hadn’t.
For example there was a paper reporting “the first time, to have seen H3+ ions in the disk of gas and dust surrounding a young star” (Brittain, S. D. & Rettig, T. W. Nature 418, 57-59; 2002), some of whose conclusions haven’t been supported by subsequent work and whose authors “promote our very tentative interpretation that the unidentified lines might be H3+“. Then there was a paper on the production of “fully functioning stem cells from adult human bone marrow” (Jiang, Y. et al. Nature 418, 41-49; 2002) which should have revolutionised stem cell research but instead s proving very difficult to employ outside the original lab.
The crucial point to my mind is that the average reader would know nothing of these subsequent developments from looking at this relatively recent issue of Nature, even the online version. Indeed the stem cell and H3+ articles were featured on the cover. You need to be working in the area or do the kind of extensive research that Jim performed to find out what has turned out to be gold and what merely glistered.
Jim has talked to a number of people like Harry Collins, at Cardiff University, Michael Ronemus, from Cold Spring Harbor Protocols; Jacques Distler, at the University of Texas, oh yes and me, asking what might be done to improve the situation. As you would expect I wanted much more integration of the literature with discussion about the literature. That way papers can be continuously assessed and statused. Papers whose promise was not fulfilled will not mislead, slow burners can attain their due prominence.
Rather pleasingly the people Jim spoke to had broadly similar views to mine so I can leave Jim to sum up:
“Forty years ago, the Nobel-prizewinning immunologist Peter Medawar declared that all scientific papers were frauds, inasmuch as they describe research as a smooth transition from hypothesis through experiments to conclusions, when the truth is always messier than that. Comments, blogs and trackbacks, by expanding the published realm beyond the limits of the traditional paper, may make the scientific literature a little less fraudulent in Medawar’s sense, and in the more general one. They could also help the many frustrated scientists struggling to reproduce claims when, perhaps, they should not be bothering. Replication, for all its conceptual importance, is a messy, social business; it may be that it needs a messy, social medium.“