Nature troubled by Reproduction
There was a very nice News Feature in Nature last week on the problems of reproducibility in scientific papers. This isn’t the topic of ‘reproducible documents’ on which I have posted before but rather the question of repeatability of scientific research. The feature isn’t freely available I’m afraid but I can do some summarising here.
Nature’s report Jim Giles (he of the ‘Wikipedia only a little bit less accurate than Britannica‘ story and recent winner of the Association of British Science Writers award for best feature in 2005) did a bit of investigation of a four year old issue of Nature. He looked at the research articles and saw how they had been cited and how they had been followed up.
“impact of the papers has also been within the normal range: a handful have been referenced hundreds of times each, but most have notched up only a few tens of citations.”
Jim went a bit deeper than citation though, going back to the authors of the papers, and others in the fields to ask whether the results hade been reproduced or independently replicated. Some had been repeated, some had been corroborated (you can’t repeat the description of a fossil!) but some hadn’t.
For example there was a paper reporting “the first time, to have seen H3+ ions in the disk of gas and dust surrounding a young star” (Brittain, S. D. & Rettig, T. W. Nature 418, 57-59; 2002), some of whose conclusions haven’t been supported by subsequent work and whose authors “promote our very tentative interpretation that the unidentified lines might be H3+“. Then there was a paper on the production of “fully functioning stem cells from adult human bone marrow” (Jiang, Y. et al. Nature 418, 41-49; 2002) which should have revolutionised stem cell research but instead s proving very difficult to employ outside the original lab.
The crucial point to my mind is that the average reader would know nothing of these subsequent developments from looking at this relatively recent issue of Nature, even the online version. Indeed the stem cell and H3+ articles were featured on the cover. You need to be working in the area or do the kind of extensive research that Jim performed to find out what has turned out to be gold and what merely glistered.
Jim has talked to a number of people like Harry Collins, at Cardiff University, Michael Ronemus, from Cold Spring Harbor Protocols; Jacques Distler, at the University of Texas, oh yes and me, asking what might be done to improve the situation. As you would expect I wanted much more integration of the literature with discussion about the literature. That way papers can be continuously assessed and statused. Papers whose promise was not fulfilled will not mislead, slow burners can attain their due prominence.
Rather pleasingly the people Jim spoke to had broadly similar views to mine so I can leave Jim to sum up:
“Forty years ago, the Nobel-prizewinning immunologist Peter Medawar declared that all scientific papers were frauds, inasmuch as they describe research as a smooth transition from hypothesis through experiments to conclusions, when the truth is always messier than that. Comments, blogs and trackbacks, by expanding the published realm beyond the limits of the traditional paper, may make the scientific literature a little less fraudulent in Medawar’s sense, and in the more general one. They could also help the many frustrated scientists struggling to reproduce claims when, perhaps, they should not be bothering. Replication, for all its conceptual importance, is a messy, social business; it may be that it needs a messy, social medium.“
I’ve been talking for years about how valuable a Journal of Null Results would be. Granted, it wouldn’t be interesting necessarily, but a repository or database where one could look up someone’s failed methods would prevent a lot of unnecessary reproduction of failed attempts.
Also, the ability to add comments (especially if authors’ identities could be verified) to a research publication would be a revolution in peer-reviewed publications. If the PLoS is planning on adding that functionality, it will win many converts I believe.
Well Chris, I’m still mad that you got your picture in there and I didn’t.
But I was pleased with the depth in which Jim Giles analyzed the issue, and I also felt that all of us expressed fairly similar views. The Medawar quote was a decent (if a bit overwrought) summary statement. Too many researchers waste far too much time—often in parallel with many other of whom they’re not even aware—struggling to reproduce claims.
‘Replication’ does need a social component beyond the occasional scientific meetings and e-mails and phone calls. As you alluded to, within a given subfield there’s almost always an informal understanding of what’s turned to gold and what’s rubbish. But placing this discussion in a more formal (but not overly), err…format and getting scientists to be aware of it, to be able to easily access it if they’re looking for it, and even just stumble across it sometimes is what needs to happen now. Throw out whatever cliché you want—’paradigm shift’ comes to mind—but that’s what needs to occur in the way scientists approach publication and the subsequent analysis and re-analysis (and so on).
What I’m hoping to do at CSH Protocols (and I don’t mean for this to be an advertisement) is to create this on a small scale with a (relatively) non-controversial subject—the technical aspects of individual experiments. Hopefully efforts like this and PloS One will, over time, help change the way people look at scientific research, from a set of self-contained, smoothly flowing publication units to a set of messy little experiments that don’t have to have grand, sweeping consequences. in other words, something that better reflects the reality of the bench.
There will still be lots of dead ends. But why spend years travelling them?
Michael Ronemus
Michael,
There are certainly a few people thinking along the same lines as us at the moment. One of the most important challenges will be to integrate the various collaborative approaches to scientific communication. They need to talk to each other seamlessly. I’m putting a lot of faith in the fact that we are going to be working on an open source publishing platform to help achieve this.
Chris