PLOS has posted a report, along with accompanying data, on qualitative research about how researchers assess credibility and impact of research in different contexts…
A research article is an orderly summation of a complex and circuitous process. It is characterized by detailed planning, iterative trial and error, meticulous execution and thoughtful analysis. As a summary, articles are invaluable, but detailed insight into processes and procedures is required to truly understand and reproduce research. Detailed public methodological documentation can contextualize results, subvert bias, enhance reproducibility, and increase efficiency across the entire scientific ecosystem. In recent years, the research community has gained a deeper appreciation for the value of detailed methodological documentation. Here’s why:
Methods are research
Developing a methodology and then executing it is arguably the largest component of research. It’s the day-to-day business of science, the mechanism for capturing data that can later be used to investigate a research question. Depending on the field and approach, methods can take many forms, from a recipe-like protocol, to a script, to a database of behavioral stimuli. But whatever the specifics, a clear and complete method is key to running a consistent and reliable study, reproducing the work in future, and understanding the results as a reader.
Just as with research articles, restricted access to methods creates inefficiencies and slows progress. Researchers expend time (and funding) developing similar approaches, repeating experiments and pursuing dead ends, unaware of related work. Out-dated procedures remain in use long after they’ve been superseded because practitioners cannot access the latest advances. And there is an opportunity cost. Researchers rely on other research for inspiration, direction and advancement. When the products of research remain hidden, promising new leads may be left unexplored.
Methods are lasting
Methods are highly transferable. More than any other research artifact, methods have the potential for adaptation or reuse in different contexts and across a broad range of research questions and disciplines. For evidence, look no further than researchers’ frequently expressed frustration with “citation trails” in which an article methods section cites the methods section of an earlier work, and that work cites another work, and so on, often going back years, or even decades. Similarly, stand-alone methods articles are often among the most highly-cited publications, and they continue to receive citations over a longer period of time than standard research articles.
Results are subjective—methods less so
In order to generate conclusions, researchers make choices about how to analyze data and interpret the patterns they observe. These analytical and interpretive choices are valuable, but subjective. They benefit from the researchers’ intuition, inspiration and expertise—but those same factors can also introduce bias and lead to misinterpretation. The unencumbered, unprocessed artifacts of research—such as methods and raw data—are much less likely to be influenced either positively or negatively by personal interests, insights or opinions. They are closer to neutral—a clear accounting, rather than an argument with a personal perspective.
Methods are insufficiently described
A research article is not an instruction manual. Often, published research cannot be reproduced, not because the original work was flawed, but simply because the article format is not conducive to a really in-depth description of the process.
Do your methods pass the Golden Rule test?
Methods serve authors, readers and the research community
Clear, complete, and open methods increase credibility and support lasting impact. Documenting and sharing methodologies has interrelated scientific and reputational benefits for individuals and the community.
- Making methods public creates a positive impression. Having the option to review detailed methods increases readers’ trust, whether or not they consult the documentation.
- Researchers can more easily reproduce results with detailed open methods. Authors who want to apply the method in their own research can do so more efficiently if the approach is described in detail and easy to find online.
- Strong, easy-to-follow methods are more likely to be used in future research, and by extension more likely to be cited, bringing fresh eyes to the original and helping it to remain relevant over time.
Under the best circumstances, these three factors can feed off one another in an expanding cycle of trust, reuse, and readership.
The future of methods as permanent assets
This combination of relevance, usefulness, potential for reuse, and demand suggests that methods deserve a more formalized, permanent place in the scientific record. Methods should be reviewed and validated, indexed and archived, searched for and cited. They should be treated as stand-alone research artifacts, rather than supplementary material of interest only within the context of a research article.
Alongside their enormous potential to positively impact the research community, sharing methods has a comparatively low barrier to entry for authors. From a practical perspective, process documentation often already exists in some form (for example, in instructions for internal use, or as part of a funding application), providing a convenient basis for a publication. More importantly, detailed, published methods are a natural extension of existing practice. Some discussion of methods has always been standard in the sciences. Increasing depth and detail, establishing norms, implementing peer review, and formalizing publication are all important and logically consistent extensions, analogous to the evolution of scientific discourse from published correspondence in the 18th century into the modern peer-reviewed research article. It will take time, community discussion and experimentation to refine and establish new norms for communicating methods—but the notion of communicating methods at all does not inspire the same heated debate as other aspects of Open Science practice.
How can we communicate methods more effectively?
The scholarly community is developing new ways to document, preserve and communicate methods for future use. In a future post, we’ll explore some of them.