PLOS is a non-profit organization on a mission to drive open science forward with measurable, meaningful change in research publishing, policy, and practice.

Building on a strong legacy of pioneering innovation, PLOS continues to be a catalyst, reimagining models to meet open science principles, removing barriers and promoting inclusion in knowledge creation and sharing, and publishing research outputs that enable everyone to learn from, reuse and build upon scientific knowledge.

We believe in a better future where science is open to all, for all.

Journals
More Content
PLOS BLOGS The Official PLOS Blog
Browse all PLOS Blogs
Browse all PLOS Blogs

There’s more to research than citations: Understanding knowledge-sharing practices with Open Science Indicators

PLOS Open Science Open Science Indicators

If you’ve followed PLOS before, you know we’re committed to improving the ways in which research outputs and contributions are evaluated. Citations have long-been the primary method to evaluate research output but this measure alone fails to capture other ways in which research makes an impact, or the practices that make it impactful.

Open science practices and open science monitoring have an important role to play in our ongoing efforts to recognize a broader range of research contributions. They not only present a more complete picture of the diverse activities and expertise behind a research study or project, they also make those contributions visible and accessible. 

And yet for many years we lacked rigorous, accessible tools for monitoring the growth and impact of open science practices at scale. This was the driving force behind our decision to develop Open Science Indicators (OSIs). While we designed OSIs with our own goals for open science enablement in mind,we knew that others faced similar challenges. By sharing our learnings publicly and being open to community input, we assumed we could help other related efforts. 

After many new developments, collaborations, and feedback from diverse stakeholders, OSIs are becoming a valuable resource for the wider scholarly community. We’ve just released our latest update in partnership with DataSeer which now includes over 166,000 article records from PLOS and a range of comparator journals. The latest release offers full integration of indicators for protocols and study registration as well as a new method for preprint detection. For those who haven’t yet explored the data, or are considering open science monitoring solutions of your own, we want to share what we’ve learned and the possibilities for the future.

Why do OSIs matter? 

The benefits of open science practices are wide-ranging and well-evidenced: they improve research integrity and reproducibility, make visible a broader and more diverse range of research contributions, and have the potential to increase both the academic and social impact of research. These benefits are enjoyed by a wide variety of stakeholders, including researchers, universities, funders, publishers, policymakers, journalists, teachers, medical professionals, and the general public.

When we created OSIs, we wanted to be able to measure trends in open science and the effects of our activities to advance key practices — which tools and services were researchers using? How did open science practice vary from journal communities to regional differences? Were our policies effective? In order to realize our vision of a better future where science is open to all, for all, we first needed to understand where we are at currently and be able to measure the effectiveness of different initiatives that aim to move the needle. 

What knowledge have we gained from OSIs so far?

  1. A FAIR foundation for monitoring open science practices 
    Our work began with a set of principles which would underpin our approach and pave the way for others. We knew we wanted OSIs to be transparent, inter-operable and valuable in order to facilitate further discussions and collaboration around open science monitoring in the scholarly community. Many of the detailed requirements for some indicators were created in consultation with the research community. Based on feedback from others, we’ve also made updates to our underlying methods along the way to improve the accuracy of the results.

    Our work on OSIs has enabled us to contribute to and integrate with other global monitoring efforts including the Open Science Monitoring Initiative (OSMI) who have just published their Principles of Open Science Monitoring. We look forward to conducting a self-assessment to evaluate the extent to which OSIs align with these principles.  

  1. Better context for open science behaviors
    There is a wealth of information for each individual article represented in the dataset opening up more pathways to analyze trends. Not only do OSIs offer a consistent framework for measuring the number of papers sharing code, for example, but we can even more granularly look at code sharing trends as a percentage of articles which generated code. This enables users to more accurately focus on the behaviors they want to evaluate.

    The dataset extends beyond PLOS including a set of comparator articles from other publishers. It also contains information about the mechanisms of sharing and chronology. This extra detail enables metaresearchers and others to use the data to get a more nuanced understanding of specific author behaviors.

  1. Ability to understand trends for a specific audience cohort
    OSIs enable data users to track engagement with different practices over time and across academic disciplines and geographic regions. These insights enable stakeholders to identify a baseline of open science behaviors for a given research community, understand what’s working well, and identify opportunities for improvement. 

    From institutional leaders to funding program officers to academic librarians, open science monitoring is a task that falls on many shoulders. OSIs can save time and resources by indicating where further analysis and new solutions are needed, or act as a comparison to insights gathered from other monitoring solutions and needs assessments.
  1. Improved solution design for open science tools and services
    OSIs have been key to measuring the success of publishing solutions we’ve implemented at our own journals. From deciding to implement a preprint server solution of our environmental journals to adapting our approach to increase data repository usage, OSIs have helped validate areas for improvement and iterate to find the solution that works for our audiences. 

  1. Effective policy design and compliance monitoring
    Insights from OSIs can be used to design policies that are well-suited to specific research communities based on their current level of engagement with open science practices. The data also give policymakers and officers tools and methods for monitoring compliance and measuring success.

Toward a more collaborative, integrated approach to open science monitoring

What questions about open science behaviors spark your interest? How can the data drive your decisions? 

We invite everyone – researchers, librarians, funders, policymakers, institutional leaders and other publishers– to explore our data and consider what open science monitoring could mean for you. 

Ultimately, we envision a world in which open science monitoring becomes more ubiquitous, helping to improve how the value of research and the contributions of researchers are assessed. We hope that deeper understanding of open science practices will contribute to a broader view of what it means to conduct rigorous, trustworthy, impactful research. 

Related Posts
Back to top