PLOS has posted a report, along with accompanying data, on qualitative research about how researchers assess credibility and impact of research in different contexts…
Authors: Ruth Schmidt is an Associate Professor at IIT’s Institute of Design and Anna Hatch is the program director for DORA. PLOS supports DORA financially via organizational membership, and PLOS is represented on the steering committee of DORA.
In part one of our series we showed how objective comparisons are not equitable . In our second part of this series we showed how individual data points can accidentally distract from the whole. In part 3 we explored how we gauge value by association. In our final installment we get into how incumbent processes and perceptions have the advantage.
The deep legacy traditions held by many academic institutions can be powerful, contributing a sense of continuity and identity and reducing the need to continually reinvent the wheel. But when these organizational habits and norms become too normalized, and old processes and structures persist because “that’s how we’ve always done this,” they can create an unhealthy sense of insularity that excludes new ideas and people.
One contributing factor to this is Confirmation bias, which describes peoples’ tendencies to dismiss evidence that doesn’t fit their initial judgments or preconceptions, even if these presumptions are grounded in subjective experiences and limited data. Failing to gather and consider counter-evidence makes us more likely to fall into old ways of thinking, such as cherry-picking information from a CV to confirm the view we already have, or dismissing potential warning signs because a candidate has already been accepted as a good fit.
Status quo bias, sometimes known as “effort aversion,” reflects our tendency to take the path of least resistance unless there are strong reasons not to. This can lead to situations where people stick with recognizably flawed processes because the effort to fix them or adopt new ones is perceived as too much effort. For example, the continued use of citations to indicate research quality is driven in part by the fact that it’s perceived as familiar and easier, not necessarily better, than using alternate indicators of real-world impact. For academic institutions — never known as stealthy and agile entities even in the best of times — this can cause significant paralysis, keeping demonstrably subpar structures in place long past their usefulness.
What can institutions do?
Make it easy. Sometimes the biggest barriers to adopting new behaviors are actually smaller than one might expect, such as not knowing where to start or a lack of clarity about the value of doing things differently. Making the benefits of new behaviors concrete, salient, and simple to grasp makes both starting and follow-through easier.
Question what you “know.” Sticking with old assumptions tends to reward those with more traditional backgrounds, which comes at the expense of attracting new or more diverse talent. Productively pushing back on assumptions — even assigning the role of playing devil’s advocate during deliberations — can force all involved to be crisper and more defined about what qualities matter, and why.
Broaden notions of success. New processes can only go so far if definitions for success (and who’s invited to the table) aren’t also revisited. Setting, publicizing, and adhering to goals that look beyond traditional norms or value alternate experiences can broaden the diversity of individuals under consideration.
Incumbent systems, processes, and structures are inherently advantaged, partly due to familiarity and partly because our tendency toward effort aversion tells us it’s almost always easier to stick with the old, however flawed, than introduce something new. This is especially true when those in positions of seniority have benefited from these legacy systems as their careers have progressed, and when in many cases entire academic careers spent at an individual institution may mean limited insight into even how to do things differently. But the need to introduce more equitable decision-making into RPT decisions to increase the presence of minoritized candidates, researchers, and faculty is very real, and will not happen without a combination of serious intent and structural support. Without active intervention, one model of current post-doc to faculty transitions has indicated that faculty diversity will not otherwise significantly increase until 2080. This is too little, too late.
To encourage the adoption of more equitable hiring, and RPT processes, the authors are collaborating on a series of tools for DORA to assist institutions in experimenting with new processes, indicators, and principles. Available here.
We thank Stephen Curry for very helpful comments. We also thank Stephen, Olivia Rissland and Stuart King for and the accompanying briefing document on the DORA webpage.