This post is by Research Data Management Advisor Karen Abel
For July’s Open Lunch, Professor of Psychology Daryl O’Connor, open research activist, took us on a tour of the current research landscape, considering challenges posed and cultural and practical changes underway. He homes in on three key opportunities for researchers to engage in open research practices.
The scientific world, Daryl suggests, has changed dramatically. Within the psychological sciences, a key moment occurred in 2015 with the publication of an article in Science by Nosek showing that only 36% or so of psychological science studies could be replicated. This ‘springboarded’ an understanding of the ‘replication crisis’. Daryl has been involved as an advocate in moving forward the open psychological science debate for many years but the implications, he stresses, are the same for all the sciences, and indeed, we might add, for all areas of research. There are many different factors contributing to this ‘reproducitility crisis’ and Daryl cites Marcus (2017) for capturing effectively a variety of questionable research practices, with examples such as P hacking and ‘HARKing’ – hypothesizing after results are known. Whilst highlighting these threats is a useful exercise, Daryl is keen to turn the focus to opportunities and specifically, in this talk, his focus is on three opportunity areas: pre-registration; registered reports; reproducibility of meta-analyses.
Essentially, this is about planning well in advance. But it is worth looking at the differences between confirmatory and exploratory research, the former enabling hypothesis testing, the latter being hypothesis generating. The problems occur when we present exploratory findings as confirmatory research. Pre-registration, Daryl suggests, is a crucial way to help reduce that. Of course, Daryl points out, in many areas of health sciences, pre-registering is already a standard course of action, for example, those pre-registering meta-analyses and systematic reviews via the likes of Prospero. Recognising this good work is important but there is still much room for improvement. In need of further convincing, Daryl asks? Well, take a look at this great study from 2015 by Robert Kaplan and Veronica Irvin which demonstrated that the introduction of trial registration in 2000 in the US dramatically increased null findings in heart and lung research, with 57% of pre-2000 studies finding significant benefits and only 8% of post-2000. The noise and false positives generated without preregistration are particularly concerning when working in areas such as medicine.
Making Pre-Reg Easy
There are many different ways to pre-register but Daryl advocates use of As Predicted as being one of the easiest tools available. For many, there are barriers to engaging with open research practices: not understanding what is required or finding it too difficult, for example. As Predicted helps to overcome this. It involves answering 9 key questions about your research project. Your hypothesis isthen stored with its own URL and can be viewed by the public (or kept private if you prefer) but crucially has enabled you to consider and capture your hypothesis prior to your research. As Predicted, and other similar tools available via the Open Science Framework, allow researchers to share their research intentions easily and quickly and that speed is crucial in times such as these, as we have seen with the pandemic.
Acknowledging a lack of consensus on how to approach pre-registration, Daryl has been working as part of an international group, the Joint Psychological Societies Preregistration Task Force, to define a gold-standard. The outcome of this group, ‘A Template for Preregistration of Quantitative Research’ has recently been accepted for publication in the American Psychologist and the pre-print can be read here. Although it happens to be in the domain of Psychological Science, it is applicable to all areas of science. It sets out standards for aspects such as theoretical backgrounds, hypotheses and methods and guidance on issues such as conflicts of interest and analysis plans. A similar group is now working on producing a similar template for qualitative research.
Launch of Registered Reports
Registered reports is a new publication format where peer review happens before the results are known. See Chris Chambers (2019) report for more information. There are two stages of peer review. First, after an idea is developed a study design created, the first submission is made for peer review. Once an in-principal acceptance is received, data is then collected and analyze data and the results are written up ready for the second peer review. This goes a long way to reduce questionable research practices.
From a psychological scientist’s point of view, and as Chair of the British Psychological Society’s Research Board, Daryl is keen to make sure that as many journals as possible in the psychology field are using this approach. Following a pilot in 2016, in 2018 the society led the way in psychological sciences by launching registered reports across all 11 BPS published journals.
Progress since take-off?
The speed at which journals have adopted registered reports has been very impressive but even so, at 275 in total by 2020, this is an alarmingly small of journals amongst all those published. Daryl points out that there are differences in disciplines with the life and social sciences being the main adopters and other sciences having minimal take up. So, much possibility for improvement.
Registered Reports – why bother?
Daryl highlights the two major reasons for committing to registered reports. First, without them, as reported by Warren (2018) the percentage of studies where the hypothesis is supported by the research paper is estimated to be around 95%. With registered reports, the percentage drops dramatically to approximately 55%. This indicates a very high number of false positives when registered reports are not involved.
The second is so that null findings, which are of great importance, particularly in medicine, can be fully exposed.
Barriers to uptake
Daryl acknowledges the genuine and perceived barriers to uptake of registered reports, for example, lack of awareness. However, with regards to concerns about ‘stifling creativity’, Daryl argues that exploratory analysis is in no way prevented by registered reports, it simply needs to be labelled as such. Similarly, he suggests, there is no need to worry about being scooped, as your registered report will be timestamped with your name against it.
Daryl demonstrates how the problem of lack of replicability is common in the area of meta-analysis too. Gøtzsche et al were unable to replicate at least one of the standard mean differences of 37% of a random sample of 27 meta-analyses. Alarmingly, these errors often equal a negation of the conclusions or reversal of the findings of the original studies. This is a huge area of science, Daryl underlines, where we need to improve our methods. Daryl suggests reading this article by David Moreau and Beau Gamble (2020) for tips on how to raise the quality of meta-analyses.
Using Psychology Expertise to Affect Change
If many of these problems are about the behaviour of scientists then Daryl suggests we apply the expertise of a Psychologist’s understanding of behavior change to promote uptake of open science. A central model for understanding behaviour is the CON-B model (Capability, Motivation, Opportunity-Behaviour). Read Daryl’s article (2019) Science and Behaviour, for more detail on this. Expanding on the CON-B model is the Behaviour Change Wheel model devised by Michie et all (2011) in which the Sources of Behaviour (CON) are each related to Intervention Functions demonstrating what is required in practical terms for those Sources of Behavious to be affected.
So, for example, Opportunity may be affected by education, persuasion or incentivization. And, if we’re thinking about how that relates in practical terms to the uptake of open science, well something like the UKRN journal club ReproducibiliTea could operate as a persuasive communication forum. Similarly, an Intervention Function of Capability is Environmental Restructuring and here we have practical examples such as the Open Science Framework laying down infrastructure to support open science methods. Daryl urges open science advocates to identify these different components and help to increase enablement for potential open research practitioners.
The Architecture of Science is Broken
Daryl’s article (2021) Leonardo da Vinci, preregistration and the Architecture of Science: Towards a More Open and Transparent Research Culture, demonstrates how much the research landscape today is damaged and what we might do to address this. It is an excellent read and delves in further depth to some of the topics covered in this presentation. Much can be changed through bottom-up and top-down processes and Daryl highlights Marcus Manafu’s notion that what is needed is collective action.
The ‘publish or perish’ culture still exists, Daryl warns, where promotions are very much driven by the number of papers published, by quantity and not quality. As Manafu underlines, funding, appointments, promotion, tenure, prizes – every aspect of scientific endeavour is linked to individual work, not teamwork. To change this, we need to change the culture, and to change the culture we need to adopt open science practices. And these practices should be rewarded both internally, via promotional opportunities, and externally by funding opportunities.
A way of working towards this is for universities to sign up formally to the UKRN, which provides both a bottom-up and top-down approach via support networks and regular meetings married to the ability of a national consortium to influence government and universities. An exciting prospect that will become more exciting when we have more colleagues, universities and key stakeholders signing up to it!
Slow down to speed up
The ‘publish and perish’ approach promotes fast science. But, in Uta Frith’s (2020) Fast Lane to Slow Science article, she explains that: “Fast science is bad for scientists and bad for science. Slow science may actually help us make faster progress, but what can we do to slow it down?”. Daryl agrees this is what we need to move to. It is about having the incentive structures changed; it’s about promoting teamwork. As a good example, Utrecht university have dramatically changed their structure of reward.
Critically the emphasis on outputs has moved from a large volume of closed outputs with strong h-index and impact factors and high rates of funding to quality open outputs with meaningful narratives and metrics demonstrating societal relevance.
Daryl invites us to take away these closing thoughts and calls to action:
- Preregistration is very straightforward and will improve transparency.
- Registered Reports are an important tool to improve open science.
- Try to adopt at least one open science practice.
- The incentive structure and architecture of science need to be changed.
- It’s an exciting time to be a researcher!