In October we heard from Dr Madeleine Pownall to discuss the need to integrate open scholarship principles into research training within higher education and how this can be achieved. You can view a full recording on YouTube with an event summary below:
The Framework for Open and Reproducible Research Training (FORRT) is dedicated to “advancing research transparency, reproducibility, rigor, and ethics through pedagogical reform and meta-scientific research”. It is an example of one of several open research initiatives that Dr Maddi Pownall has contributed to through the course of her PhD, which she was awarded in July 2022. Others include the Society for the Improvement of Psychological Science (SIPS) and the research group RitPoSE (Research in the Psychology of Student Education)
Today though, Maddi was focussed on FORRT, a grassroots, early career researcher led organisation, the vision of Dr Flavio Azevedo, that runs lots of different pedagogical projects and community initiatives.
Sustaining progress in open research
We’ve discussed open science, research and scholarship extensively on this blog and Maddi emphasises the principles that research should be transparent, open, rigorous, accessible, replicable and inclusive. While there has been some real progress in recent years, with initiatives such as preregistration and open data sharing gaining traction, these principles still aren’t properly embedded in universities through research training.
This was the starting point for Maddi and Flavio, and FORRT. If we really want to integrate open and reproducible scholarship into the curriculum, it’s important first to consider the rationale. Certainly (some) undergraduates are the scientists of tomorrow so it is important from that point of view, but the vast majority of psychology students won’t go on to an academic career and might use their research training in all different kinds of ways.
Pedagogy and open research
So moving beyond the argument that training in open research principles is important for the progress of science, Maddi and Flavio began to consider what other tangible benefits might be for students. What pedagogical benefits might be associated with integrating these principles into the curriculum?
Make it easy
It should be as easy as possible for educators to include aspects of open scholarship in their teaching with the types of guidelines and pedagogical resources available on the FORRT website. One example is a FORRT curated set of lesson plans to introduce students to terminology and practices.
This approach doesn’t necessarily integrate open research into the curriculum however. For that you need to clearly articulate a rationale that those already engaged with open scholarship can take to their Heads of School that demonstrates what such integration will achieve so we don’t need to rely on individual enthusiasts.
Big team science
Maddi put out a request to the FORRT community for a project to explore the evidence that an open research approach has an impact on student outcomes. It is a testament to FORRT that they ended up with 75 colaborators from dozens of different organisations.
Maddi led the project with Flavio, using a big team science approach to synthesize the existing evidence from the research literature, which was both valuable and challenging to coordinate the niche expertise of 75 colleagues on a Zoom call!
The methodology was a structured narrative review and citation searching. Criteria were any kind of evidence about the impact of open and reproducible scholarship and was intentionally broad in scope. They were particularly interested in any mention of the impact of using this or that open science approach on student outcomes, whether undergraduate or postgraduate – though the vast majority were undergraduate – in any discipline in any country. As the research team was so big, they were able to be broad with a fairly loose definition of evidence, not just the peer-reviewed literature, but covering different kinds of evidence base; gray literature, Twitter and the Open Science Framework for example. They also asked for student evaluations (that had consent to share).
The next challenge was to make some sense of a vast corpus of literature and present it in a way that was useful. The team decided to break it down into three distinct domains:
- Scientific literacies
- Student engagement
- Attitudes towards science
The team was broken up to explore each of these themes in detail:
- Scientific literacies
This subset included any paper about skills, in some cases quite technical competencies but also encompassing how generally literate students are in the scientific process.
There was evidence that pre-registration can improve students’ understanding of statistics, and teaching about FAIR data can lead to a more positive appraisal of open research generally. There was also some evidence that asking students to reproduce analyses from open data sets can really enhance their understanding of the research process.
Overall a clear narrative emerged that these these kinds of open research tools in a teaching context can be useful for students literacies.
- Student engagement
There was limited reference in the published literature to engagement, motivation or enjoyment (which is itself worthy of note). The majority of evidence were from narrative summaries published on platforms such as the Open Science Framework, or Twitter threads where people talked about how much their students enjoyed particular processes or activities.
Giving students access to “messy” data can make research feel more exciting; integrating open research and, particularly, working in partnership with students enhances the sense for students that they are being real researchers, allowed into the real world of research rather than using artificial data sets.
- Attitudes towards science
The team found that things under this domain became a bit less clear. What even constitutes a positive or negative outcome in this space? One paper for instance found that if you give students a one-hour lecture on the replication crisis it will negatively impact their trust in science. Is that a bad thing? Maddi argues not and that there is a lot to be said for creating ‘critical consumers’ of research.
A more positive example from Sakura and Brown (2019) suggested that while teaching students about questionable research practices (QRPs) can also reduce students’ trust in the process and in published findings, crucially, it can also help them to identify questionable practices. Maddi suggests this is an important caveat and a common rebuttal to integrating open scholarship into teaching is that it highlights issues of trustworthiness and credibility and that students should not be exposed to these issues. On the contrary, it is important to have these discussions to improve critical skills and improve science in the future by reducing QRPs.
A final caveat on this work was that that the quality of the pedagogical evidence, whether peer-reviewed literature or more anecdotal, or from Twitter and gray literature, didn’t itself always reflect the values of methodological rigour, robustness and transparency promoted through open scholarship. Much of the research used small sample sizes and the qualitative evidence was often presented without sufficient analytical information. The research base itself needs to be more robust.
There is also potentially valuable data that the team couldn’t use, from other universities for example that couldn’t be shared due to consent and data sharing restrictions for example, which is quite reasonable. Nevertheless, it is important to incentivise sharing of good practice so it can be used to inform a rationale elsewhere. There is a real need, not just for position papers – like the recently published University of Leeds Open Research Statement – but for real evidence from robust research.
Maddi is currently working on a longitudinal project looking at students who pre-register their undergraduate dissertations to explore if it impacts on their outcomes. Preliminary analysis suggests that the only significantly impacted outcome was their understanding of open science concepts, with no significant improvement in scientific literacies. This represents Maddi’s attempt to contribute more methodologically rigorous appraisals to the literature. She is also planning a fully-fledged cluster randomised control trial, again in collaboration, that will send open research materials to different departments and institutions to explore various outcomes and to see whether introducing students to these concepts has any tangible benefits on pedagogical outcomes.
Maddi also wants to explore evidence more locally, through Curriculum Redefined, by working with the Library for example to deliver open research workshops for educators and to connect to research culture work within student education.