The UK Reproducibility Network (UKRN) is a national peer-led consortium investigating the factors that contribute to robust research, promoting training activities, and disseminating best practice.

Under the leadership of Professor Daryl O’Connor and Dr Eike Rinke, the University of Leeds has formally joined UKRN as part of our commitment to open research.

For our open lunch in November we heard from Daryl and Eike and were pleased to welcome Neil Jacobs, Head of the UKRN Open Research Programme.

You can watch a full recording of the event on YouTube:

The scientific world has changed

This is the second time we’ve hosted Daryl for lunch after he spoke about Preregistration and the architecture of open science back in Summer 2021. This time around, in his role as UKRN Institutional Lead, he was discussing some of the work Leeds will be doing with UKRN but started by highlighting an important paper from 2015 that demonstrated that replication of research results was a significant problem, within the discipline of psychological science specifically, but that is relevant to a wide range of disciplines.

Similar issues have since been demonstrated across disciplines while Psychology has been at the forefront of important developments to address the ‘reproducibility crisis’, notably preregistration and registered reports which were the subject of Daryl’s previous talk and which is increasingly used not just in Psychology but all areas of science.

Scheel et al show the positive result rates for standard reports compared with registered reports. Clearly a researcher shouldn’t expect a positive result for virtually every experiment as indicated by the standard approach, which drops to a more reasonable 40% when defining methodology in advance of data collection.

Other positive initiatives Daryl highlighted include open science badges and increased data sharing. However recent research, that Eike has contributed to, indicates that there is still a lack of awareness amongst many researchers.

Dysfunctional Research Culture

It’s increasingly recognised that the dominant research culture incentivises novel, positive outcomes and problematic behaviours like selective reporting and questionable research practices such as HARKing and P-Hacking

In order to improve science and research, it’s crucial to improve the rewards system to incentivise rigour and high quality methodology, to promote full transparency in all aspects of research. These issues are addressed by Leeds’ Research Culture work.

UKRN workstreams

Daryl highlighted two specific UKRN projects that he was especially keen to be involved with:

UKRN@Leeds

Eike Rinke is a social scientist and UKRN local network lead for the University of Leeds promoting a community across campus to bring together researchers and professional staff who share an interest in open, transparent and rigorous research practice. He has recently been elected to the Supervisory Board of UKRN.

The UKRN@Leeds community is already over 200 members and very diverse, any and all PGRs and members of staff are invited to join.

This link will take you to The Microsoft Team (UoL staff and PGRs only)

As a grassroots initiative, the local UKRN community provides academic and professional staff with information about the rapidly developing open research landscape at Leeds, across the UK and around the world

Eike and the community use the UKRN@Leeds MS Team to share the latest open research news and publications a well as job opportunities, open research events at Leeds and beyond, and links to other communities of interest.

ReproducibiliTea

ReproducibiliTea is an early career researcher led journal club format practiced at many campuses around the world. Everyone is welcome to monthly hybrid meetings during term time to discuss a nominated article or paper about a specific aspect of open research.

Disciplinary differences

Neil Jacobs emphasised how UKRN is primarily a platform to enable collaboration across the sector, and across discipliness. He echoed Daryl’s point that while there is perhaps a tendency for conversations about reproducibilty and transparency to focus on quantitative research and STEM, similar principles apply to all disciplines.

In a recent blogpost for instance Anton Howes asked “Does History have a Replication Crisis?” while UKRN have also partnered with the British Psychological Society (BPS) and the Practice Research Advisory Group (PRAG) to explore what reproducibility might look like in theatre studies, music or arts based research

Another project has recruited a “Story Associate” to explore how to retain the benefits of accessible narrative forms, whilst maintaining standards of rigour within research in a a range of different disciplines including the humanities.

From work done at the University of Surrey, UKRN have published a series of case studies across a whole range of different subject areas to show what open research looks and feels like in those different disciplines.

Research culture and integrity

Neil highlighted some quite startling figures from a 2023 research integrity report, with more than half of researchers surveyed reporting that they have not conducting a thorough review of a manuscript, while 1 in 5 did not report findings if they contradicted their theory.

Such figures might suggest that researchers are working in a culture that doesn’t necessarily reward rigorous, high quality research and that there’s a disconnect between what research culture incentivises compared to the intrinsic motivations of researchers who surely want to produce high quality work.

This is one of the aspects that the REF is trying to address for 2028, positing a greater emphasis and closer review of the culture in which research is undertaken. There is ongoing discussion around these proposals with some keen to maintain the focus on research outputs as being the measure of excellence, while Gemma Derrick has argued in Research Professional News that Opposition to changing the REF shows why change is needed.

Enhancing Research Culture: UK sector priorities and challenges (UKRN report)

Another key focus for UKRN is community support for local networks to develop a peer community of practice across the sector.

UK research needs to open up to keep up

In his own piece for Research Professional News Neil has argued that the national reputation of the UK depends on joining the global push towards transparency.

Historically the UK has been a leader in this area, but there’s lots of activity now being promoted in Europe and the US.

UNESCO Recommendations on Open Science

Neil highlighted the UNESCO Recommendation on Open Science which illustrates the broad range of things covered by open research (as open science tends to be referred to in the UK to be inclusive of the humanities).

In addition to open scientific knowledge – research publications and open access, research data, open source code and pre-registration – the other quadrants highlight open science infrastructures, engagement with societal actors – including public participation and citizen science, and dialogue with other knowledge systems.

UKRN has been working with UNESCO and with the Swiss Reproducibility Network to produce a guide for institutions who want to implement these recommendations which are signed by 194 National governments including the UK.

UKRN open research programme

The UKRN open research programme now comprises 22 universities and growing with four main strands of work:

  • training
  • reform of recruitment, promotion and appraisal
  • sharing institutional practice e
  • evaluation / meta-research / indicators

A train the trainer program is in place, designed to to build up institutional capacity and improve sustainability, so that institutions have access to a range of trainers whereby researchers and professional services staff to train their peers in a range of different aspects of open research.

UKRN are also building up a national community of trainers to share and enable good practice among those trainers.

Open and Responsible Researcher Reward and Recognition (OR4) Project

43 institutions have joined the OR4 project either as a case study institution like Leeds or part of the community of practice, to reform the ways in which staff are recruited, promoted and appraised, to ensure that open research practices are recognised in these processes.

There’s a huge diversity across the 43 institutions, from the World College of Music to the University of Cambridge and Cancer Research UK’s Scottish Institute, together representing over 80,000 researchers across the UK sector. Neil hopes that over time this will mean that researchers can be confident that the kinds of open research practices that you do at Leeds will also be recognised at any other UK University.

A more general approach to enable universities to share good practice among themselves, to find out what peer institutions are doing to support open research is being developed and Neil demonstrated an early example for Kings College London. These pages will develop over time to become more dynamic, to enable the community of of change agents across these institutions to learn from each other and share good practice over time.

Evaluation design

Finally Neil discussed methodology to evaluate progress. There’s already been a large scale survey, though we all recognise that the sector is hardly under-surveyed, so it’s important to bring some coordination to this activity, with common question sets for example, or similar sorts of survey instruments.

Currently there is the open research indicators pilots with about 15 institutions including Leeds working with solutions providers to explore the extent to which we can reliably, ethically, practically and validly measure different aspects of open research. The potential solutions providers include OpenAIRE, Digital Science, PLOS+DataSeer, Elsevier and CORE.