Open Research Advisor Nick Sheppard reports on day 2 of Reproducibility, Replicability and Trust in Science with sessions on Research quality and Infrastructure and Tools. For day 1 see previous post.

Full programme here.

Research quality

In addition to being Academic Lead for Research Improvement and Research Integrity at the University of Edinburgh, Malcolm Macleod is also on the steering group of the UK Reproducibility Network. He kicked off day 2 with a talk on institutional strategies for research improvement, arguing that we need to be more rigorous and robust in the interventions we put in place in order to increase the quality of of published work. Just because a specific intervention looks like it may work, such as publication checklists, does not mean it will.

We tend to trust pharma less than academia but the next speaker Tim Koder from Oxford PharmaGenesis asked if this is really fair. While there has been bad practise in the past, Tim suggested this has largely been eliminated, while Ben Goldacre has pointed out that academic funders and institutions actually have inferior policies and performance on trial reporting. TrialsTracker demonstrates that pharma companies disclose a greater proportion of their trials than many academic institutions (e.g. via and are increasingly publishing in peer reviewed publications with a growing proportion open access.

One initiative that is driving openness and transparency in the industry is OpenPharma, a non-profit collaboration between a number of industry partners:

Efforts to improve the scientific enterprise must include changes to graduate science education, and in a global context. This was the starting point for the next talk from Gundula Bosch from John Hopkins. Recognising that philosophy provides the framework for critical thinking, Gundula and her colleagues aim to put the ‘Ph’ back in PhD “to transform graduate science education into an interdisciplinary, collaborative grassroots approach promoting open science globally”.

The foundations of the R3ISE program at John Hopkins are rigor (sic), reproducibility and reproducibility.

Casadevall, A., Fang, F.C., (2016) Rigorous Science: a How-To Guide. mBio. Vol. 7 (6) 10.1128/mBio.01902-16

Given the increase in preprint publication during the Coronavirus pandemic and high profile retractions, the next talk from Clarissa Carneiro on the relative quality of reporting in preprints and peer reviewed articles was fascinating. Broadly they found that while peer reviewed articles were of greater quality, the difference was not as great a might be expected, and some did in fact get worse on publication! For detailed discussion see this BioArxiv preprint: Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature

Infrastructure and Tools

Ewan Birney is Deputy Director General of EMBL, Director of EMBL-EBI and Senior Scientist. The European Bioinformatics Institute maintains the world’s most comprehensive range of freely available and up-to-date molecular data resources

Ewan began his talk on Open Data and its role for good scientific practice with a quick history of scientific publication. Henry Oldenburg was secretary of the Royal Society and as founding editor of the Philosophical Transactions of the Royal Society, pioneer of modern peer review.

In modern times of course, peer review remains the gold standard of scientific credibility while sharing primary data helps to demonstrate the robustness of reported results. Simply knowing that your peers have the means to check your work means you will be more careful!

Most science relies on others’ work and open data supports serendipitous discovery and reuse in other people’s science. In biology this is not simply via theory but from actual data and knowledge. Open data also supports aggregation and meta analyses, where a single dataset is often marginal, results from aggregating many similar datasets can be much more powerful.

The next speaker was Niklas Blomberg, director of ELIXIR which coordinates and develops life science resources across Europe to help researchers more easily find, analyse and share data.

With a fascinating perspective on open science as an enabler of collaborative, distributed research infrastructure, Niklas started with the startling observation that of four papers that presented the COVID-19 genome, only one provided access to the raw data. The analyses in the other three are therefore unverifiable and irreproducible.

Niklas went on to discuss the importance of open development and sustainability of research software.

ELIXIR’s F1000 Research Gateway publishes articles that describe ELIXIR platforms, use cases, strategy documents, technology developments and reviews. 

The next speaker Anita Bandrowski presented SciScore’s Rigor and Transparency Index, a new metric of quality for assessing biological and medical science methods. Sciscore comprises around 40 algorithms to address whether authors have addressed blinding, sex, and randomisation of subjects into groups, power analysis. It generates a report containing a rigor adherence table and a key resources table based off of the “STAR” guidelines –

In addition to DOIs and ORCID, Anita also stressed that using identifiers for resources makes for more reproducible papers i.e Research Resource Identifiers (#RRID)

To find out more about Sciscore:

Software searches out reproducibility issues in scientific papers (Nature news, Jan 2020)

The final speaker of day two was Cassandra Gould van Praag, open science community coordinator for the Wellcome Centre for Integrative Neuroimaging at Oxford.

As well as the various open tools produced and maintained by the Open WIN community, to help share research outputs effectively and openly, Cassandra talked about infrastructure and community engagement via the Mozilla Open Leadership Principles