Sixth Report - Reproducibility and Research Integrity

The government's response

What the committee recommended 28

1. 1
   Insufficient quantitative evidence on UK research integrity issues hinders effective policy responses. Conclusion · paragraph 27
2. 2
   Prioritise addressing reproducibility challenges comprehensively across all academic research disciplines. Conclusion · paragraph 32
3. 3
   Mandate UK CORI to publish annual research integrity reports and a detailed strategy action plan. Recommendation · paragraph 43
4. 4
   Establish a dedicated UK CORI sub-committee to address research reproducibility challenges Recommendation · paragraph 44
5. 5
   Require UK CORI to investigate AI's impact on research reproducibility in annual reports Recommendation · paragraph 45

See all 28 recommendations and conclusions

Sources

• Sixth Report - Reproducibility and Research Integrity ↗The committee's report, HC 101, parliament.uk
• Fourth Special Report - Reproducibility and research integrity: Government Response to the Committee's Sixth report ↗The government's response, 21 July 2023, parliament.uk

Recommendations and conclusions 28

1. 1
   Insufficient quantitative evidence on UK research integrity issues hinders effective policy responses. Conclusion · paragraph 27 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

   Although qualitative evidence indicates a potentially substantial scale of research integrity issues in the UK, there is a lack of quantitative evidence, including on the relative significance of the different causes of problems. This can only hamper efforts to evaluate damage being caused to the UK research sector in terms of culture, performance, reputation and economic value—now and in the future. This in turn prevents the design of proportionate and effective solutions to any problems.
2. 2
   Prioritise addressing reproducibility challenges comprehensively across all academic research disciplines. Conclusion · paragraph 32 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

   Though the specific problems faced differ, all research disciplines are affected by the systemic issues that limit reproducibility. The established view that issues are concentrated in the social and medical sciences is outdated and tackling reproducibility challenges should be a priority in every research field, regardless of discipline and methodology.
3. 3
   Mandate UK CORI to publish annual research integrity reports and a detailed strategy action plan. Recommendation · paragraph 43 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

   Despite significant delays, we welcome the establishment of UK Committee on Research Integrity (UK CORI) as a potential answer to existing research integrity challenges. We ask UK CORI to commit to produce an annual state of the nation report on research integrity, including reproducibility, issues. These reports should contain a breakdown of the scale of these issues across different disciplines. They should also contain action plans for addressing identified issues. UK CORI should also publish an action plan detailing how it will carry out each of the steps set out in its strategy, including timelines and what resources will be committed for the activities it plans to carry out.
4. 4
   Establish a dedicated UK CORI sub-committee to address research reproducibility challenges Recommendation · paragraph 44 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

   It is disappointing that UK CORI’s recently published strategy did not mention reproducibility, especially since our inquiry highlighted that this is a major research integrity issue. UK CORI should make sure reproducibility challenges are given due attention and not overlooked in deference to other pressing research integrity issues. UK CORI should develop a sub-committee to focus on reproducibility challenges in research. This sub-committee should establish the relative weight of reproducibility within research integrity concerns, and UK CORI should use this evidence to plan its prioritisation.
5. 5
   Require UK CORI to investigate AI's impact on research reproducibility in annual reports Recommendation · paragraph 45 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

   The increased application of artificial intelligence (AI) in scientific research presents a challenge to traditional research methods. The UK Committee on Research Integrity (UK CORI) should specifically: investigate the impact of deploying AI— and other increasingly complex software—on reproducibility in research; consider whether specialised action is needed; and set out these findings in its annual reports.
6. 6
   Significant reproducibility challenges in scientific research necessitate urgent action Conclusion · paragraph 50 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

   Whilst significant reproducibility challenges are faced in research, to refer to the sum of these issues as a “crisis” risks detracting from the many successes of the UK’s scientific research base. Nonetheless, there is need for action to address the significant problems caused by the prevalence or reproducibility problems in the scientific community.
7. 7
   Develop a co-produced framework for responding to research misconduct cases effectively Recommendation · paragraph 60 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

   The UK’s lacks an established infrastructure for responding to research misconduct cases. The UK Government should lead on a co-produced framework with the UK Reproducibility Network, UKRIO and UK CORI, which sets out the roles and expectations for key actors when cases of misconduct are identified.
8. 8
   Assess benefits of establishing an additional body to investigate research malpractice Recommendation · paragraph 61 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

   The UK Government should assess the benefits that an additional body, set up to investigate malpractice, could bring to the UK’s research integrity governance architecture.
9. 9
   Foster an environment where research integrity and reproducibility are championed across the community Recommendation · paragraph 68 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

   Most reproducibility issues are, in the main, not the result of deliberate bad practice. Many of the incentives faced by individuals conducting research act against reproducibility. Whilst individuals must take responsibility for conducting work which prioritises robust analysis and transparency, and for promoting the importance of reproducibility within their research field, we recognise that aspects of the academic process, such as time pressures and the publication model, act against the promotion of research integrity and reproducibility. Therefore, the research community, including research institutions and publishers, should work alongside individuals to create an environment where research integrity and reproducibility are championed.
10. 10
    Require universities to model reproducibility culture and implement protected research time policy Recommendation · paragraph 72 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Research institutions should model a culture of reproducibility by managing inordinate pressures on academics and encouraging the prioritisation of reproducibility in research outputs. This extends to encouraging openness around mistakes and their correction. In collaboration with the Higher Education Sector, Universities UK should implement a coordinated policy on minimum protected research time for research staff.
11. 11
    Develop dedicated funding and career paths for statistical experts and software developers Recommendation · paragraph 76 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Statistical experts and software developers are insufficiently recognised and renumerated within the university research sector. Funders and universities should develop dedicated funding for the presence of statistical experts and software developers in research teams. In tandem, universities should work on developing formalised, aspirational career paths for these professions.
12. 12
    Implement stronger tests for software and statistical skills in research funding applications Conclusion · paragraph 77 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

    Research funders should implement stronger tests for the presence of adequate software and statistical skills within research teams at the outset of a funding application. Where these skills are perceived to be lacking, UKRI should consider the feasibility and cost of offering a dedicated methodological support system to research teams.
13. 13
    Increase emphasis on reproducibility and integrity in university research education and training Recommendation · paragraph 82 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Currently there is insufficient attention placed on reproducibility and research integrity training for university students and research professionals. Greater emphasis should be placed on the importance of reproducibility and research integrity in education and training at undergraduate, postgraduate and early career researcher stages. Part of this training at the undergraduate and postgraduate levels should include the routine production of replications.
14. 14
    Incorporate mandatory reproducibility training and professional development for researchers throughout their careers Recommendation · paragraph 83 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Institutions should incorporate mandatory reproducibility training and professional development plans for researchers across the course of their career.
15. 15
    Implement a trial funding programme for 'slower' science and consult researchers on grant sufficiency Recommendation · paragraph 88 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Short-term research grants place restrictive limitations on researchers, which can be to the detriment of research integrity and reproducibility. UKRI should consult with a representative sample of researchers to understand whether their grants allow them sufficient time and funding to do the work needed for ensuring their research is reproducible. UKRI should also implement a trial funding programme with an emphasis on ‘slower’ science.
16. 16
    Impose a three-year minimum contract for all post-doctoral researchers in universities Conclusion · paragraph 89 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

    Uncertainty in the academic job market, especially at earlier career stages, acts as a strong additional disincentive against the prioritisation of reproducibility by researchers. Research funders, including UKRI, should work to impose a three-year minimum contract for post-doctoral researchers in universities.
17. 17
    Implement 100% open access policies for scientific outputs by UKRI and funders by 2025 Recommendation · paragraph 98 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    The trend towards blanket open access in the communication of scientific outputs is positive. UKRI and other research funders should continue to implement open access policies until this figure reaches 100%, by the end of 2025 at the latest.
18. 18
    Mandate researchers to share open-source data and code alongside all published outputs Recommendation · paragraph 101 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Currently, research outputs are frequently published without an associated link through to their open-source data and code. This prevents other researchers assessing work for its reproducibility. In all bar the most exceptional ethical and legal situations, researchers should share their research data and code alongside published outputs.
19. 19
    Mandate deposition of research data in open-access repositories and encourage FAIR principles Recommendation · paragraph 104 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Journals should collectively encourage researchers to employ the FAIR (Findability, Accessibility, Interoperability, and Reuse of digital assets) principles within their research and should mandate the deposition of research data in open- access repositories alongside the publication of research outputs.
20. 20
    Continued emphasis on data management plans as a condition of research funding is necessary Recommendation · paragraph 106 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    We welcome UKRI’s use of data management plans. A continued emphasis on their importance as a condition of research funding is necessary.
21. 21
    Develop a pilot programme to fund replication studies, learning from NWO's model Recommendation · paragraph 115 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Providing adequate funding for replication studies is an important precondition for ensuring researchers have the resources necessary to conduct them. UKRI should learn from its Dutch equivalent, NWO, by developing a pilot programme to fund replication studies.
22. 22
    Review journal portfolios to ensure sufficient publication options for negative and confirmatory science Recommendation · paragraph 116 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Aside from some notable exceptions, publishing routes for negative and confirmatory findings are not pursued thoroughly enough by the scholarly publishing industry. Publishers should review their journal portfolios to ensure that there are sufficient options for the publication of negative and confirmatory science, in line with the proportion of submissions which demand such routes.
23. 23
    Require publishers to support academics reporting issues and ensure timely research corrections and retractions Recommendation · paragraph 123 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Publishers have a vital role in the maintenance of the scholarly record. Publishers should support academics who report issues with published research in their journals and should commit to timely publication of research error corrections and retractions where necessary—in our view this process should not take longer than two months. Publishers should also commit to timely deployment of technology to support the quality of the published record.
24. 24
    Academic reward structures disincentivise research reproducibility and integrity. Conclusion · paragraph 127 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

    Reward structures in academia disincentivise reproducibility by placing disproportionate value on secured funding and frequent publication in prestigious journals.
25. 25
    Review Research Excellence Framework criteria, ensuring transparency and considering 'originality' removal. Recommendation · paragraph 131 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    The Future Research Assessment Programme (FRAP) is consulting on reforms for the assessment of UK higher education research. It should review the Research Excellence Framework assessment criteria to assure that transparency is a prerequisite of top- scoring research. It should also consider the effects of removing ‘originality’ from the top score bracket.
26. 26
    Assess researchers on broader academic contributions, mandating 'resume for researchers' format by 2025. Recommendation · paragraph 133 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Researchers should be assessed on the broader contributions to their academic field, including time spent conducting voluntary peer review and promoting reproducibility and research integrity. Funders, led by UKRI, should move towards the exclusive use of the ‘resume for researchers’ format in funding calls by 2025.
27. 27
    Publish peer review comments to improve transparency and deter paper mills. Recommendation · paragraph 142 · addressed to Department for Science, Innovation and Technology Recommendation (the committee's words)

    Peer review should not be viewed as a binary measure of quality versus unreliability for published papers. There is a wide range of competency, depth, and rigour in the analyses carried out during peer review as time-poor academics often do not have time to conduct detailed scrutiny. Where possible, journals should seek to publish peer review comments to improve transparency in the research publication process and to deter paper mills.
28. 28
    Publishers and funders should offer a 'registered report partnership model' for research transparency. Conclusion · paragraph 143 · addressed to Department for Science, Innovation and Technology Conclusion (the committee's words)

    The ‘registered report partnership model’ offers a good opportunity for researchers to have their methodologies peer reviewed at an early stage in the research process, allowing researchers to incorporate feedback into their research plans. Within this model, funding and the principles for a study are agreed concurrently, with a guarantee of publication regardless of experiment outcomes. Publishers and funders should work together to offer a ‘registered report partnership model.’ Not only will this benefit researchers who will receive feedback on their research plans and guaranteed publication, but greater transparency in research methodologies will be achieved, which should have a positive impact on research reproducibility.