Evaluating research assessment
In working towards designing a new assessment framework for research, the funding bodies commissioned a programme of evaluation of REF 2021.
These evaluation activities were initiated in 2022, with the majority reporting in late 2022, and the remainder reporting in 2023. Further evaluations of and reflections on REF 2021 can be found on the REF 2021 website, including the reports of the expert panels.
Impact case study analysis
The funding bodies and UKRI have commissioned a project (delivered by RAND Europe) to carry out qualitative and quantitative analyses of the impact case studies (almost 7,000) submitted to REF 2021. The aim is to extract common themes and messages that will form evidence of the broad impact of higher education research on wider society.
- Download the report: Analysis of the REF 2021 impact case studies (pdf)
- Find out more about the impact of higher education research in Scotland (new reports will be published on impact case studies and funding by RAND and the Fraser of Allander Institute)
- Download the report: The impacts of research from Welsh universities (pdf)
Review of REF costs and benefits
The funding bodies have commissioned a project to understand how REF 2021 costs have been contained or grown compared to previous exercises, including understanding how the balance of costs against benefits played out in REF 2021 and the effect of changes introduced in the current exercise.
The project was delivered by Technopolis Group:
Analysis of REF outputs
A project exploring ways in which data generated by the REF may be utilised to gain insights into the health of disciplines or research areas at sector level. The project included four areas of experimental analyses of REF 2021 data: analysis of medical sub-disciplines; analysis of ageing and gerontology research; exploring research excellence within the arts and humanities; and understanding interdisciplinary research. Through this project, the funding bodies have also gained insights into the feasibility of carrying out a more granular analysis of research assessment data, or analysis of cross-cutting themes in future exercises.
This project was delivered by Technopolis Group:
- Download the Analysis of REF Outputs report (pdf).
- Download the Analysis of REF Outputs appendices (pdf).
Responsible use of technology in research assessment
A project exploring the feasibility of using artificial intelligence and machine learning approaches to streamline research assessment, from the allocation of outputs and analysis of outcomes to assessment itself..
The project was delivered by the Statistical Cybermetrics and Research Evaluation Group at the University of Wolverhampton.
Download the Responsible Use of Technology in Research Assessment report (pdf).
Reviewing the role of metrics in research assessment
The funding bodies invited an expert panel to lead a review of the role of metrics in research management and assessment. This review took a short, sharp, evidence-informed look at current and potential uses of metrics against a set of tightly-defined objectives:
- To revisit the conclusions and recommendations of the last detailed review of these questions—The Metric Tide (2015)—and assess progress against these
- To consider whether developments over recent years in the infrastructures, methodologies and uses of research metrics negate or change any of those 2015 conclusions, or suggest additional priorities
- To look afresh at the role of metrics in any future Research Excellence Framework (REF), and consider whether design changes now under consideration as part of the FRAP suggest similar or different conclusions to those reached in 2015
- To offer updated advice to research funders on the most effective ways of supporting and incentivising responsible research assessment and responsible uses of metrics
Find out more about the review and expert panel.
Download the ‘Harnessing the Metric Tide: indicators, infrastructures and priorities for responsible research assessment in UK’ report.
Understanding perceptions of the REF
A large-scale review, commissioned by Research England on behalf of the four higher education funding bodies and published by RAND Europe, collected attitudes to the REF in real time as UK institutions prepared their submissions. It gathered views via a survey (with 3,000+ researcher responses), as well as focus groups and one-to-one interviews with researchers, research managers, and institutional leads.
The review also considered the impact of changes made to the REF since the previous exercise in 2014.