These evaluation activities were initiated in 2022, with the majority reporting in late 2022, and the remainder reporting in 2023. Further evaluations of and reflections on REF 2021 can be found on the REF 2021 website, including the reports of the expert panels.
Review of REF costs and benefits
The funding bodies have commissioned a project to understand how REF 2021 costs have been contained or grown compared to previous exercises, including understanding how the balance of costs against benefits played out in REF 2021 and the effect of changes introduced in the current exercise.
The project was delivered by Technopolis Group:
Analysis of REF outputs
A project exploring ways in which data generated by the REF may be utilised to gain insights into the health of disciplines or research areas at sector level. The project included four areas of experimental analyses of REF 2021 data: analysis of medical sub-disciplines; analysis of ageing and gerontology research; exploring research excellence within the arts and humanities; and understanding interdisciplinary research. Through this project, the funding bodies have also gained insights into the feasibility of carrying out a more granular analysis of research assessment data, or analysis of cross-cutting themes in future exercises.
This project was delivered by Technopolis Group:
- Download the Analysis of REF Outputs report (pdf).
- Download the Analysis of REF Outputs appendices (pdf).
Responsible use of technology in research assessment
A project exploring the feasibility of using artificial intelligence and machine learning approaches to streamline research assessment, from the allocation of outputs and analysis of outcomes to assessment itself..
The project was delivered by the Statistical Cybermetrics and Research Evaluation Group at the University of Wolverhampton.
Reviewing the role of metrics in research assessment
The funding bodies invited an expert panel to lead a review of the role of metrics in research management and assessment. This review took a short, sharp, evidence-informed look at current and potential uses of metrics against a set of tightly-defined objectives:
- To revisit the conclusions and recommendations of the last detailed review of these questions—The Metric Tide (2015)—and assess progress against these
- To consider whether developments over recent years in the infrastructures, methodologies and uses of research metrics negate or change any of those 2015 conclusions, or suggest additional priorities
- To look afresh at the role of metrics in any future Research Excellence Framework (REF), and consider whether design changes now under consideration as part of the FRAP suggest similar or different conclusions to those reached in 2015
- To offer updated advice to research funders on the most effective ways of supporting and incentivising responsible research assessment and responsible uses of metrics
Understanding perceptions of the REF
A large-scale review, commissioned by Research England on behalf of the four higher education funding bodies and published by RAND Europe, collected attitudes to the REF in real time as UK institutions prepared their submissions. It gathered views via a survey (with 3,000+ researcher responses), as well as focus groups and one-to-one interviews with researchers, research managers, and institutional leads.
The review also considered the impact of changes made to the REF since the previous exercise in 2014.
Review of REF costs and benefits
The funding bodies have initiated a project to understand how REF 2021 costs have been contained or grown compared to previous exercises, including understanding how the balance of costs against benefits played out in REF 2021 and the effect of changes introduced in the current exercise.
The project is being delivered by Technopolis Group, who are working with institutions using surveys and other data collection tools to identify how REF submission preparation was managed within institutions. This project expects to report in the first half of 2023.
Impact case study analysis
The funding bodies have commissioned RAND Europe to carry out qualitative and quantitative analyses of the almost 7,000 impact case studies submitted to REF 2021. We aim to extract common themes and messages that will form evidence of the broad impact of higher education research on wider society.
This will be the second evaluation of REF impact case studies, with the first being undertaken following REF 2014 when impact case studies were formally introduced into the septennial assessment cycle. This project expects to report in the first half of 2023.