Intended for healthcare professionals


The Stern review of the Research Excellence Framework

BMJ 2016; 354 doi: (Published 15 August 2016) Cite this as: BMJ 2016;354:i4413
  1. FD Richard Hobbs, professor and head of primary care1,
  2. Lesley M Roberts, deputy dean2
  1. 1Radcliffe Primary Care Building, Radcliffe Observatory Quarter, University of Oxford, OX2 6GG
  2. 2Warwick Medical School, University of Warwick, Coventry, CV4 7AL
  1. richard.hobbs{at}

Proposed reforms to help navigate through the stormy seas ahead

The United Kingdom Research Excellence Framework (REF) is a 5 to 6 yearly national ranking of all higher education institutions. Assessed by national peer panels in 36 “units of assessment,” or disciplinary areas, the REF informs the government’s annual allocation of core research funding. Most of this funding is reserved for institutional units of assessment whose submitted research is judged world class (4* or 3*).1 The Stern review was commissioned by government to advise whether the REF remained relevant and robust.

Compared with other countries, the UK’s REF involves much more peer review of submissions, rather than simpler objective metrics such as research income or numbers of research students. Peer review is subjective and expensive but is considered more reliable than metrics alone.2 Reliability is important since most core funding for UK higher institutions is based on this quality ranking—£1.57 billion (€1.83bn; $2.05bn) in 2016/17.3

The UK continues to punch well above its weight in terms of global research impact. With just 0.88% of the global population,4 5 the UK contributes 3.2% of global expenditure on research, and 4.1% of global researchers,6 accounting for 9.5% of research downloads, 11.6% of citations, and 15.9% of the world’s most highly cited articles.6 We have the most productive science base among the G7 countries,6 and we should strive to protect this eminence.

The Stern review,7 published at the end of July 2016, found that the UK economy and wider society benefited substantially from investment in research and that, relative to other countries, we get good value for money.

It reports some indirect evidence that engaging in the REF process itself stimulates this global excellence.8 The conclusion was that we should build on this success, and the review makes 12 recommendations for reform, two major ones relating to research outputs (publications).

Stern proposes that all research staff be counted but sets the number of papers required at twice that staff total, with individuals contributing from zero to six papers. The rationale for this change is to reduce institutional gaming by only including a few “stellar” researchers to gain a high ranking in an otherwise unremarkable unit of assessment.

A recommendation that seems less developed is that a researcher’s publications can only be returned by the institution where they worked at the point of publication. Research is no longer “portable.” Stern says that this radical suggestion will also reduce gaming—reducing the transfer or poaching of high ranking researchers just before each REF.

However, it may be better to tackle this problem with improvements to institutions’ retention and sustainment strategies, rather than changes to REF. If researchers can’t take their publications with them when they move, important research outputs may be “lost” to the REF, devaluing the findings. This is important, since recommendation 11 is that the UK makes much better use of REF findings.

Other recommendations are laudable, but need finessing: greater flexibility to showcase interdisciplinary research; basing impact case studies on a body of work rather than specific publications; and the simplification of “environmental statements.”

Finally, the report warns that the proposed Teaching Excellence Framework should complement and integrate with REF so that the combined cost burden for universities and the Exchequer is similar to that of the current REF. Such cost neutral integration presents a serious challenge, although we welcome a move that recognises excellence in teaching as well as research. Early modelling suggests that for most institutions, excellence in both frameworks is unlikely,9 and that there is a risk that this could drive further institutional gaming.

There are stormy seas ahead for UK research in a post-Brexit world. Those charged with assessing and rewarding research excellence must navigate with great care, helped by a strong steer from the Stern review to refine our REF rather than sink it.


  • Contributions: FDRH planned and wrote the first draft, LMR edited and added content, and both authors made a final edit, FDRH acts as guarantor.

  • Acknowledgements: FDRH is partly supported as a National Institute for Health Research (NIHR) senior investigator, director of the NIHR School for Primary Care Research, director of the NIHR Collaboration for Leadership in Applied Health Research and Care Oxford, theme leader of the NIHR Oxford biomedical research centre, NIHR Oxford diagnostic evidence co-operative, and Harris Manchester College.

  • Competing interests: We have read and understood the BMJ Group policy on declaration of interests and declare the following interests: FDRH has submitted to REF returns and receives QR to his department via the block grant to Oxford University.

  • Commissioned, not peer reviewed.


View Abstract