The document provides an overview of the Research Excellence Framework (REF) process. It discusses the purpose and assessment framework of the REF, including the criteria for evaluating outputs, impact, and environment. It also outlines the key stages of the REF process, from developing submission guidelines to expert review by panels. Submissions will include staff details, up to 4 research outputs per person, impact templates and case studies, environment data and templates. The results will be published in December 2014 and help determine £2 billion in annual research funding.

1. An overview of the REF
Vicky Jones
REF Deputy Manager
Vitae: Preparing for the REF

2. Presentation outline
• Overview
• Staff
• Outputs
• Impact
• Environment
• Submissions

3. Overview:
Purpose of the REF
• The REF is a process of expert review
• It replaces the RAE as the UK-wide framework for
assessing research in all disciplines
• Its purpose is:
- To inform research funding allocations by the four UK
HE funding bodies (approximately £2 billion per year)
- Provide accountability for public funding of research
and demonstrate its benefits
- To provide benchmarks and reputational yardsticks

4. Overview:
The assessment framework
Overall quality
Outputs Impact Environment
Maximum of 4 outputs Impact template and Environment data and
per researcher case studies template
65% 20% 15%

5. Overview:
The REF process
Criteria phase Submissions phase Assessment phase
2011 2012-13 2014
• Develop and publish • HEIs submit Codes of • Panels assess
Guidance on submissions practice submissions
(Jul)
• Launch the REF • Publish outcomes
• Develop, consult on and submissions system Dec 2014
publish Panel criteria
(Jan 2012) • Submission deadline
29 Nov 2013

6. Overview:
Guidance and criteria
Comprehensive information and guidance is set out in:
• Assessment framework and guidance on
submissions (July 2011):
- Sets out the information required in submissions and
the definitions used
• Panel criteria and working methods (Jan 2012):
- Sets out how panels will assess submissions
The above documents set out the official guidelines for the REF.
These slides provide a summary of key points but do not provide or
replace the official guidelines.

7. Overview:
Submissions
• Each HEI may submit in any or all of the 36 units of
assessment (UOAs)
• Each submission in a UOA provides evidence about the
activity and achievements of a ‘submitted unit’ including:
- Staff details (REF1a/b/c)
- Research outputs (REF2)
- Impact template and case studies (REF3a/b)
- Environment data (REF4a/b/c)
- Environment template (REF5)
• A submitted unit may, but need not, comprise staff who
work within a single ‘department’ or organisational unit

8. Overview:
Assessment
• Submissions will be assessed by 36 sub-panels
working under the guidance of 4 main panels
• Panels will carry out the assessment according to the
published criteria and working methods
Sub-panel responsibilities Main panel responsibilities
• Contributing to the panel • Developing the panel criteria
criteria and working methods and working methods
• Assessing submissions and • Ensuring adherence to the
recommending the outcomes criteria/procedures and
consistent application of the
overall assessment
standards
• Signing off the outcomes

9. Staff

10. Staff:
Staff selection and circumstances
• HEIs are responsible for selecting eligible staff whose
outputs are to be included in their REF submissions
• Each HEI is required to develop, document and apply a
code of practice on the fair selection of staff
• Number of outputs can be reduced without penalty
where an individual’s circumstances have constrained
their ability to work productively or produce four outputs
in the REF period
• We have sought to make these arrangements as clear
and consistent as possible with due regard to
confidentiality

11. Staff:
Individual staff circumstances
• Up to four outputs must be listed against each
individual
• The number of outputs can be reduced without penalty
where an individual’s circumstances have constrained
their ability to work productively or produce four outputs
in the REF period
• We have sought to make these arrangements as clear
and consistent as possible, with due regard to
confidentiality

12. Outputs

13. Outputs:
Research outputs
• Panels will assess the quality of research outputs
through a process of expert review
• All forms of output that embody research will be
assessed on an equal footing
• Panels will assess the quality of outputs, not the
contribution of individual researchers
• A co-authored output may be listed against one or
more individuals that made a substantial research
contribution to it (no more than twice within the same
submission)
• Institutions may request ‘double-weighting’ for outputs
of extended scale and scope

14. Outputs:
Additional information
• Several sub-panels will make use of citation data as a
minor component to inform peer-review
• HEIs will be provided access to the Scopus citation data
(in the relevant UOAs) through the REF submission
system
• Several sub-panels invite additional information from the
HEI to inform judgements
• Panels will not use journal impact factors, rankings or
lists or the perceived standing of the publisher

15. Outputs:
Assessment criteria
The criteria for assessing the quality of outputs are
originality, significance and rigour*
Four star Quality that is world-leading in terms of originality, significance
and rigour
Quality that is internationally excellent in terms of originality,
Three star significance and rigour but which falls short of the highest
standards of excellence
Two star Quality that is recognised internationally in terms of originality,
significance and rigour
One star Quality that is recognised nationally in terms of originality,
significance and rigour
Quality that falls below the standard of nationally recognised
Unclassified work. Or work which does not meet the published definition of
research for the purposes of this assessment
* Each main panel provides descriptive account of the criteria

16. Impact

17. Impact:
Definition of impact
• Impact is defined broadly for the REF:
an effect on, change or benefit to the economy, society,
culture, public policy or services, health, the
environment or quality of life, beyond academia
• Panels recognise that impacts can be manifest in a
wide variety of ways, may take many forms and occur
in a wide range of spheres, in any geographic location
• Panels provide examples of impact relevant to their
disciplines, intended to stimulate ideas - not as
exhaustive or prescriptive lists

18. Impact:
Some examples of impact
Enhanced professional
Improved health or standards, ethics, guidelines Improved risk
welfare outcomes or training management
Public debate has
been shaped or
Improved quality, More effective Improved business informed by research
accessibility or efficiency of a management or performance
public service workplace practices
A social enterprise
Research has enabled initiative has been
Changes to the Production costs have
stakeholders to challenge created
design or delivery of reduced
conventional wisdom
the school curriculum Improved forensic
Enhanced preservation, Improved access to methods or expert
Policy debate or decisions conservation or presentation justice, employment systems
have been influenced or of cultural heritage or education
shaped by research Improved management or
Research has informed conservation of natural
Organisations have Jobs have been public
created or protected resources
adapted to changing understanding, values, attitud
cultural values es or behaviours
Enhanced corporate The policies or activities of Changes to
Levels of waste have
social responsibility NGOs or charities have been legislation or
reduced
policies informed by research regulations
New forms of artistic Changes in Enhanced technical
A new product has
expression or changes to professional practice standards or
been commercialised
creative practice protocols

19. Impact:
Submission requirements
• Sets out the submitted unit’s general
Impact template approach to supporting impact from
(REF3a) its research:
• Approach to supporting impact during
20% of the the period 2008 to 2013
impact
sub-profile • Forward strategy and plans
• Specific examples of impacts already
achieved, that were underpinned by
Case studies
the submitted unit’s research:
(REF3b)
• 1 case study per 10 FTE staff
80% of the submitted (plus 1 extra)
impact • Impacts during 2008 to 2013;
sub-profile underpinned by research since 1993

20. Impact:
Case studies
• Each case study should:
- Clearly describe the underpinning research, who undertook
it and when
- Provide references to the research and evidence of quality
- Explain how the research led/contributed to the impact
- Clearly identify the beneficiaries and define the impact
- Provide evidence/indicators of the impact
- Provide independent sources of corroboration
• All the material required to make a judgement should be
included in the case study
• Submitted case studies need not be representative of
activity across the unit: pick the strongest examples

21. Impact:
Assessment criteria
The criteria for assessing impacts are reach and significance*
Four star Outstanding impacts in terms of their reach and significance
Three star Very considerable impacts in terms of their reach and
significance
Two star Considerable impacts in terms of their reach and significance
One star Recognised but modest impacts in terms of their reach and
significance
The impact is of little or no reach and significance; or the impact
Unclassified was not eligible; or the impact was not underpinned by excellent
research produced by the submitted unit
* Each main panel provides descriptive account of the criteria

22. Environment

23. Environment:
Environment template
• Each submission to include a completed template:
- Overview
- Research strategy
- People, including:
- staffing strategy and staff development
- research students
- Income, infrastructure and facilities
- Collaboration and contribution to the discipline or research
base
• The ‘panel criteria’ request specific types of evidence
under each heading, and indicate how much weight they
will attach to each component

24. Environment:
Environment data
• All submissions to include data on:
- Research doctoral degrees awarded (REF4a)
- Research income (REF4b)
- Research income in-kind (REF4c)
• Definitions are aligned with HESA returns; the data
relate to the ‘whole unit’ - not just submitted staff
• Some sub-panels request specific additional data, to
be included within the environment template (REF5)
• Data will be considered by panels alongside the
narrative information provided in the relevant section
of the environment template

25. Environment:
Assessment criteria
The criteria for assessing the environment are
vitality and sustainability*
Four star An environment that is conducive to producing research of
world-leading quality, in terms of its vitality and sustainability
An environment that is conducive to producing research of
Three star internationally excellent quality, in terms of its vitality and
sustainability
An environment that is conducive to producing research of
Two star internationally recognised quality, in terms of its vitality and
sustainability
An environment that is conducive to producing research of
One star nationally recognised quality, in terms of its vitality and
sustainability
Unclassified An environment that is not conducive to producing research of
nationally recognised quality
* Each main panel provides a descriptive account of the criteria

26. Submissions

27. Pre-submission
• Codes of practice
Submission by Response from funding body by
27 April 2012 6 July 2012
31 July 2012 12 October 2012
• Requests for multiple submissions or case studies requiring
security clearance
Request by Response from REF team by
27 April 2012 8 June 2012
28 September 2012 9 November 2012
7 December 2012 18 January 2013
• Survey of submission intentions
Invitation and guidance Online survey open
July 2012 Early Oct – early Dec 2012

28. Overview:
The submission system
• All submissions must be made through the REF
submission system:
 Pilot available to all HEIs: Sep 2012
 Open for submissions: Jan–Nov 2013
• Each HEI to set up system users and user permissions
• All data may be entered onto the system and/or bulk
imported
• HEI contacts have seen a demonstration of the system
• User guidance and support will be provided

29. Access to REF4 data
• We will provide HESA data on research income and
doctoral degrees awarded in stages:
- May 2012: Data for 2008-09, 2009-10 and 2010-11
- Apr 2013: Data for 2011-12
- Doctoral degrees data for 2012-13 will be available from
HESA when the student record is submitted
• Data on research income-in-kind will be provided by
the Research Councils and health research funders
• Institutions will need to allocate these data to the
appropriate UOAs; or use their own sources
• The submission system will validate submitted data
against the data we provided, at HEI level
• stem

30. Further information
www.ref.ac.uk
(includes all relevant documents)
Enquiries from staff at HEIs should be directed to
their nominated institutional contact
(see www.ref.ac.uk for a list)
Other enquiries to info@ref.ac.uk