Keynote 4 - Analysis of RAE 2008 and Prospects for REF - Jonathan Adams, Evidence, Thomson Reuters

1. EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009

3. Indicators vs profiles

4. Decision support, not substitution

5. What variables are relevant in ...

6. Academic

7. Economic

8. Policy

9. Social ... Impact

12. Alberta journal of educational research

13. American journal of education

14. Asia Pacific journal of education

15. Australian journal of early childhood

16. British educational research journal

17. Chinese education and society

18. DidacticaSlovenica - pedagoskaobzorja

19. Durham and Newcastle research review

20. Egitimvebilim - education and science

23. Steady but volatile improvement

24. What is used to represent excellent research outcomes?

25. RAE2008 grade spread

26. Previously among studies by Evidence HEFCs/UUK – Maintaining Research Excellence Recognition of a peak supported by a research platform Evolution of 3 to 4 to 5 grade units HEFCE – Role of QR funding Strategic significance of the block grant Universities UK – Excellence and Diversity Regional network of research competence Importance of an evidence base for research policy OST/DIUS/BIS Diversity as a critical element in research policy

27. RAE2008 – a novel outcome with dispersed rewards Gareth Roberts’ proposals on research profiling Problem of the ‘cliff edge’ Recognising dynamic excellence in the research base Unexpected differences between panels The problem of interpreting methodology and standards The absence of a commonly understood standard? The dispersed map of reward-able research Decisions about funding allocations 2* = 1 (x 3 to) 3* = 3 (x 2.33 to) 4* = 7 Greater relative rewards for modest gains than for very expensive world-class excellence (cf 2001)

28. How can we explore the spread of excellence? We have Panel outcomes from RAE2008 – but opinions differ, so ... Bibliometric data on a consistent basis from 1981 We split the HE sector into crude groups All HEIs Russell Group 1994 Group Universities with a shorter history of research investment And we created an elite ‘golden triangle’ band Oxford (C12th), Cambridge (1209), UCL (1836), Imperial (C19th) , LSE (1895)

29. Excellence is linked to selectivity

30. And excellence is concentrated

31. The problem with simplistic indicators They don’t really express the complexity of research performance Average impact (e.g. ‘crown indicator’) can be very misleading Research Council studies reveal error of interpretation Skewed data, median much smaller than average Lots of papers are not cited The interesting bit is about how much is really, really cited lots So we we prefer Impact Profiles®

32. UK background and ‘golden triangle’ This is the small but critical excess of really highly cited research output

33. Even smaller differences separate the UK and USA research profiles

34. Excellence extends to other institutions

35. But the differences become very fine

36. And if we distil further, other complexity is revealed

37. Older institutions have no monopoly on quality, but the peak is distinctive For the 1994 Group, there are relatively fewer low-cited papers and relatively more high-cited papers than for many in the Russell Group

38. What does this imply? There is a very concentrated peak of exceptional excellence UK international competitiveness is associated with selectivity that increased concentration There is a regional network of exceptional quality, much in dynamic institutions created in the 1960s The balance that supports diversity & dynamism and the concentrated support of international excellence is very fine Model first, meddle second

39. EVOLVING RESEARCH EVALUATION RAE2008 OUTCOMES AND REF PROSPECTS JONATHAN ADAMS 10 NOVEMBER 2009