The document discusses issues with national and international university league tables. It notes that while league tables provide information for various stakeholders, the criteria and metrics used do not fully capture university quality and can distort behaviors. Both national and international league tables utilize different methodologies and data sources that are often flawed or biased. While league tables are influential, their results should be interpreted carefully given the limitations of ranking complex institutions on simplistic metrics.

1. Pride and prejudices:
Problems with national
and international league
tables
Dr Paul Greatrix, Registrar,
University of Nottingham
1

2. League Tables
• Background
• Who wants to know?
• Who are the providers of this valuable
information?
• Mad, bad and dangerous
• National and international dimensions
• Measuring the unmeasurable?
2

3. Background
They sell papers...
• ...and, some would suggest, create 3
perverse incentives

4. Who wants to know?
• The Government • Parents, teachers,
• The State advisors
• Employers – national and
• The Funding Councils
international
• Potential Students • Overseas sponsors
• Alumni • Jo Public
• Journalists • The Universities
• Dangerous obsessives
4

5. Who’s responsible for providing this
valuable data?
• The Times • Shanghai Jiao Tong
University
• Sunday Times
• QS
• The Guardian
• US News and World Report
• Complete University
Guide • HEFCE
• Others...
• Financial Times
• and, of course,
• THE
Government
5

6. Invaluable information
• For all stakeholders
• Intelligent decision-making
• Better than prejudice
• Reflecting the realities of
the market place
• We have a right to know
• Employers have the right to
know
6
• We aren’t stupid

7. League tables are a bad thing...
“The silly season that marks the publication of
University league tables is nonsensical and illogical.
As any New Scientist knows, letters into numbers;
quality into quantity won’t go. League tables are
simplistic, divisive and undermine the qualitative
nature of a University’s work”
7

8. But we’ll use them anyway!
“Having said that, I’m not
ashamed to report that we came a
very creditable 79th overall, with
my own department rating a
particularly good score for
research
- and as I remarked to the Dean,
you can’t get much better than
that.” 8

9. Dangerous...
• Criteria used do not reflect quality of education
• Historical data
• Variation over time
• Scores are institutional averages – mask strengths
• Distorting effect of weightings and scalings and
data manipulation
• Many of criteria used are inter-related
(ie not independent)
9

10. ... extremely dangerous...
• Apples and elephants and
paperclips
• Perverse incentives
• Hugely political
• Open to manipulation
• Delivered by journalists
• Spurious precision
• Serious consequences for
universities, departments,
staff and students 10

11. The Times 2011
1 Oxford 11 York
2 Cambridge 12 Bath
3 LSE 13 Bristol
4 Imperial 14 Sussex
5 UCL 15 Edinburgh
6 St Andrews 16 Nottingham
6 Durham 17= Leicester
7 UCL 17= Sheffield
8 Warwick 19 Southampton
9 Lancaster 20 Loughborough
10 Exeter 11

12. Sunday Times 2011
1 Cambridge 11 Loughborough
2 Oxford 12 Newcastle
3 Durham 13 Sheffield
4 LSE 14 Imperial
5 Bath 15= Nottingham
6 St Andrews 15= York
7 UCL 17 King’s
8 Warwick 18 Lancaster
9 Exeter 19 Sussex
10 Bristol 20 Glasgow
12

13. The Guardian 2011
1 Cambridge 11= Exeter
2 Oxford 11= Sussex
3 St Andrews 13 SOAS
4 LSE 14 Bath
5 UCL 15 York
6 Warwick 16 Edinburgh
7 Lancaster 17 Leicester
8 Durham 18 UEA
9 Loughborough 19= Nottingham
10 Imperial 19= Surrey
13

14. Complete University Guide 2011
1 Cambridge 11 Bristol
2 Oxford 12 York
3 Imperial 13 Edinburgh
4 LSE 14 Southampton
5 Durham 15 Exeter
6 St Andrews 16 King’s
7 UCL 17 Nottingham
8 Warwick 18 SOAS
9 Lancaster 19= Loughborough
10 Bath 19= Sussex
Wednesday, 4 April 12 14

15. International value-added
• HE is now a global business
• Global branding assisted by competitive
ranking
• International benchmarking increasingly
important, especially in research
• Student recruitment is increasingly
international...
• ...mobile students are increasingly
choosy
• It’s all good healthy fun 15

16. Just as dangerous...
• Again, the criteria used do not reflect
quality of education
• Archaic and irrelevant data
• Major biases to large, English-
speaking, research- and science-
intensive universities
• Institutional scores are extraordinarily
broad brush
• Distorting effect of weightings,
scalings and data manipulation
• Far from comprehensive surveys of
peers and employers
16

17. Times Higher Education World Rankings 2011
4 Oxford 48 Manchester
6 Cambridge 56 King’s
8 Imperial College 66 Bristol
17 University College 83 Durham
London 85 St Andrews
36 Edinburgh 99 Sussex
47 LSE
17

18. QS World Rankings 2011
1 Cambridge (1) 64 LSE (80)
5 Oxford (6) 67 Birmingham (59)
6 Imperial (7) 72 Sheffield (69)
7 UCL (4) 74 Nottingham (73)
20 Edinburgh (22) 75 Southampton (81)
27 King’s (21) 93 Leeds (85)
29 Manchester (30) 95 Durham (92)
30 Bristol (27) 96 York (88)
50 Warwick (53) 97 St Andrews (95)
59 Glasgow (77)
18

19. Shanghai Jiao Tong ARWU 2011
5 Cambridge (5) 53 Edinburgh (54)
10 Oxford (10) 68 King’s (63)
20 UCL (21) 70 Bristol (66)
24 Imperial (26) 85 Nottingham (84)
38 Manchester (44) 97 Sheffield (88)
19

20. Another approach:
Ecole des Mines de Paris Rankings
1 Harvard 11 U of Pennsylvania
2 Tokyo University 12 Columbia
3 Keio University 13= Stanford
13= Tohoku U
4 HEC (France)
13= Nottingham
5= Kyoto
16 MIT
5= Oxford
17 Institute for Study of
7 Ecole Polytechnique Politics, Paris
8 Waseda 18 U St Gallen
9 ENA (France) 19 U Sao Paolo
10 Seoul National University 19 Northwestern U 20

21. Another Approach: Webometrics
1 Massachusetts Institute of Technology
2 Stanford University
3 Harvard University
4 University of California Berkeley
5 Cornell University
6 University of Michigan
7 California Institute of Technology
8 University of Minnesota
9 University of Illinois Urbana Champaign
10 University of Texas Austin
21

22. Webometrics Europe
1 University of Cambridge, World Rank: 28
2 Swiss Federal Institute of Technology WR: 40
3 University of Oxford WR: 42
4 University of Helsinki WR: 58
5 University of Oslo WR: 60
6 University of Edinburgh WR: 65
7 University College London WR: 68
8 Utrecht University WR: 71
9 University of Manchester WR: 82
10 Royal Institute of Technology (Sweden) WR: 84
22

23. International approaches 1
Shanghai Jiao Tong: Ranking of World Universities
– 6 indicators covering: quality of education;
quality of faculty; research output;
performance relative to size
– Includes: Nobel and Fields winners among
alumni and staff; highly cited researchers;
articles in Nature and Science; articles in
citation indices.
23

24. International approaches 2
24

25. Other tables
• ALOHA, PISA, OECD
• China – number of billionaires
• Just research performance: ranking of
scientific papers
• Most conservative, best parties, most interns
• European initiative – U-Multirank
• Islamic universities
25

26. .
Guess the
League
Table!

27. Which league table?
1) Loughborough University
2) University of Sheffield
3) University of East Anglia
4) University of Cambridge
5) University of Dundee
6) University of Oxford
7) University of Glasgow
8) University of Leeds
9) Aberystwyth University
10) University of Southampton

28. Which league table?
1) Leeds Metropolitan University 26.72
2) University of Liverpool 26.63
3) University of Wales Institute, Cardiff 26.14
4) Bath Spa University 24.95
5) University of Glamorgan 22.26
6) University of Bath 21.97
7) University of Brighton 21.98
8) Buckinghamshire New University 21.59
9) Newcastle University 21.2
10) De Montfort University 21.0

29. Which league table?
1 University of Glamorgan 10.9
2 University of Wales Institute, Cardiff 10.6
3 University of Brighton 9.5
4 Leeds Metropolitan University 9.4
5 Aberystwyth University 9.1
6 Manchester Metropolitan University 8.4
7 University of Dundee 8.2
8 University of Liverpool 8.1
9 University of Strathclyde 7.7
10 University College London 7.6
29

30. Which league table?
1 University of Nottingham, UK
2 Northeastern University, US
3 University of Connecticut, US
4 University College Cork, Ireland
5 Linkoping University, Sweden
6 University of California, Berkeley, US
7 University of California, Los Angeles, US
8 Washington University In St. Louis, US
9 University of California Merced, US
10 University of Bath, UK
30

31. Which league table?
1 Greenwich
2 Sheffield Hallam
3 Kingston
4 Westminster
5 East London
6 Central Lancashire
7 Leeds Met
8 Wolverhampton
9 Coventry
10 Middlesex
31

32. Which league table?
1 Webb Institute 70.9
2 Carleton College 61.3
3 Princeton University 60.3
4 Middlebury 60.1
5 Amherst 59.5
6 Williams College 57.6
7 Centre College 56.7
8 Indiana Institute of Technology 55.1
9 Davidson College 54.9
10 Thomas Aquinas College 52.5
32

33. Which league table?
1. Oxford 39
2. Cambridge 21
3. Manchester 8
4=. Imperial College 5
4=. Open University 5
6=. Durham 4
6=. Sussex 4
8=. St. Andrews 3
8=. Birkbeck 3
33

34. Which league table?
1) Nottingham Trent University
2) University of Gloucestershire
3) University of Worcester
4) University of Plymouth
5) Bournemouth University
6) University of Greenwich
7) Bath Spa University
8) University of East Anglia
9) University of Hertfordshire
10) Leeds Metropolitan

35. An indicator too far
(not covered in the UK tables - yet)
• Alumni giving
• Academic staff pay
• Research income
• Citations
• Brand impact
But international tables will increasingly influence
methodologies of UK tables
35

36. What is to be done
• Publish prolifically and get top RAE grades
• Be Highly Cited
• Get articles in Nature and Science
• Win a Nobel Prize or a Fields Medal
• Make sure your students all get top jobs with big multinationals (and
then win Nobels)
• Plug the institution relentlessly and cultivate peers, Headmasters/
mistresses and employers
• Improve NSS results (and SSRs) - every little helps
• Spend more on Library and IT and everything else to do with teaching
and learning
• Recruit more international students and staff (all with Nobel potential)
36

37. Conclusions
• They aren’t going to go away
• The international dimension will become increasingly
significant
• Methodologies - for both national and international tables
- are all dubious, at best
• They can and will be used by many different groups – but
can be dangerous in the wrong hands
• Handle with great care!
37

38. Finally
For a copy of the presentation and further league table
commentary and observations, visit:
http://registrarism.wordpress.com/
Also on Twitter: @registrarism
38