Talk given at Sydney University on 4 August 2015. Across many parts of our lives we are faced with the increasing availability of data to support decision making. With every element of the research process moving online, there are many new sources of data, as well as improved old sources of data, that can provide information on the performance, value and use of research and researchers. But there is a problem. The proliferation of proxy data, and their naive equation with such weakly defined concepts as “quality” and “excellence”, have lead to a reliance on rankings and quantitative measures as institutional targets. More than this the adoption of these instrumental targets has lead us away from a critical discussion of institutional values, indeed of what the institution is for. I will argue that it is only by moving away from such vague terms as “quality”, “excellence” and “impact” and focussing on institutional values and a well articulated mission that institutions of scholarship will continue to be relevant for the future. It is through interrogating the goals of the institution that the enormous potential resource of data on the research enterprise can be realised. Using the data effectively will allow us a window on how knowledge actually moves and is used. In combination with a clear sense of institutional goals this provides great opportunities for institutions to differentiate themselves from the pack.

1. No Evidence without Stories,
No Stories Without Evidence
Responsible use of metrics in research assessment
1
Cameron Neylon - 4 August 2015 - University of Sydney
@cameronneylon - http://orcid.org/0000-0002-0068-716X

2. A QUESTION OF VALUES
2Nichole Burrows: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY

3. 3https://www.flickr.com/photos/schnappi/6116266037 CC BY

4. 4
MissionMissionValues

5. Mission
5
AssessmentAssessment

6. Mission
6
AssessmentAssessment

7. 7
http://www.flickr.com/photos/drachmann/327122302 CC-BY-SA

8. More data is the answer!
8https://www.flickr.com/photos/blprnt/3261815454 CC BY

9. 9
DATA AND MEASUREMENT
Scott Akerman: http://www.flickr.com/photos/sterlic/4299631538 CC BY

10. 10http://www.flickr.com/photos/verzo/8020565592/ CC BY

11. 11http://www.flickr.com/photos/andrein/372192048/ CC BY-SA

12. More data is the answer?
12https://www.flickr.com/photos/blprnt/3261815454 CC BY

13. Types of Data
•Citations
•Usage
•Bookmarks
•Conversation

14. Citations
•Traditional and valued
•Less consistent than most think
– WoS, Scopus, CrossRef,
(Europe)PMC
•Track a very specific kind of usage
•Only track a very specific kind of

15. Usage
•Page views, downloads
•Measure usage by a wider
community
•Measure greater range of use
•Inconsistent between publishers
•Rarely get demographics of viewers

16. Bookmarks
•Academic and “public” facing sites
– People collecting literature
– Building a library
•Measure greater range of use vs cites
•Focus on use by researchers
(Mendeley)

17. Conversations/Social Media
•Twitter, Facebook, Blogs, News sites
– Sharing literature within
communities
– Conversations about literature
•Open to the public (but often
actually driven by/involving
researchers)

18. A story…
18

19. 19altmetric.com

20. 20

21. 21

22. 22

23. 0 20 40 60 80
0
5000
10000
15000
PLOS papers authored by University of Cape Town staff
Usage data from the PLoS journal website (combined HTML views and PDF downloads). Circle size correlates with CrossRef
citation counts. Red circles represent papers with "HIV" in title.
Months
Total Views
Not an obvious outlier...
23

24. ...but interesting
interactions...
24

25. 25

26. …driven by this
person...
26

27. ...who I can talk to.
27

28. 28

29. 29

30. 30See http://bit.ly/NJOyUn for the whole conversation

31. This. Is Impact.
31

32. 32
DISSECTING IMPACT
Steve Evans: https://www.flickr.com/photos/babasteve/3140688308 CC BY

33. Impact
33

34. Impact
34
Research
Economic
Cultural
Education
Health
Environment

35. Impact
35
Research
Economic
Cultural
Education
Health
Environment

36. 36
Research
Economic
Cultural
Education
Health
Environment

37. 37
Research
Economic
Cultural
Education
Health
Environment
Research
Outputs

38. 38
Research
Economic
Cultural
Education
Health
Environment
Research
Outputs

39. 39
Research
Economic
Cultural
Education
Health
Environment
Research
Outputs

40. 40
Research
Research
Outputs

41. 41
Research
Research
Outputs

42. 42
Research

43. 43
Research
Citation

44. 44
Research
Bookmark
Citation

45. 45
Research
Bookmark
Citation
Previous
co-authorship

46. 46
Research
Bookmark
Citation
Previous
co-authorship
Conference
co-attendance
Social media
conversation

47. 47

48. 48

49. 49

50. 50

51. 51

52. 52

53. 53

54. 54

55. 55

56. 56

57. 57https://www.flickr.com/photos/see-through-the-eye-of-g/5392290809 CC BY

58. 58https://www.flickr.com/photos/jenny-pics/3239638494 CC BY

59. 59https://www.flickr.com/photos/garycycles2/2575676610 CC BY

60. 60https://www.flickr.com/photos/53487196@N08/5439253136 CC BY

61. 61https://www.flickr.com/photos/jenny-pics/3239638494 CC BY

62. 62

63. “Stories that persuade with data”
Anita de Waard
63

64. “Stories that persuade with data”
Anita de Waard
64
evidence

65. TELLING A GOOD STORY
65Magenta Rose: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY

66. 66

67. 67m01229: https://www.flickr.com/photos/39908901@N06/8538482488 CC BY

68. 68Martin Howard: https://www.flickr.com/photos/martinhoward/2709364519 CC BY

69. 69

70. 70
http://ivory.idyll.org/blog/tag/skepticism.html

71. 12 13 14 15 16
0
5000
10000
15000
Usage vs. Citations for Harvard Authors 01−04/12
220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates
with number of CrossRef citations, and color with PLOS journal. Data collected April 11, 2013.
Age in Months
Total Views
71

72. 12 13 14 15 16
0
5000
10000
15000
Usage vs. Mendeley for Harvard Authors 01−04/12
220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates
with number of Mendeley bookmarks, and color with PLOS journal. Data collected April 11, 2013.
Age in Months
Total Views
A Collaboratively-Derived Science-Policy
Research Agenda
72

73. 12 13 14 15 16
0
5000
10000
15000
Usage vs. Facebook for Harvard Authors 01−04/12
220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates
with number of Facebook activity, and color with PLOS journal. Data collected April 11, 2013.
Age in Months
Total Views
73

74. 12 13 14 15 16
0
5000
10000
15000
Usage vs. Wikipedia for Harvard Authors 01−04/12
220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates
with number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013.
Age in Months
Total Views
74

75. 12 13 14 15 16
0
5000
10000
15000
Usage vs. Wikipedia for Harvard Authors 01−04/12
220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates
with number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013.
Age in Months
Total Views
Afghanistan's Ethnic Groups Share a Y-Chromosomal
Heritage Structured by Historical Events
75

76. 76Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY

77. 77Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY
There are four pillars to our research vision:
•an unstinting commitment to research
excellence
•a willingness to harness that excellence to
address some of the vital national and global
challenges of our time
•a desire to engage with the communities for
whom our research has real meaning and
consequences and from whom we can learn –
whether in the private or public sector, local or
global
•new initiatives to develop, nurture and support
our researchers; that is, to develop not only
brilliant minds, but also research leaders.

78. “Qualities” not “Quality”
78

79. 79
Academic quality is highly context - specific, and it is
sensible to think in terms of research qualities, rather
than striving for a single definition or measure of quality.
The central problem identified here is that academic
quality is a complex notion that cannot easily be
reduced to quantification – the use of proxy variables
runs the risk of misrepresenting the qualities of research
contributions and may lead to unintended
consequences.
As PLOS noted in its submission to this review, “it is
unclear whether any unique quality of research
influence or impact is sufficiently general to be
measured”.
Wilsdon et al. (2015) - The Metric Tide

80. 80
MissionMissionValues

81. RESPONSIBILITY USE OF METRICS
81Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY

82. RESPONSIBILITY USE OF METRICS
82Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY
indicators

83. Develop the stories…
83https://www.flickr.com/photos/jblmmwr/8027451065 CC BY

84. …into a shared mission
84Daniel Spiess: https://www.flickr.com/photos/deegephotos/5984387881 CC BY-SA

85. Get informed…
85

86. Sign DORA
86

87. …and mean it
87

88. Above all…
88

89. 89Joe Flintham: https://www.flickr.com/photos/joeflintham/5631590491 CC BY-SA

90. It’s just good scholarship…
90

91. 91

92. 92https://www.flickr.com/photos/ell-r-brown/4655401891 CC BY

93. @cameronneylon
cn@cameronneylon.net
http://orcid.org/0000-0002-0068-716X
93