Presentation slides from the panel 'Trust in Metrics' at AoIR 2019 in Brisbane, Australia.

1. @qutdmrc
AoIR 19, Brisbane, Australia, 3 October 2019 | @kimosman
Kim Osman
Do scholars trust metrics as an indicator of research
quality?
The Rise of Altmetrics in Response to the Engagement and
Impact Agenda

2. Amplifying Public Value: Scholarly
Contributions’ Impact on Public
Debate

3. @qutdmrc
● Engagement & Impact Assessment
Trial
● Impact beyond academic and industry
circles (we are tax-payer funded!)
● Visibility of scholars in public debate
● Take-up in public policy making at all
levels of government
● Trusted voices and opinions
Public Value of Research
@qutdmrc

4. @qutdmrc
Key terms
● ‘Metrics’ are compromised of bibliometrics (citations, h-index) and
altmetrics.
● ‘Altmetrics’ (Priem et al., 2010) are the metrics generated through
public communication on amplifier platforms and other digital media.
They include measures such as reads, views, listens, shares,
republications and comments.
● ‘Amplifier platforms’ are those digital media sites specifically designed
to translate and publicly disseminate research to a general audience –
sites like The Conversation, Medium.com, and perhaps even Wikipedia
(Osman & Cunningham 2019).

5. @qutdmrc
Engagement & impact on digital media
‘Sharing science and research content on social media also rivals sharing content
about health and fitness as well as entertainment and celebrity news.’ Hargittai,
Füchslin and Schäfer (2018, p7)
Scholarly contributions in crowded social media spaces matter.

6. @qutdmrc
Trust in altmetrics?
● Altmetrics have a long history as scholars sought to engage in public
communication outside the tightly controlled spheres of mainstream media
and academic publishing.
● Existing literature suggests that altmetrics can generate a larger degree of
trust than traditional citation metrics.
The online, public nature of these tools exposes and reifies scholarly
processes once hidden and ephemeral. Metrics based on these activities
could inform broader, faster measures of impact, complementing
traditional citation metrics. (Priem, Piwowar, & Hemminger, 2012).

7. @qutdmrc
Altmetrics and scholars
● Our qualitative research elicits rich stories from our interviewees and is
important to contextualise the “move towards digital monitoring and
quantification [that] has significant implications for shaping academic
work practices and identities.” (Lupton, Mewburn & Thomson, 2018)
● Metrics as “automated processes of persona production” (Marshall,
Barbour & Moore, 2018)
What is the role of metrics in shaping scholars’ online public
communication?
How do scholars understand impact, and what drives them to
engage with the public on issues related to their research?
@qutdmrc

8. Contributor Interviews

9. @qutdmrc
Interviews
28 interviews
● 20 Australian and 8 Canadian scholars
● 17 high performing authors (based on available The Conversation
metrics)
● 7 early career scholars
● 4 communication professionals
● 3 non-users

10. @qutdmrc
Female Male Non-conforming

11. @qutdmrc
‘I don’t believe in all honesty that academia in terms of the
current model has any influence on practice at all.
Because I don’t think the people really bother reading our
papers and when they do it’s great, but we produce a
massive piece of work and put it into a couple of articles and
the publish them academically. You’re missing out on a real-
world audience.
So I automatically go to The Conversation as an outlet for
publishing or at least alerting people outside of the academy.’

12. Canada

13. @qutdmrc
The value of altmetrics (Canada)
‘From my institution? No…. But maybe for potential funders. We have a
big grant from [a venture capitalist] for a new centre based on our work.
We do, I guess, use The Conversation to connect to people outside of
academia with our work. Now the university is getting together a press
release for this grant, it’s big…. So maybe indirectly I get recognition.’
‘I don’t trust that the metrics from The Conversation will help my career at
all. I’m teaching intensive with tenure. I don’t write because I need to…
well I do need to. I need to get the message out to parents. I see it as an
extension of my teaching.’

14. Australia

15. @qutdmrc
An ethical imperative to engage
‘I just use those platforms because they’re useful to get the message
out. But it’s actually trying to get out a message that’s important in terms
of society as opposed to well, will this help me with my academic career
and will I be on the telly and be a celebrity? It’s genuinely about “this is
appalling”, if you read the report it’s genuinely appalling .... And it’s a good
way of getting good academic work out into the light.’

16. @qutdmrc
The value of altmetrics (Australia)
‘the obvious thing from that is … straight eyeballs or clicks. But you know that gives
you no quality indicator. However, arguably some of the more traditional measures,
your citation rates and so on, aren’t necessarily quality indicators, either.’
‘I mean everything seems to get driven by these inane… I mean we have pro Vice
Chancellors whose entire job is calculating, you know, counting beans and dividing by
the number you first thought of and that stuff.’
‘They [altmetrics] don't actually count for anything often and you often don't even
bother to write them down anywhere because they’re not quantifiable in terms of the
formal way in which research dissemination is mapped.’

17. @qutdmrc
Value and trust is increasing
‘Although you know to say there’s no formal recognition for it, it is brought
to the attention of Heads of School and pro-Vice Chancellors and people
up the line, so that’s useful. For me the main value I see in it for them is
just the ability to talk to normal people. To get out of the science-y
mindset. And as I say, I always say, “I think this is going to help your
writing”.’
‘Look one hundred percent sure altmetrics count, no. But as the years go
on, especially in the last few years, I'm increasingly confident they will.’

18. @qutdmrc
On openness, reach and republication
‘I’ve had contact from the Wall Street Journal, The New Scientist saying
they saw the stuff in The Conversation. Interview on such and such, you
get a lot of contacts like that. A lot of contacts from other researchers who
have seen it and republished it, and a lot of contacts not always favourable,
from the general public.’ [Laughs]
‘I mean who would have thought Mamamia would publish a long essay by an
academic …? But they do. And those sorts of audiences are really important, I
think, for academics.’
‘So sometimes you do get a much wider reach than you think.’

19. @qutdmrc
Altmetrics can evidence public engagement
‘Looking at the matrix of The Conversation to see where it was going
was really interesting. To see how many reads it was getting, it was
amazing.’
‘Because to me, you know, it takes me a couple of hours to write a
Conversation article and I get hundreds of thousands of reads and
lots of comments. It takes me six months to a year to write a [journal
article] and approximately 200 people will cite it. So … if I want my
opinion to go further there’s no question where I should put my effort.’

20. Conclusion

21. @qutdmrc
Metrics & the engagement and impact agenda
● Metrics need to be augmented with narrative and qualitative indicators
of research quality, there was some trust in the ability of altmetrics to
better reflect public engagement with scholars’ work over and above
traditional metrics.
● There are implications for how metrics shape Australian scholars work
practices.
● Scholars demonstrate a moral imperative to engage, and institutional
support is necessary for scholars to increase engagement and impact
using amplifier platforms.

22. @qutdmrc
References
Hargittai, E., Füchslin, T., & Schäfer, M. S. (2018). How Do Young Adults Engage With Science and Research on
Social Media? Some Preliminary Findings and an Agenda for Future Research. Social Media + Society, 4(3),
205630511879772. https://doi.org/10.1177/2056305118797720
Lupton, D. (Ed.), Mewburn, I. (Ed.), Thomson, P. (Ed.). (2018). The Digital Academic. London: Routledge.
Marshall, P. D., Barbour, K., & Moore, C. (2018) Academic persona: The construction of online reputation in the
modern academy in Lupton, D., Mewburn, I., Thomson, P. (Eds.). (2018). The Digital Academic. London:
Routledge.
Osman, K. & Cunningham, S. (2019). Engagement and impact through ‘amplifier platforms’, Australian Universities
Review, 61(2), pp 42-48.
Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the Wild: Using Social Media to Explore
Scholarly Impact. Retrieved February 22, 2019, from https://arxiv.org/html/1203.4745?
Priem, J., Taraborelli, D., Groth, P. & Neylon, C. (2010). Altmetrics: a manifesto. Retrieved from
http://altmetrics.org/manifesto/
Image Credits:
https://dataportal.arc.gov.au/Landing
www.analytics.theconversation.com
vectorstock.com

23. Thank you!
@kimosman
@qutdmrc