Demonstrating knowledge mobilization and accountability are increasingly prominent features of the scholarly landscape; scholar-practitioners need to understand and strategically manage available indicators of impact. At the same time, traditional scholarly metrics and indexing are converging with social media, resulting in new approaches for measuring scholar-practitioner influence. The emerging scene challenges libraries to support scholars, practitioners and students to engage with an evolving environment in which much may be gained or forfeited depending on how reputation is curated. For librarians to assist scholars in this new altmetrics environment, more needs to be known about how students and faculty are or are not engaging with emerging tools available to them. This presentation gives an overview of the considerations, perceptions, and issues related to the use of altmetrics by graduate students and scholar-practitioners at VIU and Royal Roads University.
4. Social Media + Traditional
Metrics
Altmetrics
New metrics based
on the Social Web for
analyzing, and
informing
scholarship
(altmetrics.org)
+ =
Creative Commons Flickr users: macloo, Wiley Asia Blog
5. Altmetrics Tools
• http://scholar.google.ca
• http://www.mendeley.com
• http://impactstory.org
• http://www.plumanalytics.com
• http://orcid.org
• http://altmetric.com
• http://article-level-metrics.plos.org/alt-metrics/
• http://citedin.org/
• http://klout.com/home
• http://www.carboncapturereport.org/
• http://sciencecard.org/ (“this is not the end…”)
• Reader Meter
10. What Altmetrics Aren’t:
Be all and end all of impact measurement
Useful to compare between individuals
Throwing out traditional metrics
11. Why VIU/RRU?
Applied scholarship
Professional grad programs
Faculty role, teaching and scholarship –
non traditional
Faculty who apply scholarship to
practice in the community (knowledge mobilization)
Library engagement with emerging models of scholarly
communication
12. Research Questions
What issues do scholar-practitioners & grad students face
when trying to establish, grow, & measure scholarly
presence on the web?
How do scholar-practitioners & grad students perceive &
negotiate issues related to scholarly presence on the web?
How can librarians assist scholar-practitioners to create,
discover & manage online reputation using traditional &
emerging tools?
13. Methods
22 participants
45 min interviews with each
participant
Faculty were selected via
VIU’s Annual Report on
Research & Scholarly
Activity, presence in
altmetrics tools
Students were referred by
interested faculty
14. Issues & Themes
Factors that influence participation/ interest
Perceived benefits/drawbacks of tools
How to make decisions about degree/nature of sharing
Discipline-related themes
Library role/value added
15. “I’ve Searched For Myself in
Google...”
All participants had at least run their names through Google for
professional purposes
Some already knew of Google Scholar profiles, some didn’t. Some
very strategic.
General awareness of impact in relation to the concept of journal
impact factor
Different disciplines favour different tools and services
16. How Participants Saw
Themselves
Stage of career
Field of scholarly activity
Value that institution may/not place on research & publishing in
faculty role
In/formality of institutional requirements around research in
promotion
Relative emphasis on research, teaching, service
Dependence on traditional impact measures to obtain, retain funding
17. How Participant Approach Publishing
Disciplinary dependent
Publishing as formula, game, or ritual hazing
Idea of disseminating research beyond academe –
knowledge mobilization & community impact
Accountability to institution, profession, funder
Knowledge of, interest in Open Access initiatives
18. Engagement with Tools
Awareness of tools
Time
Time (again)
Which tools to use for what purpose
Stage of development of those tools
How the tools are viewed within their field
Advantages and disadvantages of engaging with or reporting out
results from social media
Creative Commons Flickr user: *casserpillar*
19. Participant Response
“I would book an appointment tomorrow”
“I think I got more out of this than you did”
Some tools more mature than others
Cool factor
Skepticism
How to make comparisons across disciplines
Caution about being reduced to just numbers
Promise of measuring value they’ve not been able to capture
20. Managing Scholarly Profile
Broad awareness of strategies, tools complements field-
specific understanding that scholars may have
Guides, self help
Workshops – department, faculty
Integration with research methods grad curriculum
Custom consultation
I want all of you who are doing research to think about yourselves as the centerpoints here - and that your research is moving outwards in waves into society. As scholar-practitioners, we all want to make waves - or at least ripples - in our fields.
I’m working with my librarian faculty colleague Dana McFarland and Rosie Croft, University Librarian at RR, to explore how scholar-practitioners make waves online, how we can measure how big the waves are, how far those waves reach, and how they interact with one another.
Before I get into our research project, I want to set the scene around how influence of scholarly-practitioner work has been measured in the past and the environment we’re currently in to give you some context.
Right now there’s a push for...
Taxpayer accountability, Knowledge Mobilization = universities pressured to demonstrate ROI in economic and societal terms,
trying to get at community impact/relevance piece and measure it.
What impact is work having on diverse audiences including scholars, practitioners, clinicians, educators & general public???
At the same time, technology has improved...
Speed of technology - we have means of instantaneous distribution (vs. printing) and the technology to capture impact immediately - days vs. years
Products are digital - datasets, software, photos, videos, blog posts
WoK
conditions led to need for tools to be created that could show a more nuanced understanding of impact than currently available via journal rankings and citation counts.
We needed something that would show us which scholarly products are read, discussed, saved and recommended as well as cited.
SocMed developed and combined with traditional journal metrics (journal impact rank, article citation counts) to form altmetrics (alternative metrics)
“Altmetrics" only dates from 2010, when Jason Priem, doctoral candidate at the School of Information and Library Science at the University of North Carolina at Chapel Hill, first used it in, fittingly, a tweet.
Led to an influential manifesto written by Mr. Priem and three other researchers.
Looking at first 4
-notable that some, like reader meter, die
-discipline themed too
-proprietary systems
-Interest in nature of the broader index – conference proceedings
-Interested to see co-author/colleague profiles
-Interest in automatic nomination of content/publications offered by scholar profile
-Publications they had no idea were being tracked or were even had a record of any kind
-Interest in item level reader analytics, novelty of this as an alternative measure to the citation
-Reticence among scientists to see Mendeley as a real contender with ISI/endnote
-One researcher still types out all his citations manually and never used a citation manager
-value of cross platform aggregation perceived
-inclusion of both scholarly and social metrics and utility in demonstrating knowledge mobilization
-questions about reliability of the data, normalization
Be all and end all of impact measurement - can be gamed, aren’t comprehensiveNot useful to compare between individuals - instead use a story to tell of ways in which your work is being used in unexpected places
We were intrigued by early discussions of altmetrics at research-focused institutions.
It occurred to us that these indicators could have as much to offer in teaching and practice-focused settings – if not more. Traditional impact measures have been exclusive in nature, and less relevant to applied, practical, emerging and interdisciplinary fields. Between those and SEO there has been a gaping hole.
This slide notes characteristics shared by VIU and RRU that we thought might predispose our scholarly communities to benefit particularly from the promise of altmetrics – use/impact/value beyond the academy, emphasis on application and practice, non-traditional approaches
3 main research questions:
What issues do scholar-practitioners, grad students face when trying to establish, grow, measure scholarly presence on the web?
How do scholar-practitioners, grad students perceive, negotiate issues scholarly presence on the web?
How can librarians assist scholar-practitioners to create, discover & manage online reputation using traditional & emerging tools?
-22 semi-structured interviews (half from RRU, half from VIU), 5 interviews with grad students
45 min interviews with each participant
Faculty were selected via VIU’s Annual Report on Research & Scholarly Activity, presence in altmetrics toolsStudents were referred by interested faculty
-results of the project represent a snapshot of our faculty and student interest in these services
We asked participants about the following issues and themes:
Factors that influence participation/ interest in online tools to grow & measure scholar-practitioner influence
Perceived benefits/drawbacks of tools
How to make decisions about degree/nature of sharing
Discipline-related themes
Library role/value added
Getting into what we found… we started out trying to get a sense of participants’ general ideas and practices around tech tools and citation analysis.
All participants had at least run their names through Google for professional purposes
Some already knew of Google Scholar profiles, some didn’t. Some very strategic.
General awareness of impact in relation to the concept of journal impact factor
-not so clear on what an h-index is
-Different disciplines favour different tools and services
environmental scientist, no idea what WOS or scopus is (put people on the spot a bit though)
Where participants had awareness of tools, tended to be narrow and deep – field-specific –
Whether or not participants felt that measuring impact & new altmetrics tools were important or not depended on several variables:
Stage of career:
-saw importance for young/new scholars even if they themselves weren’t interested
-there are those for whom this has no relevance – not going somewhere else, not going for advancement, not researchers
-Accordingly, may try to “assign” activity related to managing profile to juniors
-varied importance by discipline
-Institutional values, requirements for promotion, emphasis of institution
-Need traditional sources for funding
-Those on committees of granting bodies did not think these kinds of measures would be taken into account
- one exception perceives that her tri-council agency already acknowledges and values her dissemination of work in non-traditional ways.
Diverse perspectives on the publication of research emerged:
-highly disciplinary dependent - scientists not surprisingly concerned first and foremost with publishing in high impact journals.
-Predisposition to see role/value of quantitative measures, standards and disambiguating tools, if in certain disciplines (chemists seeing Orcid number analogue to CAS number)
-one researcher referred to the “formula” that gets articles published, easier to be accepted on subsequent submissions, publishing to build resume more than any other reason
-some researchers cared most about their blog/webpage and getting info out that way. Saw it as being far more meaningfully influential
How ppl publish is governed somewhat by accountability to institution, what profession they’re in, & source of funding
Knowledge/interest in OA
Whether or not people engaged with altmetrics/soc media tools and to what extent depended on a number of factors:
Awareness
-part of what we asked about was the interviewees’ use of social media and whether or not they use it to drive dissemination of their research and/or measure dissemination of their research
-very different answers that didn’t seem to fall along generational, cliched digital native/immigrant lines – some people philosophically averse to social media, some completely embrace it, some keep professional and personal divided, some couldn’t imagine why they would divide them
-Many were using the usual in various ways: Linked-in quite popular (generally despised), Facebook somewhat, twitter less so, some passionate bloggers, some failed bloggers. Interest in how to measure value of these efforts meaningfully.
Time
-almost all noted and/or feared the huge amount of time needed to discover and set up profiles, pages, blogs, etc.
Time
-almost all noted and/or feared the huge amount of time needed to keep the info in such services up to date
What tools for what purpose
Stage of development of tools
How tools are viewed within their fields
-fastidious aversion to self-promotion (uber-canadian? Discipline culture?); self-promotion vs research promotion – may be useful approach to create clear distinction between these in social media/networks
-Questions about integrity/quality of the data/feedback from such sources – Facebook, etc traffic isn’t necessarily positive commentary; use of work for purposes not intended (J’s issue); conflicting reports of usage, etc., -- similar may be said of traditional measures – look at the # of citations the “vaccines cause ADD” study got
Advan/Disadvtges of engaging with or reporting out results via social media
-frustration or uncertainty about how metrics may be used or abused: accounts hacked, spammed, used against them in online public spaces, ppl picking up research for own purpose (climate change deniers), used to disadvantage in funding or advancement decisions, particularly in while they are emerging – really more indicators than measures
-Concern about engaging in social media leading to flame wars
-misapprehension by unsophisticated readers regarding the research
outright enthusiasm for being able to present research output in diverse ways to engage with researchers/non-academic audiences directly
-plum analytics particularly impressive – appreciated for powerful visual representation of indicators
-Varied perception of utility/importance in academic settings where metrics may not be key to career progression: even so, perceived use to support evaluation and maintenance of grants, recruitment of students, demonstrate value to community, constituency, funders – internal or external
Our research suggested a number of ways that the library can play a role in assisting scholar-practitioners with the set up and management of scholarly profiles.
Broad awareness of strategies, tools complements field-specific understanding that scholars may haveGuides, self helpWorkshops – department, facultyIntegration with research methods grad curriculumCustom consultation