This workshop is the 3rd in a series of 4 titled "Maximize your impact" offered by the IUPUI University Library Center for Digital Scholarship. Faculty must provide strong evidence of impact in order to achieve promotion and tenure. Having strong evidence in year 5 is made easier by strategic dissemination early in your tenure track. In this hands-on workshop, we will introduce key sources of evidence to support your case, demonstrate strategies for gathering this evidence, and provide a variety of examples. These sources include citation metrics, article level metrics, and altmetrics as indicators of impact to support your narrative of excellence.

1. Gather evidence to demonstrate
the impact of your research
Maximize Your Impact Workshop Series
Heather Coates, MLS, MS
Digital Scholarship & Data Management Librarian
IUPUI University Library Center for Digital Scholarship

2. Overview
Products
Levels of Metrics/Evidence
Types of Metrics/Evidence
Examples
Data Sources & Tools

3. Overview

4. IUPUI
Values
Civic
Engagement
Collaboration
Diversity,
Equity,
Inclusion
Economic
Development
(Indiana)
Interdisciplinary
Work &
Publication
Open Access
Public
Scholars/
Scholarship
R&CA in Urban
Environment
RISE
Translational
Research
https://academicaffairs.iupui.edu/PromotionTenure/IUPUI-Guidelines

6. Why do we need metrics?
• P&T review committees can’t read everything
• P&T review committee members are not experts in your field
• Models for producing scholarship differ by discipline and research
methods used
• Some metrics are decent indicators of scholarly use (e.g., citation)
• Some metrics are decent indicators of public discussion (e.g.,
Twitter, Facebook, news media attention)
• Some metrics are decent indicators of reuse (e.g., data citations)

8. Metrics are only one kind of evidence
How do we measure…
• Relationships developed with practitioners and patients
• Affecting change in how stakeholders see an issue
• The long-term benefits of community partnerships to document
and preserve local history
• Increases in the civic literacy of adults across central Indiana
• Increased understanding of an issue by the state legislature
resulting from ongoing policy discussions

9. Consider
Leiden Manifesto for research metrics:
http://www.leidenmanifesto.org/
San Francisco Declaration on Research Assessment (DORA):
http://www.ascb.org/dora/
“Do not use journal-based metrics, such as Journal Impact Factors (JIFs), as
surrogate measures of the quality of individual research articles, to assess an
individual scientist’s contributions, or in hiring, promotion, or funding
decisions.”

10. Citation-based Metrics
Journal Level
Metrics
Output/Article
Level Metrics
Author Level
Metrics

11. Citation Metrics: Levels of Evidence
• Article/Item-level:
– citation counts, Field Weighted Citation Impact (Scopus),
Eigenfactor Article Influence Score, Relative Citation Ratio (NIH -
https://icite.od.nih.gov/)
• Journal-level:
– Journal Impact Factor, 5-year JIF, Eigenfactor
• Author-level:
– h-index, i-10 index (Google Scholar)

12. Web Metrics
View Use
Social Media
Metrics
Share Recommend Discuss
Altmetrics

13. Altmetrics
• Supplement traditional metrics
• Scope extends beyond the formal published scholarly record
(journals & books)
• Timeframe: immediate to short-term impact
• Sources: focus is online attention, interaction, usage
• DOES capture and link to qualitative data (ex: blog snippets,
tweet content)

14. Altmetrics Sources
• Publishers
• Aggregation services
– Impact Story:
http://impactstory.org/
– Altmetric.com:
http://www.altmetric.com/
– Plum Analytics:
https://plu.mx/g/samples
• Subject repositories
– PubMedCentral (PMC)
– arXiv
– SSRN
• Institutional repositories
– IUPUI ScholarWorks:
http://scholarworks.iupui.edu
– IUPUI DataWorks:
http://dataworks.iupui.edu

15. Aggregator Sources
• News media
• Social media platforms
• Blogs
• Twitter
• Reference managers (e.g., Mendeley)
• other online conversations captured include YouTube,
Wikipedia, Reddit, F1000, Q&A, Policy Docs
https://www.altmetric.com/about-our-data/our-sources/

16. Using altmetrics to demonstrate impact
• Complementary/Supplemental
– More immediate (before citations accrue)
– Indications of impact outside your discipline and academia
– Reflect engagement with and impact upon a broader audience, beyond
academia and community of practice
• Flavors of impact (Priem et al, 2012)
– “popular hit” – highly tweeted and shared on non-academic social media sites
– “expert pick” – good F1000 ratings and subsequent citations, few shares or
social media mentions

17. https://becker.wustl.edu/impact-assessment

18. Impact Indicators
Research
Output &
Activities
Biological materials, collaborations, data, databases, repositories, techniques & procedures, grey
literature, invention disclosures, mobile apps, patents, trainees, etc.
Advancement
of Knowledge
Books/chapters (inclusion in bibliographies, library ownership, textbook use), change in
understanding (paradigm shift, lead to new approach), citations (first & second generation
citations, countries and institutions represented), conference themes, new centers/institutes
Clinical
Implementation
(or TRIP)
Biological materials, study cited in clinical decision aid, clinical/practice guidelines, diagnostic
application, instruments, quality measure guidelines (gov’t or NPO), reporting requirements
Legislation &
Policy
Committee participation, study cited in guidelines, study cited in policy, study cited in enactment
of standards
Economic
Benefit
Findings cited in reduced costs for delivery of healthcare services, findings result in enhancement
of existing resources and expertise, license agreements for use of IP, spinoff or startup company
Community
Benefit
Public awareness of risk factors, patient decision materials, cited in public/private insurance
benefit plan

19. Examples

20. Citation Metrics Altmetrics
Product Title Journal/ Source
Year of
Publication
JIF [Quartile
Rank] Citations
Field-Weighted
Citation Impact
Citation
Benchmarking
Mendeley
reads Twitter Blogs
Dietary inferences from dental occlusal
microwear at mission San Luis de Apalachee
American Journal of
Physical Anthropology 2005 2.1 [Q3] 22 1.51 55th percentile 47 0 0
A comparison of calcium to zoledronic acid for
improvement of cortical bone in an animal
model of CKD
Journal of Bone and
Mineral Research 2014 6.8 [Q1] 16 3.9 86th percentile 16 1 0
Reference-point indentation correlates with
bone toughness assessed using whole-bone
traditional mechanical testing Bone 2013 4.5 [Q1] 48 6.28 97th percentile 53 2 0
In vivo reference point indentation reveals
positive effects of raloxifene on mechanical
properties following 6 months of treatment in
skeletally mature beagle dogs Bone 2013 4.5 [Q1] 27 3.47 94th percentile 19 0 1
Structure and function of platyrrhine caudal
vertebrae Anatomical Record 2010 1.4 [Q3] 23 2.39 84th percentile 31 0 1
Functional correlates of fiber architecture of
the lateral caudal musculature in prehensile
and nonprehensile tails of the platyrrhini
(primates) and procyonidae (carnivora) Anatomical Record 2009 1.5 [Q3] 23 2.04 80th percentile 103 0 1
Contours of the hominoid lateral tibial condyle
with implications for Australopithecus
Journal of Human
Evolution 2006 3.3 [Q2] 24 1.16 60th percentile 40 0 0

21. Product Title
Item
Views
Item
Downloads
Physician Workforce Collection 735 1363
Mental Health Workforce 819 1004
Nursing Workforce 662 1258
Oral Health 577 849
Primary Care Workforce 284 512
Pharmacy 447 432
Full Collection 3545 6019
How is this being
disseminated to local and
statewide communities of
practice?
Qualitative feedback on
these reports

22. Data literacy instructional materials (shared in 2014) Views Downloads Favorites
Email
Shares
Data Management Lab: Session 1 Slides 700 4 2 0
Data Management Lab: Data management plan instructions 376 2 0 0
Data Management Lab: Data mapping exercise instructions 807 4 0 0
Data Management Lab: Data mapping exercise example 5407 5 0 1
Data Management Lab: Session 2 slides 432 2 0 0
Data Management Lab: Session 2 - Documentation Instructions 327 2 0 0
Data Management Lab: Session 3 Slides 645 2 0 0
Data Management Lab: Session 3 Data Review Checklist 402 3 0 0
Data Management Lab: Session 3 Data Entry Best Practices 384 2 0 0
Data Management Lab: Session 3 Data Coding Best Practices 270 2 0 0
Data Management Lab: Session 4 Slides 590 6 0 0
Data Management Lab: Session 4 Review Outline 538 3 0 0
Data Topics Series: Ensuring data quality 550 11 0 1
Data Topics Series: Preventing data loss 461 2 0 0
Data Topics Series: Practical Data Management Plans 400 2 0 0

23. http://opensyllabusproject.org/

24. Chronicling Hoosier
https://katiemartinpapers.wordpress.com/2016/08/07/abstract-analysis-chronicling-hoosier/

25. Top 5 Non-Article Outputs
Title Impact Evidence
Huntington Study Group website Community
Receives more than 2,000 unique visitors
per month.
HSG participant database Community 4300 registered potential participants
Research sites Community 63 research sites in 11 countries
Physician's Guide to the
Management of Huntington
Disease Clinical Implementation/TRIP Downloaded more than 2,400 times
UHDRS Clinical Implementation/TRIP Downloaded more than 13,000 times

26. Data Sources & Tools

27. Sources of Data
• Web of Science
• Journal Citation Reports (JCR)
• Scopus
• Google Scholar Citations
• ImpactStory
• Altmetric Bookmarklet
• Publisher

30. Google Scholar

31. Gathering Data
• Gather manually
– Free
– Time-consuming
– Messy & redundant
– Tailored to your scholarship &
argument
– Include unusual or field-
specific sources
• Use an aggregator
– Minimal cost (individual)
– Less time-consuming
– Generic, broad presentation
– Heavy emphasis on major
social media channels

32. https://impactstory.org/u/0000-0002-4356-3837

33. The problem is not a lack of
data, but how to evaluate,
select, and make sense of it.

34. Using metrics responsibly
• What is your case? Your story?
– What statements do you need to support?
• Present the data in context
• Provide a range of evidence – metrics + qualitative evidence

35. Resources
• Sarli, C. C., Dubinsky, E. K., & Holmes, K. L. (2010). Beyond citation analysis: a model for
assessment of research impact. Journal of the American Medical Library Association.
• Piwowar, H. & Priem, J. (2013). The power of altmetrics on a CV. Bulletin of the Association for
Information Science & Technology. https://www.asis.org/Bulletin/Apr-
13/AprMay13_Piwowar_Priem.html
• Roemer, R. C. & Borchardt, R. (2012). From bibliometrics to altmetrics: A changing scholarly
landscape. College and Research Libraries News, 73(10), 596-600.
http://crln.acrl.org/content/73/10/596.full
• Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of
ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003.
doi: 10.1371/journal.pbio.1002003
• Steven Roberts website & P&T dossier, both incorporating altmetrics:
http://faculty.washington.edu/sr320/?p=2806
• Altmetric webinar: Applied Altmetrics:
http://godigitalscience.com/view/mail?iID=Y9PXAVDJMH5JVAUPJU79

36. Images
• Publish: https://thenounproject.com/search/?q=article&i=74762
• Evaluate: https://thenounproject.com/search/?q=peer+review&i=210141
• Article: https://thenounproject.com/search/?q=article&i=74762
• Book: https://thenounproject.com/search/?q=book&i=547809
• Binary code: https://thenounproject.com/search/?q=binary+code&i=177042
• Data stream: https://thenounproject.com/search/?q=data+stream&i=160443
• Q&A: https://www.readytalk.com/blog/readytalk-admin/question-how-do-i-manage-the-qanda-
portion-of-my-webinar

38. Heather Coates
Digital Scholarship & Data Management Librarian
University Library Center for Digital Scholarship
hcoates@iupui.edu
https://ulib.iupui.edu/digitalscholarship/impact