Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
June 18, 2014
NISO Virtual Conference: Transforming Assessment: Alternative Metrics and Other Trends
Assessing and Reporting Research Impact – A Role for the Library
- Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
Similaire à Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
Similaire à Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine (20)
Indexing Structures in Database Management system.pdf
Assessing and Reporting Research Impact – A Role for the Library - Kristi L. Holmes, Ph.D., Director, Galter Health Sciences Library, Northwestern University, Feinberg School of Medicine
1. Assessing and Reporting Research
Impact – A Role for the Library
Kristi L. Holmes, PhD
Director, Galter Health Sciences Library
Associate Professor, Preventative Medicine-Health and Biomedical Informatics
Northwestern University, Feinberg School of Medicine
http://orcid.org/0000-0001-8420-5254
NISO Virtual Conference:
Transforming Assessment: Alternative Metrics and Other Trends
June 18, 2014
2. overview
• Impact metrics - the ususal suspects and more
• A view from the library
• Identifying needs
• Roles played by libraries
• Developing services
• Moving this space forward on the local and larger
level
3. impact metrics – why do we care?
• Quantify and document research
impact
• Justify future requests for
funding
• Quantify return on research
investment
• Discover how research findings
are being used
• Identify similar research projects
• Identify possible collaborators
• Determine if research findings
are duplicated, confirmed,
corrected, improved or
repudiated
• Determine if research findings
were extended (different
human populations, different
animal models/species, etc.)
• Confirm that research findings
were properly
attributed/credited
• Demonstrate that research
findings are resulting in
meaningful health outcomes
• Discover community benefit as
a result of research findings
• Progress reports
• Tenure
• Promotion dossiers
https://becker.wustl.edu/impact-assessment/model
4. alternative metrics
• Twitter
• Facebook
• Blogs
• Research Highlights
• Google+
• Main Stream Media
• Reddit
• Forums
• Q&A – StackExchange
• Pintrest
• LinkedIn
• FigShare
• F1000 Reviews
• GitHub
Thelwall M, Haustein S, Larivière V, Sugimoto CR (2013) Do Altmetrics Work? Twitter
and Ten Other Social Web Services. PLoS ONE 8(5): e64841.
doi:10.1371/journal.pone.0064841
Konkiel S (2013) Altmetrics: A 21st-Century Solution to Determining Research
Quality. Information Today. Available at
http://www.infotoday.com/OnlineSearcher/Articles/Features/Altmetrics-A-stCentury-
Solution-to-Determining-Research-Quality-90551.shtml
Articles
Datasets
Slides
Software
Webpages
5. Understanding the impact of research is a massive task.
How do we understand the impact of a published work?
A person?
A research center?
A university?
What IS impact?
6. Ioannidis JA, Khoury MJ. Assessing Value in Biomedical Research: The PQRST of Appraisal and Reward. JAMA. Published online June 09, 2014.
doi:10.1001/jama.2014.6932.
• We need to move assessment toward
“desired outcomes: research that is
productive, high-quality, reproducible,
shareable, and translatable (PQRST)”
• Productivity metrics should reward high-
influence science rather than least
publishable units and decrease publication
bias against negative results.
Research impact and assessment
7. go beyond counts!
https://becker.wustl.edu/impact-assessment
Investigate and adapt
frameworks to help put
things into context
More and more organizations and efforts are
considering the impact of their work through
frameworks.
The Becker Model
Frameworks from the IOM, CDC, and NIEHS
Snowball Metrics
Discipline-specific apporaches
and more…
9. The Becker Model
• Provides a supplement to publication analysis to provide a more
robust and comprehensive perspective of biomedical research
impact.
– reporting templates, glossary of resources and terms, examples of
relevant indicators of impact across the research process, and readings
• Straightforward framework for tracking diffusion of research outputs
and activities to locate indicators that demonstrate evidence of
biomedical research impact
– individual, core, and institutional-level; modify for different disciplines
• Guidance for quantifying and documenting research impact as well
as resources for locating evidence of impact.
• Strategies for enhancing the impact of research
10. strategies for enhancing the impact
of research
https://becker.wustl.edu/impact-assessment/strategies
Repetition, consistency, and an
awareness of the intended
audience form the basis of most
the strategies.
The strategies focus upon
Preparing for Publication,
Dissemination, and Keeping
Track of Your Research.
Optimizing discoverability and
access of your research is the
surest way to enhance its
visibility and impact.
Suggestions for researchers and
recommendations to reach out to
their library for assistance.
11. Understanding the impact of research is a massive task.
How do we understand the impact of a published work?
Aperson?
Aresearch center?
Auniversity?
What IS impact?
How do we scale up and operationalize this process?
How do we support it locally from the library?
12. leverage ongoing efforts on the local
level and beyond
• ORCID
• CASRAI
• CERIF
• VIVO-ISF ontology
• NISO Altmetric work
• CV/Biosketch
• Large-scale semantic search
• Institutional
repositories
• Research
networking systems
• Institutional
research
information and
management
systems
• Analytical and
benchmarking
efforts
• Tracking and
evaluation
13. VIVO lends itself to research
assessment and impact
• Facilitates a more robust research
ecosystem
– Diffusion of research products
– Discovery of shared products
– Reproducibility
– Translation
– Enables credit and visibility by
showcasing individual
achievements and expertise
• Supports team-based science and
collaboration
• Allows better large-scale understanding
of the research enterprise
– Temporal relationships (career
development, time from publication to
research synthesis)
– Peer comparisons
– Strategic planning and visualization
– Identify emerging trends
• Enables more efficient means of
collecting & representing meaningful
outputs en masse
• New experimental protocols
• Datasets
• Instrumentation
• Software code
• New diagnostic criteria
• New standard of care
• Curriculum guidelines
• Measurement instruments
• Continuing education
materials
• Clinical/practice guidelines
14. VIVO lends itself to research
assessment and impact
• Facilitates a more robust research
ecosystem
– Diffusion of research products
– Discovery of shared products
– Reproducibility
– Translation
– Enables credit and visibility by
showcasing individual
achievements and expertise
• Supports team-based science and
collaboration
• Allows better large-scale understanding
of the research enterprise
– Temporal relationships (career
development, time from publication to
research synthesis)
– Peer comparisons
– Strategic planning and visualization
– Identify emerging trends
• Enables more efficient means of
collecting & representing meaningful
outputs en masse
• New experimental protocols
• Datasets
• Instrumentation
• Software code
• New diagnostic criteria
• New standard of care
• Curriculum guidelines
• Measurement instruments
• Continuing education
materials
• Clinical/practice guidelines
VIVO can help operationalize this
process:
Enhance profiles by incorporating meaningful
outputs to enhance content and value; can
facilitate dissemination and discovery of
scholarship
The ontology is the key…
15. • Plum Analytics gathers metrics across
five categories—usage, mentions,
captures, social media and citations.
• Metrics are gathered around artifacts
• Collected information is displayed in
visualizations, dashboards, and widgets.
• Customized for the institution or
organization.
ARTIFACTS
articles
blog posts
book chapters
books
cases
clinical trials
conference papers
datasets
figures
grants
interviews
letters
media
patents
posters
presentations
source code
theses / dissertations
videos
webpages
http://www.ebscohost.com/newsroom/stories/plum-analytics-becomes-part-of-ebsco-information-services
IR integration
17. IR integration
Benefits?
1. Create a feedback loop for researchers that
gives them timely insight into the impact of
their research right from the start
2. Easy access to advanced metrics from
PlumX can help build buy-in from
researchers, increasing repository support
by your contributors
3. Aggregated metrics and reports from the
PlumX dashboard puts the repository at the
center of communicating about the impacts
of your organization’s research
Courtesy of Plum
18. a spectrum of possibilities…
http://commons.wikimedia.org/wiki/File:Linear_visible_spectrum.svg
Integrate
altmetrics
in your IR
Lead or
support
RNS project
(VIVO, etc)
Run citation
reports
ORCID
services and
support
Encourage
prudent use of
online tools
Welcome deposit of
alternative outputs
in your IR
Keep
current
Collaborate with
TT office to
track outputs
Develop
standard
reports/visualiz
ations/analyses Hire an impact
and evaluation
librarian
Workshop on
strategies for
enhancing
research impactRun
publication
data reports
Establish a
consultation
service
Help
researchers
obtain DOIs
and understand
data options
Understand
motivations
of funding
agencies
19. local success depends on
stakeholder engagement
• Stay up to date on scholarly issues and the scholarly
workflow
• Stay up to date on funding/reporting requirements
• Brainstorm to identify and understand motivations of
stakeholders
• Anticipate need and present solutions
• Call on advocates
• Think beyond ‘business as usual’
• Be persistent!
20. Acknowledgements
Support:
• Northwestern University
Clinical and Translational
Sciences Institute, NIH
award 8UL1TR000150-05
• VIVO - DuraSpace
Thanks:
• Cathy Sarli, MLS, AHIP
• Karen Gutzman – NLM Fellow
• Andrea Michalek at Plum
@amichalek
• VIVO Community @VIVOcollab
• Galter Health Sciences Library
Questions:
@kristiholmes
kristi.holmes@northwestern.edu
Thank
you!
Notes de l'éditeur
How do we scale up and operationalize this process
Our work in this area really orbits around efforts to develop a library-based framework for understanding meaningful research impact, The Becker Model.
Project with Cathy Sarli, originally motivated by work she did for one of our researchers at Washington University, Dr. Mae Gordon.
Our work in this area really orbits around efforts to develop a library-based framework for understanding meaningful research impact, The Becker Model.
Project with Cathy Sarli, originally motivated by work she did for one of our researchers at Washington University, Dr. Mae Gordon.
Consistency enhances retrieval.
A lot of the suggestions lie directly in our wheelhouse – makes this a great opportunity for the library
How do we scale up and operationalize this process
Multiple platforms
VIVO can play a big role in helping us operationalize this process – as well as leverage the data at the enterprise level for higher-order analyses, visualizations, and strategic planning efforts.
Much of this is not possible to tease out from bibliographic or grant data
Applying the pathways, we are able to uncover outputs and impacts that may not be visible if one only follows the papers – things related to
Diagnosis, clinical care, clinical environment.
Cost-effective interventions
Even materials and improvements related to how we educate doctors to practice medicine
VIVO can play a big role in helping us operationalize this process – as well as leverage the data at the enterprise level for higher-order analyses, visualizations, and strategic planning efforts.
Much of this is not possible to tease out from bibliographic or grant data
Applying the pathways, we are able to uncover outputs and impacts that may not be visible if one only follows the papers – things related to
Diagnosis, clinical care, clinical environment.
Cost-effective interventions
Even materials and improvements related to how we educate doctors to practice medicine
KH
Let’s use Plum Analytics as a good example for a deep dive on the types of things you can track…
Subjects
Trade Names
Manufacturers
Publisher
Abstract
Author Keywords
Index Keywords
Date
Grant Funding Support
Language
DOI
Source
Document Type
Peer Review Status
Authors
Author Affiliations
Corresponding Author
Group Authors
Chemicals
Institutions
Countries
References
Citation Counts
Just a few notes of thanks.
First to Cathy Sarli, a dear colleague and collaborator. We work together on the Becker Model.
Cathy, Karen Gutzman (NLM second year fellow) and I work together at Becker Library and we are also members of the WU ICTS Tracking and evaluation team. We’re in the midst of a pilot project to apply the Becker Model to ICTS investigators and look forward to learning more from this process.
Andrea Michalek from Plum – very cool work – for their work with VIVO and information for this presentation
Finally, the VIVO team and open source community – a fantastic group of friends and colleagues from all over the world doing very cool work and spreading Linked Open Data love wherever they go.
Librarians may be interested to read more about a complementary effort, Linked Data for Libraries, a recent award from the Mellon Foundation to Cornell, Stanford, and Harvard Libraries.