Beth Bernhardt (speaker), Iain Hrynaszkiewicz (speaker)

1. MEASURING RESEARCH IMPACT ON THE WEB
Charleston Conference, November 2013
Iain Hrynaszkiewicz
Outreach Director, Faculty of 1000
iain.hrynaszkiewicz@f1000.com
http://f1000.com
@iainh_z

2. SOME PROBLEMS WITH SCHOLARLY COMMUNICATION
http://blog.f1000.com/2013/10/04/vitek-tracz-science-interview/
1.
2.
3.
4.
Individual papers and scientists judged on journal-based
metrics
Closed, pre-publication peer review
Lack of access to original research – and data
Lack of credit for many of scientists’ contributions
The Seer of Science Publishing
Science 4 October 2013:
Vol. 342 no. 6154 pp. 66-67
DOI: 10.1126/science.342.6154.66
http://www.sciencemag.org/content/342/6154/66.full.pdf

3. SOME PROBLEMS WITH SCHOLARLY COMMUNICATION #1
“I am sick of impact factors and so
is science.”
Prof Stephen Curry, Imperial College,
Aug 2012
http://occamstypewriter.org/scurry/201
2/08/13/sick-of-impact-factors/
“[Journal Impact Factor is] a poor
indicator of citations to specific
papers or of the future performance
of individual researchers”
Nature Materials 12, 89 (2013)
“Citations are heavily gamed and are
painfully slow to accumulate, and overlook
increasingly important societal and clinical
impacts.”
Priem et al., PLoS ONE 7(11): e48753
“The widely held notion that high-impact
publications determine who gets academic
jobs, grants and tenure is wrong.”
Dr Michael Eisen, UC Berkeley, Feb 2012
http://www.michaeleisen.org/blog/?p=911

4. http://am.ascb.org/dora/
Themes of DORA recommendations:
1.Eliminate the use of journal-based metrics in funding, appointment, and promotion
considerations
2.Assess research on its own merits not the journal
3.Capitalize on the opportunities provided by online publication such as exploring
new indicators of significance and impact
DORA has been signed by >9000 individuals and nearly 400
organisations. Original signatories included American Association for the
Advancement of Science (AAAS), American Society for Cell Biology, Howard Hughes
Medical Institute, Faculty of 1000, Public Library of Science

5. IMPACT FACTORS AND CITATION METRICS
Advantages
Disadvantages
Reproducible
Slow – delay of up to 2 years
Transparent calculation (kind of)
Data not publicly available
Curated – some human filtering
Can be manipulated
Predictor of journal quality
Poor predictor of paper and researcher
quality

6. I, Cawi 2001 [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons
http://commons.wikimedia.org/wiki/File:Internet-Sign.jpg

7. “ALTERNATIVE” METRICS
Source: Wikipedia, http://en.wikipedia.org/wiki/Alternative

8. ALTMETRICS – WHY ARE THEY IMPORTANT?
“For all new grant applications from 14 January, the US National Science
Foundation (NSF) asks a principal investigator to list his or her research
“products” rather than “publications” in the biographical sketch section.
This means that, according to the NSF, a scientist's worth is not dependent
solely on publications. Data sets, software and other non-traditional
research products will count too.”
Heather Piwowar, UBC, in Nature Jan 2013
http://www.nature.com/nature/journal/v493/n7431/full/493159a.html

9. ALTMETRICS – WHY ARE THEY IMPORTANT?
“Metrics --- such as the number of article citations, your h-index or others
(such as those available at ImpactStory.org) --- can be useful in making the
case that the publication or scholarly work was significant. The Faculty
Promotions Committee discourages the use of journal-based metrics (such
as journal impact factors), since it is the quality and importance of the
research contribution itself that is the key.”
Excerpt from UC Denver promotions handbook guide to dossier preparation

10. TOOLS FOR SHARING RESEARCH PRODUCTS...

11. AND TOOLS TO INTERACT WITH THEM

12. Source: http://impactstory.org/collection/v7b25c

13. Source: http://www.altmetric.com/details.php?citation_id=938211

14. (ALT)METRICS – ADVANTAGES AND DISADVANTAGES
Advantages
Disadvantages
Fast – data available immediately
Heterogeneity across tools
Transparent (mostly)
Can lack context/meaning
Lots of open data and tools available
Can be manipulated
All research products tracked
Not always reproducible (ephemeral)
Much broader picture of impact
Emergence and disappearance of tools

15. WHAT’S OFTEN MISSING?
By Taco Hoekwater; Wolfgang Schuster; Xan; svg from Lumu [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
http://upload.wikimedia.org/wikipedia/commons/0/0d/ConTeXt_Unofficial_Logo.svg

16. HOW F1000PRIME WORKS
•
Faculty includes over 5000 peer-nominated scientists and clinical
researchers and ~5000 Associates
•
Faculty Members select, rate and comment on the most
interesting and important research articles (2-3% of the life science
literature) from ~3,700 journals
•
Assigns one of three positive ratings: Exceptional (3 stars), Very
Good (2 stars) or Good (1 star)
•
Text also serves as a short, expert, plain English recommendation
written for a global readership
•
Adds relevant classifications (e.g. changes clinical practice)
•
Publishes about 1500 recommendations per month (>143,000
published to date)

17. EXAMPLE RECOMMENDATION

18. F1000, ALTMETRICS AND RESEARCH ASSESSMENT/IMPACT
The Wellcome Trust published in PLOS ONE an analysis of
F1000Prime, finding that expert review highlights important
papers that bibliometric indicators alone would miss
(http://bit.ly/F1000Wellcome).
A 2009 study by the MRC concluded that F1000Prime
recommendations are an indicator of future citation impact
(http://bit.ly/F1000future).
Collection of studies of F1000Prime:
http://www.mendeley.com/groups/3748101/studies-involvingfaculty-of-1000-s-f1000prime/
Also, studies have shown that tweets and number of readers on
Mendeley predict citations.
How much does all this matter?

19. SOME PERSPECTIVE. WHAT IS IMPACT?

20. IS THIS IMPACT?
Source: http://www.jmedicalcasereports.com/mostviewed/alltime

21. THIS IS IMPACT
Source: http://www.ncbi.nlm.nih.gov/pubmed/15555666

22. SUMMING UP
•
Impact factors are not good for assessing individual papers and
individuals
•
The internet has given us a wealth of new data and tools to give
a broader picture of the impact of research
•
We are still trying to understand the meaning of many new
metrics – we need data with context
•
Some “alternative” metrics are beginning to influence research
assessment
•
All metrics – citation and non-citation metrics – have advantages
and disadvantages