1. Finding Insights in ALMS
for Research Evaluation
Richard Cave, @richardcave
I.T. Director, PLOS
Charleston Conference 2013

2. Article-Level Metrics at PLOS
• Article-Level Metrics (ALM) allow the
comprehensive analysis of the post-publication
activity around a paper, employing usage
stats, citations and altmetrics.
http://article-level-metrics.plos.org/alm-info/
• Started in 2009 at PLOS
• Lots of data, tracking over a dozen sources
2

3. ALMs provide the flexibility to look at a wide
range of numbers
3

4. Authors want to see the impact of their article
4

5. What about funders and institutions?
• Funders want to monitor the research that they
support
– Wellcome Trust spends approximately £650 million on
research per year. What’s the impact?
– Médecins Sans Frontières – how do they know the
reach of their article?
• Institutions want to see how their researchers are
making a change
– Stanford University – do librarians know what research
was published by Stanford researchers?
5

6. Numbers are
GREAT
but what’s the
impact of the
research?!?
6

7. How easy it for institutions and funders to
track research impact?
•
•
•
•
It’s not easy!
Aggregate data for all articles? Hard to do
Follow tweets? Not likely
News/blog stories? Hard to find correct sources
7

8. ALM Reports
almreports.plos.org
Launched June 18
Allows researchers, institutions &
funders to:
• Create a report of the ALMs for a
set of PLOS articles
• View a summary of the metrics
along and data visualizations.
8

9. But more important, provides a way to provide
context of ALMs with search
Search based on:
• Keyword
• Author name & country
• Institutional affiliation
• Publication date
• Subject areas
• Funder
9

10. For researchers at your institution, can
show impact for all of their articles
10

11. For institutions and funders - Get specific!
(author_affiliate:"Stanford University") AND
(financial_disclosure:"National Institutes of Health" )
(author_affiliate:"Stanford University") AND
(financial_disclosure:"Wellcome Trust")
(author_affiliate:"Stanford University") AND
(financial_disclosure:"National Cancer Institute")
(author_affiliate:"Stanford University") AND
(financial_disclosure:"National Cancer Institute" AND
financial_disclosure:"Doris Duke Foundation")
11

12. Page views and citations over time
Standford University + National Cancer Institute
Total usage includes page views and downloads from PLOS and PMC. Bubble size correlates with
Scopus citations and bubble color with the PLOS journal.
12

13. Page views and Mendeley bookmarks over
time
Standford University + National Cancer Institute
Total usage includes page views and downloads from PLOS and PMC. Bubble size correlates with
Mendeley bookmarks and bubble color with the PLOS journal.
13

14. Article usage and citations by subject
category
Standford University + National Cancer Institute
Rectangle size correlates with total views (HTML and PDF) from PLOS website. Color intensity
correlates with number of Scopus citations.

15. Research collaborations
Standford University + National Cancer Institute
15

16. Wellcome Trust evaluation
Adam Dinsmore at Wellcome Trust:
• Identified and exported the entire corpus of PLOS articles
associated with the Wellcome
• Analysed the ALMs associated with them and benchmarked
against the sector average
• Produced a consumer report
Findings:
• Wellcome Trust associated PLOS articles are viewed, saved and
cited more often than the average for the biomedical research
sector
• They are discussed by the blogosphere more often than the sector
average
http://article-level-metrics.plos.org/files/2013/10/Dinsmore.pptx
16

17. For Publishers
• Improve Business Intelligence
– Optimize publicity efforts with more targeted campaigns (Call For
Papers, conference presence, etc.)
– Launch new collections of papers in areas of interest
– Find new fields growing at the intersections between communities
of interest
• Deepen Editorial Capacities
– Direct editorial resources to promote new research of interest in
growing fields
– Identify promotional opportunities for papers and authors (writing
press releases, front matter content, editorial overviews, etc.)
– Uncover potential new authors
• Deliver a Richer Product
– Maximize front matter visibility
– Create more targeted editorial overviews
– Enhance press reviews and promotional content for high-impact
papers
17

18. But beware of dragons
• Data issues = Garbage in, garbage out
• Author names
• Funder names
• “Welcome Trust” vs. “Wellcome Trust”
• Institutional names
• A dozen ways to enter “University of
California, Berkeley”
• Will be helped in the future by:
• ORCID
• FundRef
18

19. ALM info
• General information on ALM
http://article-level-metrics.plos.org/
• ALM Reports
http://almreports.plos.org/
– Future: ALM Reports will be open source but will have
limitations (no search functionality)
• ALM Application is Open Source (Ruby on Rails)
https://github.com/articlemetrics/alm
– Recently launched at Copernicus Publications
– OJS has a beta trial
19