The National Center for Computational Toxicology at the EPA investigates impact metrics to measure the performance and value of its work. NCCT's key products include scientific publications, research data, software tools, collaborations, training, and scientific leadership. NCCT is exploring alternative metrics like downloads, views, citations, reuse, and social media mentions to better quantify the impact of these products beyond traditional bibliometric measures. An integrated website is being developed to aggregate these alternative metrics for NCCT's work across multiple platforms to demonstrate its large scientific impact.
Investigating Impact Metrics for Performance for the US-EPA National Center for Computational Toxicology
1. Investigating Impact Metrics for Performance
for the US EPA National Center for
Computational Toxicology
Antony Williams*, Monica Linnenbrink, Kevin Crofton,
Russell Thomas and Jeffery Edwards
National Center for Computational Toxicology
U.S. Environmental Protection Agency, RTP, NC
August 21-25, 2016
ACS Fall Meeting, Philadelphia, PA
http://www.orcid.org/0000-0002-2668-4821
The views expressed in this presentation are those of the author and do not necessarily reflect the views or policies of the U.S. EPA
2. Who is NCCT?
• National Center for Computational Toxicology – part of EPA’s
Office of Research and Development
• Research driven by the EPA’s Chemical Safety for
Sustainability Research Program
– Develop new approaches to evaluate the safety of chemicals
– Integrate advances in biology, biotechnology, chemistry, exposure
science and computer science
• Goal - To identify chemical exposures that may disrupt
biological processes and cause adverse outcomes.
• Can we quickly and cheaply evaluate thousands of chemicals
for potential risk?
1
3. What are our “products”?
• Our scientific publications and presentations
• Our research data
• Our software applications, code, algorithms and models
• Our collaborations
• Our trainings – postdoctoral researchers
• Our Scientific Leadership
2
4. What are our products?
• Our Scientific Publications and Presentations
https://cfpub.epa.gov/si/si_lab_search_results.cfm?fed_org_id=1267&SIType=PR&TIM
SType=Journal&showCriteria=0&view=citation&sortBy=pubDateYear
3
6. What are our products?
• Our Data
https://www.epa.gov/chemical-research/downloadable-computational-toxicology-data
5
7. What are our products?
• Our collaborations
https://www.epa.gov/chemical-research/collaborative-agreements-computational-toxicology-research
6
8. What are our products?
• Our scientific leadership
https://www.epa.gov/chemical-research/toxicity-forecasting
7
9. How should our impact be measured?
• Classical measures of impact – publications (How many,
where published, how many times cited)
• The world of AltMetrics (http://altmetrics.org/manifesto/)
8
10. Altmetrics impact
• Scientific Publications and Presentations – how many
downloads, views, shares, tweets, embeds,
• Research data – downloads, uses, publications about
and citations to data
• Software applications, code, algorithms and models –
number of visitors, geography of visitors, number of
searches, amount of data downloaded, integration to
applications (web service access), resuse of code,
algorithms and models
9
11. Altmetrics impact
• Collaborations – funding of external work, provision of
ToxCast plated materials,
• Training – postdoctoral researchers – what careers do
they have when they leave NCCT?
• Scientific Leadership – contributions to EPA policy
documents
10
12. Scientist AltMetrics
• Many Social and Altmetric platforms in recent
years…
• …and publishers use these for tracking, assisting
distribution and for amplification 11
13. Article Metrics: e.g. PLoS
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0062325
12
28. Integrating Data
• Tools can provide BOTH scientist level
and article level metrics and activities
• NCCT wants to aggregate the activities
of the Center – including through our
scientists and our publications. But also all
of our other “products”…
• Aggregation into one website is underway
27
32. Data & Tools: ToxCast Data Download Web
0
50
100
150
200
1-Sep-2015 1-Oct-2015 1-Nov-2015 1-Dec-2015 1-Jan-2016 1-Feb-2016
ToxCast Data Download Webpage Traffic (Daily)
Pageviews Unique Pageviews
0
500
1,000
1,500
2,000
2,500
September '15 October '15 November '15 December '15 January '16 February '16
ToxCast Data Download Webpage Traffic (Monthly)
Pageviews Unique Pageviews
What we would like…
MOCKUP: Data Usage
33. Data & Tools:ToxCast Data Download Web Demographics
Top Users by Country
1. United States 1,715
2. Germany 170
3. United Kingdom 148
4. Canada 120
5. France 119
6. China 117
7. Japan 84
8. Italy 78
9. South Korea 60
10.Switzerland 56
1
23
4
5
6 78 9
10
What we would like…
MOCKUP: Geographic Access
34. Interest and Use: 2014 Stakeholder Workshop Survey Results
60%
13%
27%
ToxCast Data
Very Satisfied
Satisfied
Neither Satisfied or
Dissatisfied
Dissatisfied
13%
34%
20%
33%
iCSS Dashboard
Very Satisfied
Satisfied
Neither Satisfied or
Dissatisfied
27%
66%
7%
Workshop
improved
understanding of
data/tools and
use of them
Strongly Agree
Agree
Undecided
Disagree
Strongly Disagree
What we would like…
MOCKUP: Aggregating Feedback
40. Work in Progress
• Investigating VIVO Open Source
• Data model in development
• ORCiDs for people, DOIs for papers
• Will integrate AltMetric, PlumX and
Kudos embeddable widgets
39
• Integrate existing
EPA tracking
systems – e.g. EPA
Science Inventory
41. Our center – A BIG Impact!
We are out to measure it…
40