Presentation to the Earth Science Information Partners (ESIP) in Bloomington Indiana 27 July 2017. Presentation covers value and economic impact studies by Charles Beagrie Ltd and our CESSDA SaW cost -benefit advocacy toolkit. A particular focus given to Earth Sciences.

1. Earth Science Information Partners (ESIP) Workshop July 2017
Neil Beagrie (Charles Beagrie Ltd)
Measuring the Value and
(Economic)
Impact of Research Data Services
Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark

2. About us
• Charles Beagrie Ltd – independent consultancy in
digital archive, library, science and research
sectors.
• Neil Beagrie consultant specialising in cost-benefit
analysis, research data curation and digital
preservation, e.g. KRDS projects.
• Prof John Houghton economist specialising in
applying economic methods to information
technology policy, science and technology policy,
and Open Access.

3. Presentation Overview
• Previous work(note focussing on cost-benefit)
• ESDS Impact Study
• ADS/BADC/EBI Impact Studies
• CESSDA Cost-Benefit Advocacy Toolkit
• Key Challenges and Lessons
Informal – happy to take questions as we go

4. Why measure value and
(economic) impact?
• Advocacy: proving (to non-researchers) that current services
or future services are a worthwhile investment;
• Complementary to other approaches such as data/article
citation analysis;
• Can show more of the impact on research, teaching &
learning, etc.
Note however:
• It is challenging to do;
• Methods well-established but impact data hard to obtain and
to use.

5. Previous Work
Big Science and Innovation Study for BIS July 2013
• Desk review of c. 100 studies internationally;
• 3 studies highlighted to BIS as being particularly good
examples of ‘good practice’ in the measurement of
economic impacts:
• Berkeley Lab 2010
• Human Genome Project 2011
• Economic and Social Data Service (ESDS) 2012

6. The ESDS Study

7. Best Practice from ESDS study
• Applies range of
methods;
• Includes counter-
factual;
• Data collection:
users & depositors,
research & teaching;
• Data weighting -
survey value
responses weighted
to reflect the overall
pattern of use from
weblogs;
• Case studies/ KRDS
benefits illustrate
benefits and impact
pathways.

8. ESDS Study:
Returns on Investment (ROI)
ROI - service Benefit/cost ratio of
net economic value to
ESDS operational costs
ROI - research data creation
scenarios -Increase in returns
from data re-use
facilitated by ESDS (counterfactual)
5.4 to 1
Up to 10 to 1

9. ESDS Study:
Researcher Efficiency Gains
Impact of using ESDS data and services on research efficiency
(after Beagrie et al 2012, p77, Figure 15)
Economic and Social Research Council © 2012 CC-BY licensed

10. The Value and Impact Studies

11. Economic & Social Data Service (ESDS)
Archaeology Data Service
British Atmospheric Data Centre
European Bioinformatics Institute
Value + Impact Analyses
of Data Services
John Houghton + Neil Beagrie 4 joint studies to date.
5th in progress. Methods applied to:

12. Approaches and Methods
• We combined quantitative and qualitative approaches.
• All studies used:
• Desk-based analysis of existing literature and reports;
• Existing management and internal data collected by the data
services; and
• Original data collection in the form of online surveys, together with
semi-structured interviews.

13. Indicative Scale (currently £1.00 = $1.31)

14. The Value and Impact of the
British Atmospheric Data Centre

15. Key FindingsThe economic analysis shows that:
• increase in research efficiency estimated as at least £10 million per
annum.
• Increase in ROI from the data re-use facilitated by BADC estimated
as between £11 million and £34 million over thirty years (net present
value) from one-year’s investment.
The qualitative analysis in the user survey showed that:
• 81% of the academic respondents reported that BADC was very or
extremely important for their academic research, and
• 53% of respondents reported that it would have a major or severe
impact on their work if they could not access BADC data and
services.
In the depositor survey:
• 52% suggested that being unable to deposit data with BADC would
have a major or severe impact on them.

16. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
CESSDA SaW Toolkit

17. ESDS Study: Context
• UK Data Archive one of the largest European
social science data archives: perhaps the largest
• UK Data Archive established for 40 years: has
built up collections and users over time
• Only economic impact study for any social
science data archive to date
• How to extend work for other social science
archives?

18. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
CESSDA-SaW Task 4.6
Capturing and communicating the value and
economic impact of social science data services.
Develop a benefit/cost advocacy programme and
supporting tools; assembling an evidence base to support
the negotiation with ministries and funding organisations;
support advocacy with other core stakeholders such as
data creators and data users.
Toolkit published April 2017.

19. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
CESSDA-SaW
User Requirements Survey

20. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Survey Q2
THE DATA SERVICE'S CURRENT STAFFING IS
APPROXIMATELY:

21. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Toolkit Components
• Factsheets
• ROI, Benefits, Costs
• Worksheets
• Benefits Summary for a Data Archive
• Archive Development Canvas
• Case studies
• ADP, FSD, LiDA, UKDS
• Selected External Tools
• CCeX, KRDS, CDMA, ESDS Impact, ADP surveys , etc
• User Guide

22. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
The Factsheets

23. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
ROI Factsheet (1)

24. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
ROI Factsheet (2)

25. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Counter-factuals – “Costs of Inaction”
“Ideally, economic impact assessments should estimate the
counterfactual – i.e. what would occur in the absence of the
facility…However, counterfactuals are rarely addressed in the
[c.100] studies reviewed due to lack of data. We found two
exceptions that address this issue partially. One is the
evaluation of the economic impacts of ESDS (2012) which
partially explores the counterfactual through a users’
survey…Another exception is a review of economic impacts of
large-scale science facilities in the UK (SQW, 2008) …
however, this estimation is not done rigorously and relies
mostly on the estimation of the local benefits.”
Big Science and Innovation - Report to BIS - Technopolis
2013

26. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
ROI Factsheet (3)

27. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Benefits
Factsheet (1)

28. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Benefits
Factsheet (2)

29. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Benefits Factsheet (3)

30. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Costs
Factsheet (1)

31. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Costs
Factsheet (2)

32. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Bringing it all together:
The Archive Development Canvas

33. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.

34. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
The Archive Development Canvas (detailed)

35. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
The Archive Development Canvas (mapped)

36. Key Challenges and Lessons

37. EBI Report: User population
• How ‘actual direct users’ were estimated from web stats:
• Unique host log of 10.8 million (Annual figure for 2014) adjusted
to reflect key considerations:
• People behind unique IPs in web stats usually over-estimated by a
factor of 7 (Formitchev, 2010)
• EMBL-EBI accessed by organisational IPs (survey: 4185 responses
from 3622 unique IPs): Factor in 1.16 per unique IP
• Respondents accessed 9 services on average – 10.8 million figure
captured across EMBL-EBI resources
• Total user population estimate: 198,000 unique, actual
direct users

38. Online Surveys
(BADC)
• Two independent online surveys were conducted by CB
Ltd;
• Survey questionnaires developed iteratively - small group
of BADC customers helped test pilot versions;
• Given the nature of subject and diverse customer base
substantial effort was needed to design questionnaires;
• Significant effort spent to reduce the likely burden on
questionnaire recipients.

39. Online surveys (BADC)
Significant time spent on:
• wording of the invitations to participate
• advance notice of the surveys via websites, news feeds, social
media and email shots
Five £50 Amazon vouchers offered in a draw for participants
Surveys had high response rates and completion rates

40. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Toolkit challenges & lessons
• diverse scale / type of organisations
• complexity of topic / local resources

41. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Effort Grading Levels
hours days months

42. This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement number 674939.
Design criteria
• Easy for overloaded individuals/ smaller
services
• Short documents
• Good Infographics
• Synthesis
• Making existing tools easier to use/tailored to
(social science) data services

43. Conclusions
• Value and (economic) impact analysis doable but
challenging;
• Difficulties and critical points for economic analysis include:
• Limited knowledge of real user populations;
• Survey sample and survey design;
• Complexity of mature data services and their resources;
• "Costs of inaction" data - counter-factuals for data
services.
• CESSDA SaW toolkit approach adaptable to other disciplines
and can tailor for/utilise sector specific data and needs.

44. Questions?