This document summarizes a workshop on methodology and indicators for assessing research and innovation held by the Data4Impact project. The project uses big data techniques to track the impacts of publicly-funded health research. In the morning, the project presented its analytical model and indicators for tracking outputs, academic impacts, economic impacts, and societal/health impacts. Discussion groups provided feedback on the indicators. In the afternoon, a panel of policymakers and funders discussed the relevance of the indicators and opportunities to improve the methodology. The workshop concluded by inviting participants to a future workshop on applying big data in scientometrics.

1. Workshop on
Methodology & Indicators
24 June 2019
Brussels, Belgium
Workshop report
Data4Impact receives funding from the European Union’s Horizon 2020 Research and
Innovation programme under grant agreement No 770531.
Disclaimer: The content of this publication is the sole responsibility of OpenAIRE and
Data4Impact consortiums and can in no way be taken to reflect the views of the
European Union.

2. 1.Introduction
Recent economic and societal developments have created a strong need for additional evidence on the
performance of research and innovation (R&I) systems, and the societal impact they bring. However, this
increased demand is not being met by current R&I indicators as data collection typically stops at the end
of research funding, while most results and impacts materialise in the medium- and long-term of the R&I
lifecycle.
Data4Impact, a H2020 project funded by the European Commission, addresses this problem by
capitalising on the latest developments in big data technologies and analytics (Natural Language
Processing and Machine Learning) in order to provide better R&I monitoring and impact assessment.
Combining data from different sources, the consortium has applied their methodology on publicly-funded
health research in EU member states, and has developed a set of indicators aimed at examining the
academic, economic and overall societal impact of such activities.
The project’s end-product will be a data and visualisation tool that can be used to:
- identify and monitor the direct links between research activities and the resulting technological
innovations and impact, as well as to
- examine the evolution of different health-related topics over time as outcomes of research and
technology, and with regard to the public interest.
This hands-on, interactive workshop focused on the methodology and indicators developed over the
course of Data4Impact. The project consortium presented a series of indicators on the performance and
societal impact of 40+ research programmes in the health domain, and opened the floor to discussion on
the selected approach and results.
2.Summary of the Morning Session
2.1. Introduction of Data4Impact
Data4Impact coordinator Vilius Stanciauskas (PPMI) introduced the project. The key message was that
building on data harvested from PubMed, OpenAIRE, Lens.org, PATSTAT, clinical guidelines repositories,
company websites, social media and media platforms, EC monitoring data and other databases,
Data4Impact enables policymakers, funders, experts, researchers and the public in general to ‘Ask less
& know more’ in the context of advanced research assessment.
2.2. Data4Impact Analytical Model Of Societal Impact Assessment
(AMOSIA)
Following that, Alexander Feidenheimer (Fraunhofer ISI) provided an overview of the Data4Impact
Analytical Model Of Societal Impact Assessment (AMOSIA) which has been developed over the course of
the project. The analytical model is structured around four distinct phases of the research lifecycle,
including input, throughput, output, and impact. Relying on novel big data techniques such as web
scraping, crawling, and mining, as well as text analysis methods such as Natural Language Processing and
deep learning, Data4Impact has gathered data for each analytical phase.

3. Figure 1. Data4Impact Analytical Model Of Societal Impact Assessment
2.3. Tracking of Research Outputs
The primary purpose of the following presentations by Data4Impact consortium was to present the
indicators which have been developed over the course of the project for each of these phases.
Ioanna Grypari (ATHENA RC) discussed the approach used for tracking of research outputs which
includes using entity extraction on project reports in order to automatically identify and monitor
different analytical phases during a project lifecycle.
2.4. Academic, Economic & Societal Impact
Then, the consortium including Ioanna Grypari (ATHENA RC), Vasilis Antonopoulos (Qualia), Vilius
Stanciauskas (PPMI) and Gustaf Nelhans (University of Borås) switched to discussing what happens after
the end of research, i.e. the impact phase.
The concept of topic modelling was introduced which provides another, rich level of analysis besides
the standard ones (such as organisations, projects, and funders) and allows for comparisons across
different fields and setups.
The academic, economic, and societal (health) impact indicators were presented, as well as metrics
on the societal relevance of research (what people care about). The consortium described their
approach to identifying these impacts, as outcomes of specific research activities, using big data
approaches.
Please refer to the Data4Impact booklet which demonstrates linkages between data collected across
the three dimensions of impact, including academic, economic and societal/health impact.
For more information on the workshop presentation, click here.

4. 3.Summary of the Afternoon Session
3.1. Group discussions
Data4Impact invited the participants to two parallel sessions, focusing on the Data4Impact methodology
and indicators in the areas of:
• Academic Impact & Societal Relevance of Research
• Economic Impact & Societal & Health Impact
The following table summarises the key Data4Impact indicators which were presented to the discussion
groups as well as their assessment of the utility and credibility of indicators in each of the focus areas.
Table 1. Summary of Data4Impact indicators & their assessment by the audience
Level Indicator Description Relevance &
Credibility
Comments
Academic
impact
Funding priorities Topic size in PubMed (absolute and normalised) High Topic modelling depends
highly on the selection of
text input, for instance
abstracts tend to be
more generic/less
interesting compared to
full texts
Discussion on the best
ways to identify
breakthrough research
through topic modelling
(i.e. identify new words
which would not be
assigned to a common
topic)
Emerging areas (which
are rare but growing)
should be included
Interest in predictive
analytics, i.e. which
indicators predict a
successful project
Distribution of topics per funder (normalised) High
Distribution of funders per topic (absolute) High
Timeliness of research
performed
Rate of topic growth between 2012-2018
compared to 2005-2011
High
Share of funding allocated to top-10% fastest
growing topics
High
Funding exclusivity Share of funding allocated to top-10% smallest
research topics by size (i.e. investment in small-
niche topics)
High
Number of funders per topic whose output
share exceeds 3% globally
High
Share of funding allocated to research topics
with less than 5 funders whose share exceeds
the 3% mark (i.e. investment in topics where
few other funders invest)
High
Technological
value/significance of
patents
Analysis of the extent to which commonly patent
forward citations (i.e. citations a patent receives
from subsequent patent filings) are used
High Sometimes patents lead
to publications, start
from patents and track
them
Economic
impact
Economic and innovation
performance of companies
Estimated share of enterprises with evidence of
innovation activities
High To be clarified
Estimated share of highly innovative enterprises Medium
Estimated share of enterprises with evidence of
licensing activities (incl. patent/ trademark
license agreements)
Medium
Estimated share of enterprises involved in
activities related to acquisitions
Low
Estimated share of enterprises with evidence of
private investment/capital attracted
High
Continuity of innovation
activities
Estimated overlap between project activities in
FP7 & identified company innovations
High To be clarified
Number of newly CE-marked devices and
medical technologies that could be directly
Medium

5. linked to R&I activities in the EU Framework
Programmes
Societal/
health
impact
Impact on public health Citations of publications in clinical guidelines High Most positively assessed
indicator
Societal
awareness/relevance of
research
Rank of research topic based on number of news
articles, blogs, posts, tweets, etc. discussing a
given topic
Medium Need for information on
the type of mentions (i.e.
positive vs negative,
disagreement vs
consensus)
Discussion on how to use
the extracted
information
Congruence of research
funding with societal
priorities
Rank similarity of most discussed research
topics versus actual spending in the topics
Low N/A
Newly launched medicines
and medicinal products
Number of human medicinal products or orphan
medicines that could be directly linked to R&I
activities in the EU Framework Programmes
High N/A
Strength of link based on the number of
mentions of product names and their active
substances in EC monitoring data
High N/A
The discussion leads from the consortium invited the audience to consider whether any other indicators
could be relevant in the context of research assessment. The Academic Impact & Societal Relevance of
Research discussion group suggested to explore the possibility to identify spot terms (methods,
technologies, tests) which are changing over time but can be shared across disciplines (and across topic
models in Data4Impact). This could be particularly relevant if detected in the early stage. Moreover, the
Commission officials enquired whether Data4Impact could help the Commission measure its funding
strategy. For instance, the officials enquired if the current strategy which is mainly based on large and
consolidated projects/research (as opposed to funding smaller ones) seems appropriate given the
project results. Moreover, questions were raised on whether it would be possible to detect the most
‘productive’ funded projects or determine which are the small projects with very large ROI.
3.2. Summary of a Panel Discussion
The workshop programme concluded with a panel discussion involving the policymakers,
representatives of the funding organisations analysed in Data4Impact as well as experts working with
social media analysis, including:
● Eleni Chatzichristou (European Commission, ERCEA)
● Olivier Le Dour (European Commission, DG RTD, Unit E1 – Strategy)
● Shane O’ Reilly (Science Foundation Ireland)
● Michael Taylor (Digital Science)
The panel discussion was moderated by Vilius Stanciauskas, Director of Research & Policy Advice at
PPMI and the coordinator of Data4Impact.
The panel discussion focused on the following themes:
● Relevance of developed indicators in evaluating R&I activities
● Lessons learned from the use case in terms of possibilities and limitations of big data
● Transfer of knowledge and methodology to other and programmes

6. Some of the key conclusions of the panel discussion were:
- Big data methods presented in the project can effectively help measure the impact of funding
schemes, quality of the evaluation process (i.e. if funded research leads to appropriate results and
innovativeness), impact on researcher careers.
- Big data can go beyond the already well known and well funded research topics, to identify the
seeds of less funded but potentially innovative research, which could gain from increased funding.
- Burden for researchers/funding administrators can be reduced with the use of big data methods.
- The growth of policy documents provide a leading indicator of where research is going to.
- Some leading examples of the organisations which are good at identifying the areas that need
funding and research are the World Health Organisation and Doctors Without Borders (MSF).
- The key challenge is getting stuck when it comes to the limited number of open sources where
reliable complete data to work with can be found.
- The work done by extracting information from corporate websites was truly impressive in terms
of scale and ambition and that work was to be lauded. Data4Impact should also consider
extracting press releases, company reports and financial documents since these are places where
companies state their business and also provides possibility for funders to trace if the research
they have funded has resulted in tangible results in terms of medical products, drugs or other
relevant outcomes.
- There is a need to take Data4Impact even further:
o By adding comparative/conclusive elements to see which areas stand out and other
indicators that affect decision-making more directly, for instance can Data4Impact show
how long does it takes for the EU funding to provide a drug for cancer treatment in the EU
market?
o By addressing the fact that research in some fields take longer to produce an impact than
in others.
o By finding ways to identify breakthrough research at its early stages.
o By curating and making the results available to the public as soon as possible (and on ad
hoc basis when they are needed). This is essential as the results might be outdated if
published much later.
o By providing direct qualitative links between inputs and impact (i.e. learning to read the
narrative underlying the data).
4.Invitation to the Upcoming Workshop
If you are interested in learning more about Data4Impact, we would like to invite you to attend our
upcoming Workshop on Application of Big Data in Scientometrics which will take place on 2
September 2019 at the ISSI 2019 Conference in Rome, Italy. The workshop will focus on the
application of big data techniques to improve the monitoring of R&I activities and assessment of their
impact. Over the course of the project, Data4Impact has developed an online analysis platform showing
a series of results and indicators for 40+ research programmes in the health domain. This prototype
platform will be presented at the ISSI conference for the first time. You may find more details on the
Data4Impact workshop in the PDF file attached and the conference website.
5.Follow Up
If you are interested in learning more about Data4Impact methodology, results or the online analysis
platform the project is currently developing, please contact Sonata Brokeviciute, administrative
coordinator of Data4Impact, at sonata@ppmi.lt.

7. List of Workshop Participants
Name Organisation
Alexander Feidenheimer Data4Impact/Fraunhofer ISI
Ana Catarina Duarte European Commission, DG RTD, Unit E3 - Public
Health Sector
Antonella Canalis European Commission, Bio-Based Industries Joint
Undertaking
Athina Karvounaraki European Commission, DG RTD, JRC, Unit B7 –
Knowledge for Finance, Innovation and Growth
Barbara Gonzalez Lobo European Commission, ERCEA
Davide Innocenti European Commission, ERCEA
Dimitra Dirmiki European Commission, Fuel Cells and Hydrogen
Joint Undertaking
Efstratios Kontopoulos Information Technologies Institute/Center for
Research & Technology Hellas
Eleni Chatzichristou European Commission, ERCEA
Eric Fribourg-Blanc European Commission, Electronic Components &
Systems for European Leadership Joint
Undertaking
Eric Iversen Nordic Institute for Studies in Innovation,
Research and Education
Evelyn Obele Federal Ministry of Education and Research
Fenja Scheu Karlsruhe Institute of Technology
Ferenc Marofka European Commission, DG RTD, Unit E1 - Strategy
Giorgio Clarotti European Commission, DG RTD, Unit E1 - Strategy
Gregor Kyi European Commission, Eurostat
Gustaf Nelhans Data4Impact/University of Borås
Ioanna Grypari Data4Impact/ATHENA RC
Haris Papageorgiou Data4Impact/ATHENA RC
Janna Van Belle Technopolis Group
Joao Martins European Commission, ERCEA
Johan Eklund Data4Impact/University of Borås
Johan Fröberg Swedish Research Council
Juan Fernández de Guevara The Valencian Institute of Economic
Research/University of Valencia
Judit Sandor European Commission, Shift2Rail Joint
Undertaking
Laszlo Helmle European Commission, DG RTD, Unit E1 - Strategy
Natalia Manola Data4Impact/ATHENA RC
Magnus Eriksson Minso Solutions AB
Maksym Tsytlonok European Commission, ERCEA
Martina Kadunc European Commission, DG RTD, Unit A5 - Better
Regulation and Innovation Principle
Michael Taylor Digital Science

8. Olivier Le Dour European Commission, DG RTD, Unit E1 - Strategy
Paloma Martin European Commission, REA
Paola Paiano European Commission, Bio-Based Industries Joint
Undertaking
Pascal Dissard European Commission, ERCEA
Petros Gkotsis European Commission, DG RTD, JRC, B3 –
Territorial Development
Ralph Reimann Austrian Research Fund
Sebastien De Harlez Clarivate Analytics/Web of Science Group
Sergio Di Virgilio European Commission, DG RTD, CSC, Unit J5 -
Common Service for Horizon 2020 Information
and Data
Shane O'Reilly Science Foundation Ireland
Sofie Bekaert VIB Grand Challenges Program
Sonata Brokeviciute Data4Impact/PPMI
Tatjana Guznajeva Technopolis Group
Thu-Van Nguyen Stifterverband
Vasilis Antonopoulos Data4Impact/Qualia
Vilius Stanciauskas Data4Impact/PPMI