Presentation at the International Conference on Scientometrics and Bibliometrics. Kiev, Ukraine, June 4, 2019.

1. Scientometrics for research assessment
Ludo Waltman
Centre for Science and Technology Studies (CWTS), Leiden University
International Conference on Scientometrics and Bibliometrics
Kiev, Ukraine
June 4, 2019

2. Centre for Science and Technology Studies (CWTS)
• Research center at Leiden University in the Netherlands
• Science and technology studies, with a strong emphasis
on scientometrics
• About 50 staff members
• Our mission: Making the science system better!
1

3. Outline
• Research assessment and scientometrics
• Scientometric data sources
• Scientometric indicators and tools
• Scientometric analysis of Ukrainian science
• Recent developments
2

4. Research assessment
and scientometrics
3

5. Research evaluation: Why, what, and how?
• Why evaluate?
– Advocacy
– Allocation
– Analysis
– Accountability
• What to evaluate?
– Countries
– Research institutions
– Research groups
– Individual researchers
• How to evaluate?
– Expert judgment
– Scientometrics 4
Adam et al. (2018)

6. Leiden Manifesto
• Leiden Manifesto provides 10
principles for responsible use of
scientometrics in research
assessments
• Used by research institutions to
develop internal policies for use
of scientometrics
5

7. Scientometric data
sources
6

8. Scientometric data sources
• Web of Science (Clarivate Analytics)
– “With the Web of Science platform, you can access an unrivalled breadth of world-class research
literature linked to a rigorously selected core of journals”
– Consists of multiple citation indices (SCIE, SSCI, AHCI, ESCI, CPCI, BCI, …)
– Content selection by Editorial Development team
• Scopus (Elsevier)
– “Scopus is the largest abstract and citation database of peer-reviewed literature”
– Content selection by Content Selection and Advisory Board
• Dimensions (Digital Science)
– “Re-imagining discovery and access to research: grants, publications, citations, clinical trials,
patents and policy documents in one place”
– Based on Crossref and PubMed data, enriched with data from publishers
– Content selection by end user
– Basic web interface is freely accessible; data is freely available for scientometric research
7

9. Scientometric data sources
• Google Scholar
– Most comprehensive scientometric data source
– Low data quality and easy to manipulate
– No possibilities for large-scale data access
• Microsoft Academic Graph
– Data can be accessed through API and Microsoft Azure
– ODC-BY license (free to share, create, and adapt as long as you attribute)
8

10. Comparison of data sources
9
Overlap of publications in different data sources
(1996–2017); WoS includes only SCIE, SSCI,
AHCI, and CPCI

11. Scientometric
indicators and tools
10

12. Journal impact factor
11

13. h-index
12
h-index is arbitrary and inconsistent

14. Field-normalized citation impact indicators
13
0 1 2 3 4 5 6 7 8 9 10
0
5
10
15
20
25
30
35
publication age in years
averagenumberofcitationsperpublication
biochemistry & molecular biology
cardiac & cardiovascular systems
chemistry, analytical
surgery
economics
physics, applied
mathematics

15. Beyond citation impact
• Scientific collaboration
• Collaboration with industry
• Open access publishing
• Gender balance
• Mobility
14

16. University rankings
15www.researchresearch.com/news/article/?articleId=1368350

17. VOSviewer
16

18. Leiden algorithm
17

19. Micro fields of science
18
Social sciences
Biomed.
& health sciences
Math. & comp.
science
Physical
sciences & eng.
Life & earth sciences

20. Scientometric analysis
of Ukrainian science
19

21. Scientometric analysis of Ukrainian science
• Based on Web of Science (SCIE, SSCI, AHCI) data
• Only international literature; no local literature
• Time period 2014–2017
20

22. Publication output and citation impact
21

23. Micro fields of science
22
Social sciences
Biomed.
& health sciences
Math. & comp.
science
Physical
sciences & eng.
Life & earth sciences

24. Micro fields of science: Contribution from Ukraine
23
Social sciences
Biomed.
& health sciences
Math. & comp.
science
Physical
sciences & eng.
Life & earth sciences
Share of
publications
from Ukraine
Nuclear
accidents

25. Micro fields of science: International comparison
24
Ukraine
Poland
Germany
Russian

26. Nuclear accidents micro field
25

27. Nuclear accidents micro field: Contribution from
Ukraine
26
Share of
publications
from Ukraine

28. Nuclear accidents micro field: Contribution from
Ukraine
27
Share of
publications
from Ukraine

29. Recent developments
28

30. Open citations
• Initiative for Open Citations (I4OC) advocates open availability of citation
data
• More than half of all citations in Crossref have been opened
29

31. New journal: Quantitative Science Studies
• New high-quality journal established in January 2019 as a result of Journal of
Informetrics flip
• Owned by the International Society for Scientometrics and Informetrics and
published by MIT Press
30

32. Full-text data
• Increasing availability of full text of scientific publications for scientometric
research
31

33. More diverse journal indicators
• Call for more diverse journal
indicators (not just citation impact
indicators) covering the different
functions of journals
• Coordination by a governing body
that brings together different
stakeholders
32

34. CWTS Scientometrics Summer School
• Summer school for PhD students and researchers in scientometrics
• Previous editions in Leiden (Summer 2018) and Haining (Spring 2019)
• Next edition in Leiden (Summer 2020)
33

35. Thank you for your attention!
34