PhD defense about the appropriate use and validity of bibliometric indicators at the individual level. The presentation considers the familial relationships between indicators, the extent concepts being measured are captured by the indicator model as well as the general characteristics of the indicators and the profile of the researchers they are designed to measure

1. Measure up!
The extent author-level bibliometric indicators are
appropriate measures of individual researcher
performance
Lorna Wildgaard
Royal School of Library and Information Science
Faculty of the Humanities, CPH U
27th November 2015

2. OVERVIEW
1. Background
2. Characteristics and effects of ALI
(Papers 1, 2, 4)
3. The appropriateness of ALI across different disciplines
and different academic seniorities
(Papers 3, 5, 6, 7)
4. Concept definintion in the construction of ALI
(Chapter 6 & Appendix B, http://tinyurl.com/nj4mvca)
5. Conclusions
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CORRECTION

3. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
“ Bibliometrics is the
application of various
statistical analyses to study
patterns of authorship,
publication, and literature use.”
(Lancaster, 1977).
“[…] to analyse the structure of science, measure science, and indicate
the production, citations and collaboration of researchers, institutions
and countries.” (De Bellis 2014; Pritchard 1969).

4. 4
1980s 1990s 2000s 2010s1970s1960s1950s
de Solla Price
growth model
Namilov
interdisciplinary
approach to indexing
scientific literature
Merton Social structure of scientific communication
Haitun; Yablonsky distribution of citations
Garfield
SCI & IF
h-index
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

5. 5
Managerial
Teaching
Outreach
Societal
Technical Skills
Funding/Grants
Productivity
Quality
Relevance
Visibility
Reputation
Scientific
Counting,
adding,
dividing,
multiplying
an authors’
publications
& citations
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

6. DEFINTITIONS OF AU, P & C IN ALI
Authorship, publication and citation is different from
discipline to discipline, from time to time and location to
location and has implications in indicator development
(Bošnjak and Marušić, 2012)
What constitutes a publication [author and citation]
should be clearly defined to ensure representative
operationalization in the indicator and the extraction of
meaningful relationships
(Wouters 1999; Skupin 2009; Colledge 2012)
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

7. EVALUATION
OUTPUT
DATA
ALI
MODEL
MOTIVATION TO PUBLISH
MOTIVATION TO CITE
INTERPRETATION OF ALI
LEGITIMATE LINK TO REAL
WORLD
PERFORMANCE OF RESEARCHER
IN SYSTEM REDEFINES WHAT
SUCCESS IS
EXOGENOUS VARIABLES
WHAT IS MISSING
INDEXING POLICY
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
THE PROBLEM

8. 1. What are the characterisitics of ALI of academic
performance?
2. To what extent are ALI appropriate in the
evaluation of researchers from different
disciplines and different academic seniorities?
3. To what extent are the concepts being measured
defined in the construction of ALI?
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
RESEARCH QUESTIONS

9. 9
ALI GIVE A SNAP-SHOT OF SELF
IMAGE AND CORE PERSONALITY
TRAITS
COMPARING RESEARCHERS
CAN EXPOSE THE
INDIVIDUAL
INDIVIDUALS USE ANY DATA TO
INCREASE ALI SCORES, TO INCREASE
THEIR SUBJECT VALIDITY & SELF-
WORTH
ALI BRING OBJECTIVITY TO THE
EVALUATION & REDUCE GENDER;
CULTURAL AND RACIAL BIAS
DOCUMENTING BEING OUT-
PERFORMED IS DETRIMENTAL TO
A RESEARCHER’S SELF-DEFINITION
ALI DO NOT ADD CONTEXT BUT
CAN ADD REDUNDANT
INFORMATION
SUCCESS IS DEFINED AS WELL
IN THE SYSTEM
EXTERNAL CHARACTERISTICS OF ALI

10. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
INTERNAL CHARACTERISTICS OF ALI

11. ALI are designed to measure particular aspects of
the effect of a researcher’s work – over time, to
field, as quality, ranking all or selected works, co-
authorship etc.
Judgements based on ALI can lead to assumptions
about the productivity and impact of a researcher,
which can be unsubstantiated, and affect the
psychological character of the individual.
ALI have to be methodologically sound
PRELIMINARY ANALYSIS: RESULTS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

12. FAMILY CHARACTERISTICS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
METHOD:
Compute simple indicators on publication and citation
data of 750 researchers (585 m, 165w)
4 disciplines, 5 academic seniorities
Correlation (Kendall’s Tau) as measure of association
between seniority and ALI, discipline by discipline
ALSCAL IBM SPSS v.19
ENDPOINT
Suitability of indicators in different disciplines and
seniorities

13. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
FAMILY CHARACTERISTICS
DISCIPLINE CENTRAL INDICATOR S-STRESS % VARIANCE
ASTRONOMY hg 0.37 25
ENVIRONMENTAL
SCIENCE
h, h2 0.37 24
PHILOSOPHY IQP 0.38 47
PUBLIC HEALTH g 0.49 38

14. METHOD:
Publication and citation data for 512 researchers
22,143 papers, 423, 371 citations (WOS)
52,227 papers, 746,985 citations (GS)
For each researcher 17 ALI calculated
Agreement: determined by matching rank positions
Variability of rank position determined by standard deviation
of the difference in scholar rank position, calculated from the
matched pairs
END POINTS:
agreement in ranking between WOS & GS
RANKING CHARACTERISTICS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

15. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
hg produced highest level of agreement in rankings in WOS
& GS, 0.8-0.9 tau across disciplines and seniorities
(Patel 2013)

16. hg fulfills its potential as a ranking indicator (Alonso et al
2010) yet this tells us nothing about the excellence of the
researcher.
h and g cannot be rationally combined to produce
indications of research excellence (Franceschini & Maisano
2011).
Indicator rankings inform of the researcher’s visibility in the
citation index not in their academic community (Bar-Ilan
2008; De Battisti 2012; Farhadi et al 2013).
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
RANKING CHARACTERISTICS

17. METHOD:
Same data as previous studies
Explorative statistical study of distribution; two step cluster;
ordinal regression: odds ratios
IBM SPSS v.22
ENDPOINT: Identify disciplinary & seniority appropriate
indicators
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
GROUPING CHARACTERISTICS

18. CLUSTER 1
below IQR, high nnc, sc, low
P & C. Adjusting for academic
age mg, m.quotient,
currency of cites improved
scores.
CLUSTER 2
Median IQR, high scores on
collaboration indicators
CLUSTER 3
Top IQR,
highest scores
normalized for
field, one SIG
paper
CLUSTER 4
Extreme outliers,
highest C, cited quickly,
highest htype scores

19. ASTRO HEALTHPHILENVIRO
h2 SUM PP
TOP
PROP
Q2 e
GROUPING CHARACTERISTICS
Ordinal regression:
Academic age is statistically significant for cluster placement
Within cluster rank position determined by the ratio P:h

20. 1. Simple ALI are complex!
2. Basic descriptive statistics can be more informative
3. There are families of indicators
4. ALI scores are highly influenced by the database,
academic age and the fit of the data to the model
5. ALI are more appropriate in some disciplines than
others
6. Different ALI produce stable rankings within and
across databases.
7. ALI can be easily manipulated
8. Scores can be due to chance rather than practical
importance.
EMPIRICAL ANALYSIS: CONCLUSIONS
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

21. Knowing all these practical challenges, what
properties should a well constructed indicator
contain?
METHOD:
Based on papers 1-7
Validate ALI using Gingras’ criteria
Analyze operationalization of AU, P, C in ALI
ENDPOINT:
Recommendation of a set of disciplinary dependent
indicators of scientific publication output
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
VALIDITY ANALYSIS

22. THE 3 CRITERIA OF PROFESSOR YVES GINGRAS
1. Adequacy of the indicator to the object it
measures
-quantity of items a researcher publishes is an
indicator of production
2. Sensitivity to the intrinsic inertia of the object
-changing values in the time interval correspond to
the speed and direction of change in the object
3. Homogeneity of the dimensions of the indicator
-h-index is a heterogeneous indicator: it mixes
quantity of papers published with citation counts
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

23. 10
28
HOMOGENEITY
ADEQUACY INERTIA
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
17 did not
fulfill any of
the criteria
n=69 ALI
4 1
7
20

24. OPERATIONALIZATION OF AU, P, C
”THE LOGIC GRID”
METHOD:
• Extract definitions and hypotheses in ALI
• Matrix of definitions of AU, P, C & aim of ALI
AIM:
Identify in a simple way the properties of ALI
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

25. HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER
CITATION
6 6 6 5 2
PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF.
2 1 1 1 1 20
PUBLISHED SCIENTIST AUTHOR
AWARD
WINNER
PUBLISHED
& CITED
SENIORITY NO DEF.
AUTHOR 12 11 10 7 5 2 5
PAPER IN WOS
PAPER IN OTHER
INDEX
PAPER
OBJECTS WITH
CITATION
EXPRESSION
PUBLICATIONS 30 15 6 1 1
EXCELLENCE EFFECT INDEPENDENCE COMPARE RANK CURRENCY
MEASURE
9 7 7 6 6 4
GROWTH QUALITY DISTRIBUTION
QUANTITY &
QUALITY
CAREER DURABILITY
3 3 2 2 1 1

26. LOGIC GRID
http://tinyurl.com/ofm7h8s

27. • ALI are designed for specific author and
publication types
• Majority of ALI designed for papers in WoS
• Some indicators are purely theoretical
• Inherent bias in all ALI is that they are
designed for researcher profiles befitting the
hard sciences
• Lack of definitions = methodological
deficiency
If developers cannot define the variables
they are measuring, how can we?
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

28. 1. Transparency
Know which data is used to compute the indicator
Understand the math and the inferences
2. Demographics
Be aware if demographics affect the ALI scores
3. Motive
ALI must fit the objectives of the evaluation
4. Diversity
Choose ALI that fit the discipline
5. Openess
Make the limitations of ALI explicit, use
supplementary ALI
PROPERTIES OF A WELL CONSTRUCTED ALI
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

29. RQ 1: What are the characterisitics of ALI of academic
performance?
ALI are complex in character. They use different:
• disciplinary perspectives,
• objectives,
• operationalization of variables,
• requirements to data, and
• mathematical models that favour different disciplines
and seniorities.
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CONCLUSIONS
one size does NOT fit all.

30. RQ 2: To what extent are ALI appropriate in the
evaluation of researchers from different disciplines and
different academic seniorities?
• No seniority ALI were identified
• There is a disconnection between the performance of
researcher on the CV and in ALI.
• Some ALI are more appropriate in some disciplines
than others as grouping and ranking indicators or as
indicators that isolate unique information
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CONCLUSIONS

31. RQ 3: To what extent are the concepts being measured
defined in the construction of ALI?
• Many indicators are poorly or only partially defined
• Decisions based on these ALI could turn out to be less
effective than expected
• If we do not know how the developer has done the
measurement, we cannot repeat it, reflect on the
meaning associated with the concept and compare
results to previous findings
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
CONCLUSIONS

32. THANK YOU!!
ACUMEN partners: especially
Judit Bar-Ilan, Frank van der
Most & Paul Wouters.
My supervisors: Jesper
Schneider, Peter Ingwersen &
Birger Larsen
PhD/Masters group: Jesper,
Niels-Peder, Maria, Ole, Rikke,
Sara, Sille, Lisa Sutherland &
BADASS
Researcher/teacher support:
Johanne Maibohm, Susanne
Acevedo, Pia Dithmar &
Ragnhild Riis.
My boys:
Kim, Zander & Balthazar
My opponents: Henk Moed
Rodrigo Costas & Jeppe
Nicolaisen