The dos and don'ts in individudal level bibliometrics

   ’  

 1
,  2
1Centre for R&D Monitoring and Dept MSI, KU Leuven, Belgium
2Centre for Science and Technology Studies, Leiden University, The Netherlands

 Introduction 
In the last quarter of the 20th
century, bibliometrics evolved from a
sub-discipline of library and information science to an instrument for
evaluation and benchmarking (G, 2006; W, 2013).
• As a consequence, several scientometric tools became used in a context for
which they were not designed (e.g., JIF).
• Due to the dynamics in evaluation, the focus has shied away from macro
studies towards meso and micro studies of both actors and topics.
• More recently, the evaluation of research teams and individual scientists
has become a central issue in services based on bibliometric data.
• The rise of social networking technologies in which all types of activities are
measured and monitored has promoted auto-evaluation with tools such as
Google Scholar, Publish or Perish, Scholarometer.
G  W, The dos and don’ts, Vienna, 2013 2/25

Introduction
There is not one typical individual-level bibliometrics since there are
diﬀerent goals, which range from the individual assessment of proposals or
the oeuvre of applicants over intra-institutional research coordination to the
comparative evaluation of individuals and benchmarking of research teams.
As a consequence, common standards for all tasks at the individual level
do not (yet) exist.
☞ Each respective task, the concrete field of application requires a kind
of flexibility on the part of bibliometricians but also the maximum of
precision and accuracy.
In the following we will summarise some important guidelines for the use
of bibliometrics in the context of the evaluation of individual scientists,
leading to ten dos and ten don’ts in individual level bibliometrics .

 Ten things you must not do … 
1. Don’t reduce individual performance to a single number
• Research performance is influenced by many factors such as age,
time window, position, research domain. Within the same scholarly
environment and position, interaction with colleagues, co-operation,
mobility and activity profiles might diﬀer considerably.
• A single number (even if based on sound methods and correct data)
can certainly not suﬀice to reflect the complexity of research
activity, its background and its impact adequately.
• Using them to score or benchmark researchers needs to take the
working context of the researcher into consideration.

Ten things you must not do …
2. Don’t use IFs as measures of quality
• Once created to supplement ISI’s Science Citation Index, the IF
evolved to an evaluation tool and seems to have become the
“common currency of scientific quality” in research evaluation
influencing scientists’ funding and career (S, 2004).
• However, the Impact Factor is by no means a performance measure
of individual articles nor of the authors of these papers (e.g., S,
1989, 1997).
• Most recently, campaigns against the use of the IF in individual-level
research evaluation emerged on the part of scientists (who feel
victims of evaluation) and bibliometricians themselves (e.g.,
B  HL, 2012; B, 2013).
◦ The San Francisco Declaration on Research Assessment (DORA) has
started an online campaign against the use of the IF for evaluation of
researchers and research groups.

3. Don’t apply (hidden) “bibliometric filters” for selection
• Weights, thresholds or filters are defined for in-house evaluation or
for preselecting material for external use.
Some examples:
◦ A minimum IF might be required for inclusion in oﬀicial publication
lists.
◦ A minimum h-index is required for receiving a doctoral degree or for
considering a grant application.
◦ A certain amount of citations is necessary for promotion or for
possible approval of applications.
This practice is sometimes questionable: If filters are set, they should
always support human judgement and not pre-empt it.
☞ Also the psychological eﬀect of using such filters might not be
underestimated.

4. Don’t apply arbitrary weights to co-authorship
A known issue in bibliometrics is how to properly credit authors for their
contribution to papers they have co-authored.
• There is no general solution for the problem.
• Only the authors themselves can judge their own contribution.
• In some cases, pre-set weights on the basis of the sequence of
co-authors are defined and applied as strict rules.
• The sequence of co-authors as well the special “function” of the
corresponding authors do not always reflect the amount of their real
contribution.
• Most algorithms are, in practice, rather arbitrary and at this level
possibly misleading.

5. Don’t rank scientists according to 1 indicator
• It is legitimate to rank candidates who have been short-listed, e.g.,
for a job position, according to relevant criteria, but ranking should
not be merely based on bibliometrics.
• Internal or public ranking of research performance without any
particular practical goal (like a candidateship) is problematic.
• There are also ethical issues and possible repercussions of the
emerging “champions-league mentality” on the scientists research
and communication behaviour (e.g., G  D, 2003).
• A further negative eﬀect of ranking lists (as easily accessible and
ready-made data) is that those could be used in decision-making in
other contexts than they have been prepared for.

6. Don’t merge incommensurable measures
• This problematic practice oen begins with output reporting by the
scientists them-selves.
◦ Citation counts appearing in CVs or applications are sometimes based
on diﬀerent sources (WoS, SCOPUS, Google Scholar).
• The combination of incommensurable sources combined with
inappropriate reference standards make bibliometric indicators
almost completely useless (cf. W, 1993).
• Do not allow users to merge bibliometric results from diﬀerent
sources without having checked their compatibility.

7. Don’t use flawed statistics
• Thresholds and reference standards for the assignment to
performance classes are proved tools in bibliometrics (e.g, for
identifying industrious authors, uncited or highly cited papers).
◦ This might even be more advantageous than using the original
observations.
• However, looking at the recent literature one finds a plethora of
formulas for “improved” measures or composite indicators lacking
any serious mathematical background.
• Small datasets are typical of this aggregation level: This might
increase the bias or result in serious errors and standard
(mathematical) statistical methods are oen at or beyond their limit
here.

8. Don’t blindly trust one-hit wonders
• Do not evaluate scientists on the basis of one top paper and do not
encourage scientists to prize visibility over targeting in their
publication strategy.
◦ Breakthroughs are oen based on a single theoretical concept or way
of viewing the world. They may be published in a paper that then
aracts star aention.
◦ However, breakthroughs may also be based on a life-long piecing
together of evidence published in a series of moderately cited papers.
☞ Always weight the importance of highly cited papers versus the
value of a series of sustained publishing. Don’t look at top
performance only, consider the complete life work or the research
output created in the time windows under study.

9. Don’t compare apples and oranges
• Figures are always comparable. And contents?
• Normalisation might help make measures comparable but only like
with like.
• Research and communication in different domains is differently
structured. The analysis of research performance in humanities,
mathematics and life sciences needs different concepts and
approaches.
◦ Simply weighting publication types (monographs, articles, working
papers, etc.) and normalising citation rates will just cover up but not
eliminate differences.

10. Don’t allow deadlines and workload to compel you to drop
good practices
• Reviewers and users in research management are oen overcharged
by the flood submissions, applications and proposals combined with
tight deadlines and lack of personnel.
◦ Readily available data like IFs, gross citation counts and the h-index
are sometimes used to make decisions on proposals and candidates.
• Don’t give in to time pressure and heavy workload when you have
responsible tasks in research assessment and the career of scientists
and the future of research teams are at the stake and don’t allow
tight deadlines to compel you to reduce evaluation to the use of
“handy” numbers.

 Ten things you might do … 
1. Also individual-level bibliometrics is statistics
• Basic measures (number of publications/citations) are important
measures in bibliometrics at the individual level.
• All statistics derived from these counts require a suﬀiciently large
publication output to allow valid conclusions.
• If this is met, standard bibliometric techniques can be applied but
special caution is always called for at this level:
◦ A longer publication period might also cover diﬀerent career
progression and activity dynamics in the academic life of scientists.
◦ Assessment, external benchmarking and comparisons require the use
of appropriate reference standards, notably in interdisciplinary
research or pluridisciplinary activities.
◦ Special aention should be paid to group authorship (group
composition and contribution credit assigned to the author).

Ten things you might do …
2. Analyse collaboration profiles of researchers
• Bibliometricians might analyse the scientist’s position among
his/her collaborators and co-authors. In particular, the following
questions can be answered.
◦ Do authors preferably work alone, work in stable teams, or prefer
occasional collaboration.
◦ Who are the collaborators and are the scientists rather ‘junior’, ‘peers’
or ‘senior’ partners in these relationships.
• This might help recognise the scientist’s own role in his/her research
environment but final conclusion should be drawn in combination
with “qualitative methods”.

3. Always combine quantitative and qualitative methods
At this level of aggregation, the combination of bibliometrics with
traditional qualitative methods is not only important but indispensable.
• On one hand, bibliometrics can be used to supplement the
sometimes subjectively coloured qualitative methods by providing
“objective” figures to underpin, confirm arguments or to make
assessment more concrete.
• If discrepancies between the two methods are found try to
investigate and understand what the possible reasons for the
diﬀerent results could be.
☞ This might even enrich and improve the assessment.

4. Use citation context analysis
The concept of “citation context” analysis was first introduced in 1973 by
M and later suggested for use in Hungary (B, 2006).
• Here citation context does not cover the position, where a citation is
placed in an article, or the distance from other citations in the same
document. It covers the textual and contentual environment of the
citation in question.
• It is to be shown that a research results is not only referred to but is
used indeed in the colleagues’ research and/or is scholarly
discussed. ⇒ The context might be positive or negative.
“Citation context” represents an approach in-between qualitative and
quantitative methods and can be used in the case of individual proposals
and applications.

5. Analyse subject profiles
Many scientists do research in an interdisciplinary environment. Even
their reviewers might work in diﬀerent panels. The situation is even
worse for “polydisciplinary” scientists.
In principle, three basic approaches are possible.
1. Considering all activities as one total activity and “define” an
adequate topic for benchmarking
2. Spliing up the profile into its components (which might, of course,
overlap) for assessment
3. Neglecting activities outside the actual scope of assessment
It depends on the task, which of the above models should be applied.
More research on these issues is urgently needed.

6. Make an explicit choice for oeuvre or time-window analysis
The complete oeuvre of a scientist can serve as the basis of the individual
assessment. This option should rather not be used in comparative
analysis.
• The reason is diﬀerent age, profile, position and the complexity of a
scientists career.
Time-window analysis is more suited for comparison, provided, of course,
like is compared with like and the publication period and citation
windows conform.

7. Combine bibliometrics with career analysis
This applies to the assessment on the basis of a scientist’s oeuvre.
• Bibliometrics can be used to zoom in on a scientist’s career. Here the
evolution of publication activity, citation impact, mobility and
changing collaboration paerns can be monitored.
• It is not easy to quantify the observations and the purpose is not to
build indicators for possible comparison but to use bibliometric data
to visually and numerically depict important aspects of the progress
of a scientist’s career.
 Some preliminary results have been published by Z  G (2012).

8. Clean bibliographic data carefully and use external sources
Bibliometric data at this level are extremely sensitive. This implies that
also input data must be absolutely clean and accurate.
• In order to achieve cleanness, publication lists and CVs should be
used if possible. This is important for two reasons:
◦ External sources help improve the quality of data sources.
◦ Responsibility with the authors or institutes is shared.
• If the assessment is not confidential, researchers themselves might
be involved in the bibliometric exercise.
• Otherwise, scientists might be asked to provide data according to a
given standard protocol that can and should be developed in
interaction between the user and bibliometricians.

9. Even some “don’ts” are not taboo if properly applied
There is no reason to condemn the oen incorrectly used Impact Factor
and h-index. They can provide supplementary information if they are
used in combination with qualitative methods, and are not used as the
only decision criterion.
Example:
• Good practice (h-index as supporting argument):
“The exceptionally high h-index of the applicant confirms his/her
international standing aested to by our experts.”
• estionable use (h-index as decision criterion):
“We are inclined to support this scientist because his/her h-index
distinctly exceeds that of all other applicants.”

10. Help users to interpret and apply your results
At any level of aggregation bibliometric methods should be
well-documented. This applies above all to level of individual scientists
and research teams.
• Bibliometricians should support users in a transparent manner to
guarantee replicability of bibliometric data.
• They should issue clear instructions concerning the use and
interpretation of their results.
• They should also stress the limitations of the validity of these results.

 Conclusions 
• The (added) value of or damage by bibliometrics in individual-level
evaluation depends on how and in what context bibliometrics is
applied.
• In most situations, the context should determine which bibliometric
methods and how those should be applied.
• Soundness and validity of methods is all the more necessary at the
individual level but not yet suﬀicient. Accuracy, reliability and
completeness of sources is an absolute imperative at this level.
• We recommend to use individual level bibliometrics always on the
basis of the particular research portfolio. The best method to do this
may be the design of individual researchers profiles combining
bibliometrics with qualitative information about careers and
working contexts. The profile includes the research mission and
goals of the researcher.

 Acknowledgement 
The authors would like to thank I R and J G for
their contribution to the idea of a special session on this important issue
as well as the organisers of the ISSI 2013 conference for having given us
the opportunity to organise this session.
We also wish to thank L W and R C for their
useful comments.

The dos and don'ts in individudal level bibliometrics

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (17)

Similaire à The dos and don'ts in individudal level bibliometrics

Similaire à The dos and don'ts in individudal level bibliometrics (20)

Dernier

Dernier (20)

The dos and don'ts in individudal level bibliometrics