Bibliometrics and scientometrics

Bibliometrics and
scientometrics
Part I: the classical performance indicators

Pablo Achard - NCCR Affective Sciences

Tuesday, February 24, 2009

1.What are we talking about?
2.Basic indicators
3. Performance indicators at the
micro level
4.Performance indicators at the
meso and macro level
P. Achard “Bibliometrics and scientometrics”


Deﬁnitions
“Bibliometrics” is introduced by Pritchard in 1969:
“the application of mathematical and statistical
methods to books and other media of
communication”

“Scientometrics” is the science of measuring and
analyzing science. As such, it includes the bibliometrics of
scientiﬁc books and articles; but it also takes into
account funding, demography, geography, etc.


What is it used for
1. Science and Technology Studies (epistemology, science
sociology, science history,…) and the scientiﬁc study of
idea spread

2. Biblioeconomics

3. Science management

4. Rankings (used by journalists, students, scientists,…)


What is it used for

Lindberg, PhD Thesis


Why is it becoming so
important

1. Availability of large databases

2. Increased use of management tools in the research
administration

3. Globalization of the education market


What is this talk focusing on?

The managerial use of performance indicators at the
micro level (individuals, groups), and at the meso and
macro levels (from departments to continents)


What is this talk focusing on?

The managerial use of performance indicators at the
micro level (individuals, groups), and at the meso and
macro levels (from departments to continents)

Therefore we will not deal with important topics like:
- The study of idea spread
- Demographics, education, collaborations
- Links between Research and Development, public

understanding of science


Direct input indicators

Number of scientists
•

R&D budgets
•


Direct output indicators

Number of publications / books / abstracts
•

Number of patents
•

Number of PhDs
•


Inﬂuence indicators
Measure the “passive” effects

Number of citations
•

Number of downloads
•

Pagerank of a website
•


Number of citations ≠ quality
The number of citations measures the reception

“if a paper receives 5 or 10 citations a year throughout
several years after its publication, it is very likely that its
content will become integrated into the body of knowledge
of the respective subject field; if, on the other hand, no
reference is made at all to the paper during 5 to 10 years
after publication, it is likely that the results involved do not
contribute essentially to the contemporary scientific
paradigm system of the subject field in question”

(Braun et al. 1985)


Number of citations ≠ quality

Book evaluation vs citations:
“The J-shaped distribution of
citedness” (Nicolaisen 2002)


Number of citations: issues


Normalized indicators

Number of citations / number of expected citations for
similar publications

The “Crown indicator” of Leiden University: Normalized
by publication type, year and ﬁeld


The myth of delayed
recognition

Papers highly cited only after a period of 5 years = 60

out of 450’000 published in 1980 or 0.013%



IF

IF

Statistics matter: the mean
value is a very rough
description of a distribution

(W. Glänzel, 2003)

Methodological pitfalls
Field delineation

Aggregation level

Time frame

Counting scheme

Data quality


Warning 1: Mating...
You wouldn’t mary someone based on his/her picture
on a mating website

But if you look for partners on a mating website, you
should care about the picture you upload
(using Photoshop is a matter of personal ethics)


Warning 1: Mating...
You wouldn’t mary someone based on his/her picture
on a mating website

But if you look for partners on a mating website, you
should care about the picture you upload
(using Photoshop is a matter of personal ethics)

Apply the same rules with evaluating someone’s

scientiﬁc value and bibliometrics


Warning 2: Distortion...
1. Funding agencies and science policy makers want to
get the best science for their money. But there are
more scientists than one can know and more ﬁelds
than one can understand.

2. They fund / hire preferentially projects / scientists
with the best score on a metric that correlates with the
‘quality’ they are looking for.

3. Scientists adapt their publication behavior to

increase their score on this metric.


Individual performance
indicators

Number of publications / books / abstracts

Number of patents

Problem: no indication of the inﬂuence of the work

Distortions: multiplication of small papers (Least

Publishable Unit); “honorary” authors


indicators

Number of citations received

Problems: one article can make it all (highly skewed); too
long to build

Distortions: self-citations; citation exchange


indicators
H-index:
quot;A scientist has
index h if h of his or
her Np papers have
at least h citations
each and the other
(Np - h) papers have
fewer than h
citations eachquot;


indicators
H-index:
Problems: very strong
quot;A scientist has
correlation with the
index h if h of his or
number of publications;
her Np papers have
perfect to compare
at least h citations
scientists... at the end of
each and the other
their career!
(Np - h) papers have
fewer than h
Distortions: same as
citations eachquot;

publication numbers


indicators
Many variants of the h-index


indicators
Number of publications x IF of the journal in which they
where published

Problems: highly noisy; very ﬁeld-dependent; IF doesn’t
determine future citation


The ﬁeld dependency of IF

(Leydesdorff, 2008)


IF-citation correlation?

Number of citations vs
IF of the journal in
which articles were
published for each and
every article of 4
researchers

(Seglen, 1997)


indicators
Number of publications x IF of the journal in which they
where published

Problems: highly noisy; very ﬁeld-dependent; IF doesn’t
determine future citation

Distortions: editors “hot topics”; worse at the

macroscopic level


How to increase the IF of your
university?


university?
Just cut the departments with low citation rates!


university?
Just cut the departments with low citation rates!

Do you think I’m kidding?


indicators

www.phdcomics.com

University rankings
Shanghai = alumni awards score + staff awards score +
highly cited score + Nature and Science articles +
articles + a mixture of all this/number of faculty

Times Higher Education = peer-review score +
employer-review score + staff/student score + citation/
staff score + international student score

UNIGE ranking 2007 = 105 ranking 2008 = 68


University rankings
Shanghai = alumni awards score + staff awards score +
highly cited score + Nature and Science articles +
articles + a mixture of all this/number of faculty

Times Higher Education = peer-review score +
employer-review score + staff/student score + citation/
staff score + international student score

UNIGE ranking 2007 = 105 ranking 2008 = 68

Let’s have a look at more serious indicators!

Publications per country


Publications vs GDP


Citations vs GDP (normalized)


Field dependency (again)

Clinical medicine (MED)
Biomedical research (BRE)
Biology (BIO)
Chemistry (CHE)
Physics (PHY)
Mathematics (MAT)
Engineering (ENG)
Earth and space sciences (ESS)

(Glänzel, 2003)


Conclusions?

“The use of a single index crashes the multidimensional
space of bibliometrics into one single dimension.”
(Wolfgang Glänzel)

“Not everything that can be counted counts,
and not everything that counts can be counted.”
(Albert Einstein)


Bibliometrics and scientometrics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Recently uploaded

Recently uploaded (20)

Bibliometrics and scientometrics

Editor's Notes