1. The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm
BIJ
18,4 Peer evaluation to develop
benchmarking in the public sector
Lucio Cappelli
490 ` `
Facolta di Economia, Universita di Cassino, Cassino, Italy
Roberta Guglielmetti
`
Universita di Roma, Rome, Italy, and
Giovanni Mattia, Roberto Merli and Maria Francesca Renzi
`
Universita degli Studi Roma Tre, Rome, Italy
Abstract
Purpose – The urgency to strengthen the effectiveness and efficiency of public administrations has
led to the adoption in the public sector – in Italy as well as in other countries – of tools and models
inspired by the total quality management (TQM) approach, such as the common assessment
framework (CAF). A parallel need was felt within public structures to train people, the “peers”, who
aside from being able to implement the self-evaluation activities of their own administration, also
needed to be equipped with the necessary skills to complete external evaluation activities, namely
“peer evaluations”, with the ultimate aim of spreading quality management culture and best practices
in the field through an approach based on benchmarking. The purpose of this paper is to present the
results of a survey designed to determine the training requirements that a “peer” should acquire in
order to perform “evaluation” activities.
Design/methodology/approach – The paper has strong empirical connotations and is essentially
based on, first, a questionnaire given to evaluators/self-evaluators to identify the problems emerging
with the application of the CAF model in administrations that have adopted it. In this sphere a great deal
of attention has been paid to – in addition to the preparation of the questionnaire – the delicate task of
codifying the answers to open questions proposed to interviewees; and second, the investigation
inherent in the current educational choices in Italy in the TQM arena, specifically addressing the public
sector.
Findings – Apart from the analysis of data obtained from the investigation, presented with
descriptive statistics, the paper identifies the training content necessary: first, to place “peers” in a
position to be able to autonomously and fully carry out their evaluation work on the basis of the CAF
model; and second, to render the evaluation activities systematically comparable among the various
administrations.
Research limitations/implications – At the present time the research deals with the Italian
context. However, future investigations involving different European countries could be carried out,
taking the present results as a starting point for a benchmarking activity.
Practical implications – The paper puts forward the creation of a “network” of actors who manage
all the “peer evaluation” activities, from the provision of the training pack and monitoring the “peer”
exchange process, up to benchmarking initiatives and the dissemination of best practices.
Originality/value – The paper presents original data and information in order to identify the
training needs of individuals that actually use CAF, starting from the assumption that the training of
evaluators is the primary condition for promoting the adoption and diffusion of CAF in Italian public
administrations and thus maximizing its benefits.
Benchmarking: An International
Journal Keywords Peer evaluation, Benchmarking, Public administration, Common assessment framework,
Vol. 18 No. 4, 2011 Total quality management
pp. 490-509
q Emerald Group Publishing Limited Paper type Research paper
1463-5771
DOI 10.1108/14635771111147605

2. 1. Introduction Peer evaluation
Modernization of the public administration process, which today is still a fundamental
requirement for many countries, is framed within the intense debate that started at the
end of the 1970s, and developed during the 1980s and 1990s, around the problem of
changing government systems and public sector management. A key reference can be
found in the scientific management philosophy known as New Public Management, that
has certainly influenced – in different ways, to both acclaim and criticism – the Public 491
Administration change processes of several countries, notably Anglo-Saxon countries
and first amongst these the UK, in turn drawing research inspiration from the practical
experiences of these same countries (Boston et al., 1996, Denhardt, 1981, Dunleavy et al.,
2006, Osborne and Gaebler, 1993, Simon, 1976). In Italy, the start of the debate on Public
Administration reforms was greatly delayed with respect to the precursory countries:
for example, we recall that precisely in the 1980s – while other countries were
undertaking harsh structural reforms of their public sectors – the Italian public debt
explosion was talking place. Reinforcing the urgency of a radical change of the complex
Italian Public Administration was brought about the European unification process and
especially participation to the single currency, with the resulting constraints related to
debt and public deficit. These problems, which a good part of European countries share
with Italy, have been brought to the fore by some recent events (think of the Greek crisis
and the difficult situations in Portugal and Spain). In this context, numerous instruments
have been identified as useful references for the innovative management of public sector
organizations. Amongst these several models inspired by total quality management
(TQM), such as common assessment framework (CAF, to be discussed extensively
below), have become widespread.
This present work, which has as its central theme several issues specifically linked to
the use of CAF, draws its inspiration from two surveys developed by the authors: it thus
has a strong empirical connotation and the aim of identifying the key factors that can
promote the diffusion of the CAF instrument – such as internal and external
benchmarking activities aimed at improving public organizations – largely but not
exclusively referring to Italy. Both these objectives, as will be ascertained, can be
attained by means of the “peer evaluation” instrument.
In wanting to find an initial and concrete public act to motivate the process of change
in the Italian public sector we should probably start with Law no. 59 of March 15, 1997,
the so-called “Bassanini Law” that urges institutions to be more responsible towards
citizens in terms of transparency, efficiency and effectiveness. The aim of the process is
to strengthen awareness that the competitiveness of a country depends on the quality of
its public administration. In particular, through the concept of quality, the public utility
process supplied to citizens is linked to the organizational and managerial performance
of institutions. This is demonstrated on one side by the creation of initiatives leading to
the improvement of the service provided, to the transparency of management, to
rewarding the best performing administrations and, on the other side, by the
development of benchmarking (Camp, 1991) as a tool to compare best practices among
institutions.
The December 19, 2006 Directive “for quality-oriented public administrations” is
linked to this context of transformation and change. It aims to produce a substantial
increase in the number of administrations engaged in quality improvement, both on a
central and local level, with a focus on areas and contexts where citizen satisfaction

3. BIJ and trust in public administrations is quite low. The Directive promotes the introduction
18,4 of the TQM approach in the self-evaluation process as a standard managerial procedure
in public administrations. Self-evaluation can be defined as a systematic analysis of the
results of the organizations, leading to the identification of their strengths and
weaknesses in order to implement specific improvement actions.
The spread of self-evaluation, considered as the essential requirement for continuous
492 performance improvement and benchmarking, is supported by the adoption of
internationally defined models of excellence such as the EFQM model thoroughly tested
in the public and private sector. Within the public sector, the CAF (2006) is used more
and more often. This is a common European framework leading to the spread of quality
culture in the public sector and originates from the EFQM model. It is composed of nine
criteria, five Enablers and four Results, which can be used by the organization to initiate
the self-evaluation process. CAF is the reference point used by the Department of Public
Function to develop competition in Italy, entitled “Quality in Public Administration
Award”, and is dedicated to all public administrations that apply the CAF model. The
winner is the organization that achieves the highest scores in CAF-based assessments.
CAF is also the reference point used by the Department of Public Function to implement
peer evaluation in the public sector as a natural systematic process. It provides
European institutions with a common system of references and language to carry out
comparisons as well as promoting the definition of plans to improve critical processes in
administrations. The CAF model can therefore be regarded as a powerful tool at the
disposal of public administrations in the context of systematic benchmarking activities.
To activate a successful benchmarking process, institutions need to submit the
results of their self-evaluations for external validation, adopting the most suitable
options according to the needs of the organization, such as certification, participation in
international and national awards and peer-to-peer evaluations.
Peer evaluation is one of the tools available. This kind of activity, expressly indicated
in the Department of Public Function’s Action Plan, promotes the spread of quality
culture through the adoption of appropriate benchmarking policies, benchlearning and
the identification of best practices to be diffused in the public sector. Peer evaluation is
carried out by a taskforce of experts, the peers, who are in charge of evaluating quality
management performance in the organization. In order to be considered external
auditors, peers must have adequate knowledge, skills and preparation on TQM
principles and on the CAF model. Training is therefore a key factor to activate peer
evaluation in the public sector.
This paper presents some results of a research project carried out in cooperation
with the Department of Public Function. The main objectives of the research are:
(1) identification of the training needs of those “peers” that actually have to
implement evaluation activities; and
(2) designing a network of different actors able to assume the management
function – on a strategic, coordination and operational level – of the set of “peer
evaluation” activities.
2. Methodology
As we already mentioned, the paper has empirical bases. In fact, in order to achieve the
research objectives indicated above, two parallel field-investigations were carried out:

4. (1) The first relies on a questionnaire-based survey; such a tool was administered to Peer evaluation
evaluators/self-evaluators belonging to organizations where CAF was already
adopted, with the aim of defining the sectors where training requirements are
most needed.
(2) The second intended to identify the training offer based on TQM content
existing today and targeted at the public sector.
493
The research endeavours to define a training format aimed at:
.
guaranteeing a common and shared package of knowledge and skills;
.
facilitating the creation of a stakeholder network;
.
consolidating the peer evaluation system; and
.
speeding up the process of change indicated above.
The present paper highlights some of the most interesting results and proposals
deriving from the entire research project, based on the assumption that CAF is
considered the reference model to implement “peer evaluation”.
It is worth recalling that literature on CAF has produced numerous and significant
contributions since the earliest years of the model’s dissemination (2000). However, this
is almost entirely about works dealing with issues such as:
.
research inherent to the spread of CAF in European countries (Staes and Thijs,
2005; Engel and Fitzpatrick, 2003);
.
research on the dissemination of CAF in specific public administration sectors
(Dearing et al., 2006; Dipartimento della Funzione Pubblica, 2006); and
.
interpretation – and opinions – of the model in its concrete application in public
administration context (Heino et al., 2007; Hellenic Ministry of the Interior, 2006;
Conti, 2004).
There are practically no contributions focusing research on actors that carry out
evaluations and self-evaluations with the CAF model with the specific objective of
identifying the knowledge and skills necessary for the implementation of this activity.
Up to today – with specific reference to the situation in Italy, but probably common to
other European countries – training of evaluators has been based on “spot” initiatives,
neither systematically planned nor managed. Understanding the training needs of those
who practice concrete evaluation activities seems to be the first step in bridging this gap
and to establish a systematic process of “creating” evaluators in the context of a more
comprehensive approach based on “peer evaluation”. This paper – through the two
empirical surveys on which it is based – has the objective of defining both an
appropriate training course for “peers” and putting forward the constitution of a
network of public subjects for the management of all “peer evaluation” activities.
The carrying out and some significant results obtained in both investigations –
A and B – are described below.
3. First investigation (A): functioning and results
The first investigation has been developed on a sample of evaluators/self-evaluators,
in order to define the areas where training and support interventions are necessary.
The analysis of training needs is the first step of the entire training process to highlight

5. BIJ the gaps that a professional/operator needs to bridge to continuously adjust his knowledge
18,4 and skills. This is a basic step in defining each training objective and is pivotal in achieving
effective interventions. In Italy, ISFOL is a point of reference for concepts, models, schemes
and skill exemplification. The proposed model refers to expertise as a dynamic system of
knowledge, skills, attitudes and capabilities: it is achieved not only through knowledge, but
also through “know-how” and “know-how-to-be”. The training objective is to fill the gap
494 between actual skills, knowledge and capabilities and the optimum working standards as
established by the organization. In this research, in order to evaluate this gap, an
investigation among civil servants with direct experience of the CAF model was carried
out. The objectives of the investigation required drafting a questionnaire as the survey tool
to analyze peer evaluators in terms of knowledge, know-how and know-how-to-be.
The questionnaire was structured on the “gap model” and aimed to establish the
gap between the ideal evaluator/self-evaluator and actual knowledge, skills and
relational aspects.
The structure of the questionnaire was based on the following criteria:
. to refer to the knowledge and skills (Boyatzis, 1992; Hay Group, 1993) required of
evaluators/self-evaluators and to criticalities in the use of the CAF model (CAF,
2006);
.
to carry out a critical analysis with appropriate systematic sorting of the
knowledge and skills held by evaluators/self-evaluators; as a result of this
analysis, an initial set of knowledge and skills of reference was proposed to the
interviewees on each of the CAF criteria;
.
to define a picture of the evaluation/self-evaluation experiences acquired by
interviewees;
.
to also duly consider the “immaterial” aspects of the evaluation, such as the
relational aspects that somewhat impact on the outcome of the process;
.
to obtain direct indications on training programmes for evaluators/
self-evaluators, investigating the balance between theory and application in
room and on the job training, live or distance training; and
.
to maximize personal and free contributions of interviewees through the ample
use of open questions.
The questionnaire was composed of six sections with a total of 31 questions: seven of
which were open questions and the remaining were multiple choice and scale questions
(intensity, ratios, semantic differentials). The questionnaire was submitted as a pre-test
to 15 target individuals and some minor optimization adjustments were made as a
consequence.
It should be pointed out that the open questions in reality were more than seven:
a particular question requires in fact an answer (open) for each of the nine criteria of the
CAF model. When taking this into consideration, open questions total 15.
As known, open questions pose significant additional problems for the elaborations
of answers. The main issues arising from open questions are:
.
for people filling in the questionnaire: the difficulty in summarizing answers; and
.
for people analyzing the questionnaire: the difficulties in respect of codification,
risking loosing relevant information.

6. Nonetheless, on one side the interpretation and codification of open questions requires Peer evaluation
additional work in terms of time and commitment but on the other they are rather
useful to investigate the subject in-depth, thus acquiring ulterior information. In fact,
a large quantity and variety of comments and observations on criticalities/difficulties
encountered by evaluators/self-evaluators was collected. In order to organize the
informative content a number of steps were followed:
(1) differentiate the multiple concepts contained in the same answer in order not to 495
lose the informative content;
(2) homogenize the language used by different interviewees to express similar
concepts;
(3) interpret the answers through synthetic yet representative expressions;
(4) codify answers via a homogeneous classification system, suitable to implement
the statistical descriptive analysis;
(5) eliminate redundancies (overlapping codification); and
(6) group the codified answers into tailor-made macro-categories. Each macro-area
contains those expressions codified with the same meaning.
With this approach it was possible to maximize the amount of information and to
proceed with the statistical analysis of the results in order to establish the critical
difficulties to focus on.
It should be specified that the choice to use descriptive statistical tools is coherent
with the type of information and data that existed. Indeed, it should be stressed that the
main difficulty – which required an intense work of interpretation, homogenization
and classification of answers – was in achieving a simple and instant representation of
a complex situation of a mass of highly heterogeneous information.
The questionnaire was made up of 27 closed questions investigating 6 themes:
knowledge and skills; evaluation experiences; use of the model; improvement plans and
designs; rational aspects; evaluation training programmes.
Opinions were requested on the importance, utility and difficulty of each of the
above-mentioned themes, depending on the specific investigation area. It should be
stressed that the analysis of the results of open and closed questions was performed
collectively in order to maximize the survey’s informative value. The population to
whom the questionnaire was submitted was identified through a Formez database and
was made up of 160 CAF managers and evaluators trained during the two editions of the
“Quality in the Public Administration Award” that took place in Italy in 2005 and 2007.
The survey was submitted via e-mail. A total of 78 questionnaires were compiled and
returned, representing 49 percent of the population. The sample was mainly constituted
of individuals who worked as report writers (41 percent), followed by self-evaluators
(32 percent) and evaluators (26 percent). The interviewees took part in the following
quality training programmes: CAF Evaluator Course (30 percent); ISO 9000 Audit
Course (25 percent); “The paths of Quality” Laboratory projects (15 percent). In the
evaluation team, 71 percent of interviewees worked as team member, while 29 percent
worked as “team leaders”. In accordance with the research aims, simple statistical tools
were adopted with the purpose of defining the spread of evaluation and self-evaluation
approaches within public administrations as much as possible.

7. BIJ The following statistical instruments were used to analyse the results of open
18,4 questions, following their codification:
.
bar chart, to identify the frequencies of distribution for each codified expression;
.
Pareto chart, to identify priority criticalities/difficulties according to the standard
80-20 rule; and
496 .
cause-and-effect chart (or Ishikawa chart (Ishikawa, 1985)), to represent the
primary and secondary causes of a given effect of the study.
Since in this context it is not possible to provide an overview of the processing of
answers, selection criteria based on a double principle were adopted:
(1) the “object” of the survey principle, the CAF model; and
(2) the principle based on the “role” of interviewees: evaluators and self-evaluators.
It is important to point out that the data analysis is based on interviewee responses.
From the “object” principle point of view, the most relevant element is the particularity
of the CAF model, which can be divided into two parts: as mentioned above, the first
area includes the criteria defined as Enablers, the second criteria are defined as Results.
The Enablers examine the managerial approaches used by the organizations and how
they are applied in order to achieve their planned objectives, whereas the Results focus on
indicators and measurements to demonstrate what the organization is achieving.
Analyzing the Results and comparing them with the planned objectives allows
implementing continuous improvements through innovation and benchlearning processes.
The hypothesis to be verified is whether the model’s dichotomy is confirmed in the
criticalities/difficulties indicated in the survey, so that they are differentiated when
referring to Enablers rather than Results in the CAF model.
From the “role” principle point of view, the key element lies in the different
experiences of interviewees: as evaluators and self-evaluators. Evaluators work in
institutions that they do not belong to, carrying out evaluations starting out from
documentation (the so-called “application”) of the organization to be assessed, and
ending with the so-called “site visit”. Self-evaluators carry out assessment activities in
the institutions they belong to, producing the respective documentation. Of interest is
understanding whether different criticalities/difficulties emerged in the survey and to
what extent they are represented by the two categories of interviewees.
Figures 1 and 2 analyze the data gathered according to the CAF model.
Through a comparison of the two models, it is possible to note that in the case
of Results the criticalities/difficulties are clustered around a few micro-categories.
In particular, the macro-category “measurement indicators” represents 50 percent of the
total, followed by “data analysis” and by “model use” with 15 percent. In the case of
Enablers, the most important macro-categories are “model use” with 25 percent,
followed by “evaluation documents” and “evaluations criteria” with 20 percent and
“quality approach” with 15 percent. It seems clear that in the evaluation of Results the
most relevant aspect in terms of difficulties is linked to the availability of measurements
and/or identification of suitable indicators, followed by the interpretation of data to
analyze. In the evaluation of Enablers, the most relevant factor is the difficulty to apply
the model in the specific context of the organization, followed by the difficulties
regarding evaluation documents, such as drafting and interpreting the application,

8. 100.00 Peer evaluation
240 90.00
80.00
Cumulative percentage
200
70.00
160 60.00
Counts
50.00
120
40.00 497
80 30.00
20.00
40
10.00
0 0.00
se
s
ria
ch
ps
s
e
n
ge
e
t
am
g
si
t
or
m
m
io
in
en
u
hi
oa
ka
vi
ite
at
he
Ti
te
at
n
el
m
ns
ai
pr
lin
ic
te
in
cr
sc
n
od
cu
Tr
io
nd
Si
io
ap
rd
n
el
n
M
do
at
at
io
oo
tI
tio
y
od
el
Figure 1.
u
at
lit
en
n
lc
al
lr
ua
M
io
u
ua
Ev
m
al
na
na
al
at
Main difficulties
Q
re
Ev
Ev
io
u
so
su
al
at
er
Ev
ea
iz
encountered in “Enablers”
rp
an
M
te
rg
In
O
100.00
280 90.00
Cumulative percentage
240 80.00
70.00
200
60.00
Counts
160 50.00
120 40.00
30.00
80
20.00
40 10.00
0 0.00
s
is
e
e
ge
ria
ts
am
or
us
m
ys
en
ka
ite
at
he
te
al
el
m
Figure 2.
lin
ic
cr
sc
n
an
od
cu
nd
tio
n
el
n
M
do
a
tio
ti
tio
ua
at
od
Main difficulties
en
n
D
ua
al
ua
M
tio
m
Ev
al
al
ua
re
encountered in “Results”
Ev
Ev
su
al
Ev
ea
M
the evaluation of the single criteria and the general approach to quality. We should point
out that some participants worked as both evaluators and self-evaluators but were not
taken into account in this context.
The data regarding knowledge (Table I) is consistent with the results of Enablers in
the model, underlining the need for training processes focused on TQM content and
CAF model principles.
Intersecting the data on open questions with some data on closed questions, shown
in Figures 1 and 2, the necessity to provide adequate training on quality, especially
through the creation of appropriate competencies (Table II), is confirmed and
consistent with the above-mentioned CAF Results.
A further in-depth examination of CAF in Table III shows the average difficulties
encountered in each of the nine CAF criteria.
The most relevant information is:

9. BIJ
List of knowledge Average usefulness (1-10)
18,4
TQM principles and tools 9.13
CAF/EFQM models 9.12
Audit techniques 8.22
Interpersonal relationship management 8.19
498 P.A. Organization 8.18
Communication (internal and external) 8.03
Benchmarking 7.77
Laws on P.A. modernization and simplification 7.64
Management control 7.54
Regulations ISO 9001, 9004, 14001, 19011, SA8000 7.36
IT 6.95
Table I. Statistics 6.88
Average usefulness Marketing 6.14
of knowledge Economic theory 5.92
List of skills Average usefulness (1-10)
Setting-up a system of indicators 8.94
Representation and reading of processes 8.87
Improvement tools and methods 8.81
Audit interview management 8.58
Gathering, reading and quantitative data processing 8.35
Strategic planning 8.18
Evaluation tools for personnel / executives 8.12
Planning and analysis of people satisfaction surveys 7.96
Table II. Project management 7.96
Average usefulness Mapping of personal skills 7.64
of skills Identification of training needs 7.52
List of criteria Average difficulty (1-10)
Company results 6.05
Results on key performances 5.68
Processes 5.12
Citizen/client oriented results 4.96
Personnel results 4.68
Policies and strategies 4.63
Table III. Partnership and resources 4.59
Difficulty of CAF criteria Leadership 4.36
(closed question) Personnel 4.07
.
Criteria 3 “Personnel” is the least difficult to evaluate. This result may be linked
to the fact that some personnel management methods are without doubt used in
all organizations, thus making their evaluation easier.
. Among the five easiest criteria to be evaluated, four belong to factors. This
evidence further strengthens the hypothesis of a clear distinction between

10. Enablers and Results in the CAF model, consistent with several Peer evaluation
criticalities/difficulties highlighted by the investigation.
The disaggregated data of interviewees, evaluators or self-evaluators is analyzed below.
It is important also in this case to understand whether criticalities/difficulties are
different with reference to the two groups or whether, to the contrary, the “role” is
irrelevant. 499
Figures 3 and 4 report the data on evaluators and self-evaluators.
It is evident that criticalities/difficulties are clearly different in the two groups. The first
three micro-areas for evaluators are “measurement indicators” with 30 percent,
“evaluation documents” with 15 percent and “model use” with 10 percent. On the other
hand the first three macro-areas for self-evaluators are “interpersonal relationships” with
20 percent, “evaluation scheme” and “measurement indicators” with 15 percent. Therefore,
the main difficulties in the evaluation process for evaluators lie in the availability of data
160 100.00
150
90.00
140
130 80.00
120
Cumulative percentage
110 70.00
100
60.00
90
Counts
80 50.00
70
40.00
60
50 30.00
40
30 20.00
20
10.00
10
0 0.00
s
n
ts
e
e
e
a
ch
am
ps
s
is
t
si
or
ge
us
im
m
ri
io
en
ys
hi
oa
vi
ite
at
he
te
at
ka
T
al
el
um
ns
pr
ic
te
lu
cr
sc
n
lin
an
od
tio
nd
io
Si
ap
va
oc
n
n
M
at
a
io
el
la
ti
E
io
D
y
at
lu
od
re
at
lit
en
at
D
va
lu
Figure 3.
lu
ua
M
al
m
va
E
va
on
re
Q
E
su
E
rs
Difficulties for evaluators
pe
ea
M
er
t
In
100.00
180 90.00
Cumulative percentage
160 80.00
140 70.00
120 60.00
Counts
100 50.00
80 40.00
60 30.00
40 20.00
20 10.00
0 0.00
ps e rs ts se h t s s e on ng ffer visit
hi m to en
m
tea el u ac men lysi ge
Ti
m
ati ini
ns he ca m ro e a ka o te
o ns
c
nd
i u
tio
n od pp rov a an l lin alu Tra ning Si
ati o ti a oc M a t ev i
lr
el ati en ion
d
alu al ity Imp Da o de r ia T ra Figure 4.
a lu em luat Ev ite
on va r Qu M
Cr Difficulties for
rs E su va
pe ea E self-evaluators
ter M
In

11. BIJ and measurements, and in the interpretation of the documents provided by the institution
18,4 to be assessed. Instead, for self-evaluators the main difficulty lies in internal relationships
resulting from the evaluation activity: management of the relationship with executives
and colleagues is quite sensitive and critical for those in charge of collecting proof,
information and data to draft evaluation documents according to the CAF model.
The second macro-category for evaluators is the model’s “evaluation scheme”. It is
500 interesting to note that the macro-category “time” is in fourth position for evaluators,
while it is clearly less relevant for self-evaluators. Therefore, overall, the data
disaggregated into evaluators and self-evaluators confirms the initial hypothesis of a
clear distinction between the criticalities/difficulties of the investigation. This
distinction, as well as the dual structure of the model, needs to be duly considered in
order to identify the basic format of the training content.
From this stratification, we were able to obtain useful information in aid of
designing a customized training format.
4. Second investigation (B): functioning and results
The second investigation focused on the mapping and analysis of the training content
on TQM principles and models for civil servants, leading to defining a scenario to
verify the state of the art in the training offer.
The aim was to verify the current situation on TQM training in public
administrations through the identification of the main training agencies in Italy and
the courses available on this subject.
In detail, the analysis focused on:
.
Identification of the training agencies that thus far offer training courses on these
subjects.
.
Analysis of the training courses offered by universities and Higher Learning
Public Administration Institutions with an in-depth examination of contents
(modules, educational units, hours dedicated to TQM issues).
Attention was focused on two types of offers considered significant for P.A.: 1st and
2nd Level Masters and courses provided by Higher Learning P.A. Institutions.
The research activity was carried out in September 2008 and was designed as follows:
.
setting-up the work methodology and defining the survey’s scope;
.
research of existing TQM training offers in Italian Universities; and
.
research of the existing TQM training offers in P.A. Schools and Colleges.
The starting point was to verify the availability of a database on Italian universities.
Through consulting the Ministry’s web site, and in particular the Statistics Office’s, it
was possible to draft a list of educational offers in Italian universities in the academic
year 2006/2007. However, after a more in-depth analysis, this database was quite
limited due to fact that the Masters are activated on a yearly basis.
A new exercise was carried out to identify the Masters offered by universities and
Higher Learning P.A. Institutions for the academic year 2007-2008. The starting point was
Cineca’s list of Italian universities (www.cineca.it). According to this source, in June 2008,
Italian Universities numbered 84. A census survey was conducted on the universities’ web
sites, intersecting university, faculty and department data available online.

12. Through this survey of web sites, it was possible to collect basic information on Peer evaluation
Masters and to classify them into a single dataset according to the following
parameters: University, Faculty, Number of 1st Level Masters offered, Number of
2nd Level Masters offered, Masters not available online, Public Administration targets,
Quality contents, Web site, Notes.
With the aim of verifying how quality culture is spread among trainers, a detailed
desk analysis of University Masters was carried out, selecting Masters presenting both 501
of the following characteristics:
.
targeting Public Administration internal personnel (graduates/executives/civil
servants); and
.
examining issues related to TQM.
For these Masters a schema was created with the following characteristics (Table IV).
Up to 4th July 2008, the offer provided by Universities in Italy (public and private)
was 1,166 1st Level Masters and 933 2nd Level Masters, for a total of 2,099 Masters.
In particular, University Masters (1st and 2nd Level) targeted at public administration
and offering contents on quality are 85. It is worth noting that six among those are
MIMAPs (Master in Public Administration Innovation and Management) offered by the
Universities of Cassino, Catania, Foggia, Roma Tre, Roma Tor Vergata, Salerno.
A total of 49 percent of the 85 Masters identified are based in central Italy, while
North and South Italy share the remaining quota.
Figure 5 shows the distribution of the 85 Masters with reference to the Faculties
they belong to; in 14 out of 85 Masters it was not possible to identify the Faculty.
TQM issues are mainly dealt with and offered to Health Sector Public Adminstrations
through the Faculty of Medicine, namely 28 percent of the total, and the Faculty of
Economics with 24 percent. The other faculties show lower percentages.
Out of the 85 Masters, 33 are 1st Level Masters and 50 are 2nd Level Masters (it was
not possible to identify the type of level for 2 of these). Therefore, the 1st and 2nd Level
Masters directed at Public Adminstrations with contents on quality are, respectively,
3 percent and 5 percent of all Masters offered in Italy.
With reference to the 85 Masters, the average cost for the 1st Level is e 3,180 versus e
4,100 for the 2nd Level. With reference to the target, Figure 6 shows that the 85 Masters
are mainly directed at the graduate/executive/civil servant category. About 23 out of
85 Masters specifically target Health Sector professionals: this could be due to the fact
that it is mandatory for health workers to acquire 5 credits per year. Only five Masters
target managers from the School Sector.
University Credits Hours on quality issues
Faculty Duration (in months) Didactic units for each module
Name of Master Places available/total hours Cost
Web address Total modules Target
Master headmaster Details of modules on quality Contacts (mail, telephone) Table IV.
Level Academic year Master classification

13. BIJ 25
18,4 20
Count 15
10
502
5
0
e
y
es
g
es
y
er
y
w
in
om
in
ar
og
La
th
nc
nc
ic
er
lin
ol
O
Figure 5.
on
ie
ie
ed
ne
ci
ip
sc
sc
Ec
M
gi
So
isc
Faculty offering masters al
n
En
io
rd
ic
at
lit
te
with contents on
uc
In
Po
Ed
quality in P.A.
Faculty
60
50
Counts
40
30
20
10
0
Graduates/
executives/
civil servants
Health workers
School
executives
Graduates and
public sector
professionals
Figure 6.
Target
Target
Strong diversity emerged in the language used to define the specific training contents.
Only for 74 out of 85 Masters was it possible to extract information on the modules
dedicated to quality.
Through interpretation and synthesis, the content diversity was brought back to
12 key categories: Figure 7 shows how contents are spread in the different educational
proposals. It should be pointed out that information on module content was not
available for all 85 Masters and that modules focusing on EFQM/CAF excellence
models are almost entirely non-existent.
The educational offer proposed by P.A. Schools and Colleges (henceforth referred to
as Schools) was simultaneously taken into account with the Masters provided by
Italian Universities.
Owing to the fact that an official list of these Schools is not available, a survey was
carried out using a list of central institutions (Constitutional Bodies, Ministries and
Agencies). Overall, 35 institutions were taken into account and 68 P.A. Schools and
Colleges were identified.
The Schools forming part of the survey can be divided into two different types:
(1) general Schools, providing training for all civil servants; and
(2) specialized Schools, providing training for employees of specific public
institutions.

14. 30 Peer evaluation
25
20
Counts
15
10
5
0
503
s
.
ce
e
ly
m
n
t
..
ip
t
s
t..
en
en
se
el
ic
io
r.
pp
te
sh
ur
ke
od
rv
es
em
m
ct
fo
ys
er
so
Su
ar
se
fa
rn
oc
em
ts
ds
ov
ad
re
m
tis
ve
pr
in
en
ho
Le
pr
nc
an
sa
d
go
d
y
em
an
im
et
lle
an
um
lit
er
em
y
ua
n
ag
lit
ce
m
y
n
-h
io
lit
io
stu
ua
Q
an
Ex
tiv
ng
at
ua
at
Eq
m
ic
Co
ita
ni
iz
Q
un
y
an
an
nt
lit
Figure 7.
m
ua
Pl
rg
ua
m
O
Q
Q
Key contents on quality
Co
Contents
From the analysis emerged that the Schools (general and/or specific) offer 319 courses,
which can be classified as follows:
.
training courses: training activity aimed at strengthening the knowledge of
general or specific content, not directly linked to the attainment of qualifications;
.
qualification courses: training activity aimed at the achievement of a specific
professional qualification;
.
refresher and retraining courses: training activities aimed at the adjustment of
professional knowledge liked to a specific qualification;
.
degree course: training activity recognized by institutions (set up in cooperation
with universities) and granting the attainment of 1st or 2nd Level Degrees;
.
Master: training activity recognized by institutions granting the attainment of
higher education degrees; and
.
conferences, seminars, other activities: spot training activities aimed at
popularizing and spreading specific issues.
It should be highlighted that the survey on courses offered by Schools, and carried out
following the criteria listed above, presents some gaps. This is largely due to the fact
that the courses offered adopt non-uniform presentation criteria.
Figure 8 shows the training offer of Schools according to the above-mentioned
classification. According to the data, 79 percent are training courses while 6 percent are
Masters. The Master category is composed of university masters and training courses
referred to as Masters by the Schools. The latter are different from the former in terms
of duration and attainable qualifications.
A further area of investigation concerned the distribution of courses offered in
comparison with their target. The courses target:
.
top levels (executives, directors, managers, high levels);
.
operative levels (technicians, employees, medium-low levels, officers, doctors,
secretaries, trainers); and
.
top levels and operative levels.

15. BIJ This information was obtained in 106 out of 319 courses. The results are shown in
18,4 Table V.
As expected, the stratification analysis demonstrated that General Schools focus on
top level training while Specialized Schools target operative levels.
An in-depth analysis was conducted on the contents of courses focusing on quality.
The information available was limited due to the fact that the Schools do not freely
504 disclose their programme contents. Thus, a qualitative approach was adopted, listing
the contents of modules and didactic units. The information collected was arranged
according to key contents in line with the research carried out on university Masters.
The most widespread key contents are marketing and communications, leadership,
human resource management, customer satisfaction, quality improvement,
e-government, organization and processes, strategic planning, quality management
system, quantitative decision-making methods.
Through a comparison of the mapping of Masters offered by Italian Universities
and P.A. Schools and Colleges, the similarity in subjects dealt with became evident.
Furthermore, the absence of initiatives leading to the presentation of topics on
excellence models was confirmed.
5. Research implication
In accordance with the research aims, building a taskforce of people working in the
public sector capable of implementing peer evaluations requires integrating the
information collected in both the explorative and descriptive phases.
Research on training proposals addressed to Public Administrations in Italy
emphasizes the segmentation of the educational offer due to the absence of a common
vision and shared objectives. This results in the impossibility of creating a taskforce of
public employees able to independently spread quality culture in the public sector,
according to a common vision and disengaged from consultancies.
In particular, the first analysis of the needs expressed by public employees taking
part in the survey and with past experience in evaluation and self-evaluations,
Refresher and retraining
2%
2% Degree course
5% 6%
6% Training
Figure 8. Master
Classification of the Qualification
training offered by P.A.
Institution of Seminars, congresses
Higher Learning and other activities
79%
Profile Frequency (%)
Table V.
Targeted attendees Operative 51
of P.A. Schools and Top level 41
Colleges courses Top and operative levels 8