The Review looks at the various components of assessment and evaluation frameworks. These include:
Student assessment; Teacher appraisal; School evaluation.
Education system evaluation;
Other types of evaluation (programme evaluation, evaluation of school leadership etc)
OECD Reviews of Evaluation and Assessment in Education: Czech Republic
1. OECD Reviews of
Evaluation and Assessment in Education: Czech Republic
By Paulo Santiago, Alison Gilmore, Deborah Nusche and Pamela Sammons
www.oecd.org/edu/evaluationpolicy
Conference, Prague
The main conclusions and policy
recommendations
Presentation by
Paulo Santiago
Directorate for Education, OECD
15 March 2012
2. Outline of presentation
Part 1
The OECD Review on Evaluation and Assessment Frameworks for
Improving School Outcomes and the Review of Evaluation and
Assessment in the Czech Republic
Part 2
Conclusions and Recommendations
• The Evaluation and Assessment Framework
• Student assessment
• Teacher appraisal
• School evaluation
• Education system evaluation
3. PART 1
(1) The OECD Review on Evaluation and
Assessment Frameworks for Improving School
Outcomes and;
(2) the Review of Evaluation and Assessment in
the Czech Republic
4. OECD REVIEW - OBJECTIVES
●Purpose: To explore how systems of evaluation and assessment can be used to
improve the quality, equity and efficiency of school education.
A Review of national approaches to evaluation and assessment in school education
●The Review:
Synthesises research-based evidence on the impact of evaluation and
assessment strategies and disseminate this knowledge among countries.
Identifies innovative and successful policy initiatives and practices.
Facilitates exchanges of lessons and dialogue.
Identifies policy options for governments to consider.
5. OECD REVIEW - SCOPE
• The Review looks at the various components of assessment and evaluation
frameworks. These include:
• Student assessment;
• Teacher appraisal;
• School evaluation;
• Education system evaluation;
• Other types of evaluation (programme evaluation, evaluation of school
leadership etc)
• Comprehensive approach: Investigation of each component individually, as
well as the coherence of the framework as a whole (including the links between
the different components).
6. OECD REVIEW - KEY ISSUES
Key policy issues for analysis
– Governance: Designing a systemic framework for evaluation and assessment
– Procedures: Ensuring the effectiveness of evaluation and assessment procedures
– Competencies: Developing competencies for evaluation and for using feedback
– Use of results: Making the best use of evaluation results
– Implementation: Implementing evaluation and assessment policies
7. OECD REVIEW - BROAD FEATURES
●Oversight by the Group of National Experts (GNE) on Evaluation and Assessment
https://community.oecd.org/community/evaluationpolicyinschools
●Countries working collaboratively with the Secretariat
Countries do part of the work of the Review, a national co-ordinator is nominated, dialogue within
countries is encouraged
●Countries exchanging lessons and experiences, in particular through GNE
meetings
●Collection of views and perspectives from a range of stakeholders
Through their participation in GNE meetings and their interaction with Country Review expert
teams
●A diverse set of external experts involved in the Country Reviews
Involvement of top academics and policy makers to bring expertise into the Review
●Ensuring a range of principles are respected
Independence of analysis; inclusiveness in terms of the views and perspectives which are collected;
extensive reviews of the literature; evidence-based policy advice; site visits to interact with
school realities; contextualisation of policy advice.
8. OECD REVIEW - METHODOLOGY
Analytical strand
• Identifying the key questions for analysis and the background information needed
from countries
• Reviewing the literature and evidence on the impact of evaluation and
assessment procedures
• Gathering data on countries’ policies and practices (Country Background Reports)
Country Review strand
• Country Reviews provide specific advice to individual countries.
• OECD-led Review Team with external experts
• The scope and focus is determined by the country in consultation with the
Secretariat
Synthesis report
• Comparative report to analyse policy options and highlight good practices across
countries.
Collaboration with other OECD activities and international agencies
• INES (in particular NESLI), PISA, TALIS, CERI’s ILE , Longitudinal Information Systems and
Governing Complex Education Systems project, CELE’s on evaluating quality in
educational facilities, GOV’s work on Public Sector Evaluation.
• EC, Eurydice, the World Bank, Standing International Conference of Inspectorates(SICI),
UNESCO, BIAC, TUAC.
9. OECD REVIEW - PARTICIPATION
●Twenty four systems are preparing a Country Background
Report:
Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile,
Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Korea, Luxembourg, Mexico,
Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia and Sweden.
●Twelve countries also opted for a Country Review:
Australia, Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico,
New Zealand, Norway, Portugal, Slovak Republic and Sweden.
●All OECD Member countries and Observers to the Education Policy
Committee are members of the Group of National Experts on
Evaluation and Assessment
10. REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC
●Objectives:
1. To provide insights and suggestions for policy development in the Czech
Republic;
2. To inform the wider international community about (a) the main features of the
Czech Republic’s evaluation and assessment policies; and (b) innovative and
effective approaches to evaluation and assessment in the Czech Republic; and
3. To inform the final synthesis report from the project.
●Structure of visit (29 March – 5 April, 2011):
• Review team with 4 members: 2 OECD Secretariat members; and researchers
from New Zealand and the United Kingdom;
• Interacted with about 200 individuals; had about 45 meetings amounting to
about 50 hours of discussions; visited 6 schools in Prague, Ostrava and Liberec
• Discussions were held with a wide range of national, regional and local
authorities; education officials; the Schools Inspectorate; parents’ organisations;
representatives of schools and school directors; teacher educators; and
researchers with an interest in evaluation and assessment issues.
11. REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC
●Relevant features:
1. External perspective: looking from a distance;
2. Independent perspective: no review team member has a vested interest in
the system;
3. Not an examination: qualitative analysis that seeks to provide an input of a
specific nature to the internal debate;
4. Bring together research evidence, data and information available in a broad
comprehensive comparative framework.
●Country Review report released on 30 January, 2012 and
available at: www.oecd.org/edu/evaluationpolicy
14. The Evaluation and Assessment Framework
Strengths
There are common references at the national level to provide the basis for
evaluation and assessment
4-year Long-term policy objectives with indicators; Framework Education
Programmes
There are good conditions for adapting learning to local needs
School Education Programmes: curriculum innovation; collaborative work
Regional 4-year Long-term policy objectives
Responsibilities across the evaluation and assessment framework are well
articulated
MEYS, CSI, Regions and municipalities, schools
15. The Evaluation and Assessment Framework
Strengths (continued)
There is a range of initiatives to strengthen evaluation and assessment in
the school system
National standardised tests, common part of the school-leaving examination,
external school evaluation, school self-evaluation mandatory, national
indicators on education
There is an “open door” climate among teachers
There is a good principle of supporting policy work with specific expertise
Institutes with specialised expertise, data collections, OECD projects
There are some reporting requirements:
Education database, inspection reports
16. The Evaluation and Assessment Framework
Challenges
The evaluation and assessment (E&A) framework needs to be completed
and made coherent
o There is no integrated E&A framework
o The E&A framework is incomplete (formative assessment of students,
moderation of marks, systematic teacher appraisal, school self-evaluation
incipient, no framework for school leadership appraisal)
o Some articulations within the E&A framework are not sufficiently developed
There is little attention to equity and inclusion in the evaluation and
assessment framework
The Framework Education Programmes are not perceived as specific
enough to guide teaching and assessment
Little shared understanding about what constitutes adequate, good and excellent
performance
It is unclear that the students are at the centre of the evaluation and
assessment framework
17. The Evaluation and Assessment Framework
Challenges (continued)
There is a narrow understanding of the purposes of evaluation and
assessment
Perceived as instrument to hold stakeholders accountable, to “control” and assess
compliance with regulations
There is a need to strengthen competencies for evaluation and assessment
across the system
The articulation between levels of government and the support from the
centre are limited
o Concerns about the lack of systematic application of national directions,
inconsistency of practices and little capacity or commitment to developing
quality frameworks.
o Weak articulations between the different decision-making levels.
o Limited provision from the centre of tools and guidelines to assist evaluation and
assessment activities
There are challenges in the implementation of some evaluation and
assessment initiatives
18. The Evaluation and Assessment Framework
Policy recommendations
Better articulate learning goals
o Establish clear goals for education and make equity and inclusion more prominent
o Clarify reference points and criteria for quality in evaluation and assessment
Integrate the evaluation and assessment framework
Develop a strategic plan or framework document that conceptualises a
complete evaluation and assessment framework and articulates ways to
achieve the coherence between its different components
Strengthen some of the components of the evaluation and assessment
framework
Teacher appraisal, appraisal of school leaders, formative student assessment, school
self-evaluation
Further develop some articulations within the evaluation and assessment
framework
19. The Evaluation and Assessment Framework
Policy recommendations (continued)
Build on some key principles to effectively implement evaluation and
assessment
o Place the students at the centre of the evaluation and assessment framework
o Ensure a good emphasis on the improvement function of evaluation and
assessment
o Communicate the rationale for evaluation and assessment
o Recognise the importance of school leadership
o Establish an implementation strategy
Develop evaluation and assessment capacity across the school system
Improve the articulation between levels of government and assure
support from the centre
21. Student Assessment
Strengths
Assessment is seen as part of the professional role of teachers
o Teachers play the major role in assessing and reporting
o Schools decide and publish assessment criteria
o Variety of approaches
An external dimension to assessment was introduced
o National component of school-leaving examination
o Common assignments for the apprenticeship certificate
o Addresses the need for checks and balances to ensure reliability in the
application of standards
There is an increased focus on student outcomes
o Including a move to identify expected minimum standards of achievement for
students at key points in their education
22. Student Assessment
Challenges
Approaches to learning and assessment remain markedly traditional
o Traditional approach to the organisation of classrooms
o Little emphasis in assessment practices on providing student feedback
Assessment for learning is not systematically used in Czech schools
o Feedback often understood as ‘summative assessment done more often’
Summative assessment of students raises some concerns
o Assessment is often norm-referenced
o Achievement mixed with effort and motivation
There is limited consistency of student assessment across schools and
classes
The national-level support for teacher-based student assessment is
limited
o Lack of guidance to assess against FEPs (e.g. in the form of exemplars),
insufficient assessment tools for teachers, limited professional development.
23. Student Assessment
Challenges (continued)
The national standardised tests entail a range of limitations and risks
o Development of the standards being rushed
o May be more appropriately regarded as specifications for the national tests,
rather than standards
o Potential negative consequences of high-stakes use: (i) ‘teaching to the test’;
(ii) cross-curricular competencies ignored; (iii) classroom time spent on
preparation of the test; (iv) schools selecting the students taking the test.
The assessment of students leads to little interaction among teachers
o Low levels of teacher moderation of student assessments
Multiple purposes to school-leaving examinations raise some concerns
o To provide a certificate of achievement to students
o To compare performance of schools: need other sources of information to
achieve this and value-added techniques
24. Student Assessment
Policy recommendations
Develop educational standards covering the breadth of student learning
objectives prior to developing national standardised tests
o There is a need for clear external reference points in terms of expected levels of
student performance at different levels of education
o National tests not to be the vehicle to develop standards
Limit the undesired effects of national standardised tests
Reflect further on the purposes of the national tests; should be trialled first; minimise
negative impacts
Develop a broad strategy for student assessment and strengthen the role of
formative assessment
Assessment for learning; provide tools supporting the assessment of students; broad
range of assessment approaches; formative ass.
Build teachers’ capability for student assessment
25. Student Assessment
Policy recommendations (continued)
Develop a range of tools at the central level to support teacher-based student
assessment
e.g. marking rubrics; exemplars illustrating student performance, assessment tools
Put in place moderation processes to ensure the consistency of student
summative assessment
Student assessment should be criterion-based rather than norm-referenced
Ensure student assessment is inclusive
e.g. students with special needs, avoid biases by socio-economic background and
minority status (e.g. Roma students)
Build capacity of markers of external tests and examinations
27. Teacher Appraisal
Strengths
The principle that teachers should be evaluated is widely accepted
o Teacher appraisal takes place in all schools and is an important and normal
part of school activities
Teacher appraisal is focused on evaluating classroom teaching
The importance of teacher professional development is recognised in the
legislation
Some structures for co-operation and exchange among teachers are in
place
o E.g. Subject commissions bringing together all teachers teaching a particular
subject; lesson preparation; exchange of pedagogical approaches
28. Teacher Appraisal
Strengths (continued)
There are plans to develop teaching standards and a new career system
for teachers
The link between teacher appraisal and pay increments has potential to
incentivise high performance
o Teachers may be awarded additional pay increments and bonuses that are
determined by the school leadership team
o However, there are important questions regarding the transparency of how
salary rewards are implemented
29. Teacher Appraisal
Challenges
There is currently no shared understanding of what constitutes high quality
teaching
o Lack of a national framework defining standards for the teaching profession
o There are no uniform performance criteria or reference frameworks against
which teachers could be appraised
Teacher appraisal is not systematically implemented for all teachers
The quality and extent of teacher appraisal approaches in individual schools depend
on the capacity and leadership style of the school principals.
There is little tradition for educational leadership in schools
Many school leaders have not been sufficiently prepared for their wide range of
tasks, in particular leading teaching and learning processes in the school.
30. Teacher Appraisal
Challenges (continued)
There are tensions between the accountability and improvement functions of
teacher appraisal
o Teacher appraisal traditionally been conceived as a summative and
accountability-oriented process
o There are risks involved in trying to achieve both the accountability function and
the improvement function of teacher appraisal in one single process
The link between teacher appraisal and rewards is not transparent
Links between teacher appraisal and professional development could be
enhanced
o The provision of prof. development appears fragmented and not systematically
linked to teacher appraisal
o There is scope to better link teacher professional development to school
development and improvement
31. Teacher Appraisal
Policy recommendations
Develop a professional profile or standards for the teaching profession
A clear and concise statement of what teachers are expected to know and be able
to do is a key element in any teacher appraisal system as it provides a credible
reference to make judgements about teacher competence
Strengthen teacher appraisal for improvement purposes (developmental
appraisal)
o Strengthening regular school-based formative appraisal with a professional
development focus which is separate from the more summative appraisal
processes.
o To be validated externally by Schools Inspectorate
Further enhance the role of educational leadership
32. Teacher Appraisal
Policy recommendations (continued)
Consider establishing a system of teacher certification to determine career
progression
Access to each of the key stages of a career could be associated with formal
processes of summative appraisal that complement the regular formative
appraisal
The different career stages should match the different levels of expertise reflected
in teaching standards
Need a stronger component external to the school to validate the process and
ensure that practices are consistent across the country
Three core instruments: classroom observation, self appraisal and documentation
of practices in a simplified portfolio.
Ensure links between developmental appraisal and appraisal for certification
Ensure appropriate articulation between teacher appraisal and school
evaluation
34. School Evaluation
Strengths
External school evaluation is established
o Clear commitment to external accountability based around school evaluation
with a regular cycle of external school evaluations
o External school evaluation valued by stakeholders
The external evaluation model embodies a number of features of best
practice
o Well structured and systematic
o Set of publicly-available criteria for external inspection is drawn up every year
o Publication of inspection reports
o A range of sources of information
o Thematic reports
Schools facing greater challenges benefit from some follow-up
The CSI undertakes a follow-up inspection to assess whether improvements were
undertaken to address the challenges previously identified.
35. School Evaluation
Strengths (continued)
Classroom observation is part of school evaluation processes
Placing learning and teaching at the heart of the evaluation process sends clear
signals about what matters.
There is a new emphasis on schools’ self-evaluation
Has the potential to encourage schools and principals to place a greater emphasis
on school improvement and development planning.
School leadership is promoted in school evaluation
There is an explicit recognition that the processes of self-evaluation and external
evaluation are hugely dependent on a principal’s capacity to stimulate engagement,
to mobilise resources and to ensure appropriate training and support.
36. School Evaluation
Challenges
External school evaluation seems to have limited emphasis on school
improvement
o The accountability function tends to emphasise compliance with legislation
rather than the promotion of school improvement.
o Advice is only given to “weaker” schools which are identified as those that do not
meet the minimum standards as set by law.
o There is not enough guidance about what will lead to school improvement and
little attention is paid to identifying and disseminating best practice
There are a number of limitations in external school evaluation
Difficult to take account of the socio-economic context of the school; not enough
emphasis on pedagogical aspects, Criteria used in the CSI external evaluations are
not stable enough.
There is little emphasis on student results/progress
Limited ability to assess quality of learning and student progress.
37. School Evaluation
Challenges (continued)
School self-evaluation needs to be strengthened
Its penetration across the school system remains at an early stage of development
The use of data for school development is limited
The evaluation by organising bodies has a limited scope and impact
There is no full recognition of the role of school leaders and their appraisal is
limited
o While the school principal has a key role in the system and considerable
responsibilities, this has not as yet been translated into a dedicated career
structure
o Limited preparation for the role of school principal
o Little recognition and financial reward
o The evaluation of school principals, conducted by organising bodies, is largely
absent except in terms of the financial aspects of budget management.
38. School Evaluation
Policy recommendations
Strengthen external school evaluation
o Strengthen focus on school improvement and move away from the current
“compliance” driven model.
o Provide advice for improvement to all schools evaluated.
o The school evaluation framework, the criteria and questions governing
judgements and the methods employed should all focus much more directly on
the quality of learning and teaching and their relationship to student outcomes.
Improve the alignment between external and self-evaluation and raise the
profile of self-evaluation
Criteria used in both processes sufficiently similar as to create a common language
Give stronger emphasis to the follow-up to external evaluation
Improve the capacity of schools to engage in school evaluation
39. School Evaluation
Policy recommendations (continued)
Improve the instructional leadership skills of school principals
Plan to use data on student results effectively
Any publication of results of school performance in students’ school-leaving
examinations and/or national tests should be presented in ways that take account
of intake differences including, for example, the socio-economic background of
students.
Strengthen the evaluation of school principals
Improve the scope and role of organising bodies in school quality
improvement
o The regional and municipal authorities should strengthen their role in supporting
school improvement.
o Collaboration and networking amongst schools could be encouraged to help
develop and spread good practice and enhance teachers’ professional skills.
41. Education System Evaluation
Strengths
An Education Indicators Framework is established
It involves well-established procedures for data collection in close articulation with
schools.
There is a concern to assess the progress of the education system towards
pre established objectives
Principle of establishing educational objectives and the subsequent monitoring of
the progress towards achieving them
The qualitative analysis of thematic reports provides valuable information for
system monitoring
Richness of contextualised qualitative analysis
The participation in international surveys is instrumental for system
evaluation
42. Education System Evaluation
Challenges
There is little emphasis on the evaluation of the education system
o Limited policy attention thus far and there is no comprehensive strategic
approach to it
o The current narrow approach to system evaluation does not allow a broad
enough assessment of the extent to which student learning objectives are being
achieved
The absence of student performance data is a major gap in system
monitoring
Presently there is no mechanism for the Czech Republic to monitor at a national
level the achievement of its students against learning objectives specified in the
Framework Education Programmes
There are key information gaps at the system level
There are no measures on students’ socio-economic background; There are gaps in the
data collected from schools; Little emphasis on investigating inequities in the system;
Limited information on the teaching and learning environment
43. Education System Evaluation
Challenges (continued)
It is not possible to monitor student outcomes over time and across schools
o Mostly due to the absence of national data on student performance
o School-level results of national tests might be disclosed with no account for
schools’ particular contexts. This can considerably distort considerations about
the effectiveness of each school
Monitoring at the region and municipality levels is faced with considerable
challenges
o Regions have a limited intervention in quality assurance with their main tool
being the evaluation of school principals
o No overview of the different quality assurance systems in the regions, including
strategies for school improvement
o Limited articulation between Regions and municipalities
System-level information is not fully exploited
Little analysis to inform educational planning; Limited use to inform school
management; No comprehensive information system; Systematic sharing of data
between schools is limited
44. Education System Evaluation
Policy recommendations
Raise the profile of system evaluation within the evaluation and assessment
framework
The challenge for system-level evaluation is to ensure that the measures of system
performance are broad enough to capture the whole range of student learning
objectives – policy making at the system level needs to be informed by high
quality data and evidence, but not driven by the availability of such information.
Develop national student performance data for system monitoring
o Design national standardised tests for national monitoring and as a pedagogical
tool [closely aligned with student learning objectives; can measure only a limited
range of student learning objectives; publish test results at the school level is
premature]
o Develop strategies to monitor a wider range of curricular areas and broader
outcomes [develop sample-based surveys]
o Ensure that national monitoring covers broader outcomes [e.g. higher-order
thinking skills and cross-curricular competencies]
45. Education System Evaluation
Policy recommendations (continued)
Prioritise efforts to meet information needs for national monitoring
o Develop measures of the socio-economic background of students
o The data collection from schools needs to be improved
o Give more prominence to the analysis of inequities in the system
o Improve the information on the teaching and learning environment
Explore ways to more reliably track educational outcomes over time and across
schools
Make meaningful comparisons across schools if test results are published at the
school level
Strengthen the role of regions and municipalities in quality improvement
Optimise the reporting and use of system-level data
o Strengthen the analysis for educational planning and policy development
o Improve feedback for local monitoring
o Integrate available data and facilitate access by key agencies
o Facilitate the sharing of student information across schools
46. Thank you
(paulo.santiago@oecd.org)
(www.oecd.org/edu/evaluationpolicy)