Presentation by Nick Malyshev, Regulatory Policy Division, OECD, at the 9th Conference on Measuring Regulatory Performance - Closing the Regulatory Cycle: Effective ex post Evaluation for Improved Policy Outcomes which took place in Lisbon on 20-21 June 2017. Further information is available at www.oecd.org/gov/regulatory-policy/measuring-regulatory-performance.htm.

1. PROCESSES AND INSTITUTIONS FOR
EFFECTIVE EX POST EVALUATION
9th OECD Conference on Measuring Regulatory Performance
Lisbon, 20-21 June 2017
Nick MALYSHEV, Head of Division, Regulatory Policy, OECD
Christiane ARNDT, Regulatory Policy Division, OECD
Introduction to closed-door workshop

2. The scope of evaluation has progressively been expanded over the years
Tying up the policy cycle’s ends through evaluation is challenging
Tensions characterise the organisation and execution of evaluation
A more cohesive strategy for the use of evaluation instruments is needed
• Offsetting approaches simplify assessment methodologies and in principle also the
choice of which regulations to repeal, but they may not help increase net societal
welfare.
• Many approaches focus on a specific sets of regulatory costs , which is a legitimate
choice…
• ..but neglecting regulatory benefits would fundamentally distort the rationale and
implementation of most public policy choices
• Accounting for indirect impacts is a fundamental challenge in any regulatory
assessment
On a systemic level, evaluations are critical in supporting policy learning.
Some emerging trends in ex post evaluation

3. Countries tend to focus on ex ante
evaluation
Note: Based on data from 34 countries and the European Commission.
Source: OECD Indicators of Regulatory Policy and Governance (iREG) 2015, www.oecd.org/gov/regulatory-policy/measuring-regulatory-performance.htm.
Composite indicators: Average scores for primary laws
0
0.5
1
1.5
2
2.5
3
3.5
4
RIA Ex Post Evaluation
iREG score
Methodology Systematic adoption Transparency Oversight and quality control

4. 0
0.5
1
1.5
2
2.5
3
3.5
4
Methodology Systematic adoption Transparency
Oversight and quality control OECD average
Few countries have adopted a systematic
approach to ex post evaluation
Composite indicator: Ex post evaluation for primary laws
Note: The vertical axis represents the total aggregate score across the four separate categories of the composite indicators. The maximum score for each
category is one, and the maximum aggregate score for the composite indicator is four.
Source: OECD (2015), OECD Regulatory Policy Outlook 2015, www.oecd.org/gov/regulatory-policy/indicators-regulatory-policy-and-governance.htm.

5. The majority of ex-post evaluation reviews has
been principle-based

6. A focus on administrative burdens, competition
and compliance costs

7. Countries do not assess underlying policy
objectives systematically
9
2
10
0
5
10
15
20
25
30
35
Do ex post evaluations contain by default an assessment of whether the underlying policy goals of regulation have been
achieved?
Number of jurisdictions
All ex post evaluations Ex post evaluations regarding major primary laws Some ex post evaluations
Note: Based on data from 34 countries and the European Commission.
Source: OECD Indicators of Regulatory Policy and Governance (iREG) 2015, www.oecd.org/gov/regulatory-policy/measuring-regulatory-performance.htm.

8. Who prepares ex-post evaluations

9. What role does regulatory policy play in supporting the
soundness and coherence of various policy narratives?
• How do we move away from a procedural and
administrative approach to evaluation to one that helps
policy-maker find sound and feasible solutions?
• How do we discern noise from signal in public
consultation? And when should policy makers response
to the voice of stakeholders?
• How do we ensure that businesses/citizens willingly
comply with the law because they understand the
benefits it provides rather than the costs that it imposes?
What role for evaluation in the post truth era?

10. Breakout session 1
Institutionalizing ex post evaluation to feed back into the regulatory
process
Source: Arndt, C., S. Hermanutz, C. Kauffmann and R. Schultz (2016), “Building regulatory policy systems in
OECD countries”, OECD Working Paper on Regulatory Policy, No. 5.
» What oversight arrangements and allocation of institutional
responsibility are needed to support high-quality ex post evaluation?
How can the contribution of institutions for evaluation at arm’s
length from government be maximised?
» How can effective oversight mechanisms support a feedback loop to
ensure insights from evaluation lead to better policy outcomes?
» How can ex ante and ex post evaluation be linked more effectively to
avoid regulation being adopted without a prior analysis of the
existing regulatory framework in a given sector?

11. » How can effective stakeholder engagement help target scarce
resources for evaluation exercises where they yield the highest
benefits?
» Which tools and methods of engagement can ensure stakeholders’
inputs are effectively taken into account when laws and regulations
are being reviewed while avoiding “consultation fatigue”?
» What are realistic objectives for stakeholders’ contribution to the
evaluation process and what are the limitations? How can
transparency of stakeholder help ensure better policy outcomes?
Breakout session 2
Engaging stakeholders effectively for a targeted approach in
evaluation

12. » How can countries align budgetary and regulatory tools and public
sector audit to facilitate joined-up approaches to evaluation?
» Which institutions are best placed to prepare evaluations across
different policy instruments and provide outcome-driven policy
recommendations?
» Which methodological approaches can underpin the evaluation of
the cumulative impacts of different policy instruments against
counterfactuals to provide tangible policy conclusions?
Breakout session 3
Developing a more systematic approach to evaluation across policy
instruments