So what difference does it make? Assessing the impact of participation, transparency and accountability IDS Research Fellow, John Gaventa World Bank Institute Seminar November 22, 2010

1. John Gaventa
World Bank Institute Seminar
November 22, 2010
So what difference does it
make?
Assessing the impact of
participation, transparency
and accountability

2. Taking a Citizen – Led Approach:
10 Years of DFID – funded research on
Citizenship, Participation and
Accountability (www.drc-citizenship.org).
First phase, much attention was on dynamics of
state-society relationship
Second phase, more on how citizens mobilise and
empower themselves, often outside of the state
In this presentation draw from this work, but also
highlight two recent projects and their
implications for voice and accountability
– Mapping outcomes of citizen engagement
– The impact of transparency and accountability

3. Active citizens build effective states
- not (only) the other way around
• Much has been learned about citizens view the state,
and about the state-society relationship
• Citizens can help to build democratic institutions,
legitimacy, responsiveness, capability, accountability
• ‘societal opportunities’ create possibilities of political
reform
• But we need more focus on the society side:
• how to ‘build’ active, empowered citizens
• how active citizens mobilise to change
development policies, build responsive states, and
do things for themselves

4. What difference does citizen engagement
make? The plea for evidence
The idea that good governance cannot be achieved without
the active involvement of citizens and civil society
actors has gained growing consensus in recent years.
Many donors and NGOs now support "participatory
governance", "social accountability" or "demand for
good governance" programmes aimed at promoting the
active involvement of citizens/CSOs in public decision-
making and holding government accountable [...] I'm
currently involved in a research project to gather
evidence of the results and/or impact of such
initiatives.
E-mail to author from World Bank consultant, 2009
.
‘Our number one challenge is to demonstrate what
difference citizen engagement makes.’
- Representative of large donor agency 2008.

5. What difference does citizen engagement
make?
Results from a meta-synthesis of 100 case studies in
20 countries
Drawing from widely accepted
approaches of systematic review,
qualitative case study analysis and
synthesis
Coding of 800 ‘outcomes’
Evidence points to largely positive
contributions
With risks of negative outcomes

6. Distribution of positive and negative outcomes
0
10
20
30
40
50
60
70
80
90
Construction of
citizenship
Practices of citizen
participation
Responsive and
accountable states
Inclusive and
cohesive societies
Total
Percentage
Positive
Negative

7. Positive Negative
Increased civic and
political knowledge
Greater sense of
empowerment and
agency
Increased knowledge
dependencies
Disempowerment and
reduced sense of
agency
Outcome 1
Better Citizens

8. Outcome 2
More Effective Participation
Positive Negative
Increased capacities for
collective action
New forms of participation
Deepening of networks and
solidarities
New capacities used for
‘negative’ purposes
Tokenistic or ‘captured’
forms of participation
Lack of accountability
and representation in
networks

9. Positive Negative
Greater access to state
services and resources
Greater realisation of
rights
Enhanced state
responsiveness and
accountability
Denial of state
services and
resources
Social, economic and
political reprisals
Violent or coercive
state response
Outcome 3
More Responsive and Accountable
States

10. Positive Negative
Inclusion of new actors and
issues in public spaces
Greater cohesion across
social groups
Reinforcement of social
hierarchies and social
exclusion
Increased horizontal
conflict and violence
Outcome 4 More inclusive and
cohesive societies

11. Types of outcomes and types of engagement
Citizens engage in multiple ways
– Local associations
– Social movements and campaigns
– Formal governance spaces
– Multiple forms of engagement
Local associations and movements are particularly
important for positive outcomes
Multiple forms of engagement are particularly
important for realising responsive and accountable
states
Beyond Putnam – not just the density but the nature
of the quality and quantity of the association that
counts

12. Distribution of positive and negative
outcomes across type of citizen engagement
Outcomes sorted by type of citizen engagement
(n=828)
Outcome
type
Local
associations
(n=324)
Social
movements
and
campaigns
(n=233)
Formal
participatory
governance
spaces
(n=153)
Multiple
(n=118)
Positive 90% 71% 55% 68%
Negative 10% 29% 45% 32%
Total 100% 100% 100% 100%

13. Types of citizen engagement
(n=100)
Positive outcomes
sorted by outcome
categories
Local
association
s
(n=29)
Social
movements
and
campaigns
(n=29)
Formal
participator
y
governance
spaces
(n=19)
Multiple
(n=23)
Construction of
citizenship
36% 35% 33% 29%
Practices of citizen
participation
26% 24% 30% 22%
Responsive and
accountable states
29% 33% 25% 44%
Inclusive and
cohesive societies
9% 8% 12% 5%
Total 100% 100% 100% 100%
Distribution of positive outcomes across types of citizen engagement

14. The relationships of outcomes to context
Grouping of countries across regime (Polity
IV, Freedom House, Economist Intelligence
Unit|)
Positive outcomes are not linearly
associated with level of democratisation –
highest proportion of positive outcomes
are found in most and least democratic
countries
Associations are particularly strong for
least democratic settings

15. Distribution of positive and negative outcomes across country types
0
10
20
30
40
50
60
70
80
90
100
Tier 1 Tier 2 Tier 3 Total
Percentage
Positive
Negative

16. Types of citizen
engagement
Positive Outcomes in Tier Three countries
(n=273)
Construction
of citizenship
(n=96)
Practices
of citizen
participatio
n
(n=66)
Responsiv
e and
accountabl
e states
(n=83)
Inclusive
and
cohesive
societies
(n=28)
Local
associations 89% 92% 83% 78%
Social
movements and
campaigns 0% 2% 2% 0%
Formal
participatory
governance
spaces 6% 0% 4% 11%
Multiple 5% 6% 11% 11%

17. Implications
1. Citizen engagement makes a difference,
but not always. We need to understand
more the factors that lead to positive vs
negative change
2. These gains do not emerge automatically:
pay more attention must be paid to
building citizenship e.g. of empowerment,
as a component of other action
3. Only through empowered citizens will
accountability relationships likely occur.

18. Key Findings
4. Citizen engagement makes a difference
across regimes, not just in more
democratic settings.
5. The role of associations in fragile
settings is particularly important for
building cultures of citizenship
6. Citizen engagement is often met by
reprisals: it is critical to protect the
democratic space for engagement if
developmental outcomes are to be
achieved.

19. Part II: What is the impact of
citizen-led accountability
initiatives?
Over a decade of rapid growth and
spread of transparency and accountability
work in development and aid circles and
development academia
Apparent promise of T&A as the cures
for many ‘evils’
As yet, little clarity about what is being
achieved, what works, how it works, and
how best to fulfil that promise…

20. ... Particular focus:
Citizen-led, demand-side and ‘social
accountability’ activities and their connection to
state actors, institutions and processes
Connections between T, A and participation
Methodological challenges of impact
assessment in the T&A field
‘effectiveness’ Vs ‘impact’
Service delivery, budget processes, FoI, natural
resource governance, aid

21. ... Particular focus:
Citizen-led, demand-side and ‘social
accountability’ activities and their connection to
state actors, institutions and processes
Connections between T, A and participation
Methodological challenges of impact
assessment in the T&A field
‘effectiveness’ Vs ‘impact’
Service delivery, budget processes, FoI, natural
resource governance, aid

22. Some evidence of impact, but highly uneven….
The positive story - in some conditions TAIs
demonstrably contribute to:
– Greater state responsiveness
– Lower corruption
– Building spaces for engagement and empowering
local voices
– Better budget utilization and delivery of services
But not always:
– Evidence is uneven and scattered
– Initiatives are new and impacts unknown
– Much focus on effectiveness rather than impact
– Positive evidence in one case not corroborated by
studies in another
How do we enhance demonstrable impact?

23. Challenge 1:
Aims, claims, assumptions and expectations
(or, against what are we assessing impact and
effectiveness?)
Aims vary: Developmental outcomes, democratic
outcomes, voice  empowerment outcomes
Assumptions vary: eg on links between T, A and
participation; about ‘citizens’; hierarchies of objectives;
how explicit/implicit; etc
The need for sharper theories of change

24. Challenge 2:
Methodological issues: How do we know what
we know?
We found some:
Quantitative surveys
Analysis of aggregated
survey data, multivariate
analysis
Experimental approaches
(RCTs)
Qualitative case studies and
case study analysis
Stakeholder interviews
Indices and rankings
We found a lack of:
Comparative studies
Ex-post long-term
evaluations
Appropriate use of
baselines
Rigorous participatory
approaches
Complexity-aware
approaches eg Most
Significant Change;
Outcome Mapping;
narrative-based
Methodological mixes

25. Methodological issues (cont.)
Limited evidence, few comparators, difficulty of
valid counterfactuals
Untested assumptions and poorly articulated
theories of change: blurry goalposts
Correlation Vs causality; attribution Vs
contribution
Indicators – what we want to measure Vs what
we can realistically measure
Whose perspectives, which impacts count?
Upward or downward accountability in the
impact assessment process itself
Complexity, contingency, uncontrollability

26. Challenge 3: Factors that make a difference
Little evidence that supports generalisations of
the kind ‘initiative x produces outcome y’....
A more useful question to ask:
Which factors (enabling and disabling) shape the
possibility that TAIs will achieve their stated
goals in a particular context?

27. Challenge 3 (cont.): Factors
State (supply side)
Level of democratisation
Level of political will
Broader enabling legal
frameworks, incentives
and sanctions
Citizen voice (demand side)
Capabilities of citizens
and civil society
Interaction of TAIs with
other mobilisation and
collective action
Embeddedness of TAIs in
broader policy processes
Linking Mechanisms

28. Challenge 3 (cont.): Factors
Thinking beyond the dichotomies (state-society;
voice–response; supply-demand)
New thinking in governance would urge us to pay more
attention to:
Multiple actors, accountability coalitions, networked
approaches to governance
Changing norms and cultures of accountability in state,
private sector and civil society simultaneously
Looking across levels and scales: Linking the local,
national, regional, international
Bringing politics back in:
– Power
– The black box of political will and political economy
– Links to parties, elections and political regimes

29. Key Lessons
The evidence base (+/-) is weak - but that doesn’t mean
that TAIs are not significant. The challenge is to deepen
the evidence and the methods for developing it.
On the state of the evidence:
• Develop new approaches to assessment, with complexity
perspective, that combine methods and approaches
• Explore further user-centred and participatory approaches
• Support comparative in-depth work across contexts and
TAIs, multi-case and other synthetic analysis
• Strengthen capacities of researchers and practitioners to
develop and build on innovative approaches
• Build into new TAIs ToCs, baselines, comparators, etc

30. Key lessons (cont.)
On factors for greater impact:
• Deepen understandings of synergies between T, A,
participation and voice
• Move beyond dichotomies to build new knowledge on
cross-cutting accountability coalitions
• Apply to T&A field the cutting-edge thinking on
governance, especially networked governance;
interaction of global – national – local; and private
sector
• Explore whether initiatives can travel across context,
method and issue

31. For links to these studies
www. drc-citizenship.org
www.ids.ac.uk
• Gaventa and Barrett, ‘ So what difference does it
make? Mapping the outcomes of Citizen
Engagement, ’ IDS Working paper 247
• Review of Impact and Effectiveness of
Transparency and Accountability Initiatives,
Rosemary McGee & John Gaventa with
contributions from Greg Barrett, Richard
Calland, Ruth Carlitz, Anuradha Joshi and
Andrés Mejía Acosta, IDS, October 2010