Exploring the Future Potential of AI-Enabled Smartphone Processors
IDS Impact Innovation and Learning Workshop March 2013: Day 1, Paper Session 1 Bruno Marchal
1. Conceptual distinctions: Complexity and Systems
Making sense of evaluation of complex programmes
IDS workshop
Brighton 26-27 March 2013
Bruno Marchal Institute of Tropical Medicine, Antwerp
3. Making sense of development programmes
3
A programme represents a set of resources that is provided to people who
decide to use them (or not) to reach the programme’s goals (or other goals)
‘Programmes are complex interventions introduced in complex social
systems’ (Pawson 2013)
A programme is all about people
4. An overview of Systems in 2 minutes
A system
=
A unit made up by and organised through relations
between elements (agents),
structures and
actions (processes)
(Morin, 1977)
4
5. System as a machine
Mechanical system perspective
Systems as living organisms
Open systems
Complex systems
5
6. Open systems
Open
Bounded
Negative entropy
Embeddedness
constant interaction
with environment and between its
open components
external & internal boundaries
requiring inputs
(resource dependency)
part of a larger system and of the
environment of other systems (co-
evolution)
6
Systems thinking - Senge, The fifth discipline (1990)
7. Complex systems
Multiple interconnected
elements
Non-linear interaction, non-
proportional effects
Negative & positive feedback
loops
(time delays)
change in 1 element can change
(the context of) all others
sensitive to initial conditions
‘Positive feedback enables a
system to escalate many tiny
changes into globally different
behaviour patterns’
(Stacey 1995)
7
8. Complex systems
Influenced by their past
evolution of complex
systems is not completely
unpredictable
path dependence
the future is boundable
8
9. 9
Emergence
- emergent behaviour
Complex adaptive systems
Adaptation = capability of learning
and evolving
• not just ‘passive’ adaptation to
environment
but essentially human capacity
to learn, adapt and survive
10. Consequences
Complex adaptive systems can only be understood as a whole
its elements, relations and history all matter
Their behaviour cannot be (fully) predicted
non-linear relations
agency - structure interaction
… but their future is ‘boundable’
sensitivity to initial conditions & path-dependence
Challenges of complexity for evaluators, planners, researchers, …
10
11. How to deal with complexity?
11
Some sense making frameworks
Stacey’s diagramme (Stacey 1995)
Stacey et al. (2000)
Snowden &
Stanbridge
(2004)
Kurtz & Snowden (2003)
12. The Cynefin framework
Kurtz & Snowden (2003)
Simple contexts
Cause-and-effect relations:
stable, clear, linear
Known knowns
Predictive models and best
practices can be identified
Straightforward
management
Structured techniques: standard
best practices, command and
control style
12
- the ordered domain of well-known causes and effects
Evaluation
Assessing impact is possible
and straightforward
Standardised approaches can
be developed, requiring
technical skills
13. Complicated contexts
Cause and effect relationships
are known, but not clear for
everyone
Causal chains spread out
over time and space
Knowing cause-and-effect
relations is difficult
Known unknowns
Effective management relies
on experts & satisfying good
practices
Evaluation
Problem can be deconstructed in
sets of simple problems
But this requires expert evaluators
13
- the ordered domain of knowable causes and effects
14. Complex contexts
Cause-and-effect relations exist (multiple – non-linear)
Patterns emerge, but impossible to predict in most cases
Retrospective coherence: we can understand why events happen
only after the facts
Unknown unknowns
Expert opinion is of limited use
because it is based on understanding of and experience with hard-to-know,
but in essence predictable patterns
Pattern matching, trajectory tracking, fail-safe experiments, …
14
- the unordered domain
16. The way forward for evaluation and research
Accepting uncertainty
Complex issues requires “a willingness to be uncertain at times and to
know that being uncertain is crucial to the process” (Zimmerman et al.
2012)
Expertise is relative
Reflexivity / ability to decontextualize experience and recontextualise
knowledge and know how
Reflexive practice
Kolb’s experiential learning
Learning organisation theory
16
17. Capturing emergence
Dealing with unknown unknowns
Alterations of the planned intervention, parallel events, context
elements, etc.
Use of wide range of observation and collection methods
(see longitudinal approaches like processual analysis, Pettigrew 1990)
Dealing with the social interaction that leads to emergent behaviour
Flexible and adaptive designs that allow learning
17
18. Figuring out causal attribution
Complex problems can only be understood a posteriori
Ex post, plausible explanations based on demonstrating mechanisms
What is the relative contribution of the intervention to the observed
outcome?
Contribution analysis (Mayne 2001)
Qualitative comparative analysis (Ragin 1999)
‘Hindsight does not lead to foresight, because external conditions and
systems constantly change’ (Snowden & Boone 2007)
18
19. Dealing with social complexity
Developmental evaluation (Patton 2011)
Focus on complex situations and interventions
Continuous adaptation of the evaluation to the evolving intervention,
monitoring and documentation of changes in time
(emergent evaluation design)
Theory-driven approaches
Theories of change
Theory-based evaluation
Realist evaluation
19
20. Realist evaluation
Pawson & Tilley (1997)
Generative causality
Actors have a potential for effectuating change by their very nature
(agency)
Structural and institutional features exist independently of the actors
(and researchers)
Both actors and programmes are rooted in a stratified social reality,
which results from an interplay between individuals and institutions
Causal mechanisms reside in social relations as much as in individuals,
and are triggered by the intervention only in specific contexts
20
21. Realist evaluation = theory-driven
Middle range theory = bridge
between knowledge and empirical
findings, and between cases
MRT is ‘specified’ in a process of
cumulative testing
Plausible explanations
RE indicates in which specific
conditions a particular programme
works (or not) and how
(psychological, social or cultural
mechanisms)
21
22. Conclusion
Cynefin framework can help to make sense of when problems,
interventions or situations are likely to be complex
Practical consequences: each zone calls for specific evaluation
approaches and capacities
Currently, much attention to modelling approaches
Need for innovative approaches to better deal with social
complexity
Theory-driven approaches allow a peek in the black box
Realist evaluation
22
23. References
Kurtz C & Snowden D (2003) The new dynamics of strategy: sense-making in a complex and
complicated world. IBM Systems Journal 42: 462-483
Marchal B et al. (2013) Complexity in health. Consequences for research & evaluation,
management and decision making. Working Paper, Institute of Tropical Medicine, Antwerp
Mayne J (2001) Addressing attribution through contribution analysis: using performance
measures sensibly. Canadian Journal of Program Evaluation 16(1): 1-24.
Pawson R & Tilley N (1997) Realistic Evaluation. London: Sage
Pawson R (2013) The science of evaluation: a realist manifesto. London, SAGE Publications
Pettigrew A (1990) Longitudinal field research on change: theory and practice." Organization
science 1(3)
Ragin, C (1999) Using qualitative comparative analysis to study causal complexity. Health Serv
Res 34(5 Pt 2): 1225-1239
23
24. References
Senge P (1990) The fifth discipline. New York: Currency Doubleday
Snowden D & Boone M (2007) A leader's framework for decision making. Harvard Business
Review
Snowden D & Stanbridge P (2004) The landscape of management: creating the context for
understanding social complexity. Emergence: Complexity and Organisation 6(1-2): 140-148
Stacey R (1995) The science of complexity: an alternative perspective for strategic change
processes. Strategic Management Journal 16: 477-495
Stacey R et al. (2000) Complexity and management. Fad or radical challenge to systems
thinking? London, Routledge
Zimmerman B, Dubois N, Houle J, Lloyd S, Mercier C, et al. (2012) How does complexity
impact evaluation? An introduction to the special issue. The Canadian Journal of Program
Evaluation 26: v-xx.
24