Realist Evaluation : some thoughts about the theoretical foundations
Signaler
Partager
•1 j'aime•494 vues
Realist Evaluation : some thoughts about the theoretical foundations
•1 j'aime•494 vues
Signaler
Partager
Recommandé
Contenu connexe
Tendances
Tendances(20)
En vedette
En vedette(14)
Similaire à Realist Evaluation : some thoughts about the theoretical foundations
Similaire à Realist Evaluation : some thoughts about the theoretical foundations(20)
Realist Evaluation : some thoughts about the theoretical foundations
- 1. THEORY AND PRACTICE OF REALIST EVALUATION Some thoughts about the theoretical foundations Valéry Ridde, Emilie Robert, Anthony Lacouture July, 2; 2015 WHO Meeting : Realist evaluation of national strategic planning and policy processes equitesante.org
- 3. 3
- 4. RESEARCH QUESTION • WHAT ? Equitesante.org 4 • WE ALREADY KNOW WHAT WORKS…
- 5. FROM WHAT TO…. HOW Equitesante.org 5 FOR WHOM WHY TO WHAT EXTENT IN WHAT CIRCUMSTANCES
- 6. PUBLIC HEALTH AND COMPLEXITY « Multiple entangled actions, acting in often non-linear relationships, organized in a specific context and understood as open systems, researcher who observe in fact itself part of it » Blaise et al. 2010 Equitesante.org 6
- 7. 7 CHARACTERISTICS OF COMPLEX INTERVENTIONS Volitions Equitesante.org 7 Implementation Contextes Time Outcomes Rivalry Emergence Pawson, 2013
- 8. CRITICAL REALISM Equitesante.org 8 POSITIVISM CRITICAL REALISM RELATIVISM QUESTIONS Is the policy or intervention (cost)-effective? What works for whom under which conditions? How do actors experiences, power and understand? DISCIPLINES Epidemiology Welfare economics Political science Policy analysis Organizational studies Socio-anthropology DESIGN Deductive Fixes Abductive Pragmatic/MM Inductive Flexibles Gilson, 2012; Robson, 2002
- 9. CRITICAL REALISM • Reality exists outside the human constructions, but it is perceptible only through our senses Equitesante.org 9 Maxwell, Pawson, Bhaskar, etc.
- 10. GENERATIVE CAUSALITY • We are not looking for immutable law vs sucessionist notion of causality • Importance of mechanisms/context vs intervention • But regularities (semi-reg : Lawson) – semi-predictable patterns – pathway of program functioning • Patterns = CMO Equitesante.org 10 (Pawson, 2006)
- 11. C M O Equitesante.org 11 Robert & Ridde, 2014
- 12. MECHANISM DEFINITION • is hidden but real, sensitive to the context and produces outcomes • is an element of reasoning and reactions of agents in regard of the resources available in a given context to bring about changes • evolves within an open space-time and social system of relationships Equitesante.org 12 Lacouture et al, 2015
- 13. MECHANISM • Elements of reasoning of agents and their choices about how the change will be achieved through the implemented intervention • Reactions of agents in response to the resources provided by the intervention 13 Equitesante.org • Pre-existing characteristics of the individuals, localities, situations or systems in which an intervention is being set up (= CONTEXT) • Intentional measures, activities or a rational plan taken by program implementers (= STRATEGIES OF INTERVENTION) Lacouture et al, 2015
- 15. NOT A MECHANISM Equitesante.org 15 Actor’s attitudes : health workers’ motivation and satisfaction (A1), the provider–patient relationship (A2), patients’ satisfaction (A3), patients’ perceptions on health and on financial access to healthcare (A4), and health workers’ coping strategies (A5). Ridde et al, 2012
- 16. MECHANISM EXAMPLE CMOc of PR synergy related to interpreting and disseminating research findings (strategy). •As a result of the long-term collaborative relationships between academic and community stakeholders (context), locally informed and accessible scientific study findings were presented to the community (strategy). The audiences at the presentations felt comfortable and safe participating in the context of their community ownership of the project (mechanism) and made sense of the data in their own terms. As a result, new explanatory insights into the data were generated (outcome). Equitesante.org 16 Jagosh et al, 2014
- 17. THEORY-DRIVEN EVALUATION • Program/intervention theory (Weiss, Chen, etc.. ) • Causal model • Middle range theory (initial and final) Equitesante.org 17
- 18. INTERVENTION THEORY • "A program is a theory and an evaluation is its test" (Rein, 1981) • Programs are “theories incarnate”. (Pawson 2012) • “It is a small theory attempting explanation of specific treatment processes, not a large theory of general social phenomena” (Lipsey, 1993) Equitesante.org 18
- 19. 19
- 20. INITIAL PT IN UGANDA • Management practices and leadership styles (interven- tion) that are supportive of autonomy lead to positive work behaviour of volunteers, such as improved task per- formance and persistence (outcomes), because they satisfy the psychological needs of autonomy, competence and relatedness (mechanism). More specifically, such management and leadership contribute to a shift from ‘external motivation’ towards ‘internalised motivation’ for the volunteers whose initial engagement was more driven by external motives. Internalised motivation leads not only to positive work behaviour, but also to well- being. Such management and leadership furthermore allow volunteers whose initial engagement was mainly driven by internal motives to remain engaged and perform well. Equitesante.org 20 Vareilles et al, 2015
- 21. MIDLE RANGE THEORY • « Middle-range theory involves abstraction, of course, but they are close enough to observed data to be incorporated in propositions that permit empirical testing. » Merton, 1967 • Level of abstraction = useful + testable/data Equitesante.org 21
- 22. INITIALE MIDLE RANGE THEORY • "Hospital managers of well-performing hospitals deploy organisational structures that allow decentralisation and self-managed teams and stimulate delegation of decision-making, good flows of information and transparency. Their HRM bundles combine employment security, adequate compensation and training. This results in strong organisational commitment and trust. Conditions include competent leaders with an explicit vision, relatively large decision-making spaces and adequate resources." Equitesante.org 22 Marchal et al, 2010
- 23. INITIALE MIDLE RANGE THEORY • "‘A pluralistic health system harbours a web of accountability relationships between actors who combine the roles of account-holder and accountor.  • Public accountability is actualised when actors are answerable to the public and remedial action is undertaken. Public accountability requires both answerability and enforceability in order to be actualised. The answerability or the capability of the DHMT, SRH INGOs and partnerships to inform, evaluate and report in an open manner requires transparency and clarity on whom they represent and deliver services to. Answerability is actualised through practices grounded in compliance and persuasion. • Enforceability is grounded in the capability of the public to demand accountability on the one hand and in meta-governance, i.e. the function, exercised by a state actor(s), of regulating, monitoring and sanctioning on the public’s behalf, on the other hand. • Accountability practices operate along four dimensions (social, political, organisational and the provider dimension). Each dimension has specific bundles of strategies, practices, relationships and outcomes. Accountability is embedded in vertical, horizontal and partnership governance arrangements. • Multi-level governance arrangements weaken public accountability when there is confusion over roles and responsibilities between governing actors.’" Equitesante.org 23 Van Belle, 2014
- 24. CMO : HEURISTIC FRAMEWORK Ugly circumlocution (Pawson et al, 2012) Equitesante.org 24Robert & Ridde, 2014
- 25. Original CMO for the CHW • Context – Lack of community awareness regarding safe motherhood, danger signs during pregnancy, labour, and postpartum periods and where to go for services. • Programme response/mechanism – Training and deploying trusted members of the community instills ‘knowledge’ of dangersigns during pregnancy and labour, safe delivery practices, as well as newborn care. Regular visits from community members build relationships between women and health workers, leading to ‘trust’ in the health system. • Outcome – More women aware of complications; increased care seeking. Equitesante.org 25Adams et al, 2015
- 26. Initiale CMO CONFIGURATION Equitesante.org 26 Evans et al, 2014
- 27. Final CMO CONFIGURATION Equitesante.org 27Adams et al, 2015
- 28. CMO CONFIGURATION A community experiences a high level of unemployment to which an employment training program is offered in a remote part of town (context). But the program has low enrollment and high attrition, and few people are trained (outcome). The reason is that people feel disillusioned by the lack of effort by program planners to ensure adequate public transportation to the venue (mechanism). Equitesante.org 28 Jagosh et al, 2013
- 29. Initiale CMO CONFIGURATION Participants who are motivated and share a good rapport’ (context) may ‘share ideas, experiences and support one another’ (mechanism), which may be associated with a smoking cessation and improved weight management (outcome) Equitesante.org 29 Mackenzieet al, 2009
- 30. THEORIES AND CMO volunteer motivation & work behaviour • Crowding theory • Intrinsic motivation theory • self-determination theory • Gift exchange and high- performance HRM theory • Consistency theory • Organisational commitment • Social exchange theorY participatory research assessment 30 Equitesante.org Vareilles et al, 2015 Jagosh et al, 2013 • Theory of Capacity Building • Communicative Action Theory • Empowerment/emancipati on • Collaborative Management • Partnership Synergy Theory
- 31. FROM CMOi TO CMOf Equitesante.org 31 Pawson et al, 2012
- 32. 32
- 33. 33
- 34. 34
- 35. CONTEXTUAL LAYERS Equitesante.org 35 Robert & Ridde, 2014
- 36. MECHANISMS CHALLENGES Equitesante.org 36 • a sense of self efficacy, • an increase in self esteem, • assurance that there is a system for back-up support
- 38. 38 Robert & Ridde, 2014
- 39. REALIST EVALUATION CYCLE Equitesante.org 39Vareilles et al, 2015
- 40. FROM CMO TO MRT Equitesante.org 40 Abstraction Empirical MRT1 THEORY1 C1 M1 O1 PROGRAM THEORY C2 M2 O2 C3 M3 O3 C4 M4 O4 C1 M1 O1 C1 M2 O3 C1 M1 O3 C2 M2 O1 C1 M3 O3 C1 M4 O3 MRT2 MRT3 MRT4 MRT5 THEORY2 THEORY3 STARTSTART
- 41. FROM CMO TO MRT Equitesante.org 41 Abstraction Empirical MRT1 THEORY1 C1 M1 O1 PROGRAM THEORY C2 M2 O2 C3 M3 O3 C4 M4 O4 C1 M1 O1 C1 M2 O3 C1 M1 O3 C2 M2 O1 C1 M3 O3 C1 M4 O3 MRT2 MRT3 MRT4 MRT5 THEORY2 THEORY3 STARTSTART
Notes de l'éditeur
- Complex social interventions …... ... are theories or consist of several theories. … ... involve the participation of numerous stakeholders. .. ... consist of a chain of decision processes. .. ... are not linear and are subject to feedback loops. .. ... are entrenched in several social systems and several contexts. … ... are permeable to the influence of other interventions. …. .. are open learning systems. ...
- UN DE DEUX ???? J,aime bien celui de droite car il mets en avant la question, mais on peut adapter
- La RE est fondamentalement une theory driven evaluation
- La RE est fondamentalement une theory driven evaluation
- A mechanism is hidden but real: Existing prior to the intervention, but latent, a mechanism reveals itself during implementation of the intervention within a given context. Sensitive to the context, it produces outcomes. A mechanism is an element of reasoning and reactions of agent(s) in regard of their ability to use the resources available in a given context to bring about changes : A mechanism results of the interaction between agents, intervention and structures. It reflects the logic of intervention of the various actors involved directly (e.g. stakeholders) or indirectly (e.g. patients) in the intervention. A mechanism evolves within an open space-time and social system of relationships : A mechanism is dynamic and it may be interacting with other mechanisms (family of mechanisms), with elements of context or with the effects it has itself produced (i.e. positive or negative feedback).
- A mechanism is hidden but real: Existing prior to the intervention, but latent, a mechanism reveals itself during implementation of the intervention within a given context. Sensitive to the context, it produces outcomes. A mechanism is an element of reasoning and reactions of agent(s) in regard of their ability to use the resources available in a given context to bring about changes : A mechanism results of the interaction between agents, intervention and structures. It reflects the logic of intervention of the various actors involved directly (e.g. stakeholders) or indirectly (e.g. patients) in the intervention. A mechanism evolves within an open space-time and social system of relationships : A mechanism is dynamic and it may be interacting with other mechanisms (family of mechanisms), with elements of context or with the effects it has itself produced (i.e. positive or negative feedback).
- La RE est fondamentalement une theory driven evaluation Théorie de programme : théorie descriptive de la manière dont l'intervention est censée produire ses effets, et qui inclut nécessairement (pour l'évaluation réaliste) le modèle causal. Ce modèle causal s'appuie sur des théories ou des modèles conceptuels qui constituent le point de départ de l'évaluation réaliste. C'est là où il y a confusion avec la TMP, à mon avis. C'est l'étape n°1 de l'évaluation réaliste. - Théorie de moyenne portée : théorie explicative de la manière dont le contexte influence les mécanismes mis en oeuvre par l'intervention pour produire des effets. C'est le produit final de l'évaluation réaliste.
- La RE est fondamentalement une theory driven evaluation
- Deux façons de présenter des théories de programme mais cela n,est pas suffisant, l’intérêt de la RE est d’aller en détail, notamment pour les mécanismes et les CMO ER : l’intérêt de la RE est surtout de faire le lien entre contexte et mécanisme(s) d’une intervention, ce que ne peut pas faire un théorie de programme.
- No all good meca = attention !!
- DEUX DIAPOS POUR MONTRER COMMENT NE PAS FAIRE ER : ce n’est pas un bon exemple : à enlever ! Il faut – je crois – éviter de montrer des tableaux de liste parce que cela ne montre pas les liens de causalité, et laisse penser aux gens que c’est suffisant d’en arriver là. En plus la plupart des mécanismes n’en sont pas.
- DEUX DIAPOS POUR MONTRER COMMENT NE PAS FAIRE ER : ce n’est pas un bon exemple : à enlever ! Il faut – je crois – éviter de montrer des tableaux de liste parce que cela ne montre pas les liens de causalité, et laisse penser aux gens que c’est suffisant d’en arriver là. En plus la plupart des mécanismes n’en sont pas.
- Je mettrais bien ici un de tes exemples de thèse, car cela sera un seul, aucun risque, et proche de ce que les gens connaissent ?
- Je mettrais bien ici un de tes exemples de thèse, car cela sera un seul, aucun risque, et proche de ce que les gens connaissent ? ER : tu peux mettre celui-là ; Les défaillances dans la mise en œuvre des politiques d’exemption des paiements directs et les dysfonctionnements parfois préexistants du système de santé sapent les relations entre usagers et prestataires, et contribuent à faire émerger un sentiment de défiance des usagers envers les prestataires de soins, voire envers le système de santé. Cela favorise le phénomène de contournement de certaines formations sanitaires ou le choix de prestataires privés ou de soins domestiques.
- Comment formuler des CMO
- Revoir, adapté ? ER : je me rappelle pas trop ce que cela veut dire… Pas sûre que ce soit utile.
- L,importance du contexte et dit par tous, pas seulement Pawson et son école
- ER : pas certaine de ce schéma… Ca laisse penser que le concept de « contexte » est fourre-tout. Or il faut insister sur le fait que n’est contexte qu’un élément qui déclenche un mécanisme pour produire un effet. C’est la clé – à mon avis – de la compréhension de l’approche réaliste. Il faut que le public soit capable de comprendre que l’enchaînement contexte-mécanisme-effet peut se décliner à différents niveaux d’analyse, et différents niveaux d’abstraction. La définition LA plus importante est celle de mécanisme. Le fondement Le plus important est le réalisme critique, c’est-à-dire la relation entre contexte-mécanisme-effet et les semi-régularités. Je crois que l’on devrait commencer par là, avant de parler des théories.
- ER : pas certaine que ce soit pertinent.
- Défis de disposer d’outcomes
- Défis de disposer d’outcomes
- Processus de construction des CMO ER : il s’agit ici du processus de réflexion et d’analyse, pas nécessairement de construction des C-M-O (car cela concerne également la TMP). J’ai changé la diapo de place parce que je pense qu’on devrait présenter le processus avant le niveau d’abstraction.
- PT => CMO => MRT => ER : à la place d’études empiriques, tu peux avoir théorie du programme, qui est plus collée à la réalité, sauf que la théorie du programme explicite la manière dont les concepteurs ont pensé que fonctionner l’intervention, et ne représente pas nécessairement ce qui se passe dans la réalité. C’est là toute la nuance ! De plus, la théorie du programme n’est pas explicative mais descriptive, et ne peut donc pas mettre en évidence les interactions avec le contexte.