Summary of a systematic review: Evaluation of communication activities of international and non-governmental organisations

1. Evaluation of communication
activities of international and non-
governmental organisations:
A 15 year systematic review
Presented at the EES Conference,
Helsinki, Finland, 4 October 2012,
Glenn O’Neil
June 2012
G.A.O'Neil@lse.ac.uk

2. Introduction
 The systematic review described in this
presentation is the first component of a four
component PhD
 The review set out to determine how inter-
governmental organisations (IOs) and non-
governmental organisations (INGOs) have
evaluated their communication activities and
to what extent they adhere to principles of
evaluation methodology from1995-2010
G.A.O'Neil@lse.ac.uk

3. What is a systematic review?
 Synthesizes findings from multiple
studies/reports in a balanced and impartial
way
 Uses a rigorous peer-review protocol
 Often focuses on effectiveness questions
“what works” but can go beyond this
 A review is ideally conducted by two persons
G.A.O'Neil@lse.ac.uk

4. Development of Process
review question(s)
Creation of review
protocol
Comprehensive
search for
studies/reports
Application of
inclusion criteria
Assessment against
review protocol
Synthesis of findings
G.A.O'Neil@lse.ac.uk

5. Process
 Question: how have IOs/INGOs evaluated
their communication activities and to what
extent have they adhere to principles of
evaluation methodology ?
 Protocol: criteria based on six evaluation
principles, evaluation design and coverage
 Search: contact with all eligible IOs/INGOs
(230)
 Inclusion criteria: IOs/INGOs status;
evaluation reports with communications
focus; regional or global; time period
G.A.O'Neil@lse.ac.uk

6. Review protocol (extract)
Code Organ- Type Year Title Coverage Evaluation 1 2 3 4 5 6
isation design
A1 Care INGO 2002 Lessons learned from CARE’s Global post-only No Yes No No Yes No
Communications in the
Afghanistan Crisis, Fall 2001
A2 EU IO 2007 Evaluation of Communication, Global post-only Yes Yes Yes Yes Yes Yes
Information and Visibility Actions
in Humanitarian Aid
A3 EU IO 2006 Evaluation of the Information EU post-only No Yes No Yes Yes Yes
Policy on the Common
Agricultural Policy
A23 ILO IO 2002 Evaluation of the InFocus Global post-only Yes Yes Yes No Yes Yes
Programme on Promoting the (some
Declaration of Fundamental during data)
Principles and Rights at Work
G.A.O'Neil@lse.ac.uk

7. Description of “body”
 46 eligible reports represented evaluation of 46
distinct communication activities of 22 organisations
and four coalitions
 Most organisations were represented once or twice
with the exception being the European Union
 Majority of activities were at the global level (63%)
with remaining 37% at the regional level (mainly
Europe and to a lesser extent Asia).
 The dominant sectors were social (employment,
culture and welfare), humanitarian aid and
agriculture.
G.A.O'Neil@lse.ac.uk

8. General findings
Overall compliance of 46 evaluations to the six methodology principles
1.Defining 2.Combination 3.Rigorous 4.Focus on 5.Continued 6.Link to
communication of evaluation design outcomes improvement organisational
objectives methods goals
80% 91% 54% 71% 96% 63%
 Prevalence of communications evaluation is low in
IOs/INGOs – estimated at13-30%
 Intercoder reliability was calculated for each principle
coded; percentage agreement ranged from 73% to
100% with a mean of 85%; Cohen’s Kappa (chance
categorisation) ranged from 0.23 to 1.0 with a mean
of 0.64
G.A.O'Neil@lse.ac.uk

9. Table 3
Guttman scale - compliance of 46 evaluations to the methodology principles
Codes 1 2 3 4 5 6 Scores
1 0 1 0 1 0 1 0 1 0 1 0
A2 1 1 1 1 1 1 6
A16 1 1 1 1 1 1 6
A18 1 1 1 1 1 1 6
A19 1 1 1 1 1 1 6
A28 1 1 1 1 1 1 6
A29 1 1 1 1 1 1 6
A30 1 1 1 1 1 1 6
A44 1 1 1 1 1 1 6
A45 1 1 1 1 1 1 6
A9 1 1 1 0 1 1 5
A10 0 1 1 1 1 1 5
A12 1 1 0 1 1 1 5
A15 1 1 0 1 1 1 5
A17 1 1 1 0 1 1 5
A20 1 1 0 1 1 1 5
A23 1 1 1 0 1 1 5
A24 1 1 0 1 1 1 5
A27 1 0 1 1 1 1 5
A37 0 1 1 1 1 1 5
A38 1 1 0 1 1 1 5
A39 1 1 0 1 1 1 5
A42 1 1 1 0 1 1 5
A43 1 1 0 1 1 1 5
A46 1 1 1 1 1 0 5
A3 0 1 0 1 1 1 4
A4 0 1 1 1 1 0 4
A5 1 1 0 1 1 0 4
A7 1 1 1 0 1 0 4
A8 1 1 1 0 1 0 4
A11 1 1 1 0 1 0 4
A13 1 1 1 0 1 0 4
A14 0 1 1 0 1 1 4
A21 0 1 0 1 1 1 4
G.A.O'Neil@lse.ac.uk

10. General findings
 Most compliant: a precise set of communication
activities and/or effects desired; all were conducted
externally and most used research designs other
than post-only
 Least compliant: difficulties to show link to
organisational goals; used less rigorous research
designs and most used only one evaluation method
G.A.O'Neil@lse.ac.uk

11. General findings
 Most compliant: a precise set of communication
activities and/or effects desired; all were conducted
externally and most used research designs other
than post-only
 Least compliant: difficulties to show link to
organisational goals; used less rigorous research
designs and most used only one evaluation method
G.A.O'Neil@lse.ac.uk

12. View these slides on my blog
www.intelligentmeasurement.wordpress.com
G.A.O'Neil@lse.ac.uk
@glenn_oneil
glennoneil
www.owlre.com
12