The document outlines a framework for systematically evaluating informal education and outreach programs, which often have diverse activities and diffuse audiences that make rigorous evaluation challenging. It recommends sharing best practices across agencies to improve coordination and leverage the potential for real impact through economy of scale. A variety of qualitative and quantitative data collection methods are proposed to assess programs aimed at different age groups.

1. Top moments in UNAWE programmes
Co
un
try
CountryTeachers Ch
ild
re
n
How can we summarise
and communicate impact
in systematic ways?
Grace Kimble and Dr Cecilia Scorza

2.  Framework for evaluation
 Evaluation materials:
To do
 1. Case study of your own programme- highlight
disadvantage
To email/ hand out and collect by end of Dec
2. Survey for teachers- give after training
 3. Survey for pupils 8-10- give to teachers after training
To visit and observe/listen
 4. Activity for younger pupils: before/after
 5. Observation sheet for early years (4-7)
 Optional: long term impact interviews

3. Session structure
1530 Introduction to evaluation frameworks
1600 Sample materials
1630 Workshop activity: gathering evidence/
timescales
1700 Feedback- case studies
1730 Finish

4. “"the systematic application of social
research procedures for assessing the
conceptualization, design,
implementation, and utility of social
intervention programs."
http://ideas.stsci.edu/Evaluation.shtml

6. Front end Formative Summative
Process
Impact Outcome

7. Quantitative
Large sample sizes, closed questions
Example- Rosa’s likert scales
Qualitative
Small sample sizes, in depth, allow for open
ended responses
Mark and Libby’s questionnaires

8. Quantitative
Surveys
Closed questions
Qualitative- smaller samples
Case studies
Interviews
Observations
Video analysis

9.  Descriptive statistics
 Correlational analysis
 Statistical significance
It is necessary for data to be comparable in order for analysis

10. A way to collect data that is consistent and
systematic across different types of programme

11.  Clarify Goals and Objectives- March 2012
 Create a Model of Your Program
 Choose Methodology
 Choose Data Collection Method(s)
 Analyse Data
 Report and review
 Disseminate

12. 1.1 The Program is addressed specially to disadvantaged children in different
social environments.
1.2 The program intends to stimulate via images, playful activities, experiments,
models and artistic activities the children´s awareness of astronomical objects
and the place of the Earth as part of the Universe in a way of broadening the children
´s view of the world. In doing so try to use throughout daily-life materials in the
activities, experiments and models such that children can reproduce them at home.
1.3 Awaken the awareness of the uniqueness of the Earth and of the need for
environmental preservation.
1.4 Motivate children to appreciate other cultures via stories and twinning
activities with children living in other countries, making them aware of the place in
which other children live, their common cultural features and differences.
1.5 Throughout stimulate children´s curiosity giving them the occasion and time to
formulate their own questions related to the Universe.
1.6 Develop EUNAWE-contents to be linked to daily school curricula via training
of the teachers.

13.  Clarify Goals and Objectives- March
 Create a Model of Your Program
 Choose Methodology
 Choose Data Collection Method(s)
 Analyse Data
 Report and review
 Disseminate

14. Key questions
What are our assumptions about
a)What learning is?
b)How participants might change as a result of
programme interventions?

15. http://caise.insci.org/uploads/docs/Eval_Framework.pdf

16. 1. The variety in types of programs is expansive. Informal education and outreach activities
can take place in schools, museums, the community, the media, and various other locations
where people gather information and experience the world. Almost all ACC agencies had some
type of program that was designed to generate awareness and engage the public in the agency’s
work. Further, the types of activities varied considerably across programs.
2. The nature of these programs makes it difficult to conduct rigorous evaluation because,
among other reasons:
(1) the audience for these programs is diffuse and difficult to identify;
(2) the multiple factors affecting and affected by these activities cannot be isolated for assessment;
(3) the modest scale of these efforts does not warrant a costly assessment approach.
There are examples of pre- and post- quasi-experimental evaluations of these programs, but it is
extremely challenging to carry out rigorous studies to identify causality in these programs.
3. Third, despite all of these complexities many programs share the same or similar goals. Though
the programs are varied, the Informal Education and Outreach group agreed on the importance
of better interagency coordination and information sharing. Sharing best practices across
agencies could offer significant benefits as these programs often bear a closer relationship to
outreach and informal education programs in other agencies than they do to K–12 or
postsecondary education programs in the same agency
(U.S. Department of Education. [DoEd], 2007, p. 26.).

17. “Furthermore, due to budget and time
constraints,
the evaluation of initiatives is often
limited. The interconnections between initiatives
are very rare which effectively eliminates
the possibilities for scaling-up and for the
dissemination
of the new ideas: the dynamics of
“economy of scale” and the huge potential for
real impact are simply not being exploited”.
http://ec.europa.eu/research/science-society/document_library/pdf_06/report-rocard-on-s

19. a cy
Leg Awareness
of the
Universe
(knowledge)
EUNAWE
fostering…
Scientific Intercultural
skills attitudes
v at ion
Moti
“DOMAINS OF LEARNING”

21. y
g ac Awareness
Le
of the
Universe
(knowledge)
EUNAWE
fostering…
Scientific Intercultural
skills attitudes
ion
Motivat

24.  Clarify Goals and Objectives
 Create a Model of Your Program
 Choose Methodology
 Choose Data Collection Method(s)
 Analyse Data
 Report and review
 Disseminate

25. Motivation

26. 2. Scientific skills

30.  Mixed methods are required owing to variety
in programmes and ages
 Qualitative evidence needs to be gathered:
-observation templates for children aged up to 7
-before and after activities for children 6-10
(Personal Meaning Mapping, John Falk)
-drawings for ages 6-8, surveys ages 8-10

31.  Framework for evaluation
 Evaluation materials:
To do
 1. Case study of your own programme- highlight
disadvantage
To email/ hand out and collect by end of Dec
 2. Survey for teachers- give after training
 3. Survey for pupils 8-10- give to teachers after training
To visit and observe/listen
 4. Activity for younger pupils: before/after
 5. Observation sheet for early years (4-7)
 Optional: long term impact interviews

41. Using materials

42. Task 1: Submit a case study about your programmes and
development: http://goo.gl/yJQY5
These will form the basis for identifying examples for long term impact
Task 2: Modify the pupil survey- translate. Check teacher
survey in paper and online: http://goo.gl/XejyP
Task 3: Complete the checklist spreadsheet and return
Extra: add to observation framework

48.  Clarify Goals and Objectives- March
 Create a Model of Your Program
 Choose Methodology
 Choose Data Collection Method(s)
 Analyse Data
 Report and review
 Disseminate

49.  Clarify Goals and Objectives- March
 Create a Model of Your Program
 Choose Methodology
 Choose Data Collection Method(s) –Dec 2012
 identify opportunities for evaluation
 check materials suitable for group
 Seek support
 Analyse Data- GK: Jan- Feb 2013
 Report and review- Mar 2013
 Disseminate- Oct 2013

50.  Framework for evaluation
 Evaluation materials:
To do
 1. Case study of your own programme- highlight
disadvantage
To email/ hand out and collect by end of Dec
 2. Survey for teachers- give after training
 3. Survey for pupils 8-10- give to teachers after training
To visit and observe/listen
 4. Activity for younger pupils: before/after
 5. Observation sheet for early years (4-7)
 Optional: long term impact interviews

51.  Materials returned to GK by end of Dec 2012
 Draft report by end of Feb 2013
 Identify issues in data
 Modification and addressing gaps Mar 2013
 Final report