Rotary Foundation Cadre Training: Monitoring and Evaluation

Monitoring and Evaluation
TRF Cadre Training
12 January 2015

CADRE TRAINING | 2
PRESENTERS
Philip Silvers
•Cadre Chairman
2014-15
•Past RI Director and
board liaison to the
Trustees
•44-year career as a
program evaluation
and survey research
professional
•10 years professor of
graduate-level
Research Methods
James Robinson
•Director, Strategy
Innovation and
Evaluation
•Former Division
Manager
Humanitarian Grants
and Future Vision
Pilot
•RI-USAID Steering
Committee
Korhan Atilla
•Cadre Technical
Coordinator 2008-
11
•Civil Engineer
•Conducted
sustainability check
site visits
•District Governor
Elect 2014-15, D-
2430

CADRE TRAINING | 3
Cadre’s Efforts to Support Sustainability
Phil Silvers

CADRE TRAINING | 4
 The New Grants Model Is Working Fairly Well
– TRF Manuals Are Helpful
– Clubs and Districts are working together
– The Cadre Is an Important Resource
• Technical assistance
• Stewardship (accountability)
 Proposal Writers and Sponsors are struggling
– Online application system is a challenge
– Effective needs assessments are not being done
– Sustainability—still an elusive concept
– Monitoring and Evaluation (M&E) Who will do it?
– Average time to grant approval: 3 months
YEAR ONE – WHAT DID WE LEARN?

CADRE TRAINING | 5
Core Effects:
Sustainable
Change/
Lasting
Results
Process
Results
(‘Doing Good’)
Training
Cultural
Experiences
Adventure
Travel
Friendships
Humanitarian
Supplies
Equipment
Externalities
(The ‘Sizzle’)
Challenges
Rotary
“Credits”

CADRE TRAINING | 6
• Real sustainability begins with the Community
Needs Assessment—by talking to the grass-roots people
before you come up with the solution.
– Need to comprehend their view of the world—their
fears, their challenges, their dreams for their kids
• Needs Assessment, Sustainability, and M&E are not
independent concepts—they must be integrated into one
piece by one team.
• Depending on what you are trying to accomplish, the
following slide gives you some options on measuring
Sustainability (or ‘red flags’ if you omit them)
BASIC PRINCIPLES OF SUSTAINABILITY

CADRE TRAINING | 7
Security—
•People
•Premises
•Equipment
Security—
•People
•Premises
•Equipment
Dimensions
of
Sustainability
Ecological
Human Capital Organizational
Systems
Behavioral Change Mechanical
Systems
New Technologies
(Tools) Cultural Ethos
Legacy
DIMENSIONS OF SUSTAINABILITY
Economic New Policy/
Legislation
Cultural Ethos/
Legacy

CADRE TRAINING | 8
Security refers to the protection against harm to
people, to premises, and to equipment. Other than
a good needs assessment, this is the most
fundamental issue in Sustainability.
Are the participants secure from coercion, threats to their reputation,
privacy, and physical/mental well-being?
Are the premises secure from unwanted intruders?
Are the equipment and materials safe from theft and vandalism?

CADRE TRAINING | 9
Economic refers to the financial viability of the
project over the long haul.
Are revenue streams planned to support the program after the grant
funding ends?
 Operating costs?
 Equipment maintenance and replacement?
Who will be responsible for managing the revenue streams?
Are contingency plans in place if the planned funding streams fail to
deliver?
Economic

CADRE TRAINING | 10
Cultural ethos refers to what is acceptable or not
acceptable behavior within society, e.g., smoking
in common areas, young girls’ access to
education, women’s role in government, getting
tested for HIV/AIDS, tribal enmities,
xenophobia.
Have project planners done a ‘force-field’ analysis of the factors
inhibiting and fostering change?
What rewards, incentives, modeling, or strategies might help shift
prevailing attitudes?
What various measures will reveal whether a change has happened?
Cultural Ethos/
Legacy

CADRE TRAINING | 11
SUSTAINABILITY
How Else Can You Say It?
 Game changer
 Systemic change
 Lasting results
 A permanent difference
 Self-perpetuating
 Continuity
 Can continue without outside help
 Can exist on its own indefinitely
 The ‘buck’ does not stop here!

CADRE TRAINING | 12
Evaluation Plan Overview
James Robinson

CADRE TRAINING | 13
EVALUATION PLAN OVERVIEW
M & E Activities
13-
14
14-
15
15-
16
16-
17
17-
18
18-
19
19-
20
20-
21
21-
22
Areas of Focus X X X
Strategic Partners X X X
Grant Model X X X
Sustainability Assessments X X X
 Multi-year evaluation
 Site visits for 5-10% of Global Grants
 Study common project types in each area of focus
 Replicate sustainability check methodology

CADRE TRAINING | 14
M & E Framework

CADRE TRAINING | 15
TERMINOLOGY
Monitoring typically refers to the routine and ongoing
process of collecting data as agreed upon before project
implementation.
Evaluation generally involves a post-project assessment
of the degree to which the project goals were met and the
impact the project activities had on beneficiaries.
A Measure is a quantitative or qualitative description of a
project’s activities or impact.
Slide 15

CADRE TRAINING | 16
Slide 16
ONLINE REPORTING

CADRE TRAINING | 17
Slide 17
ONLINE REPORTING

CADRE TRAINING | 18
Slide 18
ONLINE REPORTING

CADRE TRAINING | 19
Slide 19
Is monitoring and
evaluation data required
to close a global grant?
Yes. Sponsors must provide the minimum standard (at
least one measure per AoF) for M & E data in the final
report. This information is required to close the grant.
If Sponsors promised more M & E data than they can
provide at the time of reporting, can close the grant if
the minimum standard has been met.
How should M & E funds
budgeted in a global
grant be processed as
part of grant closure?
Sponsors can utilize up to 10% of the grant budget for
M & E.
Provide accounting of expenditures at time of final
report. If applicable, sponsors may provide a plan to
expend remaining funds post closure and report M & E
data into online system
GRANT REPORTING REQUIREMENTS

CADRE TRAINING | 20
CHARACTERISTICS OF GOOD M & E PLANS
1. Less is More
Rotarians should be modest in the number of measures
they will track…if it’s in the application, we’ll expect to
see it in the report.
Advise Rotarians to limit the number of areas of focus
they select in the application as this will multiply the
number of measures required.
Slide 20

CADRE TRAINING | 21
2. Measures match core project components
Rotarians do NOT need to measure every project
component. Instead, they should select the most
important components and focus on measuring their
success in those.
Measures should be relevant to the main goals of the
project and should indicate success in reaching the
goals.
Slide 21

CADRE TRAINING | 22
3. Mixture of output & outcome measures
Counting activities and participants is a good place to
start when developing an M/E Plan, but falls short of
demonstrating quality.
Strong M/E plans include measures that demonstrate the
quality of the project in terms of changes in the behavior
or knowledge of the beneficiaries.
Slide 22

CADRE TRAINING | 23
4. Reasonable attribution
– Measures should be closely and obviously connected to
activities conducted and the project participants.
– Grant officers can advise Rotarians on what is
reasonable attribution of project outputs to outcomes.
Slide 23
CHARACTERISTICS OF GOOD M & E PLAN

CADRE TRAINING | 24
5. Resources are dedicated to M/E activities
Eligible items include:
– Contracted labor of a local org or paying stipends to community members to
collect data.
– Consulting costs to technical experts to review and validate data.
– To purchase data sets and other information necessary to establish a
baseline.
– Purchasing equipment such as phones, GPS equipment, and laptops to
monitor activities.
– Local travel costs, such as car rental, fuel, and train fare to conduct site
visits.
– Cost of electronic surveys; meeting costs for focus groups for evaluation
purposes.
– Printing costs for manuals and other materials needed for data collection.
Slide 24

CADRE TRAINING | 25
5. Resources are dedicated to M/E activities
Ineligible items include:
– International travel to visit the project site.
– Accommodations, meals and per diems.
– Producing project promotional or recognition materials.
– Any measurement expenses that are incurred prior to the grant approval
date, such as the costs of conducting a community assessment.
Slide 25

CADRE TRAINING | 26
Sustainability

CADRE TRAINING | 27
SUSTAINABILITY
For The Rotary Foundation, sustainability means:
ensuring that grants provide long-term solutions
to community needs that the benefiting
community can maintain after grant funding ends.

CADRE TRAINING | 28
Legacy Grants;
Giving A Man A Fish
SUSTAINABILITY

CADRE TRAINING | 29
Rotary
Foundation
ensures
Sustainable
impact of
goods or
services.
Clubs and Districts
ensure sustainable
impact of goods and
services
Individuals &
organizations benefit
from sustainable
impact of goods and
services
Community or
organization ensures
sustainable impact of
goods and services
TRF ensures
sustainable impact of
goods and services
SUSTAINABILITY
Global Grants;
Teaching A Man to Fish

CADRE TRAINING | 30
Sustainability Framework

CADRE TRAINING | 31
SUSTAINABILITY FRAMEWORK
Community
Needs/
Strengths
Materials/
Technology
Funding
• Community
Needs
Assessment
• Meeting
beneficiary
needs
• Involving
community
partners
• Confirm local
funding for
long-term
support
• Compensate
project
participants
to ensure
continuity of
work
• Purchase
equipment
locally
• Involve
community in
selecting
technology
• Train community
to operate and
maintain
equipment

CADRE TRAINING | 32
Knowledge Motivation
Monitoring/
Evaluation
• Provide
training to
beneficiaries
• Create plan to
transfer
knowledge to
new
beneficiaries
• Collaborate
with local
organizations
with expertise
• Give incentives
to beneficiaries
to provide
ongoing support
• Identify leaders
committed to
outcomes
• Prepare the
community to
own the project
when funds
expended
• Develop
measurable
objectives
• Identify ways
to collect data
• Establish
baseline data
to track the
impact of the
project

CADRE TRAINING | 33
Community
Needs/
Strengths
Materials/
Technology
Funding
Knowledge
Motivation
Monitoring/
Evaluation
High incidence
of waterborne
illness among
children under
5 in a village

CADRE TRAINING | 34
Abt Sustainability Check

CADRE TRAINING | 35
Average Lifecycle Scores For MGs & GGs
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Application &
Proposal
Contract Point In
Time/Final
Lifecycle Stage
AverageScore(1-4Possible)
AverageOf All Grants
AverageOf All GGs
Averageof All MGs
Mid-Point
ABT SUSTAINABILITY CHECK

CADRE TRAINING | 36
Lifecycle Sustainability Scores; All Grants
1.00
1.50
2.00
2.50
3.00
3.50
4.00
1 2 3
Lifecycle Stage
Score(1-4Possible)
India GG25034
Kenya GG25081
India GG25027
Honduras GG25065
Honduras GG25363
India MG74944
India MG74942
Kenya MG70520
Honduras MG71320
Honduras MG71446
Honduras MG71940
Mid-Point
Cooperating
Organizations
ABT SUSTAINABILITY CHECK

CADRE TRAINING | 37
Sustainability Matrix

CADRE TRAINING | 38
SUSTAINABILITY MATRIX
Eight Factors of Sustainability Evaluated
Grant Management
Outcomes
Financial
Community
Beneficiary Organizations
Cooperating Organizations
Training
Software/Equipment

CADRE TRAINING | 39
Methodology
Assessment is comprised of eight sections
Each section has 15-30 indicator questions
Each grant is assigned a score for each indicator within
the applicable sections
 Score is between 1-4
Each grant is evaluated independently

CADRE TRAINING | 40
Grant Management
The Grant Management section focuses on overall project management
Key indicators include:
 Does the project have precise and finite objectives?
 Is there an implementation workplan including milestones and
benchmarks?
 Were sustainable activities incorporated into the original project
design?
 Were the required progress reports submitted on time?
 Was a sustainability plan included in the final report?

CADRE TRAINING | 41
Measuring Outcomes
The Measuring Outcomes section focuses on the measurability of a
project, and whether or not appropriate M&E was conducted during and
after the project
 Does the project have clear and measurable outcomes?
 Does the monitoring system include community involvement?
 Are the project outcomes shared with the community and other
stakeholders in ways meaningful for sustained project support?

CADRE TRAINING | 42
Financial
The Financial section focuses on budget planning and the financial role
of the cooperating organization
 Does the projected budget indicate sufficient revenue to fund the
workplan?
 Does the project Spending Plan (Budget) align with the
implementation workplan line items (Activity Plan)?
 Does the project budget indicate sufficient revenue to fund the
sustainability activities beyond the end of the grant?

CADRE TRAINING | 43
Community
The Community section focuses on community involvement and buy-in
for the project
 Was a community needs assessment completed? Were
recommendations made?
 Does the workplan include activities to create or promote local
community involvement and ownership?
 Do grant reports describe community relationships developed which
support the continuation of project outcomes beyond the life of the
grant?
 Does the grant identify community champions willing to assume
leadership roles for sustaining project outcomes after the life of the
grant?

CADRE TRAINING | 44
Beneficiary Organization
The Beneficiary Organization section focuses on the project’s
relationship with the beneficiary organization and the organization’s
capacity to take ownership of the project
 Do written agreements define the terms of cooperation, roles and
responsibilities of the beneficiary organizations?
 Does the documentation identify personnel willing to assume
leadership roles for sustaining project outcomes?
 Does the local Club/District demonstrate strong partnerships with
beneficiary organizations as appropriate to objectives?

CADRE TRAINING | 45
Cooperating Organization
The Cooperating Organization section focuses on the project
management plan and structure of the relationship between the project
sponsors and cooperating organization
 Does the documentation include analysis and/or justification of
cooperating organization(s) selection based on demonstrated
expertise or experience?
 Does the design integrate into the cooperating organization(s)
regular systems or operations?
 Does the cooperating organization agree in writing to participate in
program financial reviews?
 Does the documentation include a succession plan to hand over the
project to the partner organization(s) at the end of the grant lifecycle?

CADRE TRAINING | 46
Training
The Training section focuses on the training and follow up associated
with the project
 Does the training plan address gaps identified in the assessment of
beneficiaries targeted for training (doctors, teachers, unskilled labor)?
 Are the training course design (delivery and materials) locally and
culturally relevant (e.g., language, practical examples)?
 Is there a measurement process to assess learner outcomes after
training?
 Is there a plan for keeping skills up-to-date?
 Has the project resulted in demand for training outside the targeted
beneficiary group (organization, individuals)?

CADRE TRAINING | 47
SUSTAINABILITY MATRIX SECTIONS
Software/Equipment
The Software/Equipment section assesses whether culturally appropriate
technology was used during the project
 Does the documentation include a justification of why equipment is
needed?
 Does the proposed equipment respect local social and cultural
norms?
 Was the availability of equipment on the local market considered?
 Is there an inventory management system?
 Do user guidelines exist for equipment use?
 Are local spare parts and replacement equipment available?
 Is equipment being used as intended?

CADRE TRAINING | 48
Questions?

CADRE TRAINING | 49
Areas of Focus
James Robinson

CADRE TRAINING | 50
AREA OF FOCUS EVALUATIONS – FUTURE VISION PILOT
Global Grants:
Sample of all Global Grants approved during pilot, 35 site
visits across 5 areas of focus Nov 2013- Feb 2014
Nine countries on four continents
Focused on VTT and Humanitarian Projects, not Scholarships
10 Cadre and 4 Area of Focus Managers

CADRE TRAINING | 51
Pilot Evaluation - Area of Focus Site Visits
Korhan Atilla

CADRE TRAINING | 52
WHAT IS SUSTAINABILITY?
(is it QUALİTY?)
AND WHAT IT IS NOT?
(is it only QUALİTY?)

CADRE TRAINING | 53
MATTERS OF SUSTAINABILITY
1- Financial Sustainability
2- Maintenance
3- Proper Operation
4- Precautions during installation
5- Rotarian Involvement
6- Benefitting Community Involvement

CADRE TRAINING | 78
Questions?

Rotary Foundation Cadre Training: Monitoring and Evaluation

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (9)

Similaire à Rotary Foundation Cadre Training: Monitoring and Evaluation

Similaire à Rotary Foundation Cadre Training: Monitoring and Evaluation (20)

Plus de Rotary International

Plus de Rotary International (20)

Dernier

Dernier (20)

Rotary Foundation Cadre Training: Monitoring and Evaluation