Samantha Spinney, Ph.D., Manager, Child Welfare & Education, ICF Understand the impact mentoring has on students' behavioral engagement, academic achievement, and non-cognitive skill outcomes and learn best practices for designing and implementing a randomized controlled trial (RCT) in a school setting. Learn more: https://www.icf.com/

1. Assessing the Impact of Mentoring:
Lessons Learned from a Research
Study in West Virginia Public Schools
Presented by:
Samantha Spinney, Ph.D., Manager, Child Welfare & Education, ICF, Fairfax, VA
Zornitsa Georgieva, Ph.D., Research and Policy Analyst, West Virginia Higher Education Policy
Commission, Charleston, WV

2. Introduction – Session Format
Background
Study Purpose
and Design
Results
Lessons
Learned

3. 3
Background

4. Background – WV GEAR UP
2014-2021
Support to cohort students in the class of 2020 and just-in-time support for Grade
12 students
Housed at the West Virginia Higher Education Policy Commission
Directed by Vice Chancellor for Student Affairs
50 total schools served, 23 high schools

5. Background – WV GEAR UP
Vice Chancellor for
Student Affairs /
Director of WV GEAR
UP
College Access and Success
Program Director –
Southern Region
County
Coordinator
Site Coordinators
College Access and Success
Program Director – Central
Region
County
Coordinator
Site Coordinators
Community Outreach
Specialist
Coordinator of Curriculum
and Professional
Development
College Access and Success
Research and Policy Analyst
WV
Department
of Education
ICF Xcalibur

6. Background – WV GEAR UP
• College-going in West Virginia has made notable achievements in the last
decade, but still falls below the national average (55% WV vs. 69% US).
• The WV Higher Education Policy Commission received its second GEAR UP
grant for 2014-2021 to enhance college access programming in the state.
• Serves 50 schools, including 23 high schools
• Provides support to cohort students in the class of 2020
• Provides just-in-time support for grade 12 students (priority students)
• The Commission hired ICF to conduct an external evaluation of their GEAR
UP grant.
• We imbedded a mini-impact study within the overall evaluation to study the
impact of a mentoring program.

7. Background – WV GEAR UP Evaluation
Study Purpose Data Sources
Implementation
study
whether activities have been
implemented as planned
• Student, parent, and school personnel surveys
• Focus group/interviews with Site Coordinators
and another group (changes each year)
• Project activity/participation and academic data
Outcomes study program and academic outcomes
related to grant objectives
• Project activity/participation and academic data
Impact study academic preparation and
postsecondary enrollment
• Academic data
Sustainability study the extent to which services were
sustained for subsequent cohorts
of students
• Student, parent, and school personnel surveys
• Focus group/interviews with Site Coordinators
and another group (changes each year)
• Project activity/participation and academic data

8. Background – Impact Studies
What is an impact
study?
• “A study design in which
outcomes for a group
receiving an intervention are
compared to those for a group
not receiving the
intervention”*
• RCTs are considered the
gold standard
Why such an emphasis
on impact studies?
• Limited resources
• Shift to focus on evidence-
based reform in the United
States
*https://ies.ed.gov/ncee/wwc/Glossary

9. Background – Student Success Societies Program
• Among the college access programming available to students, the
Commission sought to expand mentoring options available to the
cohort students as they entered high school.
• Overall, young adults from low-income and underrepresented groups who
have a mentor are 55% more likely to enroll in college than similar students
without a mentor (Bruce & Bridgeland, 2014).
• Students with mentors also post higher school attendance rates and exhibit
better attitudes toward school (Herrera, DuBois, & Grossman, 2013).
• In fall 2016, the Commission launched the Student Success Society,
an intensive mentoring pilot program.

10. • Goals of SSS were to help students:
• The program was designed for mentors to meet with a group of 10-12 student
every week to cover a curriculum of 10 lessons (e.g., high school success, grit,
learning styles, emotional intelligence, social competencies, and academic
success).
• Mentors were expected to work with students to design a group project that
supports the overall goals and mission of WV GEAR UP.
• The Commission was committed to measuring the impact of the SSS mentoring
program with a randomized control trial (RCT).
Background – Student Success Societies Program
• Increase confidence and self-
awareness
• Encourage academic success
• Promote responsibility
• Build leadership skills
• Promote student engagement
• Prepare students to succeed in
postsecondary education

11. Identifying a control group
• Core GEAR UP services offered to all cohort students; the offer
of mentoring only to a random group (no denial of service)
• Only three schools were selected to participate in the one year
RCT
Reducing cost and burden to participants
• Limited logistical responsibility of schools
• Used existing data sources (administrative data, survey data)
Challenges & Solutions for Conducting RCTs in Schools

12. Eliminating confounding factors
• Treatment and control group students were selected from the same year and
both received GEAR UP services
• Three schools were selected (to eliminate the school confound but remain
manageable)
Reducing attrition/data loss
• Evaluators provided weekly reports on students’ survey completion status
• Program staff followed up to reduce attrition and inform ICF when students
left schools
• Research questions were designed to make use of pre-existing data
Challenges & Solutions for Conducting RCTs in Schools

13. Reducing no-shows and contamination
• Opt-out prior to assignment
• Mentors notified of treatment and control assignment
• Participation in other GEAR UP services tracked
Achieving statistical power
• Oversampling to assign more students to control than treatment
increases power to detect small effects without impacting cost
or overburdening mentors
Challenges & Solutions for Conducting RCTs in Schools

14. 14
SSS Study Purpose and Design

15. Study Purpose
To gather high-quality causal evidence about the benefits
of providing intensive mentoring support to students in
three high-need West Virginia high schools
Outcomes across multiple domains were tested to identify
evidence of promise.
Results from the RCT pilot study will be used to assess the
initial success of the mentoring program with a high degree
of internal validity and to inform its subsequent
development and statewide scalability.

16. Research Questions
What is the impact of offering intensive student mentoring upon students’
college-going self-efficacy, college-going outcomes-expectations, and grit* as
measured by annual WV GEAR UP student surveys?
What is the impact of offering intensive student mentoring upon students’
behavioral engagement as measured by total and unexcused absences, and
number of discipline referrals?
What is the impact of offering intensive student mentoring upon students’
academic achievement, as measured by grade point averages?*
*The analyses of grit and GPA outcomes are considered exploratory
due to the fact that ICF only had access to post-intervention data.

17. Outcomes
Behavioral
engagement
Disciplinary referral &
attendance rate
Academic
achievement
GPA
Attitudes toward
college-going
CGSE & CGOE
Non-cognitive skills
Grit

18. Data Sources
State &
District
Databases
•Student
demographics
•Attendance
and academic
achievement
data
SCRIBE
Data
Warehouse
System
•Participation in
mentoring and
other GEAR UP
services
GEAR UP
Annual
Student
Surveys
•Attitudes
toward
college-going
•Grit
A focused effort was made
to identify data sources
that would reduce burden
and cost.
District and state data-
sharing agreements as
well as subcontracting
arrangements made a
variety of data available
via existing systems.

19. 1. College-Going Self-Efficacy (CGSE)
How sure are you about being able to do the following?
I can find a way to pay for college.
I can get accepted to a college.
I can have family support for going to college.
I can choose a good college.
I can get a scholarship or grant for college.
I can make an educational plan that will prepare me for
college.
I can make my family proud with my choices after high
school.
I can choose college courses that best fit my interests.
I can pay for college even if my family cannot help me.
I can get good grades in my high school math classes.
I can get good grades in my high school science classes.
I can choose the high school classes needed to get into a
good college.
I know enough about computers/technology to get into
college.
2. College-Going Outcomes Expectations (CGOE)
If you do go to college, how sure are you about being
able to do the following?
I could pay for each year of college.
I could get A’s and B’s in college.
I could get my family to support my wish of finishing
college.
I could take care of myself in college.
I could fit in at college.
I could get good enough grades to get or keep a
scholarship.
I could finish college and receive a college degree.
I could care for my family responsibilities while in college.
I could set my own schedule while in college.
I could make friends at college.
I could get the education I need for my choice of career.
I could get a job after I graduate from college.
I would like being in college.
I could pick the right things to study at college.
I could do the classwork and homework assignments in
college classes.
Reliability: Cronbach Alpha [α] = 0.95 for CGSE and 0.97 for CGOE
Data Sources: Attitudes Towards College-Going

20. Grit Items
New ideas and projects sometimes distract me from previous ones.
Setbacks don’t discourage me.
I have been obsessed with a certain idea or project for a short time, but later lost
interest.
I am a hard worker.
I often set a goal but later choose to pursue a different one.
I have difficulty maintaining my focus on projects that take more than a few
months to complete.
I finish whatever I begin.
I am diligent.
The reliability of the scale, estimated using Cronbach’s α (estimated from
the data), was low (.50); caution is required when interpreting the results
of this analysis.
Data Sources: Grit

21. • ICF evaluators additionally conducted phone
interviews with mentors at RCT schools in spring
2018 to understand nuances of implementation.
• These data were not part of the RCT, but helped
to provide context regarding the SSS program.
Data Sources

22. Study Design
Design: Randomized Control Trial (RCT)
Setting: Three rural West Virginia high schools. Schools are a subset of 23 WV
GEAR UP high schools for the 2016-17 school year. Collectively, these three
schools enrolled 564 9th grade cohort students during 2016-17.
Students: A total sample of 450 Grade 9 students is included in the study.
• Treatment Group: 30 Grade 9 students in each high school randomly
assigned to receive the offer of the intensive SSS mentoring intervention
for the duration of their freshman year of high school (N = 90)
• Control Group: 120 Grade 9 students in each school randomly assigned
not to receive the offer of the SSS intervention (N = 360)
• N/A: Additional cohort students from participating RCT schools who were
not assigned to the treatment or control groups.
• Both groups of students are eligible to receive other services through WV
GEAR UP. Impact estimates will reflect the added value of offering
intensive mentoring.

23. Timeline
Plan study, and draft
WVDE research
application
(SUMMER ‘16)
Select schools,
gather/clean student
rosters
(SEPT ‘16)
Randomize students
to conditions /
inform mentors
(OCT ‘16)
Administer study
pretest survey:
baseline
(DEC ‘16)
Mentors implement
SSS program during
school year
(OCT ’16 - MAY ‘17
Gather demo. data
and 2016-17
outcome data
(AUG ‘17)
Administer study
posttest survey:
outcome
(DEC ‘17)
Conduct interviews
with mentors
(SPRING ‘18)

24. 24
Results

25. Results: Study Attrition
The
Whole
Sample
Treatment
Group
Control
Group
Summary
Outcomes Attrition
rate
Attrition
rate
Attrition
rate
Differential
attrition
WWC attrition
level
CGSE 24% 20% 26% 6% Low
CGOE 25% 21% 26% 5% Low
Grit 25% 20% 26% 6% Low
Unexcused absence
rate
6% 4% 6% 2% Low
Disciplinary referral 6% 4% 6% 2% Low
GPA 11% 11% 11% 0% Low

26. Zero hours
18%
0.1–4.9 hours
32%
5–9.9 hours
36%
10–14.9 hours
4%
15 hours or
more
10%
Program Exposure (n=90) • WV GEAR UP regional program directors
determined that at least 10 hours of
mentoring constituted a minimum effective
dose of the program
• Only 14% of treatment students met this
benchmark.
• Average exposure hours for all treatment
students = 5.5 hours; average exposure hours
for treatment students with at least 0.1 hours
or more of exposure = 6.7 hours
• These findings cast doubt on whether or not
the study results accurately estimate the
impact of the SSS program.
Results: Program Exposure

27. • School challenges varied from being unable to secure enough mentors
to engaging individuals with substantial commitment to and interest
in mentoring.
• Some mentors felt that the neediest students who were randomly
assigned to the program often did not stay with it.
• Two of the three sites encountered challenges carving out a regular
time for students to meet with mentors during the 2016-17 year. “We
just had to bring them in when we could get them, and it was really
hard to get them all together at one time.”
• Three mentors believed that mentoring sessions could seem too much
like a class, with students becoming less engaged as a result. One
mentor reported that some students chose another GEAR UP activity
that occurred at the same time as SSS because the former seemed less
like a traditional school-day class
Results: Mentor Interviews

28. No significant differences between
treatment and control groups for all
indicators
Results: Impact Analysis
Outcome
Variables
Estimated
Program
impact
Statistical
Test (alpha
= 0.05)
Standardize
d Effect Size
CGSE -0.06 Not
Significant
-0.06
CGOE -0.16 Not
Significant
-0.15
Grit 0.00 Not
Significant
0.00
Outcome
Variables
Estimated
Program
Impact
Statistical
Test (alpha
= 0.05)
Standardize
d Effect Size
Unexcused
absence rate
0.00 Not Sig. 0.05
Number of
discipline
referrals
-0.37 Not Sig. -0.19
Outcome
Variables
Estimated
Program
Impact
Statistical
Test (alpha
=0.05)
Standardize
d Effect Size
Cumulative
GPA
0.12 Not Sig. 0.11

29. 3.31 3.19 3.17
3.40 3.56 3.47
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Control group Group 1
Zero hours
Group 2
0.1-4.9 hours
Group 3
5-9.9 hours
Group 4
10-14.9 hours
Group 5
15 or more
hours
n=282 n=8 n=24 n=31 n=4 n=8
Grit
0.04
0.11
0.07
0.02 0.02
0.05
0.00
0.02
0.04
0.06
0.08
0.10
0.12
Control group Group 1
Zero hours
Group 2
0.1-4.9 hours
Group 3
5-9.9 hours
Group 4
10-14.9 hours
Group 5
15 or more
hours
n=350 n=16 n=28 n=31 n=4 n=9
Absence rate
1.37
1.75
1.93
0.19
0.00
2.22
0.00
0.50
1.00
1.50
2.00
2.50
Control group Group 1
Zero hours
Group 2
0.1-4.9 hours
Group 3
5-9.9 hours
Group 4
10-14.9 hours
Group 5
15 or more
hours
n=350 n=16 n=28 n=31 n=4 n=9
Disciplinary Referral
2.64
1.93
2.19
3.08
3.51
2.92
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Control group Group 1
Zero hours
Group 2
0.1-4.9 hours
Group 3
5-9.9 hours
Group 4
10-14.9 hours
Group 5
15 or more
hours
n=333 n=16 n=26 n=30 n=4 n=6
GPA
Results: Exploratory Analysis

30. 30
Lessons Learned

31. Lessons Learned: Overall Design Successes
Randomization
Utilization of existing data to control costs
Reduced participant burden
Low attrition for all six outcomes

32. Lessons Learned: Randomization
A Clear Connection
• Some recent research indicates that
matched mentor/mentee relationships
leads to greater outcomes than those
that are randomly selected.
• There should be a clear connection
between the student and mentor.
• If students feel "forced" to participate, it
can diminish the results and have
significant impact on the long-term
participation.
Is randomization
the right model for
evaluating
mentoring?

33. Lessons Learned: Program Exposure
For logistical reasons, all of the study’s exposure data were
collected after the fact.
This approach represents a missed opportunity to address
the low dosage issue as it was happening.
Findings point to the value of monitoring a program’s
implementation in real time to see more meaningful results.

34. • Grit scale had low reliability (0.50), which is problematic
for measuring impact and may have contributed to lack of
statistically significant differences between treatment and
control with respect to grit.
• Mentor interviews revealed that mentors observed
increased student motivation and communication,
particularly from shy students and those not involved in
many school activities. Were we using the right tools to
measure potential impacts of the SSS mentoring
program?
Lessons Learned: Measurement Scales

35. Relationship Building
• Limit the number of students assigned to mentors. Since this project was a
school-based group mentoring program, it's important that mentors have a
manageable number so they can focus on relationship building.
Match Longevity
• Consider a time frame longer than one academic year allowing
student/mentor matches to build relationships over a longer span of time.
This would also provide more exposure.
Exposure
• The more exposure students have with their mentors, the more likely the
program may have an impact. Consider time restraints placed upon school-
based mentoring programs and limits that administrators have to adhere
to. How much exposure will students have with their mentor and
mentoring peers?
Recommendations for Future Studies on Mentoring

36. Time and Space for Mentoring
• Mentor interviews revealed challenges in finding time and space for
mentoring to take place. Future studies on mentoring should ensure
that the mentoring program being studied includes time and space
for the mentoring to take place. This will help to ensure program
exposure.
Mentoring Format/Content
• Future studies may wish to look at the format/content of
mentoring—whether the mentoring program is more structured (like
a class) versus more flexible and how that may affect mentoring
impacts.
Recommendations for Future Studies on Mentoring

37. 37
Questions and Comments

38. Adjourn
Samantha Spinney
samantha.spinney@icf.com
Zorrie Georgieva
zornitsa.georgieva@wvhepc.edu