Short Paper Presentation at Learning Analytics and Knowledge Conference 2012, May 1. #LAK12 This paper presents current findings from an ongoing design- based research project aimed at developing an early warning system (EWS) for academic mentors in an undergraduate engineering mentoring program. This paper details our progress in mining Learning Management System data and translating these data into an EWS for academic mentors. We focus on the role of mentors and advisors, and elaborate on their importance in learning analytics-based interventions developed for higher education.

1. Bridging the Gap from
Knowledge to Action:
Putting Analytics in the
Hands of Academic
Advisors
Steven Lonn
Andrew Krumm
R. Joseph Waddington
Stephanie Teasley
USE Lab University of Michigan
Digital Media Commons www.umich.edu/~uselab

2. Research Setting:
M-STEM Academy
• Undergraduate engineering mentoring program
• Historically underrepresented students
• 200 Engineering students in 4 cohorts
USE Lab
Digital Media Commons 2

3. Goals of the project
• Utilize data stored in campus learning
management system to:
• Provide timely and targeted data on student
performance to M-STEM mentors
• Shorten the timespan from problem
identification to intervention
USE Lab
Digital Media Commons 3

4. Supporting M-STEM mentors
• Iteratively develop
• Metrics for comparing
students using LMS data
• Classification schemes
• Visualizations of student
performances
• Send mentors weekly updates
Photo%Credit:%h,p://teacherrogers.wordpress.com
USE Lab
Digital Media Commons 4

5. How does mentor’s use of EWS affect
student outcomes?
USE Lab
Digital Media Commons 5

6. Improved
Performance
Action
academic resources,
Face-to-Face / Email Communication study strategies
Mentor Audience Student
EWS Product
Classification Analysis
Data 6

7. Improved
Performance
Action
academic resources,
Face-to-Face / Email Communication study strategies
Mentor Audience Student
EWS Product
Classification Analysis
Data 6

8. Assignments
Gradebook
Weekly%query%of%LMS%for%courses%
that%include%an%MASTEM%student%and%
use%the%Gradebook%or%Assignments% “Presence”
tool 8

9. Measures from LMS data
• Gradebook and Assignments tools allow up-to-date
tracking of student performances
• Report student-level information for M-STEM
students
• Percent of available points earned
• Course averages (all students)
USE Lab
Digital Media Commons 9

10. Measures from LMS data
• “Presence” events serve as a proxy for effort and are
events common to all courses
• Cumulative and week-to-week “Presence”
USE Lab
Digital Media Commons 10

11. 75%
mean
25%
USE Lab
Digital Media Commons 11

12. Cumulative “Presence”
events can be highly
predictive for students’ final 75%
course grade performance
mean
25%
USE Lab
Digital Media Commons 11

13. USE Lab
Digital Media Commons 12

14. Classification scheme
• Absolute grade thresholds
• Difference from course average
• Presence cutoff
USE Lab
Digital Media Commons 12

15. Classification scheme
• Absolute grade thresholds
• Difference from course average
• Presence cutoff
Comparisons
• Grades (course average)
• Percentile Ranks (presence)
USE Lab
Digital Media Commons 12

16. Classification scheme
Student % Relative Distance Presence Percentile Rank E3
>=0.85 . . Encourage
0.75<=X<0.85 <-0.15 . Explore
0.75<=X<0.85 >=-0.15 <0.25 Explore
0.75<=X<0.85 >=-0.15 >=0.25 Encourage
0.65<=X<0.75 <-0.15 <0.25 Engage
0.65<=X<0.75 <-0.15 >=0.25 Explore
0.65<=X<0.75 >=-0.15 . Explore
0.55<=X<0.65 >=-0.10 . Explore
0.55<=X<0.65 <-0.10 . Engage
<0.55 . . Engage
USE Lab
Digital Media Commons 13

17. Mentor summary
USE Lab
Digital Media Commons 14

18. Student Detail Report
USE Lab
Digital Media Commons 15

19. Benefits of EWS use
• Contacting students
• Shortening time to intervention
• Viewing longitudinal trends
• By individual course
• Across all courses
• Contextualizing M-STEM student performance
USE Lab
Digital Media Commons 16

20. How mentors use the EWS
USE Lab
Digital Media Commons 17

21. Next Steps
• New infrastructure
• New versions
• Instructor
• Students
• Messaging system
• Recommendations (from person, from system)
USE Lab
Digital Media Commons 18

22. Improved
Performance
Action
academic resources,
Face-to-Face / Email Communication study strategies
Mentor Audience Student
EWS Product
Classification Analysis
USE Lab
Digital Media Commons Data 19

23. Conclusion
• Closing the gap between problem identification and
intervention
• Organizational capacity and the success of learning
analytics
• “Analytics” is but a small part
• Information is always subject to interpretation
• How can we scaffold interpretation and effective
action-taking?
USE Lab
Digital Media Commons 20

24. Collaborators
M-STEM ITS
• Cinda-Sue Davis • Bryan Hartman
• Guy Meadows • Jeff Jenkins
• James Holloway • Dan Kiskis
• Daryl Koch
• Mark Jones USE Lab
• Debbie Taylor • Gierad Laput
USE Lab
Digital Media Commons 21

25. Questions
Steve slonn@umich.edu @stevelonn
Stephanie steasley@umich.edu @stephteasley
www.umich.edu/~uselab
slides: www.slideshare.net/stevelonn
USE Lab
Digital Media Commons 22