Learning analytics has the potential to make the temporal dimensions of learning processes more visible using fine-grained proxies of how and when students engage with online learning activities. In this talk, Quan Nguyen will demonstrate the extent to which students actually follow the course timeline and the subsequent effect on their academic performance

1. Quan Nguyen
Postdoctoral Research Fellow, School of Information
University of Michigan
Using temporal analytics to detect inconsistencies
between learning design and student engagement
quanngu@umich.edu
@QuanNguyen3010

2. Exams
Surveys
Readings
Lectures
Assignments

3. Learning analytics over time
Update Nov 7th 2019

4. Learning analytics by institutions
Source: Web of Science, topic “learning analytics” (Nov 7th 2019)

5. Challenges in learning analytics
1. Instructional context
2. Actionable feedback
3. Temporal analytics

6. Learning design & learning analytics
Without context With context

7. Learning design & learning analytics
Learning
analytics
Learning
design
Explicit feedback
Pedagogical context
(Lockyer et al., 2013; Lockyer & Dawson, 2011; Persico & Pozzi, 2015; Mor et al., 2015)

8. Learning design & learning analytics
1. Conceptual frameworks (Bakharia et al., 2016; Lockyer et al., 2011;
Mor et al., 2015; Persico et al., 2015; Hernández‐Leo, D., 2019)
2. Empirical studies (Gasevic et al., 2015; Rienties et al., 2016)
3. Special issues (BJET 2015, JLA 2019)
4. A systematic review (Mangaroska et al., 2018)
*Note: Not a comprehensive list

9. Learning design in this context
1. Not about UX design
2. Not a prescriptive tool
3. Aim to provide insights to help instructors make their own decisions

10. Agenda
1. How do instructors design courses in
online and distance education?
2. To what extent students’ learning
patterns align with course design and its
effect on academic performance?
3. How does student demographics affect
their engagement and performance?
Pedagogical
context
Demographics
context
Engagement &
Performance

11. Study context
• Online & distance education
• Approx. 130K enrolled students, largest in Europe
• Diverse demographics
• Lots of data, back to 1990s.
• Trace data: approx. 2Billion data points each month

12. LEARNING DESIGN AT THE OU

13. LEARNING DESIGN AT THE OU
OULDI
Assimilative
Finding information
Productive
Interactive
Experiential
Communication
Assessment

14. LEARNING DESIGN AT THE OU
A learning design at a weekly level

15. Agenda
1. How do instructors design courses in online and distance education?

16. Learning design over time
Nguyen, Q., Rienties, B., & Toetenel, L. (2017). Unravelling the dynamics of instructional practice: a longitudinal study on learning design and VLE activities.
In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, LAK 17, ACM, New York, NY, USA, pp. 168–177.
37 courses

17. Learning design over time
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based
assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76, 703-714.

18. Nguyen, Q., Rienties, B., & Toetenel, L. (2017). Unravelling the dynamics of instructional practice: a longitudinal study on learning design and VLE activities.
In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, LAK 17, ACM, New York, NY, USA, pp. 168–177.
A network of learning activities
Figure 12. A weighted two-mode network of module X
Figure 13. Transformation of a two-mode network into a one-mode network

19. 04Nguyen, Q., Rienties, B., & Toetenel, L. (2017). Unravelling the dynamics of instructional practice: a longitudinal study on learning design and VLE activities.
In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, LAK 17, ACM, New York, NY, USA, pp. 168–177.
A network of learning activities

20. Agenda
1. How do instructors design courses in online and distance education?
• Assimilative, productive, and assessment were the main activity types
• Workload was not always consistent over time
• Strong ties between assimilative and productive
• Diverse combinations of activities between courses

21. Agenda
1. How do instructors design courses in online and distance education?
2. To what extent students’ learning patterns align with course design
and its effect on academic performance?

22. Learning design and student engagement
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based
assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76, 703-714.
37 courses over
30 weeks
45,190 students

23. Learning design and student engagement
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based
assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76, 703-714.
Variables 1 2 3 4 5 6 7 8
1. Assessment
2. Assimilative -.46**
3. Communication -.12** .17**
4. Information -.12** .08** .17**
5. Productive -.29** .16** .13** .17**
6. Experiential -.06* .02 -.02 -.02 .00
7. Interactive .00 .02 .05 .01 .01 .01
8. VLE per week .20** .01 .27** .05 .01 .01 .16**
9. VLE per visit .12** .10** .22** .04 .07* .04 .09** .84**
N = 37 modules over 30 weeks (1,088 data points)
* p < .05, ** p < .01

24. Learning design and student engagement
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the
designs of computer-based assessment and its impact on student engagement,
satisfaction, and pass rates. Computers in Human Behavior, 76, 703-714.
DV = VLE per week Unstandardized coefficients
(1) (2) (3) (4)
MODELS OLS FE_week FE_module FE_module
_week
Assessment 4.98** 5.09** 2.47** 2.80**
(0.57) (0.59) (0.47) (0.47)
Information 2.40 3.23 -0.72 0.15
(2.64) (2.60) (1.98) (1.94)
Communication 26.29** 26.29** 16.54** 17.44**
(2.66) (2.62) (2.16) (2.11)
Productive 1.75 1.73 -1.84 -1.83
(1.14) (1.12) (1.04) (1.03)
Experiential 3.57 4.49 -2.07 -0.99
(3.83) (3.78) (2.98) (2.91)
Interactive 11.57** 11.25** -0.33 -0.46
(2.23) (2.20) (1.81) (1.78)
Constant 95.66** 95.30** 110.6** 172.1**
(2.91) (2.85) (2.46) (10.50)
Observations 1,088 1,088 1,088 1,088
Adjusted R-squared 0.15 0.19 0.55 0.58
DV = VLE per visit Unstandardized coefficients
(1) (2) (3) (4)
MODELS OLS FE_week FE_module FE_module_
week
Assessment 0.46** 0.48** 0.02 0.05
(0.07) (0.08) (0.05) (0.05)
Information 0.13 0.20 -0.29 -0.21
(0.35) (0.35) (0.22) (0.21)
Communication 2.76** 2.78** 0.96** 1.06**
(0.35) (0.35) (0.24) (0.23)
Productive 0.46** 0.46** -0.17 -0.16
(0.15) (0.15) (0.11) (0.11)
Experiential 1.04* 1.09* 0.52 0.60
(0.51) (0.51) (0.33) (0.32)
Interactive 0.79** 0.72* -0.34 -0.39
(0.30) (0.30) (0.20) (0.20)
Constant 20.56** 24.15** 21.00** 24.66**
(0.39) (1.37) (0.91) (1.18)
Observations 1,088 1,088 1,088 1,088
Adjusted R-squared 0.08 0.10 0.67 0.69
Standard errors in parentheses
* p < .05, ** p < .01Baseline: assimilative

25. Time management in education
Before Postdoc After Postdoc
Learning takes time…
In Michigan weather

26. Research gaps
• Most studies used self-report instruments
• Recent studies started including trace data but focused on “how
much” time was spent, not “when” it was spent
• Learning design has been missing…
Wolters, C. A., Won, S., & Hussain, M. (2017).
Thibodeaux, J., Deutsch, A., Kitsantas, A., & Winsler, A. (2017)
Nawrot, I., & Doucet, A. (2014)
Jo, I. H., Park, Y., Yoon, M., & Sung, H. (2016)
Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015)

27. Temporal analytics
Week 8’s study materials
In advance Catch up or
revise
(1) in advance – material x assigned
to week t was studied during or
before week t
(2) catching up and revise – material
x assigned to week t was studied
after week t

28. Temporal analytics
• Ecology
• Level 2 (sophomore)
• 30 credits
• Online
• 2015: 182 students
• 2016: 198 students
• > 1.8M data points

29. Learning design and student engagement
Nguyen, Q., Huptych M., Rienties. B. (2018). Linking student’s timing of engagement with learning design and academic performance (best paper
award). In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK18), pp. 141-150, Sydney, Australia.

30. Student engagement and academic performance
Nguyen, Q., Huptych M., Rienties. B. (2018). Linking student’s timing of engagement with learning design and academic performance (best paper
award). In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK18), pp. 141-150, Sydney, Australia.

31. Student engagement and academic performance
Nguyen, Q., Huptych M., Rienties. B. (2018). Linking student’s timing of engagement with learning design and academic performance (best paper
award). In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK18), pp. 141-150, Sydney, Australia.

32. Student engagement and academic performance
Nguyen, Q., Huptych M., Rienties. B. (2018). Linking student’s timing of engagement with learning design and academic performance (best paper
award). In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK18), pp. 141-150, Sydney, Australia.

33. Click to edit
Master title style
Timing of engagement per study material

34. Excellent group
In advance Catching up

35. Passed group
In advance Catching up

36. Failed group
In advance Catching up

37. Student engagement and academic performance

38. Click to edit
Master title style
Variance across individuals (learners as agents)

39. Implications
Catching up
• Students: This chapter might
take more time to finish, I
better prepare ahead
• Instructors: This chapter
seemed to take students
more time than I expected, I
should examine why

40. Effect of study breaks and exam revision weeks
More study breaks = better retention?
More exam revision weeks = better retention?
Yes
No
187 courses
111,610 students
Way too many trace data…
Nguyen, Q., Thorne, S., & Rienties, B. (2018). How do students
engage with computer-based assessments: impact of study breaks
on intertemporal engagement and pass rates. Behaviormetrika. 1-18

41. What did students do during exam revision weeks?
Nguyen, Q., Thorne, S., & Rienties, B. (2018). How do students
engage with computer-based assessments: impact of study breaks
on intertemporal engagement and pass rates. Behaviormetrika. 1-18
3385 students
Same course over 3 semesters

42. Agenda
2. To what extent students’ learning patterns align with course design
and its effect on academic performance?
• Assessment, assimilative, and communication activities strongly
influenced student engagement
• Instructional context accounted for 69% of the variance in student
engagement
• High-performing students studied ‘harder’ and engaged in a more
timely manner
• Learning analytics helps identify problematic learning materials
• Learning analytics helps evaluate instructional decisions

43. Agenda
1. How do instructors design courses in online and distance education?
2. To what extent students’ learning patterns align with course design
and its effect on academic performance?
3. What can learning analytics tell us about the achievement gaps?

44. Achievement gaps
Can the achievement gaps be attributed to the differences in level of
engagement?
If I put in the same amount of effort, I should have an equal chance of
success?
Nguyen, Q., Rienties, B., Richardson, J. (in press). Learning analytics to uncover inequality in
behavioural engagement and academic attainment in a distance learning setting. Assessment &
Evaluation in Higher Education

45. Achievement gaps
401 courses
149,672 students
daily traces
Nguyen, Q., Rienties, B., Richardson, J. (in press). Learning analytics to uncover inequality in behavioural
engagement and academic attainment in a distance learning setting. Assessment & Evaluation in Higher Education
• Given the same level of engagement, BAME students were between
19% and 79% less likely to complete, pass, or achieve an excellent
grade compared to White students
• Given the same academic performance, BME students spent 4-12%
more time on studying than White students

46. Conclusion
1. Context matters in learning analytics
2. Aligning LD with LA can provide instructors with actionable feedback
3. We don’t know as much as we think we do → pedagogical
assumptions need to be tested

47. Instructors
Exams
Surveys
Readings
Lectures
Assignments

48. Agenda
1. How do instructors design courses in online and distance education?
2. To what extent students’ learning patterns align with course design
and its effect on academic performance?
3. What can learning analytics tell us about the achievement gaps?
4. What am I going to do with my life? Aka future research

49. Future plan
• Buy a good winter parka…
• Extend temporal learning analytics with time-series
models (ARIMA, LSTM), temporal network analysis
(TERGMs, SAOM, REM)
• Collaboration & applying for grants
• Publish and keep Chris happy! ☺

50. GET IN TOUCH
quanngu@umich.edu
@QuanNguyen3010
www.quan-nguyen.com