This document discusses challenges and opportunities in learning analytics. It addresses what should be measured, issues around instrumentation, capturing interventions and social networks. It also discusses challenges around scaling qualitative analysis, temporal processes, longitudinal studies, privacy, ethics and data sharing. Overall, the document advocates that learning analytics should go beyond just measuring outcomes and page access counts, and should seek to understand learning processes, contexts and the effects of students' own decisions.

1. Learning Analytics is about
Learning
Dragan Gasevic
@dgasevic

2. Growing demand for
education!

4. Scalability is possible
Low effect size of class-size
John Hattie

5. Delivery

6. Delivery

7. Scientific American, March 13, 2013
http://www.scientificamerican.com/article.cfm?id=massive-open-online-courses-transformhigher-education-and-science

8. MOTIVATION

9. Feedback loops between
students and instructors
are missing!
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational
research, 77(1), 81-112.

10. Learners
Registrations
Educators
Learning and
Collaborating

11. Learners
Registrations
Networks
Mobile
Search
Educators
Learning and
Collaborating
Networks
Videos/slides

12. Learners
Registrations
Networks
Mobile
Search
Educators
Learning and
Collaborating
Networks
Videos/slides

13. DANGER

14. Predict-o-mania
The same predictive models for
everything and everyone

16. Student diversity
http://www.census.gov/prod/2013pubs/acsbr11-14.pdf

17. Population Diversity
100%
90%
80%
70%
ACCT 1 (n = 746)
BIOL 1 (n = 220)
60%
BIOL 2 (n = 657)
50%
COMM 1 (n = 499)
COMP 1 (n = 242)
40%
ECON 1 (n = 661)
30%
GRAP 1 (n = 192)
MARK 1 (n = 723)
20%
MATH 1 (n = 194)
10%
0%
Females
International
students
Other
Living in nonlanguage at
urban
home
Part time
student
Previously
enrolled to a
course
Early access Did not access Late access

18. LMS Functionality Diversity
ACCT 1
Light Box Gallery
Forum
Course
Resource
Turn-it-in
Assignment
Book
Quiz
Feedback
Map
Virtual Classroom
Lesson
Glossary
Chat
X
X
X
X
X
X
X
BIOL 1
X
X
X
BIOL 2
X
X
X
X
X
X
X
X
X
X
COMM 1 COMP 1 ECON 1
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
GRAP 1
X
X
X
MARK 1 MATH 1
X
X
X
X
X
X
X
X
X
X
X
X
X

19. Predictive Power Diversity
100.00%
90.00%
80.00%
70.00%
60.00%
Model 1
50.00%
Moodle
40.00%
Model 1 + Moodle
30.00%
20.00%
10.00%
0.00%
All courses ACCT 1
together
BIOL 1
BIOL 2
COMM 1
COMP 1
Model 1 – demographic and socio-economic variables
* - not statistically significant
ECON 1
* GRAP 1 MARK 1
MATH 1

20. Retention is not
the only challenge
It is important, of course!
But, where is learning?

21. How do we
enhance learning
if the focus is on
outcomes only?

22. DIRECTION

23. Learning Analytics – What?
Measurement, collection,
analysis, and reporting of data
about learners and their contexts

24. Learning Analytics – Why?
Understanding and optimising
learning and the environments
in which learning occurs

25. Modern Educational Psychology
Human agency is
central to learning
Bandura, A. (1989). Human agency in social cognitive theory. American
psychologist, 44(9), 1175-1184.

26. Winne and Hadwin's model
of self-regulated learning

27. Knowledge society and
knowledge economy

28. Why does it matter?!
Challenge
Metacognitive skills
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-Regulated Learning: Beliefs, Techniques, and
Illusions. Annual Review of Psychology, 64, 417-444. doi:10.1146/annurev-psych-113011-143823

29. Why does it matter?!
Challenge
Information seeking skills
Judd, T., & Kennedy, G. (2011). Expediency-based practice? Medical students’ reliance on Google and
Wikipedia for biomedical inquiries. British Journal of Educational Technology, 42 (2), 351-360.
doi:10.1111/j.1467-8535.2009.01019.x

30. Why does it matter?!
Challenge
Sensemaking paradox
Butcher, K. R., & Sumner, R. (2011). Self-Directed Learning and the Sensemaking Paradox. Human–
Computer Interaction, 26(1-2), 123-159. doi:10.1080/07370024.2011.556552

31. Why does it matter?!
Challenge
Asking questions and critical thinking
Graesser, A. C., & Olde, B. (2003). How does one know whether a person understands a device? The
quality of the questions the person asks when the device breaks down. Journal of Educational
Psychology, 95(3), 524–536..

32. Process and context focus
for learning analytics needed
to understand learning

33. OPPORTUNITIES

34. Learning Analytics
Effects of learning context
External conditions (e.g., instructional design)

36. Cognitive presence
the extent to which the participants in any
particular configuration of a CoI are able to
construct meaning via sustained communication
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical Thinking and Computer Conferencing: A Model
and Tool to Assess Cognitive Presence. American Journal of Distance Education ,15(1), 7-23.

37. Effect size of the moderator role on
critical thinking
Cohen’s d = 0.66

38. Effect size of an intervention on
critical thinking in online discussions
d = 0.95 (non-moderators)
and
d = 0.61 (moderators)

39. Cognitive presence
TMA1
TMA2
TMA3
TMA4
Final
Control group
Cognitive Presence in Online
Discussions – Association w/ Grades
Triggering event
Exploration
Integration
Resolution
Other
-.226
-.001
.128
.201
-.028
.005
.141
.060
.027
.078
-.046
.009
.034
-.023
.113
-.050
-.037
.043
-.054
.106
-.010
.048
.113
.074
.154
** p
< 0.01; * p < 0.05

40. Cognitive Presence in Online
Discussions – Association w/ Grades
Intervention
group
Control group
Cognitive presence
** p
TMA1
TMA2
TMA3
TMA4
Final
Triggering event
Exploration
Integration
Resolution
Other
Triggering event
Exploration
-.226
-.001
.128
.201
-.028
.149
.216
.005
.141
.060
.027
.078
-.077
.197
-.046
.009
.034
-.023
.113
-.070
.163
-.050
-.037
.043
-.054
.106
.000
.223
-.010
.048
.113
.074
.154
.016
.243
Integration
.156
.396**
.417**
.338*
.454**
Resolution
Other
-.041
.219
.060
.046
.154
.050
.083
.075
.129
.088
< 0.01; * p < 0.05

41. Integration posts:
effect on final grades
100
80
60
40
20
0
Q1
Q2
Q3
p < .001, Q1 vs. Q2; Q1 vs. Q3, Q1 vs. Q4
Q4

42. Learning Analytics
Are students only driven by
assessments?
Effects of external conditions

43. Self-reflections in video annotations
Course 1
(non-graded)
Course 3
(graded)
Course 2
(graded)
Course 4
(non-graded)

44. Self-reflections in video annotations
120.00
100.00
80.00
Course 1 (non-graded)
Course 2a (graded)
60.00
Course 2b (graded)
Course 3 (graded)
40.00
Course 4 (non-graded)
20.00
0.00
Annotation total Annotation postion Annotation postion Annotation postion Annotation postion Annotation general
Q1
Q2
Q3
Q4

45. Self-reflections in video annotations
1800
1600
1400
1200
Course 1 (non-graded)
1000
Course 2a (graded)
Course 2b (graded)
800
Course 3 (graded)
600
Course 4 (non-graded)
400
200
0
Cognitive processes Perceptual processes
Positive emotions
Negative emotions
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and
computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54.

46. Learning Analytics
Effects of
students’ own decisions
Beyond external conditions

47. Learner profiles – use of LMS
Effect size .75 on
critical thinking &
academic success
3
4

48. Learner profiles – use of LMS
14
12
10
Triggering
8
Exploration
Integration
6
Resolution
Other
4
2
0
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Effect size .75 on critical thinking and
academic success

49. CHALLENGES

50. Learning Analytics
What to measure?
We don’t need page access counts only!
Wilson, T.D. (1999). Models in information behaviour research.
Journal of Documentation, 55(3), 249 - 270, doi:10.1108/EUM0000000007145

51. Instrumentation
About specific contexts and constructs

52. Instrumentation
Capturing interventions
Previous learning and (memory of) experience
Social networks (e.g., communication, cross-class)
Interaction types (e.g., transactional distances)

53. Motivation in
Information Interaction
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced
goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004

54. Motivation in
Information Interaction
Achievement goal
orientation (2x2)
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced
goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004

55. Technology and
process of self-regulated learning
Siadaty, M. (2013). Semantic Web-Enabled Interventions to Support Workplace Learning,
PhD Thesis, Simon Fraser University, Surrey, BC, Canada.

56. Scaling up qualitative analysis

57. Temporal processes
beyond coding and counting

58. Longitudinal studies

59. Generating
reports and nice visualization is
not enough

60. Building data-driven culture in
institutions
Manyika, J., et al., Big Data: The Next Frontier for Innovation, Competition, and Productivity,
2011, McKinsey Global Institute, http://goo.gl/Lue3qs

62. Privacy and ethics

63. Data sharing and mobility

64. Thank you!