Presented at the 7th International Conference of Learning Analytics and Knowledge (LAK '17) in Vancouver, Canada. Friday March 17, 2017.

1. FOLLOW THE SUCCESSFUL CROWD
RAISING MOOC COMPLETION RATES THROUGH SOCIAL COMPARISON AT SCALE
DAN DAVIS
IOANA JIVET
RENÉ KIZILCEC LAMBDA-LAB
GUANLIANG CHEN
CLAUDIA HAUFF
GEERT-JAN HOUBEN

2. MOOC LEARNING

3. FEEDBACK

4. SOCIAL CUES
SELF-REGULATED LEARNING
METACOGNITION
MOTIVATION

5. OUTLINE 2. EXPERIMENTS
3. CULTURE
4. CONCLUSION
1. DESIGN

6. 1. DESIGN
FEEDBACK · SOCIAL COMPARISON

7. SOCIAL COMPARISON THEORY
Establish social and personal worth by comparing
ourselves to others (Festinger, 1954)
Comparison with slightly-better-performing peers
has a beneficial effect on students’ grades (Huguet, 2001)

8. ROLE MODELS
learners who earned a passing grade in the
previous edition of the MOOC

9. FEEDBACK SYSTEM DESIGN

10. DESIGN GOALS
provide feedback on behavior extracted from edX’s
log traces
afford learners the ability to intuitively understand
and explore the information presented (Kay)
show learners specific ways to take action and
change their behavior
Traceable
Scrutable
Actionable

11. Learning tracker
Average graduate You kAverage graduate next week
Time on the
platform
Time watching
videos
Fraction of time
spent watching
videos while on
the platform
Videos watched
Quiz answers
submitted
Timeliness of
quiz answer
submission

12. FEEDBACK SYSTEM DESIGN
1. Embodied representation of multiple metrics
2. Numerous indicators displayed in a small space
3. Easily comparable

13. ITERATIVE DESIGN
1. Interactivity
2. Planning ahead
3. Framing
Framing
Feedback Dimension
Planning Ahead
Interactive

14. 2. EXPERIMENTS
RESEARCH QUESTIONS · RESULTS

15. RQ1
Does social comparison-based learner
feedback lead to increased achievement
and increased self-regulatory behavior
for MOOC learners?

16. Water Treatment
Urban Sewage
Responsible Innovation
Pre-Calculus
11k
8k
2k
12k
3.5%
3.8%
3.6%
1.7%
42
272
79
327
MOOC ENROLLED PASS QUIZ
QUESTIONSRATE

17. 5,460
5,483
—
Control
Treatment 1
Treatment 2
4,038
4,099
—
1,184
1,168
—
4,142
4,087
4,065
WaterX SewageX BusinessX CalcX
ALL ENROLLED

18. WaterX SewageX BusinessX CalcX
1,268
1,251
—
Control
Treatment 1
Treatment 2
771
746
—
164
160
—
1,150
1,147
1,118
ACTIVE: >5 MIN IN COURSE

19. RESULTS
COMPLETION · ENGAGEMENT · PRIOR EDU

20. COURSE
COMPLETION

21. WaterX
C 1,268 160
MOOC COND. N # PASS PASS RATE
T 1,251 188
12.6%
15.0%
UrbanX
C 771 136
T 746 165
17.6%
22.1%
BusinessX
C 164 46
T 160 54
28.0%
33.8%
OVERALL
C 2,203 342
T 2,157 407
15.5%
18.9%

22. WaterX
C 1,268 160
MOOC COND. N # PASS PASS RATE
T 1,251 188
12.6%
15.0%
UrbanX
C 771 136
T 746 165
17.6%
22.1%
BusinessX
C 164 46
T 160 54
28.0%
33.8%
OVERALL
C 2,203 342
T 2,157 407
15.5%
18.9%
**

23. WaterX UrbanX BusinessX Overall
Control
Treatment
10%
20%
30%
COURSE
COMPLETION

24. RQ1
Does social comparison-based learner
feedback lead to increased achievement
and increased self-regulatory behavior for
MOOC learners?
Providing learners a comparison of their own behavior to that
of previously successful peers will increase learner
achievement.
H1.1

25. RQ1
Does social comparison-based learner
feedback lead to increased achievement
and increased self-regulatory behavior for
MOOC learners?
Providing learners a comparison of their own behavior to that
of previously successful peers will increase learner
achievement and engagement
H1.1
Learners will change the aspects of their behavior that the
Feedback System makes them aware of.
H1.2
X

26. RQ2
Which learners benefit most from the
Feedback System?

27. PRIOR EDUCATION

28. WaterX
High 13.2
MOOC
PRIOR
Low 11.8
15.7
11.5
UrbanX
17.4
16.4
21.3
14.0
BusinessX
23.7
21.4
29.1
22.8
OVERALL
16.3
14.4
19.5
13.6
High
Low
High
Low
High
Low
EDU
0.15
0.16
0.04
0.66
0.04
0.78
<0.01
0.36
CTRL TREAT
p
GRADE GRADE

29. WaterX
High 13.2
MOOC
PRIOR CTRL TREAT
11.8
15.7
11.5
UrbanX
17.4
16.4
21.3
14.0
BusinessX
23.7
21.4
29.1
22.8
OVERALL
16.3
14.4
19.5
13.6
High
High
High
EDU
p
0.15
0.16
0.04
0.66
0.04
0.78
<0.01
0.36
<
<
<
<
GRADE GRADE
Low
Low
Low
Low

30. RQ2
Which learners benefit most from the
Feedback System?

31. RQ2
Which learners benefit most from the
Feedback System?
A:
HIGHLY EDUCATED LEARNERS

32. 3. CULTURE*
FRAMING · RESULTS
* PRE-REGISTERED osf.io/ys6au/

33. RQ3
Does feedback framed in line with a learner’s
cultural context make it more effective?

34. Framing
Feedback Dimension
Planning Ahead
Interactive

35. ON TRACK

36. ON TRACK

37. BEHIND

38. Framing
Feedback Dimension
Planning Ahead
Interactive

39. CULTURE
HOFSTEDE INDIVIDUALISM

40. INDIVIDUALISM
INDIVIDUAL AS AN INDEPENDENT
ACTOR WITH LOOSE SOCIAL RELATIONS
COLLECTIVISM

41. CLOSELY KNIT SOCIAL RELATIONS AND
GREAT CARE FOR THE WELL-BEING OF ALL
INDIVIDUALISMCOLLECTIVISM

42. CULTURE
HOFSTEDE INDIVIDUALISM
GELFAND TIGHTNESS

43. HAVE MANY STRONGLY ENFORCED RULES
AND LITTLE TOLERANCE FOR DEVIANCE
TIGHTLOOSE

44. HAVE FEW STRONGLY ENFORCED RULES
AND GREATER TOLERANCE FOR DEVIANCE
TIGHTLOOSE

45. RQ3
Does feedback framed in line with a
learner’s cultural context make it more
effective?

46. RQ3
Does feedback framed in line with a
learner’s cultural context make it more
effective?
Feedback in line with a learner’s cultural individualism index
will increase its impact on the learner
H3.1
X

47. RQ3
Does feedback framed in line with a
learner’s cultural context make it more
effective?
Feedback in line with a learner’s cultural individualism index
will increase its impact on the learner
H3.1
Feedback in alignment with a learner’s cultural tightness will
increase its impact on the learner
H3.2
X

48. WaterX
MOOC PASS RATE
13.8%
UrbanX 19.8%
BusinessX 31%
CalcX 5%

49. RAISE ACTIVITY THRESHOLD
5 MINUTES -> 1 HOUR
From N = 3,415 to N = 658

50. WaterX
MOOC PASS RATE
13.8%
UrbanX 19.8%
BusinessX 31%
CalcX 16%

51. INDIVIDUALIST FRAMING
+ Activity Threshold
Increased completion rates across cultural contexts
Control: 12.8%
Individualist Framing: 19.9%
55% Increase (t = 2.02, p = 0.04, N = 441)

52. INDIVIDUALIST FRAMING
+ Activity Threshold
Increased completion rates across cultural contexts
Control: 12.8%
Individualist Framing: 19.9%
55% Increase (t = 2.02, p = 0.04, N = 441)
Increased completion rates especially in tight cultures
Control: 12.1%
Individualist Framing: 36.3%
300% Increase (t = 2.07, p = 0.04, N = 55)

53. 5. CONCLUSION
WHAT’S NEXT

54. 1. REAL-TIME
2. FRAMING
3. PERSONAS

55. 1. REAL-TIME
2. FRAMING
3. PERSONAS

56. 1. REAL-TIME
2. FRAMING
3. PERSONAS

57. THANK YOU
BIT.LY/WIS-LEARNING-ANALYTICS
LAMBDA-LAB
DAN DAVIS
IOANA JIVET
RENÉ KIZILCEC
GUANLIANG CHEN
CLAUDIA HAUFF
GEERT-JAN HOUBEN

58. LOW CULTURAL TIGHTNESS
PREDICTS COMPLETION
p < 0.02

59. Control 1,150 45
CONDITION N # PASS PASS RATE
3.9%
Indiv. - Promotion 1,147 62 5.4%
Collect. - Prevent. 1,118 51 4.5%
OVERALL EFFECT