This study examined whether student stereotypes could be used to improve predictions of problem-solving performance in MOOCs. The researchers tested simple stereotypes based on demographics and performance, but found they did not yield accurate models of different student groups' learning. More advanced stereotypes based on patterns of problem-solving behavior also did not distinguish groups with significantly different learning models. The findings suggest stereotypes may not effectively represent the finer-grained differences in how students approach learning. Overall, the study did not find evidence that stereotype models can improve predictions of problem-solving over alternative models of individual student learning.

1. Stereotype Modeling for Problem-
Solving Performance Predictions in
MOOCs and Traditional Courses
Roya Hosseini, Peter Brusilovsky,
Michael Yudelson, Arto Hellas
1

2. Background
2
Stereotype Models
Elaine Rich (1983)
Feature-Based Models
Corbett & Andreson (1995)
InterestGoal
Knowledge
High AccuracySimple
Leverage large
volumes of data

3. MOOC: A Comeback of Stereotypes?
3
Learning Analytics:
Define demographic
stereotypes, use data
to find differences
Success in MOOC drop-
out/failure prediction
Educational DM:
Mine stereotypes
from data

4. A Simple way to Adaptive MOOC?
4
Treat learners
differently based
on demography
Recognize stereotype
on the early stages and
treat differently

5. Research Focus
• Past work: Coarse-grain prediction and
adaptation (dropped-out – completed)
• Our goal: Finer-grain adaptation – content
sequencing, navigation support
• Use stereotypes to guide
different groups of
students along different
pathways
5

6. How We Can Do It?
• A data-driven approach to personal guidance:
– Use data to learn a student model that could be used
to predict student performance on a problem
– Choose a problem (or a topic) based on predicted
performance
• How to do it better with stereotypes?
– Use large volume of course problem-solving data
– Determine learning-related stereotypes
– Learn different models for different stereotypes
– Use stereotype-specific models for advanced data-
driven personalization
6

7. Research Context: A Programming MOOC
7
Test My Code Plugin
NetBeans IDE Compiled Correctness
✗
✓
✓
✓
0%
20%
80%
100%
Snapshots
• Problem-solving (coding)
o real context
o rich data
o automated feature analysis of solutions

8. Simple Grouping
8
Demographic-based
Gender
Education
Performance-based
Low
Medium
High
#Transactions
Problems solved
% Correct

9. Assessment Approach
9
Between-Group
Within-Group
Performance Prediction
Accuracyo Do students in different groups
learn differently (student
performance models differ)?
Yes (Desired grouping)
No (No learning differences)


o Learning and comparing student models across
groups

10. 10
Group Split Score
1.0: Ideal
0.67: Expected
0.33: Sub-optimal
0.00: None
Scoring Models

11. Results for Simple Grouping
11
0
0.2
0.4
0.6
0.8
Gender Edu. #Trans. P.Solv. %Corr.
Suboptimal
Expected
Out of the five simple
grouping approaches,
only two had a non-zero
score.

12. Advanced Data-Driven Grouping
12
a
Processing
Snapshots
Mining Problem-
Solving Sequences
Grouping By
Behavior Profiles
1 2 3
A—L: long steps
(individualized)
> < =
>
=
<
0
b c
d e f
g h
j k l
i
Concept
Correctness
SPAM
_aaA_
_jjJgK_
Frequent patterns
(ac, _aa, jj_, ad, …)
Normalized vectors
of frequent patterns
C1 C2 C3 C4 C5
Hierarchical Spectral
Guerra, J., Sahebi, S., Lin, Y.-R., and Brusilovsky, P. (2014) The Problem Solving Genome: Analyzing Sequential Patterns of Student Work with Parameterized
Exercises. 7th International Conference on Educational Data Mining (EDM 2014), pp. 153-160.

13. 13
Tinkerers Movers
VS.
0.000
0.005
0.010
0.015
0.020
0.025
0.030
0.035
0.040
RatioofPatternOccurrence
Pattern
Cluster 1 (Tinkerers) Cluster 2 (Movers)
Tinkerers were
less efficient and
had lower
grades
Patterns tended
to distinguish
low/high
performers in
both groups
Both groups
included a
mixture of
strong and weak
students
Movers have more
steps of adding
code concepts and
passing more tests
Tinkerers have more
steps of changing
code concepts without
passing more tests

14. Results for Advanced Grouping
14
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
C1 C2 C3 C4 C5
None of the advanced
grouping approaches
(C1 — C5) were scored
as ideal, as we have not
found at least two groups
that were voted as
sufficiently different by
the group models of
student learning.
Expected

15. Main Take-Away Points
15
o No discernable differences in
cross-prediction accuracies in
simple and advanced student
groupings
o Stereotype models might not be
good as alternative models to
represent finer-grain student
learning

16. Main Take-Away Points
16
o Finding a useful learning-focused
stereotype, like good students or
slow students is NOT trivial.
o There might be students who
approach learning differently, but
the distinction between these
approaches are orthogonal to the
conventional dimensions that we
apply to quantify learning.

17. Contributions
• (Past) Resource usage
• (This Work) Micro patterns in problem-solving
17
Behavior Analysis in MOOC
Code Submission Analysis
• Problem-solving patterns (behavior)
• Impact of patterns on student learning/performance

18. Relevant Readings
• Concept-based analysis of MOOC program submissions
– Yudelson, M., Hosseini, R., Vihavainen, A., and Brusilovsky, P. (2014)
Investigating Automated Student Modeling in a Java MOOC. In:
Proceedings of the 7th International Conference on Educational Data
Mining (EDM 2014), London, UK, July 4-7, 2014, pp. 261-264.
– Hosseini, R., Vihavainen, A., and Brusilovsky, P. (2014) Exploring
Problem Solving Paths in a Java Programming Course. In: Proceedings of
Psychology of Programming Interest Group Annual Conference, PPIG
2014, Brighton, UK, June 25-27, 2014, pp. 65-76.
• The Problem-Solving Genome approach
– Guerra, J., Sahebi, S., Lin, Y.-R., and Brusilovsky, P. (2014) The
Problem Solving Genome: Analyzing Sequential Patterns of Student Work
with Parameterized Exercises. In:Proceedings of the 7th International
Conference on Educational Data Mining (EDM 2014), London, UK, July 4-
7, 2014, pp. 153-160.
18

19. 19
* This work used the Extreme Science and Engineering Discovery Environment (XSEDE),
which is supported by NSF award OCI-1053575. Specifically, it used the Bridges system,
which is supported by NSF award ACI-1445606, at the Pittsburgh Supercomputing Center.
Questions?
Stereotype Modeling for Problem-Solving Performance
Predictions in MOOCs and Traditional Courses
R. Hosseini, P. Brusilovsky, M. Yudelson, A. Hellas