In this paper, we study the effect of using domain knowledge structure on predicting student performance with parameterized Java programming exercises. Domain knowledge structure defines connections between elementary knowledge items. While known to be beneficial in general, it has not been used to predict performance.We compare five different approaches for this purpose: Bayesian Knowledge Tracing (BKT), Performance Factor Analysis (PFA), and three dimensional Bayesian Probabilistic Tensor Factorization (3D-BPTF), that are not able to take into account knowledge structure; and four-dimensional Bayesian Probabilistic Tensor Factorization (4D-BPTF) and Feature-Aware Student Knowledge Tracing (FAST), that can take into account knowledge structure. We approach the problem using both topic-level and question-level Knowledge Components (KCs) and test the methods on a dataset of parameterized questions. Our work is the first in the field that models students’ behavior in a four dimensional tensor. Our experiments show that, when having only the knowledge-item-level information, all of the models work similarly in predicting student performance, but adding the topic-level information that integrates knowledge items changes the performance of these models in different directions.

2. 2Shaghayegh Sahebi (Sherry)
• Programming questions
– Java problems
• Can be designed with parameterized exercises
– One question with multiple parameter sets
– Can be repeated multiple times by one student
• Authoring tool for Java questions
– Create and modify questions
– Indexing service to define concepts inside the
question

3. 3Shaghayegh Sahebi (Sherry)
Each question is generated from a template, and
students can try multiple attempts.
Students give values for specified variable, or give
the output of the code.
A question for practicing skill
nested loops

4. 4Shaghayegh Sahebi (Sherry)

5. 5Shaghayegh Sahebi (Sherry)

6. 6Shaghayegh Sahebi (Sherry)
Aggregate
(MasteryGrids
Services)
Aggregate
UM2
Other
(content
specific)
PAWS
UM Services
Content apps
Server side apps
(Apache Tomcat)
Databases (MySQL)
Client interface
MasteryGrids Interface
Content popout iframe
QuizJet
WebEx
SQLKnot
(a)
GUI calls MG
services
direct link
services calls
(b)
Aggregate uses
UM services
cbum login
Overall view of the architecture
Mastery Grids

8. 8Shaghayegh Sahebi (Sherry)
• Students can choose what question to solve
– Using social navigation support
• Adding guidance to the question
– Use the whole set of data to develop personalized
guidance
– Predict how likely the problem will be solved
– Avoid too simple and too complex problems

9. 9Shaghayegh Sahebi (Sherry)
• Predicting the student’s capability to perform
an educational task
• Assumption: the student can learn by
practicing over time by repeating
– Time sequence modeling effect on PSP
• Will present at the Problem Solving & Strategies session
on Monday
– Knowledge structure effect on PSP
• Today’s talk

10. 10Shaghayegh Sahebi (Sherry)
• Questions related to topics, concepts, or skills
– many dimensions in the data
– Structure in the data (knowledge structure)
• Traditional methods: mostly consider student’s
past performance
– Only consider correct/incorrect attempts of students
(ignoring the multidimensionality of the data)
– Bayesian Knowledge Tracing (BKT)
– Performance Factor Analysis (PFA)

11. 11Shaghayegh Sahebi (Sherry)
• Considering knowledge structure in PSP
– Feature-Aware Knowledge Tracing (FAST) [González-Brenes
et al., ‘13]
– Our suggestion: Tensor Factorization Methods

12. 12Shaghayegh Sahebi (Sherry)
• Study the effect of knowledge structure
modeling in PSP for parameterized questions
• Compare five approaches:
– Bayesian Knowledge Tracing (BKT)
– Performance Factor Analysis (PFA)
– Feature-Aware Knowledge Tracing (FAST)
– 3D and 4D Tensor Factorization (3D-BPTF, 4D-
BPTF)

13. 13Shaghayegh Sahebi (Sherry)
• Markov Model with two states
• No knowledge structure: Only one type of
knowledge component
• Guess, slip, learning, and initial knowledge
parameters
Knowledge
Structure

14. 14Shaghayegh Sahebi (Sherry)
• Regression model
• No knowledge structure
m(i, j Î KCs
,k Î items,s,f) = bk
+ (gj
si,j
+ rj
fi,j
)
jÎKCs
å
Knowledge
Structure

15. 15Shaghayegh Sahebi (Sherry)
• Extension of BKT
• Can include knowledge structure as regression
variables
fq,l,t
features L
yq,t
kq,t
timesteps T
# of skills Q
fq,e,t
features E
Knowledge
Structure

16. 16Shaghayegh Sahebi (Sherry)
• Tensors: n-dimensional arrays
• Used in collaborative-filtering recommender
systems
– Estimates each tensor as the sum of multiple rank-1
tensors
• Can be extended to as many dimensions
– Can include the data structure
– Each dimension of the data ≈ one dimension of the
tensor

17. 17Shaghayegh Sahebi (Sherry)
• Used successfully in PSP for traditional PSP
• No knowledge structure
• We use Bayesian Probabilistic Tensor
Factorization Model (3D-BPTF) [Xiong et al., 2010]
Students
Questions/ topics
…
Knowledge
Structure

18. 18Shaghayegh Sahebi (Sherry)
• Used for the first time in PSP
• Adds knowledge structure modeling
• Can be extended to more dimensions if
needed
Students
Questions Students Questions
Students
Questions
Attempt 1 Attempt 2 Attempt 3
Knowledge
Structure

19. 19Shaghayegh Sahebi (Sherry)
• From QuizJET system
• Java programming questions
• Six semesters
• 166 students
• 103 questions
• 69.04% majority class (successes)

20. 20Shaghayegh Sahebi (Sherry)
• Here, each topic can have multiple questions and
each question is related to one topic
– Two dimensions: questions and topics
• Study 1: traditional approach
– Question as knowledge unit
• Study 2: considering knowledge structure
– Topic added as knowledge unit
• 5-Fold cross validation
– 80% of students in train data, rest in test data
– User-stratified

21. 21Shaghayegh Sahebi (Sherry)
Study 1: comparing traditional approaches
(no knowledge structure)

22. 22Shaghayegh Sahebi (Sherry)
71
71.5
72
72.5
73
73.5
74
74.5
75
75.5
76
FAST with no
additional
parameters
BKT PFA 3D-BPTF
Accuracy of Traditional Models
Study 1: comparing traditional approaches
(no knowledge structure)

23. 23Shaghayegh Sahebi (Sherry)
0
200
400
600
800
1000
1200
1400
FAST with no
additional
parameters
BKT PFA 3D-BPTF
False Positive
Study 1: comparing traditional approaches
(no knowledge structure)

24. 24Shaghayegh Sahebi (Sherry)
0
100
200
300
400
500
600
700
800
FAST with no
additional
parameters
BKT PFA 3D-BPTF
False Negative
Study 1: comparing traditional approaches
(no knowledge structure)

25. 25Shaghayegh Sahebi (Sherry)
0
10
20
30
40
50
60
70
80
90
FAST with
no
additional
parameters
BKT PFA 3D-BPTF
Majority Precision
Minority Precision
BKT predicts failure better
Study 1: comparing traditional approaches
(no knowledge structure)

26. 26Shaghayegh Sahebi (Sherry)
Study 2: comparing approaches
including knowledge structure

27. 27Shaghayegh Sahebi (Sherry)
64
66
68
70
72
74
76
78
FAST 4D-BPTF BKT PFA 3D-BPTF
Accuracy of Approaches with Additional Knowledge
Structure
Study 2: comparing approaches
including knowledge structure

28. 28Shaghayegh Sahebi (Sherry)
0
200
400
600
800
1000
1200
1400
1600
1800
FAST 4D-BPTF BKT PFA 3D-BPTF
False Positive
Study 2: comparing approaches
including knowledge structure

29. 29Shaghayegh Sahebi (Sherry)
0
100
200
300
400
500
600
FAST 4D-BPTF BKT PFA 3D-BPTF
False Negative
Study 2: comparing approaches
including knowledge structure

30. 30Shaghayegh Sahebi (Sherry)
0
10
20
30
40
50
60
70
80
90
100
Majority Precision
Minority Precision
Study 2: comparing approaches
including knowledge structure
3D-BPTF predicts failure better

31. 31Shaghayegh Sahebi (Sherry)
66
67
68
69
70
71
72
73
74
75
76
FAST 4D-BPTF BKT PFA 3D-BPTF
Question as KC (No Structure)
Topic as KC (with Question
Structure)
3D-BPTF
Accuracy

32. 32Shaghayegh Sahebi (Sherry)
66
67
68
69
70
71
72
73
74
75
76
FAST 4D-BPTF BKT PFA 3D-BPTF
Question as KC (No Structure)
Topic as KC (with Question
Structure)
Accuracy

33. 33Shaghayegh Sahebi (Sherry)
• Accuracy in predicting students performance
depends on the input of the method
– When ignoring the topic of questions as KCs, all
models perform similarly
– When including topic information, in addition to
the question information, the methods that can
leverage it perform better

34. 34Shaghayegh Sahebi (Sherry)
• Adding the extra topic data in the methods
that cannot model this information decreases
the method’s accuracy
• Knowledge structure can add to the accuracy
of PSP in parameterized questions
• Tensor factorization methods are as good, or
better than the pioneers PSP methods

35. 35Shaghayegh Sahebi (Sherry)
• Include additional structure into tensor
factorization using more dimensions
• Use of other collaborative filtering methods
for PSP
• Test on other programming courses

36. 36Shaghayegh Sahebi (Sherry)
Thank You!

37. 37Shaghayegh Sahebi (Sherry)
• EM algorithm for BKT and set the initial
parameters as follows: p(L0) = 0:5 , p(G) = 0:2 ,
p(S) = 0:1 , p(T) = 0:3 . For running PFA, we use
• the implementation of logistic regression in
WEKA [3]. For BPTF and BPMF,
• we utilize the Matlab code prepared by Xiong et.
al. We experimented with different latent space
dimensions for BPTF and BPMF (5, 10, 20 and 30)
and chose the best one, which has the latent
space dimension of 10

38. 38Shaghayegh Sahebi (Sherry)
• From collaborative filtering
• Used successfully in PSP for static questions
• No attempt sequence modeling
• We use Bayesian Probabilistic Matrix
Factorization (BPMF)
1 0 0 0
1 1 0 1
0 0 1 1
0 0 0 1
Students
Questions/ topics
0.9 0
1.5 0.4
0 1.4
0 0.9
Students
KCs
0.8 0.5 0 0.3
0 0 0.5 0.8
KCs
Questions/ topics

39. 39Shaghayegh Sahebi (Sherry)

40. 40Shaghayegh Sahebi (Sherry)
Accuracy of all models is very close to each other

41. 41Shaghayegh Sahebi (Sherry)
BKT over estimates the student’s performance

42. 42Shaghayegh Sahebi (Sherry)
FAST tends to predict more failures for the students

43. 43Shaghayegh Sahebi (Sherry)
If FAST predicts a success for a student and if BKT predicts a failure
for students, their prediction is more likely to be true compared to
the other methods.

44. 44Shaghayegh Sahebi (Sherry)
FAST and 4D-BPTF perform significantly better than all other
approaches

45. 45Shaghayegh Sahebi (Sherry)
BKT and PFA perform similarly to their results in Study 1 and 3D-BPTF
on topics is slightly weaker than 3D-BPTF on questions in terms of
accuracy.

46. 46Shaghayegh Sahebi (Sherry)
• Visual, interactive, adaptive E-learning
platform
– Multi-facet social comparison
– Multi-type learning materials support
– Social navigation
– Personalized guidance
• Integration with other systems with little set
up and modification