This document presents research on estimating expertise using simple multimodal features. Three approaches to obtaining features from video, audio, digital pen data, and their combinations are discussed: literature-based, common-sense-based, and "why not?"-based features. Logistic regression and classification tree algorithms showed that features like percentage of calculator use, words mentioned, and writing speed discriminated experts from non-experts with over 80% accuracy. Estimating expertise was possible even from a small number of problems. The researchers concluded simple multimodal features can successfully identify expertise.

1. Expertise Estimation based on
Simple Multimodal Features
Xavier Ochoa, Katherine Chiluiza,
Gonzalo Méndez, Gonzalo Luzardo,
Bruno Guamán and James Castells
Escuela Superior Politécnica del Litoral

2. Download this presentation
http://www.slideshare.net/xaoch

3. How to (easily) obtain
multimodal features?
What is already there?

4. Three Approaches
• Literature-based features
• Common-sense-based features
• “Why not?”-based features

5. All approaches proved useful
Proof that we are in an early stage

6. Video: Calculator Use (NTCU)
• Idea:
– Calculator user is the one solving the problem
• Procedure:
– Obtain a picture of the calculator
– Track the position and angle of the image in the
video using SURF + FLANN + Rigid Object
Transformation (OpenCV)
– Determine to which student the calculator is
pointing in each frame

7. was
on
hat
ved
inand
format ions capabilit ies provided by OpenCV . W hile t here
were some frames in which t his mat ching was not possible
due t o object occlusions or changes in t he illuminat ion of
t he calculat or, in general t he described det ect ion t echnique
was robust and provided useful posit ion and direct ion dat a.
Video: Calculator Use (NTCU)
ing
by
ent
was
core
ven
iffion,
ex-
at h
t et
F i gur e 1: D et er m i n at i on of w hi ch st u dent i s u si n g

8. Video: Total Movement (TM)
• Idea:
– Most active student is the leader/expert?
• Procedure:
– Subtract current frame from previous frame
– Codebook algorithm to eliminate noise-movement
– Add the number of remaining pixels

9. image out put by t he Codebook algorit hm. T his magnit ude,
when comput ed for t he ent ire problem solving session, result s from summing up it s individual values obt ained from
each frame t hat compose a problem recording.
Video: Total Movement (TM)
(a) Original frame
(b) Difference frame
F i gu r e 2: R esul t s of t he C odeb ook al gor i t hm .

10. Video: Distance from center table
(DHT)
• Idea:
– If the head is near the table (over paper) the
student is working on the problem
• Procedure:
– Identify image of heads
– Use TLD algorithm to track heads
– Determine the distance from head to center of
table

11. lem
d to
cupar-
sped as
preodeant
mall
ndihere
ned
and t hen, t he average of t hese dist ances is obt ained by problem (see Figure 3). A ddit ionally, t he variance of t he average
dist ance head t o t able (SD-DHT ), was calculat ed t o det ermine if a part icipant remains most ly st at ic or varies his or
her dist ance t o t he t able.
Video: Distance from center table
(DHT)
deary
ude,
reom
F i gur e 3: C al cu l at i on of t h e di st an ce of t h e st u d ent ’ s

12. Audio: Processing

13. Audio: Processing

14. Audio: Features
•
•
•
•
•
Number of Interventions (NOI)
Total Speech Duration (TSD)
Times Numbers were Mentioned (TNM)
Times Math Terms were Mentioned (TMTM)
Times Commands were Pronounced (TCP)

15. Digital Pen: Basic Features

16. Digital Pen: Basic Features
•
•
•
•
•
Total Number of Strokes (TNS)
Average Number of Points (ANP)
Average Stroke Path Length (ASPL)
Average Stroke Displacement (ASD)
Average Stroke Pressure (ASP)

17. Digital Pen: Shape Recognition
Stronium – Sketch Recognition Libraries

18. Digital Pen: Shape Recognition
•
•
•
•
•
•
Number of Lines (NOL)
Number of Rectangles (NOR)
Number of Circles (NOC)
Number of Ellipses (NOE)
Number of Arrows (NOA)
Number of Figures (NOF)

19. Features Variation
• When the features were evaluated inside a
group two variations were usually obtained:
– Percentage of the total (e.g. Calculator Use)
– Highest / Lowest (e.g. Faster Writer, Lowest Time)

20. Next step: Find predictive
features

21. Prediction at two levels
Problem and Group

22. Analysis at Problem level
Solving Problem Correctly
• All available problems were used
• Logistic Regression to model Student Solving
Problem Correctly
• Resulting model was significantly reliable
• 60,9% of the problem solving student was
identified
• 71,8% of incorrectly solved problems were
identified

23. Variable Value for Expert s Discriminat ion Power
P CU
> 0.41
4.44
C oeffi ci entNof t h e L ogi st i c M od el P r edi ct i ng Od ds for a St u dent Sol v i n g C or r ect l y
Ps M
> 34.74
3.19
ASP Variable < 38.05
Predict or L
B 2.86
W ald df
p value exp(B )
Number of Int ervent ions (N OI )
0.0682.86
24.019 1
0.001
0.934
N OR
< 0.13
T imes numbers were ment ioned (T N M )
0.175 23.816 1
0.001
1.192
T M T M > pronounced (T CP ) 0.3292.65
T imes commands were6.25
4.956
1
0.026
1.390
Analysis at problem level
Proport ion of Calculat or Usage (P CU)
Fast est St udent in t he Group (F W )
Constant
1.287
0.924
1.654
25.622
18.889
53.462
1
1
1
0.001
0.001
0.001
a
3.622
2.519
0.191
To calculat e t he probability of correct ly solving a problem
N um b er of P oi nt s ( A N P ) : Represent s, in
s. Classificat
provided
by of point s t hat composehe following sub-setR st at ist ical ion Trees,[21] for M ac, wer
a st udent (P ) t each st roke
formula should be used: by rpa
number
in t he
software
second part of t he analysis.
St r oke T i m e L engt − 11. 7−L0.1N O I + s for N M + 0.3T C P + 1. 3P C U + 0. 9F W
h A ST
A ccount
ehe (st udent) :needed, in 0.2T
f milliseconds t hat t
avP st=
plet e each roke. + e− 11.7− 0. 1N O I + 0.2T N M + 0. 3T C P + 1.3P C U + 0.9F W
1
St r oke P at h L engt h ( A SP L ) : Represent s
umber of pixels t hat t he t raject ory of st rokes
St r oke D i sp l acem ent ( A SD ) : A ccount s for
splacement defined by t he st art ing and ending
4.2 Expert prediction
(1)
4.1 Odds of a student solving c
problem
A Logist ic regression was run wit h St udent

24. Analysis at Group Level
Expertise Estimation
• Data from group 2 was removed because
there was no expert
• Features were feed to a Classification Tree
algorithm
• Several variables had a high discrimination
power between expert and non-experts
• Best discrimination (6.53) result in 80% expert
prediction and 90% non-expert prediction

25. ASD
AST L
ASP
MD
FW
Analysis
MP
Highest value
Lowest value
Group Level
Highest value
at
Expertise Estimation
T abl e 4: C l assi fi cat i on t r ee sp l i t s w i t h nor m al i zed
and non-n or m al i zed feat ur es
Variable
FW
LP
P CU
MN
PN M
Value for Expert s
> 0.5
> 34.74
> 38.05
> 0.13
> 6.25
Discriminat ion Power
6.53
6.53
4.44
4.03
3.19
classificat ion is maint ained and plat eau at t he final value
around t he 12t h problem.

26. Expert Estimation over Problems
Plateau reached after
just 4 problems

27. Main conclusion:
Simple features could identify
expertise

28. Conclusions
• Three strong features:
– Faster Writer (Digital Pen)
– Percentage of Calculator Use (Video)
– Times Numbers were Mentioned (Audio)
• Each mode provide discrimination power to
establish expertise
• These features maintain its discriminant
power at lower levels (solving a problem
correctly)

29. Gracias / Thank you
Questions?
Xavier Ochoa
xavier@cti.espol.edu.ec
http://ariadne.cti.espol.edu.ec/xavier
Twitter: @xaoch