From Findability to Awareness: Metadata in Music and Technology Enhanced Learning.
1. FROM FINDABILITY TO
AWARENESS: METADATA IN
MUSIC AND TECHNOLOGY
ENHANCED LEARNING
Vinden en beseffen: Metadata
in muziek en E-learning
Sten Govaerts
Promotor: Prof. dr. ir. Erik Duval
Co-promotor: Dr. Katrien Verbert
1
Wednesday 12 September 12
6. METADATA
• title: Maiden Voyage
• main artist: Herbie Hancock
• artists: Freddie Hubbard, George
Coleman, Ron Carter, Anthony Williams
• record label: Blue Note, Capitol
Records
• instruments: trumpet, tenor
saxophone, piano, bass, drums.
• recording date: March 17, 1965
• studio: Van Gelder Studio, New Jersey
• version: remastered by Rudy Van Gelder
5
Wednesday 12 September 12
25. STEP 1 – DEFINE.
RQ3: How can metadata
standards be selected for specific
functionalities of applications?
9
Wednesday 12 September 12
26. METADATA SCHEMA
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<rdf:Description rdf:about="http://media.example.com/audio/guide.ra">
<dc:creator>Herbie Hancock</dc:creator>
<dc:title>Maiden Voyage</dc:title>
Dublin Core
<dc:contributor>Freddie Hubbard</dc:contributor>
<dc:contributor>George Coleman</dc:contributor>
<dc:contributor>Ron Carter</dc:contributor>
<dc:contributor>Anthony Williams</dc:contributor>
<dc:description>Maiden Voyage is the fifth album led by jazz musician
Herbie Hancock, and was recorded by Rudy Van Gelder on May 17, 1965
for Blue Note Records.</dc:description>
<dc:date>1965-03-17</dc:date>
<dc:publisher>Blue Note</dc:publisher>
<dc:type>Music album</dc:type>
<dc:format>audio/mp3</dc:format>
<dc:rights>copyrighted by Capitol Records, Inc.</dc:rights>
</rdf:Description>
</rdf:RDF>
step 1 - define. 10
Wednesday 12 September 12
27. METADATA SCHEMA
@prefix mo: <http://purl.org/ontology/mo/> .
@prefix : <#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix event: <http://purl.org/NET/c4dm/event.owl#> .
:work
Music Ontology
a mo:MusicalWork ;
dc:title "Herbie Hancock - Maiden Voyage" .
:performance
a mo:Performance ;
dc:title "Maiden Voyage" ;
mo:performance_of :work ;
mo:performer <http://dbpedia.org/page/Herbie_Hancock> ;
dc:date "1965-03-17"^^xsd:date .
event:place <http://dbpedia.org/page/Van_Gelder_Studio> ;
mo:performance_of <http://dbpedia.org/page/
Maiden_Voyage_(Herbie_Hancock_album)> .
mo:recorded_as [
a mo:Signal ;
mo:published_as :track1 ;
dc:title "Original recording at the Van Gelder Studio" ;
].
step 1 - define. 11
Wednesday 12 September 12
36. COUNTRY & CONTINENT
• useful for:
• subgenres
• popularity
• recommendations
• expensive to annotate
step 2 - create. 16
Wednesday 12 September 12
37. COUNTRY & CONTINENT
• useful for:
• subgenres
• popularity
• recommendations
• expensive to annotate
• very little existing research
step 2 - create. 16
Wednesday 12 September 12
38. MASHUP.
google maps
freebase
last.fm google app
engine
twitter last.fm
last.fm yahoo! pipes last on am/fm
website
dapper youtube
step 2 - create. 17
Wednesday 12 September 12
39. Three data sources last.fm 91%
• last.fm freebase 90%
• freebase biography 71%
• biographies combination 77%
0% 25% 50% 75% 100%
Accuracy
last.fm 7%
Evaluation: freebase 26%
ground truth data biography 55%
set of 11275 artists combination 59%
Govaerts, Sten; Duval, Erik. A Web-based approach to
determine the origin of an artist. ISMIR 2009. 0% 15% 30% 45% 60%
Coverage
step 2 - create. 18
Wednesday 12 September 12
40. Three data sources last.fm 91%
Ennio Morricone (born 10 November 1928 in Rome) is an Italian
• last.fm freebase
composer especially noted for his film scores. He has composed the 90%
scores of more than 500 films and TV series. Although only 30 of
• freebase biography 71%
these are for Western films, it is for this work which he is best known.
Morricone’s sparse style of composition for the genre is particularly
• biographies combination 77%
exemplified by the soundtracks of the classic spaghetti westerns The
Good, the Bad and the Ugly (Sergio Leone, 1966) and Once Upon A
Time In The West (Sergio Leone, 1968). 0% 25% 50% 75% 100%
[...] Accuracy
In 2003 he scored the Japanese Taiga drama about Miyamoto
Musashi, Japan’s legendary warrior. In 2005 he was on a world tour
last.fm 7%
with the Roma Sinfonietta Orchestra, touring from cities like New
Evaluation:York, Bilbao, Bratislava and Tokyo. In 2006, Morricone collaborated
with Morrissey by composing the string part for 26%
freebase
his song “Dear God
ground truth data
Please Help Me”, recorded in Rome’s Forum Music Village Studios.
In recognition of his vast body of work and dedication, Morricone 55%
biography
set of 11275 artists
received an honorary Academy Award in 2007; Morricone had
received nominations for Best Original Score in the past, but had 59%
combination
failed Erik. A Web-based approach to
Govaerts, Sten; Duval, to win.
0% 15% 30% 45% 60%
determine the origin of an artist. ISMIR 2009.
Coverage
step 2 - create. 18
Wednesday 12 September 12
41. Three data sources last.fm 91%
• last.fm freebase 90%
• freebase biography 71%
• biographies combination 77%
0% 25% 50% 75% 100%
Accuracy
last.fm 7%
Evaluation: freebase 26%
ground truth data biography 55%
set of 11275 artists combination 59%
Govaerts, Sten; Duval, Erik. A Web-based approach to
determine the origin of an artist. ISMIR 2009. 0% 15% 30% 45% 60%
Coverage
step 2 - create. 18
Wednesday 12 September 12
42. Three data sources last.fm 91%
• last.fm freebase 90%
• freebase biography 71%
• biographies combination 77%
0% 25% 50% 75% 100%
Accuracy
last.fm 7%
Evaluation: freebase 26%
ground truth data biography 55%
set of 11275 artists combination 59%
Govaerts, Sten; Duval, Erik. A Web-based approach to
determine the origin of an artist. ISMIR 2009. 0% 15% 30% 45% 60%
Coverage
step 2 - create. 18
Wednesday 12 September 12
43. Three data sources last.fm 91%
• last.fm freebase 90%
• freebase biography 71%
• biographies combination 77%
0% 25% 50% 75% 100%
Accuracy
last.fm 7%
Evaluation: freebase 26%
ground truth data biography 55%
set of 11275 artists combination 59%
Govaerts, Sten; Duval, Erik. A Web-based approach to
determine the origin of an artist. ISMIR 2009. 0% 15% 30% 45% 60%
Coverage
step 2 - create. 18
Wednesday 12 September 12
44. Three data sources
• last.fm
• freebase
• biographies
last.fm
freebase
biography
combination
71%
77%
91%
90%
}
0% 25% 50% 75% 100%
Accuracy
last.fm 7%
Evaluation: freebase 26%
ground truth data biography 55%
set of 11275 artists combination 59%
Govaerts, Sten; Duval, Erik. A Web-based approach to
determine the origin of an artist. ISMIR 2009. 0% 15% 30% 45% 60%
Coverage
step 2 - create. 18
Wednesday 12 September 12
45. Three data sources
• last.fm
• freebase
• biographies
last.fm
freebase
biography
combination
71%
77%
91%
90%
}
0% 25% 50% 75% 100%
Accuracy
Evaluation:
ground truth data
set of 11275 artists
last.fm
freebase
biography
combination
7%
26%
55%
}
59%
Govaerts, Sten; Duval, Erik. A Web-based approach to
determine the origin of an artist. ISMIR 2009. 0% 15% 30% 45% 60%
Coverage
step 2 - create. 18
Wednesday 12 September 12
53. CLASSIFICATION WITH
SEARCH ENGINES
using co-occurrence
M. Schedl, T. Pohle, P. Knees, G. Widmer, “Assigning and Visualizing
Music Genres by Web-based Co-occurrence Analysis”,
Proceedings of the 7th International Conference on Music
Information Retrieval, 2006, pp. 260-265.
step 2 - create. 22
Wednesday 12 September 12
54. CLASSIFICATION WITH
SEARCH ENGINES
using co-occurrence
M. Schedl, T. Pohle, P. Knees, G. Widmer, “Assigning and Visualizing
Music Genres by Web-based Co-occurrence Analysis”,
Proceedings of the 7th International Conference on Music
Information Retrieval, 2006, pp. 260-265.
step 2 - create. 22
Wednesday 12 September 12
58. RESULTS
• 1st results were much worse
• what happened?
• re-run the original experiment
• evaluate on the same data set: 1995 artists and 9 genres.
• different search engines: Google,Yahoo! and Microsoft.
• over time: 8 times over a period of 36 days.
step 2 - create. 24
Wednesday 12 September 12
62. WHAT TO USE?
• use Google when it’s stable else rely on Yahoo!
• when is it stable? test with a small set
• some artists get classified incorrectly on bad days
• compare the accuracy achieved with the test set to the
average.
step 2 - create. 27
Wednesday 12 September 12
63. WHAT TO USE?
• use Google when it’s stable else rely on Yahoo!
• when is it stable? test with a small set
• some artists get classified incorrectly on bad days
• compare the accuracy achieved with the test set to the
average.
Govaerts, Sten; Corthaut, Nik; Duval, Erik. Using search engine for
classification: does it still work? adMIRe WS, ISM2009.
step 2 - create. 27
Wednesday 12 September 12
64. STEP 4 – UTILISE.
28
Wednesday 12 September 12
65. STEP 4 – UTILISE.
28
Wednesday 12 September 12
66. STEP 4 – UTILISE.
Ch 5 & 6
RQ1: How can metadata be
leveraged to provide efficient and
effective access to content?
28
Wednesday 12 September 12
67. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
68. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
69. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
70. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
71. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
72. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
73. FEDERATED SEARCH.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 4 - utilise. 29
Wednesday 12 September 12
74. CONTEXT
• Personal Learning Environment:
• customizable
• re-use, creation & mashup of tools, resources
• enable users to access content
• in different contexts
step 4 - utilise. 30
Wednesday 12 September 12
75. CONTEXT
• Personal Learning Environment:
• customizable
• re-use, creation & mashup of tools, resources
• enable users to access content
• in different contexts
step 4 - utilise. 30
Wednesday 12 September 12
76. CONTEXT
• Personal Learning Environment:
• customizable
• re-use, creation & mashup of tools, resources
• enable users to access content
• in different contexts
step 4 - utilise. 30
Wednesday 12 September 12
77. WHAT’S A WIDGET ?!?
step 4 - utilise. 31
Wednesday 12 September 12
78. WHAT’S A WIDGET ?!?
step 4 - utilise. 31
Wednesday 12 September 12
80. STEP 5 & 6 – CAPTURE &
PERSONALISE.
Ch 5 & 6
RQ1: How can metadata be
leveraged to provide efficient and
effective access to content?
32
Wednesday 12 September 12
81. ATTENTION METADATA
Attention metadata does not describe
the content itself, but what people do
with the content.
step 5 & 6 - capture/personalise. 33
Wednesday 12 September 12
82. ATTENTION METADATA
Attention metadata does not describe
the content itself, but what people do
with the content.
copyright by Warner Bros. Pictures
step 5 & 6 - capture/personalise. 33
Wednesday 12 September 12
83. ATTENTION METADATA
Attention metadata does not describe
the content itself, but what people do
with the content.
copyright by Warner Bros. Pictures
step 5 & 6 - capture/personalise. 33
Wednesday 12 September 12
86. SOCIAL RECOMMENDATIONS.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 5 & 6 - capture/personalise. 35
Wednesday 12 September 12
87. SOCIAL RECOMMENDATIONS.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 5 & 6 - capture/personalise. 35
Wednesday 12 September 12
88. SOCIAL RECOMMENDATIONS.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 5 & 6 - capture/personalise. 35
Wednesday 12 September 12
89. SOCIAL RECOMMENDATIONS.
Govaerts, Sten; El Helou, Sandy; Duval, Erik; Gillet, Denis. A federated
search and social recommendation widget. SRS WS, CSCW 2011.
Govaerts, Sten; Verbert, Katrien; et al. Towards Responsive Open Learning
Environments: the ROLE Interoperability framework. EC-TEL 2011.
step 5 & 6 - capture/personalise. 35
Wednesday 12 September 12
90. PAGE RANK
HYPERLINK
WEB
SITE
Rank of node i:
A node is important if and only if many other
important nodes point to it
step 5 & 6 - capture/personalise. 36
Wednesday 12 September 12
91. OUR CASE
hare
d R1
d /s
save
saved/shared
d
en tion
Erik R2
fr i e c dis
n like
c on d
lik
ed
R3
d n
Sandy
ien ctio
fr e
c on
n lik
ed
R4
Sten R5
step 5 & 6 - capture/personalise. 37
Wednesday 12 September 12
92. THREE CASE STUDIES
1 2 3
step 5 & 6 - capture/personalise. 38
Wednesday 12 September 12
93. KU LEUVEN & EPFL
• 15 PhD students.
step 5 & 6 - capture/personalise. 39
Wednesday 12 September 12
95. KU LEUVEN & EPFL
• 15 PhD students.
• usability, satisfaction,
usefulness
• usefulness of recommendations: 11/14 pro.
step 5 & 6 - capture/personalise. 39
Wednesday 12 September 12
96. KU LEUVEN & EPFL
• 15 PhD students.
• usability, satisfaction,
usefulness
• usefulness of recommendations: 11/14 pro.
• user satisfaction: System Usability Scale (SUS) & MS
Desirability Toolkit
step 5 & 6 - capture/personalise. 39
Wednesday 12 September 12
97. KU LEUVEN & EPFL
• 15 PhD students.
• usability, satisfaction,
usefulness
• usefulness of recommendations: 11/14 pro.
• user satisfaction: System Usability Scale (SUS) & MS
Desirability Toolkit
• SUS score: 66,25%
step 5 & 6 - capture/personalise. 39
Wednesday 12 September 12
98. KU LEUVEN & EPFL
• 15 PhD students.
• usability, satisfaction,
usefulness
• usefulness of recommendations: 11/14 pro.
• user satisfaction: System Usability Scale (SUS) & MS
Desirability Toolkit
KU Leuven: high (75%)
• SUS score: 66,25%
EPFL + one KU Leuven: low (50%)
step 5 & 6 - capture/personalise. 39
Wednesday 12 September 12
99. WHY THE DIFFERENT SUS?
• 1st phase by 2 interviewers
step 5 & 6 - capture/personalise. 40
Wednesday 12 September 12
100. WHY THE DIFFERENT SUS?
• 1st phase by 2 interviewers
step 5 & 6 - capture/personalise. 40
Wednesday 12 September 12
101. WHY THE DIFFERENT SUS?
• 1st phase by 2 interviewers
step 5 & 6 - capture/personalise. 40
Wednesday 12 September 12
102. WHY THE DIFFERENT SUS?
• issues:
• distraction from unrelated widgets
• layout too dense
• height of widgets too small
• not evaluating the widget but the
whole experience...
step 5 & 6 - capture/personalise. 41
Wednesday 12 September 12
106. FESTO
• LMS (CLIX) extended with widgets
• 25 heterogeneous participants
• evaluation setup:
1. video introduction to the new system
2. period of experimentation
3. questionnaire
step 5 & 6 - capture/personalise. 43
Wednesday 12 September 12
107. FESTO
• LMS (CLIX) extended with widgets
• 25 heterogeneous participants
• evaluation setup:
1. video introduction to the new system
2. period of experimentation
3. questionnaire
step 5 & 6 - capture/personalise. 43
Wednesday 12 September 12
109. FESTO RESULTS
88% would recommend it to their colleagues
81% agreed that it helps them to be more effective
step 5 & 6 - capture/personalise. 44
Wednesday 12 September 12
110. JIAO TONG UNIVERSITY
• offers higher education for adults with a full-time job
• early prototype with
• evaluation setup:
1. task list (ETA 15 mins)
2. questionnaire (in Chinese) (partially from ResQUE)
step 5 & 6 - capture/personalise. 45
Wednesday 12 September 12
111. JIAO TONG UNIVERSITY
• offers higher education for adults with a full-time job
• early prototype with
• evaluation setup:
1. task list (ETA 15 mins)
2. questionnaire (in Chinese) (partially from ResQUE)
step 5 & 6 - capture/personalise. 45
Wednesday 12 September 12
112. RESULTS
• ran over 3 weeks, 13 respondents
• results for federated search widget:
• 69% used the federated search widget
• 56% found the recommendations relevant to the course
• 38% wanted to continue using the federated search widget
• Personal learning environment results
• 23% found it useful for the course
• 39% found it easy to use, 15% found it navigationally consistent
• 46% were satisfied with the personal learning environment
step 5 & 6 - capture/personalise. 46
Wednesday 12 September 12
113. WHY?
• an established LMS augmented with PLE features
brand new system
• cultural differences?
• slow internet connection
(long loading and querying times)
• lesson learned: the choice of evaluation environment is very
important
step 5 & 6 - capture/personalise. 47
Wednesday 12 September 12
114. WHY?
• an established LMS augmented with PLE features
brand new system
• cultural differences?
users perceive federated search as useful and
• slow internet connection
effective IF the content in the repositories relates
(long loading and querying times)
to their tasks.
[Belliston et al., 2007], [Korah et al., 2010], [Gore, 2008]
• lesson learned: the choice of evaluation environment is very
important
step 5 & 6 - capture/personalise. 47
Wednesday 12 September 12
115. WHY?
• an established LMS augmented with PLE features
brand new system
• cultural differences?
our evaluation indicates that users benefit from
users perceive federated search as useful and
• slow internet connection recommendations.
the social
effective IF the content in the repositories relates
recommendations from friends are preferred over
(long loading and querying times) to their tasks.
system recommendations. [Shinha et al., 2001]
[Belliston et al., 2007], [Korah et al., 2010], [Gore, 2008]
• lesson learned: the choice of evaluation environment is very
important
step 5 & 6 - capture/personalise. 47
Wednesday 12 September 12
118. STEP 7 – CREATE AWARENESS.
Ch 7
RQ4: How can attention
metadata be leveraged to improve
awareness and self-reflection?
48
Wednesday 12 September 12
119. PROBLEM
http://www.ypfp.org/classroom
step 5 & 7 - capture/create awareness. 49
Wednesday 12 September 12
120. PROBLEM
http://www.ypfp.org/classroom
step 5 & 7 - capture/create awareness. 49
Wednesday 12 September 12
121. STUDENT ACTIVITY METER
• visualisations of attention metadata
evaluation
• self-monitoring for learners
design
implementation
step 5 & 7 - capture/create awareness. 50
Wednesday 12 September 12
122. STUDENT ACTIVITY METER
• visualisations of attention metadata
evaluation
• self-monitoring for learners
• awareness for teachers
design
implementation
step 5 & 7 - capture/create awareness. 50
Wednesday 12 September 12
123. STUDENT ACTIVITY METER
• visualisations of attention metadata
evaluation
• self-monitoring for learners
• awareness for teachers
design
• time tracking
implementation
step 5 & 7 - capture/create awareness. 50
Wednesday 12 September 12
124. STUDENT ACTIVITY METER
• visualisations of attention metadata
evaluation
• self-monitoring for learners
• awareness for teachers
design
• time tracking
implementation
• learning resource use and recommendations
step 5 & 7 - capture/create awareness. 50
Wednesday 12 September 12
125. STUDENT ACTIVITY METER
• visualisations of attention metadata
evaluation
• self-monitoring for learners
• awareness for teachers
design
• time tracking
implementation
• learning resource use and recommendations
• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]
step 5 & 7 - capture/create awareness. 50
Wednesday 12 September 12
126. Govaerts, Sten; Verbert, Katrien, Duval, Erik; Pardo, Abelardo. The
Student Activity Meter for Awareness and Self-reflection. CHI 2012.
demo...
51 http://www.leveraas.com/
Wednesday 12 September 12
127. demogra evaluation design
negative positive
phics goal changes
•↑learnability
usability,
•↓errors
12 CS satisfaction, small usability
I. 1st iteration •good satisfaction
students preliminary issues
•usefulness
usefulness
positive
step 5 & 7 - capture/create awareness. 52
Wednesday 12 September 12
128. ITERATION TWO &
THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how
the tool can resolve these.
step 5 & 7 - capture/create awareness. 53
Wednesday 12 September 12
129. CGIAR LAK
1 2 3 4 issue for2 add- issue for add-
5 1 3 4 5
teacher ressed teacher ressed
Provide feedback to the
Provide feedback to the students
students ✔ ? ✔ ✔
Being aware of what
Being aware of whatdoing
students are students are doing ✔ ✔ ✔ ?
✗ ✗
Knowing about collaboration
Knowing and communication * and communication
about collaboration ✔ ✔
?
Knowing which documents
Knowing are used and how much used and how much
which documents are ✔ ✔ ✔
? ✗ ?
Knowing how and when
Knowing online and whenbeen used have been used
how tools have online tools ✔
Finding the students who
are not doing well
Finding the students who are not doing well ✔ ? ✔ ?
Finding the best students
Finding the best students
Knowing how much time
? ? ? ✗
? ✔ ?
students spent
Knowing Knowing if externalstudents
how much time learning spent ✔
? ? ?
resources are used
Knowing if external learning resources areIssues
used ✔Addressed by SAM
step 5 & 7 - capture/create awareness. 54
Wednesday 12 September 12
130. CGIAR LAK
issue for add- issue for add-
teacher ressed teacher ressed
Provide feedback to the students ✔ ? ✔ ✔
Being aware of what students are doing ✔ ✔ ✔ ?
Knowing about collaboration and communication ✔ ✗ ✔ ✗
Knowing which documents are used and how much ✔ ✔ ✔ ?
Knowing how and when online tools have been used ? ✔ ✗ ?
Finding the students who are not doing well ✔ ? ✔ ?
Finding the best students ? ? ? ✗
Knowing how much time students spent ? ✔ ? ✔
Knowing if external learning resources are used ✔ ? ? ?
step 5 & 7 - capture/create awareness. 54
Wednesday 12 September 12
131. CGIAR LAK
issue for add- issue for add-
teacher ressed teacher ressed
Provide feedback to the students ✔ ? ✔ ✔
Being aware of what students are doing ✔ ✔ ✔ ?
Knowing about collaboration and communication ✔ ✗ ✔ ✗
Knowing which documents are used and how much ✔ ✔ ✔ ?
Knowing how and when online tools have been used ? ✔ ✗ ?
Finding the students who are not doing well ✔ ? ✔ ?
Finding the best students ? ? ? ✗
Knowing how much time students spent ? ✔ ? ✔
Knowing if external learning resources are used ✔ ? ? ?
step 5 & 7 - capture/create awareness. 54
Wednesday 12 September 12
132. CGIAR CASE STUDY
step 5 & 7 - capture/create awareness. 55
Wednesday 12 September 12
133. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1 use for course
comparison tool design optimization more metrics
wanted
step 5 & 7 - capture/create awareness. 55
Wednesday 12 September 12
134. CGIAR CASE STUDY
want to search detect outliers
for students
more details on
good indicator for effort student
understand the workload
progress evolution
compare students
obtain course overview
increase awareness
want better 1 to 1 use for course
comparison tool design optimization more metrics
wanted
step 5 & 7 - capture/create awareness. 55
Wednesday 12 September 12
135. LAK CASE STUDY
step 5 & 7 - capture/create awareness. 56
http://maranash.blogspot.com/2011/01/conference-shout-out.html
Wednesday 12 September 12
136. LAK CASE STUDY
find students experiencing
problems and low
verify the status of the engagement
classroom activity
more
chronological course dwell time metrics
self-reflection to measure
progress and increase
motivation more data
step 5 & 7 - capture/create awareness. 56
http://maranash.blogspot.com/2011/01/conference-shout-out.html
Wednesday 12 September 12
137. LAK CASE STUDY
find students experiencing
problems and low
verify the status of the engagement
classroom activity
more
chronological course dwell time metrics
self-reflection to measure
progress and increase
motivation more data
step 5 & 7 - capture/create awareness. 56
http://maranash.blogspot.com/2011/01/conference-shout-out.html
Wednesday 12 September 12
138. demogra evaluation design
negative positive
phics goal changes
usability, •↑learnability
12 CS satisfaction, small usability •↓errors
I. 1st iteration
students preliminary issues •good satisfaction
usefulness •usefulness positive
assessing •provides awareness
19 resource
teacher •all vis. useful
II. teachers help function recomm. not
needs, use & •many uses
& TA’s useful
usefulness •90% want it
•provides awareness
assessing re-orderable
most and feedback
12 teacher parallel
addressed •many uses
III. participan needs, expert coordinates
needs are •66% want it
ts feedback, use with
indecisive •recomm. can be
& usefulness histograms
useful
57
Wednesday 12 September 12
139. ITERATION FOUR
•a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative study using a structured face-2-face interview
with 25 open questions.
step 5 & 7 - capture/create awareness. 58
Wednesday 12 September 12
140. ITERATION FOUR
•a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative study using a structured face-2-face interview
with 25 open questions.
step 5 & 7 - capture/create awareness. 58
Wednesday 12 September 12
141. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
better course monitor tool and
overview resource use
understand time
spent better
detect students course planning
for assistance
step 5 & 7 - capture/create awareness. http://www.tdpri.com/forum/telecaster-discussion-
59
forum/48431-ic503z-cap-52-ri-blackguard-book.html
Wednesday 12 September 12
142. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
better course monitor tool and
overview resource use
understand time
spent better
detect students course planning
for assistance
step 5 & 7 - capture/create awareness. http://www.tdpri.com/forum/telecaster-discussion-
59
forum/48431-ic503z-cap-52-ri-blackguard-book.html
Wednesday 12 September 12
143. ACHIEVABLE TASKS
conflicting visions on get a time
identify students doing
students doing well summary well and at risk
and at risk
all: want tocourse using it
better continue monitor tool and
9/11: give it to students
overview resource use
understand time
spent better
detect students course planning
for assistance
step 5 & 7 - capture/create awareness. http://www.tdpri.com/forum/telecaster-discussion-
59
forum/48431-ic503z-cap-52-ri-blackguard-book.html
Wednesday 12 September 12
144. demo- evaluation design
negative positive
graphics goal changes
usability, •↑learnability
12 CS satisfaction, small usability •↓errors
I. 1st iteration
students preliminary issues •good satisfaction
usefulness •usefulness positive
•provides awareness
19 assessing teacher resource
•all vis. useful
II. teachers needs, use & help function recomm. not
•many uses
& TA’s usefulness useful
•90% want it
•provides awareness and
assessing teacher most
12 re-orderable feedback
needs, expert addressed
III. participan PC with •many uses
feedback, use & needs are
ts histograms •66% want it
usefulness indecisive
•recomm. can be useful
filter & •provides time overview
search, conflicting •provides course
11 icons, visions of overview
use, usefulness
IV. teacher zooming in students •PC assist with detecting
& satisfaction
s & TA’s line chart, doing well problems
editing PC or at risk •many uses & insights
axes •100% want it
60
Wednesday 12 September 12
145. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
146. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
147. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
148. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
149. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
150. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
• to find a wide variety of new insights
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
151. CONCLUSION
• SAM is perceived useful and contributes to better
awareness.
• SAM enables:
• to detect potential students doing well and at risk
• a better course overview
• understanding student time spending
• to find a wide variety of new insights
• almost all want to continue using SAM
step 5 & 7 - capture/create awareness. 61
Wednesday 12 September 12
152. CONTRIBUTIONS.
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach
4. The design & evaluation of federated search tool with social
recommendations
5. The design & evaluation of visualisations of attention metadata to support
awareness and self-reflection.
62
Wednesday 12 September 12
153. Ch 8
CONTRIBUTIONS.
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach
4. The design & evaluation of federated search tool with social
recommendations
5. The design & evaluation of visualisations of attention metadata to support
awareness and self-reflection.
62
Wednesday 12 September 12
154. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach
4. The design & evaluation of federated search tool with social
recommendations
5. The design & evaluation of visualisations of attention metadata to support
awareness and self-reflection.
62
Wednesday 12 September 12
155. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
62
Wednesday 12 September 12
156. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach
62
Wednesday 12 September 12
157. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach RQ 2
62
Wednesday 12 September 12
158. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach RQ 2
4. The design & evaluation of federated search tool with social
recommendations
62
Wednesday 12 September 12
159. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach RQ 2
4. The design & evaluation of federated search tool with social
recommendations RQ 1
62
Wednesday 12 September 12
160. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach RQ 2
4. The design & evaluation of federated search tool with social
recommendations RQ 1
5. The design & evaluation of visualisations of attention metadata to support
awareness and self-reflection.
62
Wednesday 12 September 12
161. Ch 8
CONTRIBUTIONS.
RQ 3
1. A structured method for selecting an appropriate metadata schema
2. The design & evaluation of a metadata generator for the origin of an artist
3. The re-evaluation of a general metadata generation approach RQ 2
4. The design & evaluation of federated search tool with social
recommendations RQ 1
5. The design & evaluation of visualisations of attention metadata to support
awareness and self-reflection.
RQ 4
62
Wednesday 12 September 12