Designing and Evaluating Student-facing Learning Dashboards: Lessons Learnt (Public PhD Defence)

Sven Charleer
Designing and Evaluating  
Student-facing Learning Dashboards 
Lessons Learnt
Public PhD Defence
July 6, 2017
Augment 
Lab

LEARNING ANALYTICS
4
“The measurement, collection, analysis, and reporting of data about learners
and their contexts, for purpose of understanding and optimising learning and
the environments in which it occurs”
J. L. Santos. Learning Analytics and Learning Dashboards: a Human- Computer Interaction Perspective. PhD dissertation, KU Leuven, 2015.
G. Siemens. “Learning analytics: envisioning a research discipline and a domain of practice”. Proceedings of the 2nd
International Conference on Learning Analytics and Knowledge . ACM. 2012, pp. 4–8.
session
course
degree
year
Microlevel
intro

AUTOMATE AUGMENT
Engelbart, D. C. (2001). Augmenting human intellect: a
conceptual framework (1962). PACKER, Randall and
JORDAN, Ken. Multimedia. From Wagner to Virtual Reality.
New York: WW Norton & Company, 64-90.
VS
5
Arnold, Kimberly E., and Matthew D. Pistilli. Course Signals at Purdue:
Using learning analytics to increase student success. Proceedings of
the 2nd International Conference on Learning Analytics and Knowledge.
ACM, 2012.

LEARNING DASHBOARDS
6
“A Learning Dashboard is a single display that aggregates different indicators
about learner(s), learning process(es) and/or learning context(s) into one or
multiple visualisations.”
B. A. Schwendimann, M. J. Rodríguez-Triana, A. Vozniuk, L. P. Prieto, M. S. Boroujeni, A. Holzer, D. Gillet, and P. Dillenbourg. Understanding learning at a glance: An overview of learning
dashboard studies. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, pages 532–533. ACM, 2016.
K. Verbert, E. Duval, J. Klerkx, S. Govaerts, and J. L. Santos. Learning Analytics Dashboard Applications. American Behavioral Scientist, 57(10):1500–1509, 2013.
Perceived beneﬁts
Design guidelines
intro
awareness
(self) reﬂection
sense making
impact
data
questions
answers
behavior

7
DESIGN-BASED RESEARCHintro
External Input through collaboration with participants:
Teachers
Students
Experts

8
Multiple dependent variables

9
Rapid prototyping

Santos, J.L., Verbert, K., Govaerts, S., Duval, E.: Addressing learner issues with stepup!: an evaluation. In:
Proceedings of the Third International Conference on Learning Analytics and Knowledge, ACM (2013) 14–22
CREATING EFFECTIVE LEARNING DASHBOARDSch2
11

PATH OF EFFORT
OUTCOMES
INFLUENCES
12 https://www.ﬂickr.com/photos/lockechrisj/
LEARNER PATHSch2

RQ1: How should we visualise learner data to support students to
explore the path from eﬀort to outcomes?
RQ2: How can we promote students, inside and outside the
classroom, to actively explore this eﬀort to outcomes path?
CREATING EFFECTIVE LEARNING DASHBOARDSch2
13

TwitterBlogs
15
EVALUATIONSch2

Inquiry-Based Learning16
EVALUATIONSch2

Abstract the LA data
Provide access to the artefacts
Augment the abstracted data
Provide access to teacher and peer feedback
17
RESULTS
RQ1: What are relevant learning traces, and how should we visualise these
data to support students to explore the path from effort to outcomes?
ch2

18
RESULTS
RQ2: How can we promote students, inside and outside the classroom, to
actively explore this effort to outcomes path?
Visualise the learner path
Integrate LA into the workﬂow
Facilitate collaborative exploration of the LA data
ch2

23
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

24

25

26

27

28

29

30

32
CONTRIBUTIONSch2
Guidelines published at European Conference on Technology Enhanced Learning 
(25% acceptance) 
S. Charleer, J. Klerkx, E. Duval, T. De Laet, and K. Verbert. Creating effective learning analytics dashboards: Lessons learnt.
Adaptive and Adaptable Learning: 11th European Conference on Technology Enhanced Learning, EC-TEL 2016, Lyon,
France, September 13-16, 2016, Proceedings, pages 42–56, Cham, 2016. Springer International Publishing
14 papers 
collaboration with 
Stanford University - University of Technology Sidney - University of the Basque
Country - Murdoch University Perth - Curtin University Perth
Program Committees 
EC-TEL, LAK, Cross-LAK, ARTEL, EC-TEL Doctoral Consortium
European Commission for the weSPOT project

33
CONTRIBUTIONSch2
Austria, Bulgaria, Netherlands, Slovenia, United Kingdom
33

BALANCED DISCUSSION IN THE CLASSROOMch3
over- and under-participation
35

BALANCED DISCUSSION IN THE CLASSROOMch3
RQ3: What are the design challenges for ambient Learning
Dashboards to promote balanced group participation in
classrooms, and how can they be met?
RQ4: Are ambient Learning Dashboards eﬀective means for
creating balanced group participation in classroom settings?
36

EVALUATION SETUPch3
case study 1
# participants 12 students
deployment
1 3h session with dashboard
1 3h session without dashboard
evaluation
class discussion 
questionnaires (perceived distraction/
awareness/usefulness) 
activity/quality logging
case study 2
# participants 19 students
deployment
half 3h session without dashboard 
half 3h session with dashboard
evaluation
questionnaires (perceived importance
feedback/motivation)
activity/quality logging
37

Visualise balance in an abstract and neutral way
Add the qualitative dimension to the visualisation
Create a realistic picture of the classroom situation
38
RESULTS
RQ3: What are the design challenges for ambient LDs to promote balanced
group participation in classrooms, and how can they be met?
ch3

Ambient dashboards as support for teacher/presenter
Ambient dashboards raise awareness of the invisible
Ambient feedback information can activate students
39
RESULTS
RQ4: Are ambient LDs effective means for creating balanced group
participation in classroom settings?
ch3

40
ABSTRACT / NEUTRALch3
Group 1 Group 2
Group 5 Group 3
Group 4

CONVERGENCEch3
-0.2
0
0.2
0.4
0.6
0.8
%distancefromaverage
-0.2
0
0.2
0.4
0.6
0.8
0 10.5
% of time of feedback session passed
%ofdistancefromaverage
without visualisation
with visualisation
group 1
group 2
group 3
group 4
group 5
group 6
41

42
CONTRIBUTIONS
Published Special Issue on Awareness and Reﬂection in
Technology-Enhanced Learning, ĲTEL 
(8/21 submissions accepted) 
Charleer, S., Klerkx, J., Duval, E., De Laet, T. and Verbert, K. (2017) ‘Towards balanced
discussions in the classroom using ambient information visualisations’, Int. J. Technology Enhanced
Learning, Vol. 9, Nos. 2/3, pp.227–253.
Basis for new research collaboration 
with the University of Sidney
ch3

SUPPORTING ADVISER-STUDENT DIALOGUEch4
43
lack of data-based feedback

SUPPORTING ADVISER-STUDENT DIALOGUEch4
RQ5: What are the design challenges for creating a Learning
Dashboard to support study advice sessions, and how can they be
met?
RQ6: How does such a Learning Dashboard contribute to the role
of the adviser, student, and dialogue?
44

EVALUATION SETUPch4
design
# participants
17 study advisers (preliminary feedback) 
5 study advisers (iterative feedback)
approach
brainstorms/observations 
iterative design
evaluation
# participants 5 study advisors
deployment
Engineering Science, Engineering Science:
Architecture 
97 sessions (15-30min per session)
evaluation
15 sessions observed
questionnaires perceived usefulness
45

Data Conﬁdence
Collaboration
Adviser’s role
46
RESULTS
RQ6: How does such a Learning Dashboard contribute to the role of the
adviser, student, and dialogue?
ch4
RQ5: What are the design challenges for creating a Learning Dashboard to
support study advice sessions, and how can they be met?
Authorship
Visual Encoding
Ethics

RESULTSch4
S. Claes, N. Wouters, K. Slegers, and A. V. Moere. Controlling In-the-Wild Evaluation Studies of Public Displays. pages 81–84, 2015.
49

50
CONTRIBUTIONSch4
Published in IEEE Transactions on Learning Technologies 
11% acceptance rate, IF 2.267 
S. Charleer, A. Vande Moere, J. Klerkx, K. Verbert, and T. De Laet. Learning analytics dashboards
to support adviser-student dialogue. IEEE Transaction on Learning Technologies, 18 pages
Deployed at 
Engineering Science, Engineering Science: Architecture, Maths,
Biology, Physics, Geology, Geography, Biochemistry, Informatics,
Bio-engineering, Engineering Technology (3 campuses)
15 study advisers during 165 sessions

BEYOND THE TEXTxtra
Assistent Bachelor/Master Engineering Science courses
Advisor Master Theses
3 European Projects
51

52
PROBLEEMOPLOSSEN & ONTWERPEN 3xtra
52

53
PROBLEEMOPLOSSEN & ONTWERPEN 3xtra
53

Discover, interact Personal Space
(Mendeley)
V. Alvarez, J. Klerkx, S. Charleer, E. Duval, and D. Moor. Science 2.0 and visual data exploration using augmented reality. In Proceedings of the Workshop on Collaboration Meets
Interactive Surfaces: Walls, Tables, Tablets and Phones (CMIS), ACM International Conference on Interactive Tabletops and Surfaces (ITS), Dresden, D, October 2014, pages 1–6,
Oct. 2014
MASTER THESESxtra
56

S. Charleer, J. Klerkx, E. Duval, T. De Laet, and K. Verbert. Faceted Search on Coordinated Tablets and Tabletop: a Comparison. In Proceedings of the 8th
ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Engineering Interactive Computing Systems, Brussels, Belgium, 21-24 June 2016,
pages 165–170. ACM, June 2016
FACETED SEARCH TABLETS vs TABLETOPxtra
58

59
S. Charleer, J. Klerkx, and E. Duval. PoPI: Glyph Designs for Collaborative Filtering on Interactive Tabletops. In E. Bertini, J. Kennedy, and E. Puppo, editors, Eurographics Conference on
Visualization (EuroVis) - Short Papers. The Eurographics Association, 2015
GLYPHS FOR COLLABORATIONxtra
59

60
SUMMARYwrp
3 learning settings
7 dashboards
100+ students, 20 instructors, 17 study advisers
19 publications

62
Citation indices Since 2012
# citations 120
h-index 7
i10-index 4
CITATIONSwrp
62

63
FUTURE (ONGOING) WORKwrp
Ground work for long-term evaluations/deployments
Leiden University
Student union requests faculty deployment
KU Leuven has shown interest in dashboard university-wide

ȡ
leerkracht callcenter verkoper administratie management
actionable
non-actionable
Opleiding: MS Excel
Opleiding: MS PowerPoint
studies
8.9%
7.2%
Engels
Frans
Duits
talen
5.1%
2.4%
Antwerpen
Brussel
regio
3.9%
2.2%
leeftijd dagen werkloos
0 20
18 65 0 800
0 10 0 10
64

65
65

Designing and Evaluating  
Student-facing Learning Dashboards 
Lessons Learnt
Sven Charleer
Public PhD Defence
July 6, 2017
(╯°□°)╯︵ ┻━┻

Designing and Evaluating Student-facing Learning Dashboards: Lessons Learnt (Public PhD Defence)

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à Designing and Evaluating Student-facing Learning Dashboards: Lessons Learnt (Public PhD Defence)

Similaire à Designing and Evaluating Student-facing Learning Dashboards: Lessons Learnt (Public PhD Defence) (20)

Plus de Sven Charleer

Plus de Sven Charleer (20)

Dernier

Dernier (20)

Designing and Evaluating Student-facing Learning Dashboards: Lessons Learnt (Public PhD Defence)