Co-designing learning dashboards for scalable feedback
1. Co-designing learning dashboards
for scalable feedback
Tom Broos & Tinne De Laet
Tom.Broos@kuleuven.be
@TomBroos
Tinne.DeLaet@kuleuven.be
@TinneDeLaet
2. largest university in Belgium
founded 1425
16 faculties
→ general university
55 000 students
tuition fee for 1 year
<
1
3
average monthly
income
3. “Learning analytics is about
collecting traces that learners
leave behind and using those
traces to improve learning.”
- Erik Duval
Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 3
Learning Analytics?
4. Learning Dashboards?
4Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004
“A dashboard is a visual display of the
most important information needed to
achieve one or more objectives;
consolidated and arranged on a single
screen so the information can be monitored
at a glance.”
- Stephen Few
5.
6.
7.
8.
9.
10. Successful Transition from secondary to higher
Education using Learning Analytics
enhance a successful transition from
secondary to higher education by means of
learning analytics
design and build analytics dashboards,
dashboards that go beyond identifying at-risk
students, allowing actionable feedback for all
students on a large scale.
Achieving Benefits from Learning Analytics
research strategies and practices for using
learning analytics to support students during
their first year at university
developing the technological aspects of
learning analytics,
focuses on how learning analytics can be used
to support students.
10
www.stela-project.eu
@STELA_project
2015-1-UK01-KA203-013767
www.ableproject.eu
@ABLE_project_eu
562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD
11. STELA ♥ ABLE
11
actionable feedback
student-centered
program level
inclusive
first-year experience
institution-wide
Learning Analytics
actual implementation
12. [!] Feedback must be “actionable”.
12
Warning!
Male students have
10% less probability to
be successful.
You are male.
Warning!
Your online activity is
lagging behind.
action?
?
action?
?
14. [!] Start with the available data.
Lots of data may eventually become
available in the future …
…. already start with what is available
14
(*)
(*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication.
British Journal of Educational Technology, 44(4), 616-628.
15.
16. [!] Involve stakeholder expertise.
16
visualization experts
practitioners / end-users
researchers LA
researchers first-year
study success
Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue.
In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).
23. Study advisor – student conversations
23
Should I consider
another program?
Can I still finish the
bachelor in 3 years?
How should I compose
my program for next
year?
What is the personal
situation?
How can I help?
What is the best
next step?
25. LISSA: status
25
26 programs >4500 students
114 student advisors
training of study advisors
• Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
• http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/
• Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student dialogues. Proceedings of the 8th
International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM.
dashboards for three examination
periods
observations, interviews,
questionnaires
26. How to determine
thresholds for different
groups?
Stakeholder involvement: example 1
26
upper and lower group: clear message
middle group as small as possible
Do not overfit! (nuance)
27. Evaluation – interviews
“When students see the numbers, they are
surprised, but now they believe me.
Before, I used my gut feeling, now I feel
more certain of what I say as well”.
“It’s like a main thread
guiding the
conversation.”
“I can talk about what to do with the results,
instead of each time looking for the data and
puzzling it together.”
“Students don’t know where to look during the
conversation, and avoid eye contact.
The dashboard provides them a point of focus”.
“A student changed her
study method in June and
could now see it paid off.”
LISSA supports a personal dialogue.
the level of usage depends on the experience
and style of the study advisors
fact-based evidence at the side
narrative thread
key moments and student path help to
reconstruct personal track
“I can focus on the
student’s personal
path, rather than on
the facts.”
“Now, I can blame
the dashboard and
focus on
collaboratively looking
for the next step to
take.”
27
28. [!] Do not oversimplify. Show
uncertainty.
28
• reality is complex
• measurement is limited
• individual circumstances
• need for nuance
• trigger reflection
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
30. [!] Start with the available data.
30
data already available?
administrative (examples)
student records course grades
systems (examples)
LMS access logs advisor meetings
)
Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher
education. Journal of Research in Innovative Teaching & Learning, , 1-14.
demo: https://learninganalytics.set.kuleuven.be/static-demo-rex/ (en) or https://learninganalytics.set.kuleuven.be/demo/rex-1718jan-ir (nl)
33. Stakeholder involvement: example 2
33
Student
is
starter?
Study
efficienc
y?
Mail invitation
Student visits
dashboard
Reminder LOW GLOBAL SE
TIPS
REGULATIONS
MID GLOBAL SE
HIGH GLOBAL SE
ADVICE
INTRODUCTION
MAIL
REMINDER
SCORE
36. What can we learn from dashboard usage?
36Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and
Microinteractions. accepted for ECTEL 2018
p<1e-5 p<1e-9p-test
39. Design your own
considerations
• data
• stakeholders
• textual and visual elements
• incorporating in university practices
39
inspiration
https://learningdashboards.eu/
40. [!] Context matters!
• available data
• national and institutional regulations
and culture
• educational vision
• educational system, size of population ..
• …
Don’t just copy existing LA solutions!
40
41. Project team @
41
Sven Charleer
AugmentHCI, Computer Science department
PhD researcher ABLE
Katrien Verbert
AugmentHCI, Computer Science department
Copromotor of STELA & ABLE
Carolien Van Soom
Leuven Engineering and Science Education Center
Head of Tutorial Services of Science
Copromotor of STELA & ABLE
Greet Langie
Leuven Engineering and Science Education Center
Vicedean (education) faculty of Engineering Technology
Copromotor of STELA & ABLE
Tinne De Laet
Leuven Engineering and Science Education Center
Head of Tutorial Services of Engineering Science
Coordinator of STELA
KU Leuven coordinator of ABLE
Francisco Gutiérrez
AugmentHCI, Computer Science department
PhD researcher ABLE
Tom Broos
Leuven Engineering and Science Education Center
AugmentHCI, Computer Science department
PhD researcher STELA
Martijn Millecamp
AugmentHCI, Computer Science department
PhD researcher ABLE
Special thanks to study advisors for their cooperation, advice, feedback, and support!
Jasper, Bart, Riet, Hilde, An, Katrien, …
♥
