«Applying Learning Analytics in Living Labs for Educational Innovation». Tobias Ley

1. Applying Learning Analytics in Living
Labs for Educational Innovation
TOBIAS LEY
PROFESSOR FOR LEARNING ANALYTICS AND EDCUATIONAL INNOVATION
TALLINN UNIVERSITY, ESTONIA
EMADRID
15 JANUARY 2021
This project has received funding from the European Union’s
Horizon 2020 research and innovation programme under grant
agreement No. 669074

2. Research GroupLearning Analytics and Educational Innovation
TLU Schools involved
- School of Educational Sciences
- School of Digital Technologies
Staff
- 2 Professors
- 5 Senior Researchers
- 29 PhD students & Junior
Researchers
- 3 Administrative
(Research Coordination,
Marketing, Training Coordination)

3. WHAT MAKES A GOOD
EDUCATIONAL
INNOVATION?
- A change that leads to new practices
- Positive impact on stakeholders (students,
teachers, school leaders, researchers, society)
- Evidence-based
- Sustainable change
- Scalable change

4. Investment into ICT in
Schools?
What then ?
“... success in initiating change
does not guarantee that such
changes can be sustained
over time”
Technology and Open Educational
Resources as opportunities to reshape EU
education , Communication of the
European Commission, 2011

5. EVIDENCE-BASED!
Research first?
The linear model of
knowledge
transmission
Educational Research
Teacher Training
Teacher Practices
Student Learning

6. LIVING LABS ARE COMMON CONTEXTS
THAT ALLOW ...
- Practice-based Educational Research
- Studying the effects of new pedagogical methods
- Teacher Training
- Professional learning through “boundary
crossing” and “teacher-led research”
- Innovation Adoption
- Incubators for leading to sustainable change in
Schools
Innovation
Research
Learning

7. Living Labs for
Educational Innovation
- Research involves
practitioners in each step
- Integrates Learning
Analytics as a tool for
stakeholder decision
making
https://edulabs.ee

8. 6 Living Lab Cases
for STEM Learning
- Robomathematics
- Inquiry Learning outside the
Classroom
- Smart Sensors for STEM
- Digital Learning Resources in Math
- Robots in Early Childhood Education
- Glocally Transformative Learning

9. Demonstrating
large scale role-out
- Involvement in cases so far …
- 300+ schools
- 800+ teachers
- 5000+ students

10. - help us gather evidence for innovations in
teaching and learning “in the wild”
- be integrated into the professional development
of teachers and support decision making
- be highly flexible and stakeholder-driven
Learning Analytics
should ...

11. - teacher practice
- professional development
- school development /
educational policy
- educational research
Learning Analytics Toolbox
Graasp Google Drive
Dashboard &
Sensemaking tool
Digital Mirror
LA Pills ObservataEduLog
EL
Prolearning
CoTrackPAMEL
Sisuloome
https://edulabs.eehttp://htk.tlu.ee

12. Applying the Learning Analytics
Toolbox

13. Highlights along the
EDUlabs phases
https://edulabs.ee
Example: Collaborative outdoor inquiry learning
Mobile and Multimodal Learning Analytics
Example: Teacher training for integrating digital
learning resources in math teaching
Learning Analytics dashboards for teacher inquiry

14. Project Days
Collaborative Outdoor Inquiry Learning

15. Terje Väljataga
Senior Research Fellow in New
Learning Environments and
Technologies, School of
Educational Sciences

16. Project days for educational
innovation experimentation
Testbed for small scale trials and experiments
bringing together research and teacher education
- A 4-hour learning and teaching events with a
group of school students and their teachers for
testing and validating specific educational
innovations
- Designed and analysed in collaboration with
researchers, didactics, pre-service teachers,
educational psychologists, educational
technology and analytics experts

17. Example -
technology-enhanced outdoor
collaborative inquiry learning
- Study focus: Exploration of socio-environmental problem to form an
evidence-based opinion about the problem in the society
- Learning outcomes: Students understand the nature of the studied
socio-environmental problem and are able to formulate evidence-based
opinions about the problem
- Collaborative outdoor hands-on inquiry learning
- Use of digital technologies (Vernier sensors, smartphones, laptops)

18. Avastusrada - a
web-based location tool

19. Results:
Effects on students
- Most of the students’ conceptual understanding and the level
of scientific explanations changed; they were able to form an
evidence-based opinion according to the data the student
groups collected
- Most of the students had a positive learning and group work
experience and positive attitude towards the learning event
- Student groups were able to overcome the occurred
task-related and technological challenges by themselves

20. Did Learning Designs support
higher order thinking?
Mettis, Kadri, and Terje Väljataga. 2020. “Designing Learning Experiences for Outdoor
Hybrid Learning Spaces.” British Journal of Educational Technology 0 (0): 1–16.
https://doi.org/10.1111/bjet.13034.

21. Did contextual integration
affect higher order thinking?

22. Machine Learning for
Classifying Learning Designs
- Create an algorithm that can classify teachers’ learning
designs (cognitive load required by students, the role
played by the situated environment in the learning activity
and inquiry based learning phases involved)
- Use only textual descriptions of questions and answers
- Goal: Integrate the algorithm into Avastusrada, in order to
support practitioners while designing learning activities (ie,
by suggesting changes during the design)

23. Machine Learning for
Classifying Learning Designs
Current results
- The best performing algorithms have been the ones based on Neural
Networks (especially using new techniques, such as the BERT
transformer).
- The average accuracy of the algorithm has been over 94%, with
Cohen’s Kappa over 0.8.
Future work will investigate the support that such algorithms could offer to
practitioners design practices.

24. Highlights along the
EDUlabs phases
https://edulabs.ee
Example: Collaborative outdoor inquiry learning
Mobile and Multimodal Learning Analytics
Example: Teacher training for integrating digital
learning resources in math teaching
Learning Analytics dashboards for teacher inquiry

25. Innovation
Laboratory
Teacher training for integrating digital
learning resources in math teaching

26. Kairit Tammets
Head of the Centre for
Educational Technology and
Senior Research Fellow in
Educational Technology, School
of Digital Technologies

27. Problems addressed
- Teachers are expected to provide personalised learning experience for all
the students and work with the data in their professional practice.
- However:
- the data made available for the teachers is not contextualized
(pedagogically-informed, subject-specific);
- teachers are not used to work with the data, data is not integrated
to their practice and they need support to make meaningful
decisions based on data - teacher inquiry and data literacy
- dashboard development should be aligned with the process of
teacher noticing to better meet the needs of the teacher to
understand the impact of instruction to students’ learning

28. - Long-term Teacher
Training (6-12 months)
- Implementation of new
teaching practice
- Action research in their
own classroom
- Reflection on student
learning
The Teacher Innovation
Laboratory

29. Integrating student-activating
TEL practices in math teaching
Goal: To scaffold teachers to design, implement, monitor and sense-make of the
student-activating TEL practices (learning designs and tasks) to support students’
engagement in secondary mathematics education
Who: 21 mathematics teachers and researchers in the field of mathematics
didactics, educational psychologist, educational technology
How: 10 month intervention program, monthly co-creation sessions, individual
piloting, collective reflection
Data collection:
- Students’ process-oriented self-reports about engagement (LaPills);
- Teachers’ LDs, pre-post measures of knowledge appropriation and math
beliefs, reflections after pilots, design sessions of dashboard development

30. https://teacher.desmos.com/activitybuilder/custom/5eef8ae8dbf929782acbb598

31. https://teacher.desmos.com/activitybuilder/custom/5eef8ae8dbf929782acbb598

32. Students’self-reports -
LaPills

33. Teachers’ understanding of the math teaching
method to activate students in math classes
and teachers’ data literacy skills were
increased.
Pre-post measures of self-efficacy
demonstrated significant differences that the
training had an impact on teachers’
self-efficacy to use the method in their
practice and ability to assess its applicability.
However, students’ process
measures demonstrated
statistically not significant
differences in their active
participation at the end of
the lesson activities.

34. Conclusions
- Results indicate that students’ process measures
had an effect on teachers’ understanding about the
lesson design.
- By monitoring the process measures, teachers
possibly gained a better understanding of how to
activate students and make sense about their own
learning design and students’ learning

35. Individual view:
Self-reported
engagement and data
from DLRs
Learning design and
reflection
Class-level overview
of the interactions
with DLRs with
actionable feedback

36. Putting it all
together
The Learning Analytics Toolbox and
Infrastructure

37. María Jesús Rodríguez-Triana
Senior Research Fellow in
Learning Analytics and
Educational Data Mining
(Algorithms and Visualization),
School of Digital Technologies
Luis Pablo Prieto Santos
Senior Research Fellow in New
Learning Environments and
Technologies (Planning and
Orchestration of Learning), School
of Educational Sciences

38. Problems addressed
- Learning Analytics is tightly integrated into the decision making process
of teachers
- Design of Learning Analytics technology needs to be highly responsive
to teachers real needs
- How to build tools and infrastructure in such highly dynamic
setting?
- How to teachers react to the use of data in the classroom?

39. Scenario of use in a research
setting
s

40. CoTrack
Example: Multimodal
Learning Analytics
PAMEL
Dashboard &
Sensemaking tool
LA Pills
EduLog
EL
Prolearning
s

41. Co-Track Prototype
and Dashboard

42. How do teachers react?
- Teachers tend to trust the outcomes and grow more
confident in their decisions
- 90% would take the information into account for
intervention or reflection
- Actionability is higher if information confirms
what they had been thinking
- Received no new information: 86% intervene
- Received new information: 57% intervene

43. - The Learning Analytics Toolbox supports different stakeholders in the
different stages of evidence-based decision making process
- Addressing the classic streetlight effect problem
- Transitioning towards authentic educational practice (Project Days
and Innolabs)
- Main challenges
- technological complexity in real life settings,
- need to simplify data collection process to encourage adoption
- human factors (e.g., teacher motivation and believes, data
literacy-skills) play a decisive role in the adoption
Lessons Learned

44. Estonian State Research
Award 2020

46. Tobias Ley
Professor for Learning Analytics and Educational Innovation
Tallinn University, Estonia
http://ceiter.tlu.ee
Twitter @tobold