1.
Dr. Mutlu Cukurova
University College London
m.cukurova@ucl.ac.uk
@mutlucukurova
The Promise and Challenges
of Multimodal Learning Analytics

2.
Three conceptualisations of MMLA
1. MMLA to
Automate Human
Tasks
2. MMLA to
Augment Teaching
and Learning
Practices
3. MMLA as a
Research
Methodology

3.
“tools that have been designed and developed
to replace human tasks through analysis of
multiple modalities of data, and prediction of
the best value for a designated outcome
variable, which is conveyed through a user
interface.”
Multimodal Learning Analytics as:

4.
What Modalities of Data We Use?
0
5
10
15
20
25
30
35
40
Presenter’s pose,
gaze direction, visual
attention
Audio, dialogue’s
characteristics
Facialdata and
emotions
Posture and gesture,
body movement,
distance, motion,
Position
Learners’
concentration level,
visual attention and
habits
Learners’ gaze in off-
screen activity
Student’s
physiological data,
arousal and EDA
Students’ cognitive
load
Posture and gesture Location, position
and duration,
movement
Sitting position EEG data, brain
activity
Video and audio Eye-tracking Skin sensing In depth camera Location sensing Pressure sensing EEG sensor
Number of published articles 2010 - 2020
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2021). The evidence of impact and ethical considerations of Multimodal Learning Analytics:
A Systematic Literature Review. https://doi.org/10.35542/osf.io/sd23y

5.
MMLA as Machine Learning Tools
Human
Control
Automation through AI
Most traditional
Educational Technology
AH = Human tasks are replaced
by MMLA H ß A
Initial vision of AIED: Early
Promises of Intelligent Tutoring
Systems
High
Low High
Low

6.
Machine Learning Classification of CPS Competence
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning.
Journal of Computer Assisted Learning, 34(4), 366-377.

7.
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised
machine learning in multimodal learning analytics for estimating success in
project-based learning. Journal of Computer Assisted Learning, 34(4), 366-377.
The promise of MMLA

8.
R. Kawamura et al., Detecting Drowsy Learners at the Wheel of e-Learning Platforms With Multimodal Learning Analytics, IEEE Access, vol. 9, pp. 115165-115174, 2021, doi:
10.1109/ACCESS.2021.3104805

9.
“to support learning experiences
that may be collaborative,
hands-on, and face-to-face,
de-emphasizing the computer
screen as the primary form or
object of interaction.”
Multimodal Learning Analytics:
Worsley, M., Martinez-Maldonado, R., & D'Angelo, C. (2021). "A New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow
MMLA." Journal of Learning Analytics 8.3, 10-27.
2. MMLA to
Augment Teaching
and Learning
Practices

10.
MMLA to Augment Teaching and Learning
Practices
Human
Control
Automation through AI
Most traditional
Educational Technology
HA = Humans internalise MMLA
models H à A
Analytics and GOFAI: Changing
the operations and
representations of thought
AH = Human tasks are replaced
by AI H ß A
Initial vision of AIED: Early
Promises of Intelligent Tutoring
Systems
High
Low
High
Low

11.
17
Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial Intelligence and Multimodal Data in the Service of Human Decision-making: A case Study in Debate
Tutoring, British Journal of Educational Technology, 50 (6), 3032-3046.

12.
Models as learning affordances for humans
Kent, C., Chaudhry, M., Cukurova, M. Bashir, I., Pickard, H., Jenkins, C., du Boulay, B., Luckin, R., (2021). Machine learning models and their development process as learning
affordances for humans. International Conference of Artificial Intelligence in Education, Springer, Cham.
What do successful students look like?
What are their behaviour and personality traits?
Mental model : diagram of functionality
Initial student profiles

13.
Active
Active
Passive
Kasparova, A., Celiktutan, O., & Cukurova, M. (2020). Inferring Student Engagement in Collaborative Problem Solving from Visual Cues. Companion Publication of the 2020
International Conference on Multimodal Interaction. https://doi.org/10.1145/3395035.3425961

14.
From machine observations to learning
sciences constructs of collaboration
Interpretationsof Collaboration Competence from Ma
Observables
Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solvin
students' physical interactions. Computers & Education, 116, 93-109.
Some important components of collaboration may be interpreted through
automated machine observable data:
• synchrony,
• individual accountability,
• equality,
• mutuality.
Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students' physical
interactions. Computers & Education, 116, 93-109.
Cukurova, M. (2018). A syllogism for designing collaborative learning technologies in the age of AI and multimodal data. In European Conference on Technology
Enhanced Learning (pp. 291-296). Springer, Cham.

15.
Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students' physical
interactions. Computers & Education, 116, 93-109.
14

16.
Cukurova, M., Mavrikis, M., & Luckin, R. (2017). Evidence-Centered Design and Its Application to Collaborative Problem Solving in Practice-based
Learning Environments. Analytics4Learning. Stanford Research Institute, Menlo Park, USA.

17.
Learning sciences should drive the features.
Learner/Teachers’ needs and expectations should be met.
Kent, C., & Cukurova, M. (2020). Investigating Collaboration as a Process with Theory-driven Learning Analytics. Journal of Learning Analytics, 7(1), 59-71.
Cukurova, M., Kent, C., & Akanji, A. (2022). Identifying tertiary level educators' needs and understanding of the collaboration process analytics. https://doi.org/10.35542/osf.io/2d3ty

18.
MMLA as a Research Methodology
3.
MMLA as a Research
Methodology

19.
Ouyang, F., Xu, W., Cukurova, M. (2022). An Artificial Intelligence-driven Learning Analytics Method to Examine the Collaborative Problem-solving Process from the Complex Adaptive Systems
Perspective, International Journal of Computer Supported Collaborative Learning. http://arxiv.org/abs/2210.16059v1
Interactive Peer interaction through
communications (Int-C)
Peer interaction through behaviours
(Int-B)
Cognitive Superficial-level knowledge (KS)
Medium-level knowledge (KM)
Deep-level knowledge (KD)
Behavioural Resource management (RM)
Concept mapping (CM)
Observation (OB)
Regulative Task understanding (TU)
Goal setting and planning (GSP)
Monitoring and reflection (MR)
Socio-emotional Active listening and respect (ALR)
Encouraging participation and
inclusion (EPI)
Fostering cohesion (FC)

20.
• Behaviour-oriented collaborative pattern (Type 1) - Medium-level performance
• Communication-behaviour-synergistic collaborative pattern (Type 2) – High-level performance
• Communication-oriented collaborative pattern (Type 3) – Low-level performance
Ouyang, F., Xu, W., Cukurova, M. (2022). An Artificial Intelligence-driven Learning Analytics Method to Examine the Collaborative Problem-solving Process from the Complex
Adaptive Systems Perspective, International Journal of Computer Supported Collaborative Learning. http://arxiv.org/abs/2210.16059v1

21.
• Peer-interaction focussed
collaborative pattern (PIF Type)
• Resources-interaction focusses
collaborative pattern (RIF Type)
Zhou, Q., Suraworachet, W., Celiktutan, O., & Cukurova, M. (2022). What Does Shared Understanding in Students’ Face-to-Face Collaborative Learning Gaze Behaviours “Look
Like”?. In International Conference on Artificial Intelligence in Education (pp. 588-593). Springer, Cham.

22.
1. Methodological - ML AI is not that
intelligent - AIP
2. Logistical/practical/financial
3. Connection to learning theory and
learning design
Short-term Challenges of MMLA
Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The Promise and Challenges of Multimodal Learning Analytics. British Journal of Educational
Technology, 51(5), pp. 1441-1449, https://doi.org/10.1111/bjet.13015.
Zhou, Q., Suraworachet, W., Cukurova, M. (2021). Different modality, different design, different results: exploring self-regulated learner clusters’ engagement behaviours at
individual, group and cohort activities. https://doi.org/10.35542/osf.io/u3g4n

23.
“More Significant” Challenges of MMLA –
Particularly for the first conceptualisation
Normativity
1. What behaviours are
good/bad in education?
2. Inference based on history
3. Flourishing Diversity/ in
uncertainty
Prediction
1. What is it good
for/intervention?
2. Fairness
3. Transparency
4. Accountability
Human Agency
1. Balancing
exploration and
exploitation
2. The reward
function
3. Privacy/
surveillance
Ethics-
Human
Values
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2022). The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic
Literature Review. In The Multimodal Learning Analytics Handbook (pp. 289-325). Springer International Publishing. doi:10.1007/978-3-031-08076-0_12

24.
Students have mixed reactions to MMLA
Zhou,Q., Suraworachet, W., Pozdniakov, S., Martinez-Maldonado, R., Bartindale, T., Chen, P. Richardson, D., & Cukurova M. (2021). Investigating Students’ Experiences with Collaboration
Analytics for Remote Group Meetings. International Conference of Artificial Intelligence in Education, Springer, Cham.
Zhou, Q., Suraworachet, W., Cukurova, M. (2021). Different modality, different design, different results: exploring self-regulated learner clusters’ engagement behaviours at individual, group
and cohort activities. https://doi.org/10.35542/osf.io/u3g4n
- Were not concerned (or
faded) due to: the module
domain, invisibility, not
parts of sum assessment
- More concerning for low
contributors
- Motivational for increased
accountability for high
contributors

25.
Nazaretsky, T., Cukurova, M., Ariely, M., & Alexandron, G. (2021). Confirmation bias and trust: Human factors that influence teachers' attitudes towards AI-based educational
technology, CEUR Workshop Proceedings (Vol. 3042).
Cukurova, M., Luckin, R., & Kent, C. (2019). Impact of an Artificial Intelligence Research Frame on the Perceived Credibility of Educational Research Evidence. International Journal
of Artificial Intelligence in Education, 1-31.
Teachers have confirmation bias and unrealistic
expectations from AI-based EdTech.
"AI framing effect": when people (including
teachers) are presented with research evidence
framed as coming from AIED, they tend to judge it
as less credible compared to educational
psychology and neuroscience.
Teachers appear to be more critical

26.
How do we gain teachers’ trust in MMLA?
• Developed a survey instrument that revealed eight dimensions.
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2021). The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review. https://doi.org/10.35542/osf.io/sd23y
Nazaretsky, T., Cukurova, M., Alexandron, G. (2022). An Instrument for Measuring Teachers’ Trust in AI-Based Educational Technology. Learning Analytics and Knowledge Conference. ACM: New York.
1. Perceived Benefits
2. Lack of Human Characteristics
3. Lack of Transparency
4. Anxieties
5. Self-efficacy
6. Required Shift in Pedagogy
7. Means to Increase Trust
8. AI-EdTech vs Human Advice
Strongly
not agree
Not agree Strongly
agree
Agree
Neutral

27.
How would a CPD to increase teachers’ trust in
intelligent technologies look like?
Nazaretsky, T., Ariely, M., Cukurova, M., Alexandron, G. (2022). Teachers’ Trust in AI-powered Educational Technology and a Professional Development Program to Improve It, British Journal of Educational
Technology, DOI: 10.1111/bjet.13232

28.
What do we mean by transparency?
Transparency for whom?
Chaudhry, M. A., Cukurova, M., & Luckin, R. (2022). A Transparency Index Framework for AI in Education. Artificial Intelligence in Education. https://arxiv.org/abs/2206.03220 .

29.
Cukurova, M., Zhou, Q., Spikol, D., & Landolfi, L. (2020). Modelling collaborative problem-solving competence with transparent learning analytics: is video data enough?. In Proceedings of the
Tenth International Conference on Learning Analytics & Knowledge (pp. 270-275).
Listening Speaking
Transparent decision trees to model
Collaborative problem solving competence
Watching Making Collaborative problem-solving competence

30.
The Golden Triangle
for Value Alignment
Teachers & Learners
Tech Developers
Academic researchers
ENABLER 3
Engage educators,
trainers, researchers and
AI developers in co-
development
ENABLER 1
Train educators
and trainers
ENABLER 2
Train AI
developers
Data,
evidence
and research
Cukurova, M., Luckin, R., & Clark-Wilson, A. (2019). Creating the golden triangle of evidence-informed education technology with EDUCATE. British Journal of Educational
Technology, 50(2), 490-504.
Luckin, R., & Cukurova, M. (2019). Designing educational technologies in the age of AI: A learning sciences-driven approach. British Journal of Educational Technology,
50(6), 2824-2838. doi:10.1111/bjet.12861

31.
Human-centred design for value alignment:
How Can We Design Analytics Interfaces Aligned to Teachers' Inquiry?
Pozdniakov, S., Martinez-Maldonado, R., Shan-Tsai, Y., Cukurova, M., Bartindale, T., Chen, P., Harrison, M., Richardson, D., & Gasevic, D. (2022). The Question-driven Dashboard: How Can We Design Analytics
Interfaces Aligned to Teachers' Inquiry?. Learning Analytics & Knowledge, ACM.

32.
Human
Control
Automation through AI
Most traditional
Educational Technology
HA = Humans internalise MMLA
models H à A
Analytics and GOFAI: Changing
the operations and
representations of thought
H[A] = Human cognition (H)
extended with MMLA (A), tightly
coupled human and artificial
systems.
H[A] > (H) + (A)
AH = Human tasks are replaced
by MMLA H ß A
Initial vision of AIED: Early
Promises of Intelligent Tutoring
Systems
High
Low High
Low
2) Human-centred
3) Trustworthy
MMLA: A vision for the future
1) Learning sciences/
learning design-driven

33.
Thank you
Dr. Mutlu Cukurova
m.cukurova@ucl.ac.uk
@mutlucukurova
PhD students and Collaborators:, Wannapon Suraworachet, Qi Zhou, Ali Chaudhry, Haifa Al-Wahaby, Rose Luckin, Carmel Kent, Benedict Du Boulay, Juan I.
Asensio-Pérez, Daniel Spikol, Manolis Mavrikis, Yannis Dimitriadis, Stanislav Pozdniakov, Roberto Martinez-Maldonado, Tom Bartindale, Peter Chen, Dan
Richardson, Fan Ouyang, Xi Wu, Michali Giannakos, Zacharoula Papamitsiou, Oya Celiktutan, Anjelika Kasparova, Cristina Villa-Torrano, Abayomi Akanji