This document summarizes the findings of a study analyzing empirical learning analytics research from 2011-2014. The study found that most research examined log data from university students to visualize learning trajectories and predict success or failure. However, some innovative studies looked at informal learning communities, video/audio data, automated assessment, and error diagnosis. The document recommends that future learning analytics research incorporate more educational and psychological theories for a deeper understanding of the issues.

1. Learning Analytics: Trends and Issues of the
Empirical Research of the Years 2011-2014
Nic. Nistor, Michael Derntl, & Ralf Klamma
EC-TEL 2015 Toledo, Spain

2. 1. Introduction
Tasks and methods of Learning Analytics (LA)
(Baker & Siemens, 2015)
• predicting learning behaviour and output
• structure discovery: clustering (learner types), social network analysis
• relationship mining
• distillation of data for human judgement: monitoring, visualizing
• discovery with models
• tool development for LA
Ø Open issue…
Nistor, Derntl, & Klamma, EC-TEL2015

3. 2. Purpose
Overview of the empirical LA research
2011-2014
• educational setting
• data sources
• theoretical framework
from educational perspective
• purpose of data processing
• associated computational methods
Nistor, Derntl, & Klamma, EC-TEL2015

4. 3. Methodology
Methodology
• Analysed body of literature: 480 conference papers
– 298 papers from EC-TEL 2011-2014
– 182 papers from LAK 2011-2014
• 197 papers (71 EC-TEL, 126 LAK) were considered
relevant
Nistor, Derntl, & Klamma, EC-TEL2015

5. 3. Methodology
Methodology
• dynamic topic model using the topic modelling toolkit D-
VITA (http://monet.informatik.rwth-aachen.de/DVita?id=3001)
• Final manual analysis of 19 most central papers (12 LAK
2014, 7 EC-TEL 2014)
Nistor, Derntl, & Klamma, EC-TEL2015

6. 4. Findings
Educational setting
• University/college: 12 cases, [8–10, 12, 13, 15,
17–19, 21, 23, 25]
• School: 6 cases, [11, 14, 16, 20, 22, 24]
• Informal Settings: Total: 1 case, [26]
Nistor, Derntl, & Klamma, EC-TEL2015

7. 4. Findings
Data sources – from isolated (scalable)
cases to Big Data
• Learning management system log data: 11
cases, [8, 9, 11, 13, 14, 18–20, 23–25]
• Learner-generated data: 4 cases, [10, 12, 21, 26]
• Additionally collected data: 3 cases, [16, 17, 22]
• Available academic administration data: 1 case,
[15]
Nistor, Derntl, & Klamma, EC-TEL2015

8. 4. Findings
Theoretical framework from educational
perspective
• No Specific Educational Theoretical Framework
Acknowledged: 7 cases, [8, 13, 15, 18, 23–25]
• Linguistic/Logic Approach: 3 cases, [10, 21, 26]
• Situated Learning/Community of Practice: 2 cases, [9, 16]
• Vygotsky’s Zone of Proximal Development: 2 cases, [14, 22]
• Various Approaches: self-regulated learning [12], adaptive
learning environments [11], peer assessment [19], autism
[20], or the instructional design concept of learning
orchestration [17]
Nistor, Derntl, & Klamma, EC-TEL2015

9. 4. Findings
Purpose of data processing
• Learning Trajectory Visualization: 7 cases, [9, 10,
12, 13, 21, 23, 25]
• Predicting Learning Success or Failure/Dropout: 5
cases, [8, 14, 15, 18, 22]
• Diagnosis: 3 cases, [17, 20, 24]
• Assessment: 2 cases, [16, 19]
• Intervention (e.g., automatically adapting or
recommending learning material): 2 cases, [11, 26]
Nistor, Derntl, & Klamma, EC-TEL2015

10. 4. Findings
Associated computational methods
• Prediction: 7 cases [10, 15, 16, 18, 21, 22, 24]
• Structure Discovery: 3 cases, clustering [8, 26]
and social network analysis [13]
• Relationship Mining: 6 cases, correlation mining
[12, 17, 19], association rule mining [11] and
others with mixed approaches [20, 25]
• Visualization: 3 cases, [9, 14, 23]
Nistor, Derntl, & Klamma, EC-TEL2015

11. 6. Discussion
Summary of findings
• Mainstream:
– examine log data to visualize learning trajectories and
predict the success or failure of university students
• Innovative studies:
– informal learning in online communities
– video/audio records
– automated student learning assessment
– error/misconception diagnosis
Nistor, Derntl, & Klamma, EC-TEL2015

12. 7. Consequences
Recommendation
• Educational and psychological theories urgently
needed for in-depth understanding of
educational phenomena, and for significant
progress of upcoming LA research
Nistor, Derntl, & Klamma, EC-TEL2015

13. Thank you for your attention!
nic.nistor@lmu.de
Nistor, Derntl, & Klamma, EC-TEL2015

14. References
LAK 2014 Papers
• 6. Pistilli MD, Willis J, Koch D et al. (eds) (2014) Proceedings of Learning Analytics and Knowledge
Conference 2014, LAK ’14. ACM, New York
• 8. Bogarín A, Romero C, Cerezo R et al. (2014) Clustering for improving educational process mining. In [6],
pp 11–15
• 9. Clow D (2014) Data wranglers: human interpreters to help close the feedback loop. In [6], pp 49–53
• 10. Coopey E, Shapiro RB, Danahy E (2014) Collaborative spatial classification. In [6], pp 138–142
• 11. Fancsali SE, Ritter S (2014) Context personalization, preferences, and performance in an intelligent tutoring
system for middle school mathematics. In [6], pp 73–77
• 12. Gasevic D, Mirriahi N, Dawson S (2014) Analytics of the effects of video use and instruction to support
reflective learning. In [6], pp 123–132
• 13. Hecking T, Ziebarth S, Hoppe HU (2014) Analysis of dynamic resource access patterns in a blended
learning course. In [6], pp 173–182
• 14. Mendiburo M, Sulcer B, Hasselbring TS (2014) Interaction design for improved analytics. In [6], pp 78–82
• 15. Nam S, Lonn S, Brown T et al. (2014) Customized course advising: investigating engineering student
success with incoming profiles and patterns of concurrent course enrollment. In [6], pp 16–25
• 16. Okada M, Tada M (2014) Formative assessment method of real-world learning by integrating
heterogeneous elements of behavior, knowledge, and the environment. In [6], pp 1–10
• 17. Raca M, Tormey R, Dillenbourg P (2014) Sleepers' lag - study on motion and attention. In [6], pp 36–43
• 18. Santos JL, Klerkx J, Duval E et al. (2014) Success, activity and drop-outs in MOOCs an exploratory study
on the UNED COMA courses. In [6], pp 98–102
• 19. Vozniuk A, Holzer A, Gillet D (2014) Peer assessment based on ratings in a social media course. In [6],
pp 133–137
Nistor, Derntl, & Klamma, EC-TEL2015

15. References
EC-TEL 2014 Papers
• 7. Rensing C, Freitas S de, Ley T et al. (eds) (2014) Open Learning and Teaching in Educational
Communities: 9th European Conference on Technology Enhanced Learning, EC-TEL 2014, Graz, Austria,
September 16-19, 2014, Proceedings. Lecture Notes in Computer Science, vol 8719. Springer, Berlin
• 20. Cabielles-Hernández D, Pérez Pérez JR, Paule-Ruiz MP et al. (2014) dmTEA: Mobile Learning to Aid in the
Diagnosis of Autism Spectrum Disorders. In [7], pp 29–41
• 21. González López S, López-López A (2014) Analysis of Concept Sequencing in Student Drafts. In [7],
pp 422–427
• 22. Janning R, Schatten C, Schmidt-Thieme L (2014) Feature Analysis for Affect Recognition Supporting Task
Sequencing in Adaptive Intelligent Tutoring Systems. In [7], pp 179–192
• 23. Loboda TD, Guerra J, Hosseini R et al. (2014) Mastery Grids: An Open Source Social Educational Progress
Visualization. In [7], pp 235–248
• 24. McTavish TS, Larusson JA (2014) Labeling Mathematical Errors to Reveal Cognitive States. In [7], pp 446–
451
• 25. Vahdat M, Oneto L, Ghio A et al. (2014) A Learning Analytics Methodology to Profile Students Behavior and
Explore Interactions with a Digital Electronics Simulator. In [7], pp 596–597
• 26. Niemann K, Wolpers M (2014) Usage-Based Clustering of Learning Resources to Improve
Recommendations. In [7], pp 317–330
Nistor, Derntl, & Klamma, EC-TEL2015