Presentation at Learning & Student Analytics Conference (LSAC) 2018 "Applications of Learning Analytics to assess Serious Games"

1. Applications of Learning Analytics
to assess Serious Games
Cristina Alonso-Fernández, Ana Rus Cano, Antonio Calvo-Morata,
Manuel Freire Morán, Iván Martínez-Ortiz and Baltasar Fernández-Manjón
Learning & Student Analytics Conference (LSAC)
October 22nd 2018, Amsterdam

2. Introduction
● Learning Analytics (LA)
○ Measurement, collection, analysis and
reporting
○ Data from learners and their contexts
○ To understand and optimize learning and
its environment
● Serious Games (SG)
○ Main purpose other than entertainment
○ Applied in several domains
○ Learn, change attitude, increase awareness
○ Benefits of games applied to learning

3. Game Learning Analytics
Learning Analytics for Serious Games
● Track data while students play
● Transparent to players
● Interactions with games
● Provide feedback at real-time
● Other applications?

4. Applications of LA for SGs
● Validate and deploy games in schools
○ Conectado designed to address social problems (bullying and cyberbullying)
● Validate game design when information cannot be directly gathered from users
○ DownTown designed to improve independent life of users with ID who struggle with
communication issues
● Improve evaluation methods of SGs
○ First Aid Game previously validated; use of data mining models to predict knowledge after
playing

5. Conectado
● Goal: raise bullying and
cyberbullying awareness
● Students 12-17 years old
● Player is placed as victim
● Empathy promoted with mini-games
● Ending determined
by decisions taken
● Tool for teachers to
promote discussion

6. Conectado
● N=257 students
● High-school level
● Three educational centers in Spain
● June 2017
● 1h sessions

7. Conectado
● Pre-test and post-test to measure
awareness
○ 18 7-point Likert questions
○ Bullying and cyberbullying perception
● Interaction data captured
○ Interactions with characters
■ Dialogues
■ Actions
○ Friendship level with characters
○ Areas accessed in-game
○ Time spent in each game day
Awareness before playing
Awareness after playing
Players’ interactions

8. DownTown: A Subway Adventure
● Goal: train in using the subway
transportation system of Madrid
(Spain)
● Players 18-45 years old with ID
● Spy game
● 3D realistic perspective
● Quests to train daily skills
and social aspects

9. DownTown: A Subway Adventure
● N=51 adults with ID
● Down Madrid Association
in Spain
● May-June 2017
● 3h of playing in different
sessions

10. DownTown: A Subway Adventure
● Not possible to gather data from
users directly (no pre-post tests)
● Only interaction data captured
○ Character preferences
○ Accessibility preferences
○ Routes information
■ Choices taken
■ Times
○ Attempts

11. First Aid Game
● Goal: instruct cardiopulmonary
resuscitation (CPR) maneuvers
● Students 12-16 years old
● Three initial situations
● Questions with multiple
situations
● Videos to show the procedures

12. First Aid Game
● N=227 students
● High-school level
● One school in Madrid
● Jan-Feb 2017
● 1h sessions
● Game validated with pre-post experiment and control group in 2011 with
+300 students in 4 schools of Aragon (Spain)

13. First Aid Game
● Pre-test and post-test to measure
knowledge
○ 15 multiple-choice questions
○ First aid techniques knowledge
● Interaction data captured
○ Answers in questions
○ Interactions with
■ game character
■ game elements
○ Scores in 3 levels
○ Completion
Players’ interactions
Knowledge before playing
Knowledge after playing

14. Data Collection: xAPI-SG
https://adlnet.gov/news/
a-serious-games-profile-for-xapi

15. Results: Conectado
● Increase in cyberbullying awareness
○ From 5.72 (SD=1.26) in pre-test to 6.38 (SD=1.11) in post-test, statistically-significant effect.
○ Greater awareness of female before and after playing.
● Most players considered they learned.
● Younger players and women took longer to complete the game.
● Most players ended up in the best ending.
● Teacher dashboard to control the class at real-time
○ Monitor progress
○ Help students with problems moving forward
● Data showed some design problems (e.g. duration in first version)

16. Results: DownTown
● Most students (85.8%) reached a destination.
● Half of the mistakes (50.8%) occurred during the first 30’ of playing, once
students completed a few routes to understand the mechanics.
● Educators hypotheses were contrasted with data:
○ No difference between players who customized avatars and players who did not
○ No influence of previous transportation experience
○ Video game players completed tasks quicker and making less mistakes
● Game design validated:
○ Complexity adjusted to intellectual abilities
○ Error found in number of tasks in levels

17. Results: First Aid Game
● Predictions of post-test knowledge based on interaction data
○ Usually measured by post-test
○ Predictions compared vs actual post-test data
● Predictions carried out as
○ Pass/fail category
○ Exact score in [1-15]
● To further avoid pre-test, predictions
○ With and without pre-test information
● Predicting pass/fail: 89% precision, 98% recall, 10% MR
● Predicting scores: mean error 1.5 (SD=1.33)
● Without pre-test information, slightly worse but not significantly.

18. Summary of results
Three different experiences and results applying LA for SGs:
● Validate and deploy a SG to increase cyberbullying awareness
○ Game reached its goals for all target users
○ Teachers could obtain real-time information
● Validate a SG that trains using the subway without explicit students feedback
○ Game design validated
○ Educators hypotheses contrasted
● Improve evaluation and deployment of SG predicting knowledge after playing
○ High accurate models predict post-test information
○ Models with accurate results also without pre-test

19. Conclusions
● Different uses of GLA data for Serious Games
○ Validation of game design and mechanics
○ Verification of adequateness for all intended target users
○ Data-based testing of educators hypotheses
○ Real-time feedback to keep control of class while games are in play
○ Predictions of knowledge after playing based on interaction data
● Feedback at all stages of game design and deployment
● Importance of Learning Design: games designed to extract information

20. Main References
● Manuel Freire, Ángel Serrano-Laguna, Borja Manero, Iván Martínez-Ortiz, Pablo Moreno-Ger, Baltasar
Fernández-Manjón (2016): Game Learning Analytics: Learning Analytics for Serious Games. In Learning,
Design, and Technology (pp. 1–29). Cham: Springer International Publishing.
http://doi.org/10.1007/978-3-319-17727-4_21-1.
● Antonio Calvo-Morata, Dan Cristian Rotaru, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar
Fernández-Manjón (2018): Validation of a Cyberbullying Serious Games Using Game Analytics. (under review)
● Ana Rus Cano, Álvaro García-Tejedor, Baltasar Fernández-Manjón (2018): Using Game Learning Analytics for
Validating the Design of a Learning Game for Adults with Intellectual Disabilities.The British Journal of
Educational Technology.
● Cristina Alonso-Fernández, Iván Martínez-Ortiz, Rafael Caballero Roldán, Manuel Freire, Baltasar
Fernández-Manjón (2018): Improving serious games evaluation using data mining techniques. (under review).
● DownTown: http://downtown.ceiec.es/
● First Aid Game: http://first-aid-game.e-ucm.es/

21. Applications of Learning Analytics
to assess Serious Games
Cristina Alonso-Fernández, Ana Rus Cano, Antonio Calvo-Morata,
Manuel Freire Morán, Iván Martínez-Ortiz and Baltasar Fernández-Manjón
Learning & Student Analytics Conference (LSAC)
October 22nd 2018, Amsterdam
calonsofernandez@ucm.es