Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Celda 2019 game learning analytics for evidence based serious games final

334 vues

Publié le

The applications of data science techniques to game learning analytics data obtained from serious games can provide a more scientific approach to improve the serious games lifecycle. Honing on the game analytics data is possible to use an evidence-based approach to the design, evaluation and deployment of serious games. For instance, the use of game analytics techniques on the users gameplay interaction data can be applied to systematize the evaluation of games, and allow both teachers and institutions to make better evidence-based decisions. The talk will address some of the new possibilities offered by game learning analytics and what are the requirements (e.g. standards) for its generalization in real settings (including some of the ethical implications).

Publié dans : Formation
  • Soyez le premier à commenter

Celda 2019 game learning analytics for evidence based serious games final

  1. 1. Game Learning Analytics for Evidence-based Serious Games Baltasar Fernandez-Manjon balta@fdi.ucm.es , @BaltaFM e-UCM Research Group , www.e-ucm.es CELDA 2019, Universita degli Studio de Calgari, Sardinia
  2. 2. The Uber game – Financial Times https://source.opennews.org/articles/how-and-why-financial-times-made-uber-game/ https://ig.ft.com/uber-game/
  3. 3. Serious Games • Any use of digital games with purposes other than entertainment (Michael & Chen, 2006) • Applied successfully in many domains (medicine, military) with different purposes (knowledge, awareness) • But still is a low adoption of Serious Games in mainstream education • Serious Games considered usually as a complementary content • Mainly used for motivational purposes • No actual impact on the final mark
  4. 4. https://www.gamified.uk/2016/04/21/simulation-breaks-free-game-thinking/
  5. 5. Fake news, trolls e influencers https://getbadnews.com/
  6. 6. Learning math: fractions http://play.centerforgamescience.com/treefrog/cgs/
  7. 7. Juegos Serios Información sobre el cáncer y su tratamiento Improving adherence to the cancer treatment https://www.re-mission2.org/
  8. 8. Citizen science: use games to contribute solving “difficult problems” 8 Play to Cure™: Genes in Space - a mobile game in which players collaborate to analyse real genetic data (Cancer Research UK, n.d.) http://centerforgamescience.org/portfolio/foldit/ http://www.cancerresearchuk.org/support-us/play- to-cure-genes-in-space
  9. 9. Miller, J., Vázquez-cano, E., & Obligatoria, S. (2015). Exploring Application, Attitudes and Integration of Video Games: MinecraftEdu in Middle School. Educational Technology & Society, 18(3), 114–128. Educational version of commercial games
  10. 10. Do serious games actually work? - Very few SG have a formal evaluation (e.g. pre-post) - Usually tested with a very limited number of users - Formal evaluation could be as expensive as creating the game (or even more expensive) - Evaluation is not considered as a strong requirement - Difficult to deploy games in the classroom - Teachers have very little info about what is happening when a game is being used
  11. 11. Learning analytics • Improving education based on analysis of actual data • Data driven • From only theory-driven to evidence-based “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environment in which it occurs” (Long & Siemens, 2011)
  12. 12. Game Analytics • Application of analytics to game development and research • Telemetry • Data obtained over distance • Mobile games, MMOG • Game metrics • Interpretable measures of data related to games • Player behavior • Mainly used with “commercial purposes”
  13. 13. Business analytics
  14. 14. Game Learning Analytics breaking the game black box model to obtain information while students play. Manuel Freire, Ángel Serrano-Laguna, Borja Manero, Iván Martínez-Ortiz, Pablo Moreno-Ger, Baltasar Fernández-Manjón (2016): Game Learning Analytics: Learning Analytics for Serious Games. In Learning, Design, and Technology (pp. 1–29). Cham: Springer International Publishing. http://doi.org/10.1007/978-3-319-17727-4_21-1. •GLA is learning analytics applied to serious games • collect, analyze and visualize data from learners’ interactions with SGs Game Learning Analytics (GLA)
  15. 15. Uses of Gaming Learning Analytics in educational games • Game testing – game analytics • It is the game realiable? • How many students finish the game? • Average time to complete the game? • Game deployment in the class – tools for teachers • Real-time information for supporting the teacher • Knowing what is happening when the game is deployed in the class • “Stealth” student evaluation • Formal Game evaluation • From pre-post test to evaluation based on game learning analytics??
  16. 16. Generalize and simplify GLA for serious games Realising an Applied Gaming Eco-System
  17. 17. Game Learning Analytics (GLA) or Informagic? • Informagic • False expectations of gaining full insight on the game educational experience based only on very shallow game interaction data • Set more realistic expectations about learning analytics with serious games • Requirements • Outcomes • Uses • Cost/Complexity Perez-Colado, I. J., Alonso-Fernández, C., Freire-Moran, M., Martinez-Ortiz, I., & Fernández-Manjón, B. (2018). Game Learning Analytics is not informagic! In IEEE Global Engineering Education Conference (EDUCON).
  18. 18. Minimun Game Requirements for GLA • Most of games are black boxes. • No access to what is going on during game play • We need access to game “guts” • User interactions • Changes of the game state or game variables • Or the game must communicate with the outside world • Using some logging framework • What is the meaning of the that data? • Ethics: adequate experimental design and setting • Are users informed? • Anonymization of data could be required
  19. 19. Game Learning Analytics
  20. 20. Learning Analytics Model (LAM)
  21. 21. LAM: stakeholders, activities, outcomes
  22. 22. Experience API for Serious Games: xAPI-SG Profile Experience API (xAPI) is a new de facto standard that enables the capture of data about human performance and its context. Now it becoming an IEEE standard The e-UCM Research Group in collaboration with ADL created the Experience API for Serious Games Profile (xAPI- SG), a xAPI profile for the specific domain of Serious Games. The xAPI-SG Profile defines a set of verbs, activity types and extensions, that allows tracking of all in-game interactions as xAPI traces (e.g. level started or completed) The model https://adlnet.gov/news/a-serious-games-profile-for-xapi https://xapi.e-ucm.es/vocab/seriousgames
  23. 23. Java xApi Tracker Unity xApi Tracker C# xApi Tracker Game trackers and cloud analytics frameworks as open code (github) https://github.com/e-ucm Ángel Serrano-Laguna, Iván Martínez-Ortiz, Jason Haag, Damon Regan, Andy Johnson, Baltasar Fernández-Manjón (2017): Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces 50 (2017) 116–123,
  24. 24. Systematization of Analytics Dashboards As long as traces follow xAPI-SG format, some analysis do not require further configuration! Also possible to configure game-dependent analysis and visualizations for specific games and game characteristics.
  25. 25. Real-time analytics: Alerts and Warnings • Identify situations that may require teacher intervention • More complex and fragile, requires a cloud infrastructure • Fully customizable alert and warning system for real-time teacher feedback 07/11/RAGE Project presentation25 Inactive learner: triggers when no traces received in #number of minutes (e.g. 2 minutes) > High % incorrect answers: after a minimum amount of questions answered, if more than # %of the answers are wrong Students that need attention View for an specific student (name anonymized)
  26. 26. uAdventure: xAPI GLA in games authoring uAdventure authoring tool (on top of Unity) • Helps to create educational point & click adventure games • Open code (github) Full integration xAPI-SG game learning analytics into uAdventure authoring tool uAdventure games with default analytics Include geolocalized games https://www.e-ucm.es/uadventure/
  27. 27. Game Development Platform: uAdventure
  28. 28. Search Queries #1: Techniques - Artificial intelligence - Data mining - Machine learning - Data analysis - Deep learning #2: LA - Learning Analytics - Game Analytics - Educational Data Mining #3: Games - Serious Games - Educational Games - Computer Games - Video Games - Games-based learning - Online Games Alternative terms, synonyms, variations...
  29. 29. Analysis and research questions about GLA in SG ● RQ1: What are the purposes for which DS has been applied to LA data from SGs? ● RQ2: What DS techniques have been applied to LA data from SGs? ● RQ3: What stakeholders are the target to benefit from this information? ● RQ4: What conclusions have been drawn from these applications? ● Number of participants (N) ● Education level (e.g. primary, ages) ● Interaction data captured (e.g. times, progress, errors) ● Data format (e.g. xAPI, csv) ● Game purpose (e.g. teach) ● Game subject (e.g. maths)
  30. 30. Results: RQ1 GLA purposes ➔ Main focus: assess learning & predict performance ➔ Games are indeed useful for purposes beyond entertainment ➔ Interest now in analyzing interaction data to measure impact on players and relation to players’ in-game behaviors
  31. 31. Results: RQ2 data science techniques ➔ Linear models and cluster techniques commonly applied ➔ Classical techniques ➔ More powerful techniques (e.g. neural networks) not broadly applied yet ➔ Need of XAI
  32. 32. Results: RQ3 main stakeholders ➔ Purposes that cover interests of many stakeholders ➔ Many research done on this area ➔ Students/Learners indirect recipients of all results
  33. 33. Results: RQ4 conclusions and results Results on assessment & student profiling: ➔ GLA data can accurately predict games’ impact ➔ Performance is related to players’ characteristics Results on SG design: ➔ GLA data can validate SG design ➔ Assessment can & should be integrated in SG design ➔ Importance of SG characteristics ➔ Identified challenges when designing SG ➔ Proposed frameworks to simplify design
  34. 34. Results: Additional information Serious games used: ➔ Main focus to teach ➔ Main domain maths and science-related topics Participants in the validations studies: ➔ Small sample sizes used (<100) ➔ Primary & secondary education Interaction data: ➔ Completion times, actions & scores commonly tracked ➔ Format not reported
  35. 35. Most common methodology are pre-post experiments: Assessment with serious games Is there a significant difference between pre-test and post-test results?
  36. 36. Methodology Use GLA interaction data to predict knowledge after playing. Two steps: 1. Game validation phase: ○ create prediction models taking as input the interaction data ○ validate against actual results (post-test) 2. Game deployment phase: ○ students play and are automatically assessed based on their interactions (used as input for prediction models) ○ pre-post are no longer required We have tested this methodology with a case study.
  37. 37. Research Questions Can we predict student knowledge from pre-test + interactions with the SG? (pre+game) If we can predict it, what prediction models perform best and what variables are most relevant? Q1.2 Can we predict student knowledge solely from interaction with the SG? (game-only)Q2.1 If we can predict it, what prediction models perform best and what variables are most relevant? Q2.2 Q1.1 Is the pre+game condition (Q1.1) more effective than the game-only condition (Q2.1)?Q2.3 Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Predicting students’ knowledge after playing a serious game based on learning analytics data: A case study. Journal of Computer Assisted Learning (in press).
  38. 38. The game: First Aid Game Game to teach first aid techniques to 12-16 years old players Three initial situations: ● chest pain ● unconsciousness ● choking Game previously validated with pre-post and control group: Video-game instruction in basic life support maneuvers. Marchiori EJ, Ferrer G, Fernandez-Manjon B, Povar Marco J, Suberviola Gonźalez JF, Gimenez Valverde A. (2012)
  39. 39. Pre-post experiments + GLA data N = 227 students from a high school in Madrid (Spain) Each student completed: ● pre-test: 15 questions assessing previous knowledge about first aid techniques ● gameplay: of First Aid Game ● post-test: 15 questions assessing knowledge about first aid techniques after playing Collection of both results in pre-post test and GLA interaction data from the game (following xAPI-SG Profile).
  40. 40. Prediction models ● Predict exact post-test score (range 0-15): ○ Regression tree ○ Linear regression ○ SVR (non-linear kernels) ● Predict post-test pass/fail category (pass as 8/15 correct answers): ○ Decision tree ○ Logistic regression ○ Naïve Bayes Classifier All models tested taking as input: pre-test + game GLA interaction data (pre+game condition) only game GLA interaction data (game-only condition)
  41. 41. Pass/fail prediction Score prediction (scale [0-15]) Pre-test? Prediction model Precision Recall MR Prediction model Error mean (SD) Yes (pre+game) Decision tree 81.6% 94.2% 16.2% Regression tree 2.22 (0.55) Logistic regression 89.8% 98.3% 10.5% Linear regression 1.68 (1.44) Naïve Bayes Classifier 92.6% 89.7% 15.1% SVR (non-linear kernels) 1.47 (1.33) No (game-only) Decision tree 88.6% 92.4% 17.3% Regression tree 2.38 (0.62) Logistic regression 87.2% 98.8% 12.7% Linear regression 1.89 (1.54) Naïve Bayes Classifier 89.7% 90.6% 16.9% SVR (non-linear kernels) 1.56 (1.37) Results
  42. 42. Results Can we predict student knowledge from pre-test + interactions with the SG? (pre+game) If we can predict it, what prediction models perform best and what variables are most relevant? Q1.2 Q1.1 Yes, highly accurate results obtained to predict knowledge ● As expected, more accurate predictions when simply predicting pass/fail categories than exact scores. ● Models: logistic regression for binary pass/fail predictions and SVR for score predictions ● Variables: number of interactions with game character, game scores
  43. 43. Results Can we predict student knowledge solely from interaction with the SG? (game- only) If we can predict it, what prediction models perform best and what variables are most relevant? Is the pre+game condition (Q1.1) more effective than the game-only condition (Q2.1)? Q2.1 Q2.2 Q2.3 Yes, highly accurate results obtained to predict knowledge ● Models: logistic regression for pass/fail; SVR for score predictions ● Variables: number of interactions with game character, scores in level Yes, but only slightly. Models in game-only condition still obtain accurate results.
  44. 44. New uses of games based on GLA - Avoiding pre-test: Games for evaluation - Avoiding post-test: Games for teaching and measure of learning With or without pre-test.
  45. 45. GLA Case study: Downtown • Serious Game designed and develop to teach young people with Down Syndrome to move around the city using the subway • Evaluated with 51 people with cognitive dissabilities (mainly Down Syndrome) • 42 users with all data • 3h Gameplay/User • >120K analytics xAPI data (traces) to analyze
  46. 46. Case Study: Downtown • From user requirements to a game design and its observables • Know more about how and what is learn by people with Down Syndrome 47 Ana Rus Cano, Alvaro Garcia-Tejedor, Baltasar Fernández-Manjón (2018): Using Game Learning Analytics for Validating the Design of a Learning Game for Adults with Intellectual Disabilities. The British Journal of Educational Technology
  47. 47. Hyp 1: Users prefer to identify themselves with the avatar • REFUTED • None of the users selected the avatar with Down features despite the trainers showed them the avatar and pointed that that character was Down. • The majority of the users used the preconfigured character despite they were asked to customize the avatars at the beginning of the game session. • We are not observing significative evidences in the users’ play patterns between those who customize the character and those who don’t, but it may be significative that the majority of the users that changed the avatar were Down. 48
  48. 48. Hyp: High-Functioning users do a better performance using the game • To determine the cognitive skills and autonomy of the users we asked the trainers to complete a test about each student • 6 intelectual dimensions were measured (5-point Likert scale) • General cognitive/intellectual ability • Language and communication • Memory acquisition • Attention and distractibility • Processing speed • Executive functioning • Users were divided in two groups: Medium-Functioning (≤ 3 avg.) and High-Functioning (>3 avg.). • MF = 19 (45.2%) HF = 23 (54.8%) 49
  49. 49. Hyp 2: High-Functioning users do a better performance using the game 50 Number of MF users that played each level Number of HF users that played each level
  50. 50. Hyp 2: High-Functioning users do a better performance using the game Average time completing levels for MF Average time completing levels for HF 12:31:21 AM 12:37:02 AM 12:33:16 AM EASY MEDIUM EXPERT 12:32:44 AM 12:28:33 AM 12:36:51 AM 12:25:58 AM EASY MEDIUM HARD EXPERT
  51. 51. Hyp: ID users are engaged and motivated while learning with a videogame 52 12:01:30 AM 12:01:03 AM 12:00:59 AM12:00:59 AM 12:00:50 AM 12:01:00 AM 12:00:36 AM12:00:38 AM 0:00 0:17 0:35 0:52 1:09 1:26 1:44 1 2 3 4 5 6 7 8 • Inactivity times reduced in a 70,7% avg. from session #1 to session #8 • Positive and motivational learning environment (98,2% users show improvement and engagement performing the videogame tasks) Average inactivity time evolution avgtime game session
  52. 52. Hyp: The game design of Downtown is effective as a learning tool 53 • 100% of the trainers agree that the use of Downtown would enhance the user learning adquisition (Perceived Usefulness) • 85,8% of the users were able to follow the right path (both LF and HF) • 50,8% of the wrong path occured during the first 30 min. of playing0 20 40 60 80 100 120 140 160 180 1 2 3 4 5 6 Correct vs Incorrect Path per Game session (#correct stations vs #incorrect stations) count game session
  53. 53. Cyberbullying: Conectado game
  54. 54. Educational desing ● Adventure game- sentiments and emotions are important ● Real situations familiar for students ● Events based on user decision making (but no agression options) ● Scenarios based on research about bullying and cyberbullying ● Different roles of bullying represented ● Designed to be used at classroom
  55. 55. Game mechanics Seminario eMadrid sobre Serious gaes 2017-02-24 56 New student in school Occurs during 5 days Minigames as “nightmares” Implications of the social networks
  56. 56. 1300 12
  57. 57. Significant increase in the ciberbullying perception Wilcoxon paired test, p<0.001 5.72 6.38 Antonio Calvo-Morata, Dan-Cristian Rotaru, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018): Validation of a Cyberbullying Serious Game Using Game Analytics. IEEE Transactions on Learning Technologies (early access)
  58. 58. Validation with teachers and future teachers Evaluation with 84 teachers y 104 educational science students ● Signficant increase in their knowledge ● 99% consider that the game can be used in class VS 1% that do not agree ● 82% willing to use it in their class VS 2% would not use it ● 87% consider the game an effective tool VS 1% that do not agree Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Applicability of a cyberbullying videogame as a teacher tool: comparing teachers and educational sciences students. IEEE Access, DOI: 10.1109/ACCESS.2019.2913573
  59. 59. Simva Simva is a tool for scientific validation of serious games. Goal: to simplify the previously-identified issues: ● Connection with surveys system and learning analytics system ● Management of participants and surveys ● Data storage: questionnaires responses and traces of interactions ● Control while experiments are in play: questionnaires finished, data sent Ivan Perez-Colado, Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez- Ortiz, Baltasar Fernández-Manjón (2019): Simva: Simplifying the scientific validation of serious games. 19th IEEE International Conference on Advanced Learning Technologies (ICALT), 15-18 July 2019, Maceió-AL, Brazil.
  60. 60. Simva: simplifying serious games validation and deployment
  61. 61. Simva ● Validation of serious games is a complex, error-prone process ● Simva tool aims to simplify the possible issues ○ Before the experiments: ■ Managing users & surveys ■ Providing anonymous identifiers to users ○ During the experiments: ■ Collecting and storing all data collected ■ Relating different data from users ■ Allowing additional metadata ○ After the experiments: ■ Simplifying downloading of all data collected
  62. 62. Conclusions • Game Learning Analytics has a great potential for improving SGs • Evidence based serious games • Games as assessments (better “Stealth” student evaluation) • Games as powerful research environments • Still complex to implement GLA in SG • Increases the (already high) cost of the games • Requires expertise not always present in game developers, SME or research groups • Real time GLA is still complex and fragile (e.g. deployment is schools) • New standards specifications (e.g. xAPI) and open software development could greatly simplify GLA implementation and adoption • Ethics should drive the GLA process 63
  63. 63. 64 Thank You! Gracias! ¿Questions? • Mail: balta@fdi.ucm.es • Twitter: @BaltaFM • GScholar: https://scholar.google.es/citations?user=eNJxjcwAAAAJ&hl=en&oi=ao • ResearchGate: www.researchgate.net/profile/Baltasar_Fernandez-Manjon • Slideshare: http://www.slideshare.net/BaltasarFernandezManjon