Simplifying Serious Games Authoring and Validation with uAdventure and SIMVA

1. Simplifying Serious Games
Authoring and Validation
with uAdventure and SIMVA
Pérez-Colado Iván José, Pérez-Colado Víctor Manuel, Martínez-Ortiz Iván, Freire-Morán Manuel,
Fernández-Manjón Baltasar
Faculty of Informatics, Complutense University of Madrid
Madrid, Spain
@baltaFM
balta@fdi.ucm.es

2. Index
1. INTRODUCTION
2. AUTHORING AND VALIDATION OF SERIOUS GAMES USING
e-UCM TOOLS
3. UNIFIED AUTHORING AND VALIDATION EXPERIENCE
4. CONCLUSIONS AND FUTURE WORK

3. e-UCM work in analytics and games
● Simplifying the development and application of serious
games
● Decrease the technical knowledge required to apply games in
the classroom
● uAdventure: a game authoring platform evolved from
eAdventure
● RAGE Analytics: assessment of SGs -> using game learning
analytics (GLA)

4. uAdventure authoring tool (on top of Unity)
● Helps to create educational
point & click adventure games
● Open code (github)
Full integration xAPI-SG game learning analytics
into uAdventure authoring tool
uAdventure games with default analytics
Include geolocalized games
uAdventure: GLA in games authoring
https://www.e-ucm.es/uadventure/
https://github.com/e-ucm/uAdventure/releases/tag/v1.1.0-preview
(now running on Unity 2019 LTS)

5. Game Learning Analytics with SIMVA
● Applying GLA to serious games is complex, error-prone, and
fragile
○ Any small glitch can cause the whole process to fail
○ These issues hinder GLA adoption by non-technical users
● Simva tool aims to simplify the possible issues
○ Before the experiments:
■ Managing users & surveys
■ Providing anonymous identifiers to users
○ During the experiments:
■ Pretest-Game-Postest
■ Collecting and storing surveys and traces data (xAPI-SG)
■ Relating different data from users (GLA, tests)
○ After the experiments:
■ Simplifying downloading and analysis of all data collected

6. uAdventure: AUTHORING OF SERIOUS GAMES
1st step: Serious Game Design phase
● Game Narrative and in-game assessment
○ how students will be assessed through the game
○ having in mind the game mechanics of the uA authoring tool
○ players can explore scenarios, participate in conversations by
choosing responses, and ultimately solve logical puzzles
involving objects and interactions with other characters
2nd step: Game Development
● Creating the game with uAdventure

7. 3rd step: Validation of game effectiveness
● Pretest-game-posttest experiments
● With SIMVA
○ creation of one or more experimental users’ groups.
○ managing and tracking the state of pre- and post- game tests (surveys)
○ collecting GLA data during and after the gameplay.
● Steps of game validation using SIMVA:
1. Design and develop the game (already done)
2. Author the pre and post surveys
3. Set up an experimental group: randomly-generated tokens -> Labeling
a. ONLY allow participation of players with valid tokens
b. PDF document: list of users’ token codes -> physically distributed
SIMVA allows to control what is happening in the experiments.
A B
SIMVA: VALIDATION OF SERIOUS GAMES

8. VALIDATION OF SERIOUS GAMES
● NOW Steps in game validation using simva:
1. Analytics integration: tracker component:
a. uA: automatically generates and sends xAPI-SG traces with game events
b. Analytics server: RAGE analytics -> dashboard with real-time insights
c. SIMVA automates users access to tracking into server
d. Manual tasks (may lead to a failure):
i. register the game, create a course set up the activity
ii. Link the game using the Analytics tracking code
2. After Game development + Survey creation + Analytics setup
a. Collect Survey IDs, Collect Tracking code, Include all into the game
b. Create additional uA game login scenes that link uA with SIMVA
LOGIN - linking uA game
with SIMVA
PRE-test survey POST-test survey
Gameplay
xAPI analytics

10. UNIFIED AUTHORING AND VALIDATION EXPERIENCE
GLA configuration and setup process
Lessons learnt in the experiments:
● GLA can be challenging for less technically-oriented users
● Several critical points: small errors can compromise the
entire validation process
○ e.g. missing the tracking code will lead to full failure
Ideally: setting all up with a single "all-in-one button"
● By pressing it, any uA-authored SG would be ready to be validate
Implemented: Simple wizard to guide the three validation steps:
1. Choosing the type of study to be set up
2. Defining the number of users that will participate
3. Selecting the analysis features needed

12. UNIFIED AUTHORING AND VALIDATION EXPERIENCE
Behind the scenes: uA requests SIMVA to configure and integrate all required tools.
1. Create a teacher account in SIMVA (or use a preexisting one)
2. Generate students’ accounts using unique random tokens
3. Create a study group including both students and the teacher
4. Create, within the study group, a study of the selected type
5. If surveys included: choose one of the templates or create a new one
6. Chose analytics capabilities:
a. store the xAPI-SG traces (TRACE STORAGE)
b. RAGE Analytics that will be configured (REALTIME)
7. Create intermediate game scenes: Login -> Survey -> Gameplay -> Survey
After the experiment is carried out, all gameplay traces and survey results (if
applicable) will be downloadable from both uA and SIMVA.

13. CONCLUSIONS
● Integration of uAdventure, SIMVA, and RAGE Analytics seeks to
provide a much simpler, robust and integrated flow
● To avoid human error and simplify:
○ Management and authoring of surveys in SIMVA (No more LSS files)
○ Analytics framework setup in SIMVA (Game, tracking code…)
○ Integration of scenes that bind uA and SIMVA together has been
automated, including all end-user access control.
+ + =
SIMVA open code: https://github.com/e-ucm/simva-infra

14. ● Make GLA more accessible to data scientists
○ Making xAPI “transparent” to analysts
○ Integration with Jupyter Notebooks (python)
○ https://github.com/e-ucm/xapi-sg-processor
● Partial automation of the analysis of pre-post surveys
○ Finding and filtering outliers
● Working on a mechanism to streamline deployments
○ complete a full evaluation by following a single URL, jumping from
one activity to the next
○ simplify integration with deployments from LMS (e.g. Moodle)
WORK IN PROGRESS AND FUTURE WORK

15. THANKS!!
Baltasar Fernández-Manjón @baltaFM
balta@fdi.ucm.es
https://github.com/e-ucm/simva-infra
https://github.com/e-ucm/uAdventure/releases/tag/v1.1.0-preview