This document summarizes research on serious gaming conducted by Dr. Igor Mayer of Delft University of Technology. It provides an overview of Mayer's work in developing a comprehensive methodology for researching and evaluating serious games. Some key points: - Mayer has published several papers on developing frameworks and methods for studying serious games and game-based learning across different contexts. - His research aims to establish a science of serious gaming through developing standardized research designs, validated instruments, and a body of knowledge on both the state of the art and gaps in the field. - Mayer's work involves studying serious games from several perspectives including design, evaluation, domain applications, and theoretical lenses like modeling complex systems. He advocates comparative and multi

1. Serious Gaming Research
© Dr. Igor Mayer i.s.mayer@tudelft.nl
Faculty Technology, Policy and Management (TPM) & Signature Games
Delft University of technology, The Netherlands
Challenge the future
1

2. Challenge the future
2

3. References (1)
1. Mayer, I. S., Bekebrede, G., Harteveld, C., Warmelink, H. J. G., Zhou, Q., van Ruijven, T., … Wenzler, I. (2013). The
research and evaluation of serious games: Toward a comprehensive methodology. British Journal of Educational
Technology, n/a–n/a. doi:10.1111/bjet.12067
2. Mayer, I. S., Bekebrede, G., Warmelink, H. J. G., & Zhou, Q. (2013). A Brief Methodology for Researching and Evaluating
Serious Games and Game-Based Learning. In T. M. Connolly, L. Boyle, T. Hainey, G. Baxter, & P. Moreno-Ger (Eds.),
Psychology, Pedagogy and Assessment in Serious Games (in press) (pp. 357–393). IGI Global. doi:10.4018/978-1-46664773-2.ch017
3. Mayer, I. S., Kortmann, R., Wenzler, I., Wetters, Á., & Johan, S. (2014). Game-based Entrepreneurship Education:
Identifying Enterprising Personality, Motivation and Intentions amongst Engineering Students. International Journal of
Entrepreneurship Education (in press).
Challenge the future
3

4. References (2)
1. Mayer, I. S., van Dierendonck, D., van Ruijven, T., & Wenzler, I. (2013). Stealth Assessment of Teams in a Digital Game
Environment. In GALA 2013 Conference, Paris (pp. 1–13). Paris, France: Springer.
2. Mayer, I. S., Warmelink, H. J. G., & Bekebrede, G. (2013). Learning in a game-based virtual environment: a comparative
evaluation in higher education. European Journal of Engineering Education, 38(1), 85–106.
doi:10.1080/03043797.2012.742872
3. Mayer, I. S., Warmelink, H. J. G., & Zhou, Q. (2014). The Utility of Games for Society, Business and Politics: A Frame
Reflective Analysis. In Nick Rushby & D. Surry (Eds.), Wiley Handbook of Learning Technology (in press). Wiley.
4. Mayer, I. S., Wolff, A., & Wenzler, I. (2013). Learning Efficacy of the “Hazard Recognition” Serious Game: A Quasi
Experimental Study. In M. Ma, M. F. Oliveira, S. Petersen, & J. Baalsrud Hauge (Eds.), 4th International Conference, SGDA
2013, Trondheim, Norway, September 25-27, 2013. Proceedings (pp. 118–129). Berlin Heidelberg: Springer Verlag.
doi:10.1007/978-3-642-40790-1_12
Challenge the future
4

5. Review articles
1. Papastergiou, M. (2009). Exploring the potential of computer and video games for health and physical
education: A literature review. Computers & Education, 53(3), 603–622. doi:10.1016/j.compedu.2009.04.001
2. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of
empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686.
doi:10.1016/j.compedu.2012.03.004
3. Vogel, J. J., Vogel, D. S., Cannon-Bowers, J. A., Bowers, C., Muse, K., & Wright, M. (2006). Computer Gaming
and Interactive Simulations for Learning: a Meta-Analysis. Journal of Educational Computing Research, 34(3),
229–243. doi:10.2190/FLHV-K4WA-WPVQ-H0YM
4. Gosen, J., & Washbush, J. (2004). A Review of Scholarship on Assessing Experiential Learning Effectiveness.
Simulation & Gaming, 35(2), 270–293. doi:10.1177/1046878104263544
Challenge the future
5

6. Towards a Science of SG
(SoSG)?
Frames and
discourses
Methodology
Research designs
and data-gathering
Validated research
instruments and tools
A dynamic body of
knowledge identifying
the state of the art
and knowledge gaps.
Professional ethics of
the SG designer, the
SG advocate, the SG
seller, the SG
interventionist, etc.
Challenge the future
6

7. Game theory versus gaming
Game theory
Gaming
Rigid rule-based, closed
Free form, seminar, open
Formal, mathematical, quantifiable, economics,
Informal, social, interpretative and qualitative,
psychology
social, political intervention sciences
Experimental control, objective, separation
Semi- or non experimental, subjective, interaction
researcher and subjects, large „ n‟ , player cannot
between researcher and subjects, small „ n‟ ;
change the rules or setting
players should change the rules or setting
Theory-based, hypothesis testing, generalization,
Theory construction, exploratory, hypothesis
prediction and forecasting
formulation, constructing the future
„ Players‟ are research objects;
„Players‟ are learning subjects; they bring with
them tacit knowledge, social relations etc.
Challenge the future
7

8. Framing game related
research
Frame
Example
Research theory
Game theory as in economics, political science
Research concept
Political decision-making as a strategic game
Research object
Studying game cultures, game economics, game
politics
Design artifact
Socio-technical design etc.
Research method
Quasi-experiment
Intervention method
Therapy, learning, change or decision-method
Data-gathering method
Observation, group interview, data-modeling
Challenge the future
8

9. Strong/weak aspects of gaming as a
research method
Strong aspects
Weak aspects
Flexible, adaptable, multi-purpose,
Time and resource consuming. Difficult to
complementary
manage. No guarantee for success.
Multi-, interdisciplinary, innovative, appealing
Problematic legitimization as „science‟
Research in action, interaction with real
Dependency on others (financers, players,
stakeholder, problem owners, users, etc.
stakeholders, designers).
Deliverable by or end product.
Game takes over the research
Concrete, practical, tangible, fun
A lot of fuzz that is not research
…
…
Challenge the future
9

10. Requirements for SG Meth.
Multipurpose
d
Broad in
scope
Compar
ative
Fast
and
nontime
consumi
ng
Standar
dized
Requirements
Meth. SG
Research
Unobtru
sive
Specific
Expand
able
Flexible
Validate
d
Triangul
ated
Multileveled
Challenge the future
10

11. Underlying research questions
What is the relation between
(virtual) gaming and (professional)
learning in socio-technical systems /
multi-actor contexts?
Design-oriented research (artifact):
‘making it (better)’
Policy, management oriented
research (intervention):
‘making it work’
Domain-oriented research (energy,
rail, water, ports, tunnels etc.):
‘making it matter’
Scientific game research:
‘making it understandable’
• (How) do (virtual) gaming experiences (a)(e)ffect learning of professionals in sociotechnical systems / multi-actor contexts?
• (How) does (virtual) game-based learning (a)(e)ffect Real World (RW) policy-making in
socio-technical systems / multi-actor contexts?
• Development and testing of design and validation theories, methods & tools
• Validation studies of specific and generic game-based artifacts and events.
• Development and testing of game evaluation, measurement and feedback
theories, methods & tools
• Studies into the learning effectiveness of game-based interventions
• Studies into the transfer game-based interventions to the RW
• Studies into the design and use of SG as a research instrument, e.g. e.g. quasi
experimental, questionnaires, video observation, Q-method, etc.
• Studies using game based research in infrastructures.
• Studies on SG in a cultural, organizational, political, economic context
• Theory construction on serious gaming „as‟ / „in‟ socio-technical / multi-actor systems
Challenge the future
11

12. Group
intervie
ws focus
groups
Scenario
analysis
Q
methodo
logy
(particip
atory)
Observa
tion
Docume
nt
analysis
Inter
views
Case study
Participatory
research
methods
Serious
gaming
Path
tracking
Crowd
sourcing
Computer
simulation
Lage
schale
data
analysis
Video
observat
ion
Experimental
research
Bio &
psychoMetric
tests
Panel
research
Surveys
Quasi
experim
ental
research
Eye
tracking
Mayer, I. S., Warmelink, H., & Bekebrede, G. (n.d.). Learning in a Game-based Virtual Environment: a Comparative Evaluation in
Higher Education. European Journal of Engineering Education.
Challenge the future 12

13. Shortcomings
Few indications how to use the models, for what purpose, with what scope and under what conditions.
Few procedures how to validate the conceptual research / evaluation model.
Few research hypothesis and research designs.
Few definitions, relations and interrelations between the concepts in the model.
Few operationalization and validation of constructs.
Furthermore in the application of the models we see:
A dominance of single case studies, one game, one context of application.
Lacking information on the questionnaires used.
A focus on GBL of children in formal education; little attention to advanced-professional learning, outside education;
A focus on learning of individuals in formal training or educational context; little attention to learning of teams, groups, organizations,
networks or systems in policy or organizational context;
Challenge the future
13

14. Research framework
1. Domain application:
infrastructures
1.1 Water, Rail,
Ports, Energy,
Tunnels, etc...
Framing
Research question 1
Sub question 4
Sub question 5
m
Fra
2.1 Complex adaptive
systems, resilience,
integrated planning,
self-organization,
sense-making, etc...
2. Theory: Complex, multi
actor systems and policy
making.
ing
sub question 6
CaseExperiments,
e.g. Levee Patroller,
SimPort, Water Game,
Research question 3
Research question 2
Fra
m
ing
3.1 Design and
evaluation of (Serious)
Gaming-Simulation
3. Method: Modeling,
simulation & gaming
(MSG)
Challenge the future
14

15. Research design
MSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP
When?
Pre-game
Observation
number
O1
How?
Online survey
What?
Soc. Dem.
Involvement in
MSP
In-game
O2
O3
Paper
quest.
Paper
quest.
O4
Paper
quest.
Post-game
O5
O6
End of game
debriefing
Online
survey
Analysis of
maps
MSP
process
MSP
process
Influence
After action
review
41
41
Knowledge in
MSP
Influence in MSP
MSP in country
Valid response
Additional data
gathering
Game play
Emotions
63
50
40
38
Video registration – Observation – Data logging
Challenge the future
15

16. Quasi experimental design
(Mayer, et al. under review EJEE, BJET, 2012)
O1 =
O1 =
X1 =
O3 =
Ox1...n =
O4 =
Intervention process
Case study
O4
Learning proces
In game data logging
O3
X
O2
Transfer process
Observation
O1
Serious Gaming session
Ox1
Ox2
Ox3
Oxn
Observation 1 Learning objectives, context
Observation 2 pre-game measurement
Intervention = gaming session
Observation 3 post game measurement
In game observations / measurements
Observation 4 Learning objectives, context
Time
Challenge the future
16

17. Measuring indicators
MSP Challenge 2011, TU-Delft, ministry I&M, ICES, OSPAR, HELCOM, VASPAP
Quantitative
Qualitative
The level of engagement of the players
in the game.
The influence attributed to each
Self-reported by the participants
stakeholder, by other stakeholders.
The quality of each of the four MSPs as
The lessons and insights on the
process and outcome reported
during and after the game.
assessed by each stakeholder on
ten criteria.
The observations on how the MSP
process goes, by the game
The square nautical miles assigned to
Observed by the facilitators
facilitators.
the different spatial functions in the
The quality of each of the four MSPs.
four MSPs.
The overlap, conflicts, internal
contradictions etc. between
different spatial functions in or
between MSPs.
Challenge the future
17

18. •
References
•
Mayer, I. S., Warmelink, H., & Bekebrede, G.
(n.d.).
Learning
in
a
Game-based
Virtual
Environment: a Comparative Evaluation in Higher
Education. European Journal of Engineering
Education.
•
Mayer, I. S. (2012). Towards a Comprehensive
Methodology for the Research and Evaluation of
Serious Games. Vs games (pp. 1–15). Genoa:
Procedia Computer Science 00 (2012) 000–000.
Retrieved from www.sciencedirect.com
Challenge the future
18

19. Structural equation modelling
(Mayer, et al. under review EJEE, 2012)
.32
MOTIVATION
Would like to
play other SGs
in education.
Attitude GBL
.62
EXPSERGAME_PRE
.22
.18
Attitude GBL
.2*
EXPSERGAME_POST
.78
.42
Engagement
.28
.34
FUN
.36
Learning
expectations
ACHIEVEMENT
Quality VLE
SOCSKILLS_PRE
.2
.12*
.34
.17*
.18*
Attitude VLE
VALUEDL
.13
PROFSKILLS_PRE
.22
.2 (t-test)
.24
.25
.14
SEX
.18
.1
-.17
FREQ. DIGITAL
.16
.12*
GAMES
Learning
style
preferences
.17*
CYBIMPRESS;
.17
EXPKNOW_PRE
.24
.26
.28*
.40
RELGAME
.39
.48
PROFSKILLS_POST
.27
.27
.39
.38
.46
.39
EXPKNOW_POST
.58
LRNPREF
0.11
.3
FREQ NON-DIGITAL
GAMES
.1
FREQ SG
.53
.43
Quality
facilitator
QUAFACIL.
EFFORTPLAY
.44 .46
.41
Quality game
design
QUAGAME;
Game play
QUAPLAY;
SOCSKILLS_POST
.26
CYBCONTENT
.51
.19 .24
.27
1.1 (t-test)
.34
CYBACCESS;
.38
.42
AGE
CYBFUNCTION;
CYNCLEARNESS;
.45
.16*
-.12
Learning
satisfaction
Q game
Challenge the future
19