Gamification is a hot topic, but where is the research to back up the use of gamification? Anyone interested in gamification for learning will be interested in seeing empirical results to be better informed about whether or not gamification is appropriate for their learning environment. In this Slideshow, we look at material prepared for submission to a peer-reviewed journal, highlighting correlational results, retention, and memorization data, as well as usage data related to the gamification platform of Axonify. These slides show correlation research related to gamification and highlights empirical results, linking gamification actions to retention and learning results. Data extracted from a database of over 250,000 users of a gamified platform is examined. -How gamification impacts learner engagement -How gamification provides learning retention results -Correlations between learner engagement in a gamification platform and on-the-job performance -How gamification statistics confirm the use of gamification for learning

1. Gamification Research: What
the Numbers Reveal
By:
Karl M. Kapp, Ed.D.
Professor, Instructional Technology
Bloomsburg University
Twitter: @kkapp

2. Lynda.com Course: Gamification of LearningYouTube Video
Web Site: www.karlkapp.com
Books
Getting in touch with Karl
Twitter: @kkapp
Web: www.karlkapp.com
Email: karlkapp.com :
http://www.facebook.com/gamificationLI

3. Co-Researcher
• Deltcho Valtchanov
– An experimental psychologist at the University of Waterloo
Games Institute. His research focuses on how real and virtual
environments can influence cognitive function, and how games
and gamification can be used to motivate positive changes in
behaviors."

4. Agenda
• What did we study?
– Question Driving Research
• How Research Was Conducted
– Procedure/Methodology
– Axonify Platform
• Research Questions
• Results
• Discussion

5. Acknowledgement
• Research was graciously funded through a
grant by:

6. What are we discussing?
• Results of a research study:
– Submitted for peer-review
– Conducted to academic standards
– Not a case study
• Applied Research/Field Study
– Not experimental
– No control group
– Not exactly the same population

7. Question driving research
Is it possible that a casual game can be
used as a “hook” to motivate a learner to
engage with content that is instructional but
not part of the game play.

8. In other words…
Does a casual game encourage learners to
engage with a gamified learning platform
more than asking them questions?

9. Literature Review
• What do we know?
– Several theories support the idea of serious games
positively influencing motivation (Garris et al. 2002;
Ryan & Deci 2000; Ryan, Rigby, & Przybylski 2006 ).
– Wouters et al (2013), found in their metanalysis of 39
game studies that serious games are not more
motivating than other instructional methods used in
comparison groups.

10. Literature Review
• What do we know?
– 55% of teachers use games in the classroom on a
weekly basis (“Teachers Surveyed” 2013)
– 20% of corporations use games for learning
(“Association for Talent Development” 2014)
– Marketplace of learning games or “serious games” is
predicted grow at a compound annual growth rate of
16% reaching an estimated value of $5,448 million
by 2020 (“Serious game” 2015).

11. Literature Review
• What do we know?
– 55% of teachers use games in the classroom on a
weekly basis (“Teachers Surveyed” 2013)
– 20% of corporations use games for learning
(“Association for Talent Development” 2014)
– Marketplace of learning games or “serious games” is
predicted grow at a compound annual growth rate of
16% reaching an estimated value of $5,448 million
by 2020 (“Serious game” 2015).

12. Literature Review
• What do we know?
– Only a third of U.S. employees are engaged at work
– 50.3% report themselves as "not engaged" at work
– 16.8% reported themselves as "actively disengaged"
(Adkins 2016).

13. How was research conducted?
• Two Groups/Conditions
– Game Group
– No Game Group
• Length of Study
– 12 month period of study
• Data Collection
– In place data set
– Data mined from existing database
• Operationally Defined
– Casual Games
– Learning
– Engagement

14. Participants
• Game Group
– Retail Sales Company with 1,908 employees in a that
used casual games as motivation for learning
• No Game Group
– Retail Sales Company with 4,393 employees in a
retail chain that did not use casual games to motivate
learning.
• Note: Unequal distribution of employees between the
two major retail chains was a result of one retail chain
being much larger than the other.

15. Data Collection Axonify Platform
• Software platform was used by multiple clients who both
used casual games to engage learners and those who
did not.
• System collected and stored a variety of data related to
learner interactions on the platform which made it
possible to track and examine the data set anonymously.
• Game and no-game condition identical in every way
except for the one variable of game or no-game.
• Funded research.

27. Operational Definitions
• Learning (Recall)
– “The more one knows (remembers), the more
intellectual competencies one has to draw on for
thinking, problem solving and even
creativity…knowledge and skills are acquired through
memory” (Klemm 2007, p. 63).
– “the ability to remember an idea, material or
phenomenon in a form very close to that in which it
was originally encountered” (After Taxonomy 1956;
Seels & Glasgow 1990).

28. Operational Definitions
• Casual Game
– “In general, casual games involve less complicated
game controls and less complexity in terms of
gameplay” (Loreto & Gouaïch 2010).
• Low Barrier to Entry
• Played in Short Increments (5-20 minutes)
• Reduced Complexity
• Non-punishing game play
– Commercially popular casual games include Angry
Birds, Bejeweled, Diner Dash, Candy Crush and
Microsoft’s Solitaire.

29. Research Question One
• Does the opportunity to play a game impact level of
employee engagement?
• For the purpose of this study, level of engagement was
operationally defined as:
– Number of times a learner returns to the software.
– Number of sessions a learner spent browsing other
related parts of the platform (e.g., Report Card) and
voluntarily chooses to participate in extra learning
opportunities.

30. Research Question Two
• Does a higher level of engagement correlate with
increased learning?
• For the purpose of this study, level of learning was
operationally defined as:
– Number of correct answers provided by learners over
a 12 month period.
– Correct answer streak length
• Remember: learning is defined as: “the ability to remember an idea,
material or phenomenon in a form very close to that in which it was
originally encountered”

31. Null Hypothesis
• There will be no difference in the level of engagement
between the two groups.
• Level of learning of employees allowed to play a game
prior to being presented with content to be learned will
not be different than the level of learning by a group of
employees not allowed to play a game prior to being
presenting with content to be learned.

32. Research Question One: Results
• Does the opportunity to play a game impact level of
employee engagement?
– Using a one-way ANOVA (Analysis of Variance of the Means)
– Learners in the game condition logged in to do their training
significantly more often (M = 108.12 per year) than those in the
no-game condition (M = 71.30) p < 0.001.
– Learners were significantly (51.64%) more motivated to come
back and engage with the learning platform when they could play
a game.
Statistically highly significant is
often stated as P < 0.001 (less
than one in a thousand chance
of being wrong).

33. Research Question One: Results
• Does the opportunity to play a game impact level of
employee engagement?
– Using a one-way ANOVA (Analysis of Variance of the Means)
– Learners in the game condition logged in to do their training
significantly more often (M = 108.12 per year) than those in the
no-game condition (M = 71.30) p < 0.001.
– Learners were significantly (51.64%) more motivated to come
back and engage with the learning platform when they could play
a game.

34. Research Question One: Results
0
20
40
60
80
100
120
Games OFF Games ON
AverageNumberofLoginsin12Months
Condition
p < 0.001
108.12
71.30

35. Research Question One: Results
• Does the opportunity to play a game impact level of
employee engagement?
– A one-way ANOVA revealed that learners in the games condition
viewed their report card significantly more often (M = 9.88) than
learners in the no-games condition (M = 2.96) p < 0.001
– The analysis also revealed that learners in the games condition
volunteered to do significantly more extra training material (M =
6.28) than learners in the no-games condition (M = 3.3) p <
0.001

36. Research Question Two: Results
• Does a higher level of engagement correlate with
increased learning?
– Using a one-way ANOVA the analysis found that learners in the
game condition answered significantly more questions correctly
(M = 413.47) than learners in the no-game condition (M =
145.61) p < 0.001.

37. Research Question Two: Results
• Pearson correlation was used to determine if voluntarily
engaging with the optional learning opportunities found
in the report card and extra training correlated to
answering more quiz questions correctly across the
entire sample of learners.
– A moderate significant correlation was found between interacting
with the report card and the number of correct answers on
quizzes, r(6301) = 0.44, p < 0.001. M
– A strong significant correlation was found between engaging in
optional extra training and the number of correct quiz answers,
r(6301) = 0.67, p < 0.001.

38. Research Question Two: Results
• Pearson correlation was used to determine if voluntarily
engaging with the optional learning opportunities found
in the report card and extra training correlated to
answering more quiz questions correctly across the
entire sample of learners.
– A moderate significant correlation was found between interacting
with the report card and the number of correct answers on
quizzes, r(6301) = 0.44, p < 0.001. M
– A strong significant correlation was found between engaging in
optional extra training and the number of correct quiz answers,
r(6301) = 0.67, p < 0.001.
Pearson's r is a measure of the
linear correlation between two variables X and
Y, giving a value between -1 and +1 inclusive,
where +1 is total positive correlation, 0 is
no correlation,

39. Research Question Two: Results
• A comparison of the correct answer streaks per question,
which ranged from 0 to 2, for learners in the games
condition to those of learners in the no-games condition
using a one-way ANOVA.
– Learners in the games condition had significantly better correct
answer streaks (M = 1.71) than those in the no-games condition
(M = 1.24), p < 0.001.

40. Research Question Two: Results
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
Games OFF Games ON
AverageCorrectAnswerStreakLengthForQuestionsAnswered
Twice
Condition
p < 0.001

41. Limitations
• Two different populations
– Two different sizes
– Two different company cultures
– Unknown efforts to engage employees with platform
• One study
– Further investigation is warranted
• More research is required to determine if this result is
reproducible and what elements contribute to the results.

42. Conclusion
• It appears that the opportunity to play a
casual game prior to learning positively
impacts the level of employee
engagement.
• When a learner has a high level of
engagement with a gamified platform, they
have a higher level of content recall
(learning).

43. 1
References
Abdual Jabbar, I, A & Felicia, P. (2015, December) Gameplay engagement and learning in gam-
based learning: A systematic review. Review of Educational Research December, 85(4), 740–779
doi: 10.3102/0034654315577210
Adkins, A. (2016, January 13). Employee engagement in U.S. stagnant in 2015. Resource
document Gallup. http://www.gallup.com/poll/188144/employee-engagement-stagnant-
2015.aspx. Accessed 01 February 2016.
After Taxonomy of Educational Objectives: Handbook I Cognitive Domain (1956). In B.S. Bloom
(Ed.), Committee of College and University Examiners (pp. 201-207). New York: David McKay
Co..
Association for Talent Development [ATD]. (2014). Playing to Win: Gamification and serious
games in organizations.
Cahill, L., & McGaugh, J. L. (1998). Mechanisms of emotional arousal and lasting declarative
memory. Trends in neurosciences, 21(7), 294-299.
Bourne, L. E., Dominowski, R. L., Loftus, E. F. & Healy, A. F. (1979) Cognitive Processes 2nd
Edition. Englewood Cliffs, NJ: Prentice-Hall.
Casual Games Market Report (2007). Casual Games Association
Casual Game White Paper (2008). IGDA. International Game Developers Association.
Current and future prospects for gamification in the education sector, (2015, July 17). Resource
document Emerging Strategy. http://www.emerging-strategy.com/article/current-and-future-
prospects-for-gamified-learning-in-the-education-sector/. Accessed 02 February 2016.
Csikszentmihalyi, M. (1975). Play and intrinsic rewards. Journal of Humanistic Psychology,
15(3), 41-63.
Dicheva, D., Dichev C., Agre G., & Angelova G. (2015). Gamification in Education: A Systematic
Mapping Study. Educational Technology & Society, 18 (3), 75–88.
Dickey, D. M. (2005). Engaging by design: How engagement strategies in popular computer and
video games can inform instructional design. Educational Technology Research and Development,
53, 67–83. doi:10.1007/BF022504866.

44. 2
Eseryel, D., Law, V., Ifenthaler, D., Ge, X., & Miller, R. (2014). An Investigation of the
Interrelationships between Motivation, Engagement, and Complex Problem Solving in Game-
based Learning. Educational Technology & Society, 17 (1), 42–53.
Gagne, R. M. (1972). Domains of learning. Interchange, 3(1), p. 1-8.
Gagne, R. M. (1977). The Conditions of Learning and Theory of Instruction 4th Edition. New
York, NY: Holt, Rinehart and Winston.
Gibson, J. T. (2009). Discussion approach to instruction. In C. M. Reigeluth & A. A. Carr-
Chellman (Eds.), Instructional-design theories and models, Vol. III: Building a common
knowledge base (pp. 99-116). New York, NY: Routledge.
Gao, Y. & Mandryk, R. L, (2011) GrabApple:The design of a casual exergame. In J. Anacleto et
al. (Eds.): ICEC 2011, LNCS 6972 (pp. 35–46). IFIP International Federation for Information
Processing.
Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and
practice model. Simulation & Gaming, 33, 441–467. doi:10.1177/1046878102238607.
Gee, J. P. (2007). Learning and games. In K. Salen (Ed.), The ecology of games: Connecting
youth, games, and learning (pp. 21-40). Cambridge, MA: MIT Press.
Gibson, J. T. (2009). Discussion approach to instruction. In C. M. Reigeluth & A. A. Carr-
Chellman (Eds.), Instructional-design theories and models, Vol. III: Building a common
knowledge base (pp. 99-116). New York, NY: Routledge.
Gredler, M. E. (1997) Learning and Instruction: Theory into Practice 3rd Edition. Upper Saddle
River, NJ: Merril, an imprint of Prentice Hall.
Greenfield, P. M. (2010). Video games revisited. In R. van Eck (Ed.), Gaming and cognition:
Theories and practice from the learning sciences (pp. 1-21). Hershey, PA: IGI Global.
Hamann, S. (2001). Cognitive and neural mechanisms of emotional memory.Trends in cognitive
sciences, 5(9), 394-400.
Klemm, W. R. (2007). What good is learning if you don’t remember it? The Journal of Effective
Teaching. 7(1), 61-73.

45. 3
Loreto, I. D., & Gouaich, A. (2010) Social Casual Games Success is not so Casual. RR-10017, pp.
001-011.
Malone, T.W. (1981) Toward a Theory of Intrinsically Motivating Instruction. Cognitive Science.
5(4): p. 333-369.
Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomic model of intrinsic
motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning, and instruction:
III. Conative and affective process analysis (pp. 223-253). Hillsdale, NJ: Erlbaum.
Rebetez, C. & Betrancourt, M. (2007, March). Video game research in cognitive and educational
sciences. Cognitie, Creier, Comportament / Cognition, Brain, Behavior. Romanian Association for
Cognitive Science. ISSN: 1224-8398. XI(1), 131-142.
Seels, B. & Glasgow, Z. (1990) Exercises in instructional design. London: Merrill Publishing
Company.
Serious game market size to grow at 16.38% CAGR to 2020 in terms of value say global serious
games research reports focusing on state of play, challenges and industry prospects. (2015,
September 18). Resource document PR Newswire. http://www.prnewswire.com/news-
releases/serious-game-market-size-to-grow-at-1638-cagr-to-2020-in-terms-of-value-say-global-
serious-games-research-reports-focusing-on-state-of-play-challenges-and-industry-prospects-
528208111.html. Accessed 28 March 2016.
Skinner, E., & Belmont, M. (1993). Motivation in the classroom: Reciprocal effects of teacher
behavior and student engagement across the school year. Journal of Educational Psychology,
doi:10.1037/0022-0663.85.4.571
Teachers Surveyed on Using Digital Games in Class. (2013). Resource document Games and
Learning. http://www.gamesandlearning.org/2014/06/09/teachers-on-using-games-in-class/.
Accessed 02 February 2016.
Whitton, N. (2011). Encouraging engagement in game-based learning. International Journal of
Game-Based Learning, doi:10.4018/ijgbl.2011010106
Wouters, P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013, February 4). A
Meta-Analysis of the Cognitive and Motivational Effects of Serious Games. Journal of
Educational Psychology. Advance online publication. doi: 10.1037/a0031311