1.
INVESTIGATION PLAN
JANUARY 15, 2009
EYE MOVEMENT ANALYSIS
FOR AN AUGMENTING VIDEO
GAME DEVELOPMENT
Departamento de Comunicação e Arte
Universidade de Aveiro
Mestrado em Comunicação Multimédia
Samuel de Jesus Almeida
sja@ua.pt
3. Abstract
The video game industry generates millions of dollars every year and that income is continually growing
(Hartig, n.d.). Video game development companies, either new or strongly established, have the
constant need to develop quality video games that entertain their target audience. However, for many
video games, during the development process one of the most important aspects of product
development is overseen – usability evaluation – resulting in less enjoyable or playable games.
Nevertheless, research is being conducted and solutions that meet this problem are arising.
The project aims to deliver a set of guidelines that can be used in the video game development process;
with that in mind, the execution of a case study will take place. A set of participants will be chosen and
divided into three groups. The participants will be asked to play a video game while the eye tracker
registers their eye movements. Following their interaction with the eye tracker system, the participants
will then be questioned on their feelings and opinions about their participation through the use of a
questionnaire. The quantitative and qualitative data acquired will then be analyzed and if thought to be
relevant and applicable, written up as guidelines for augmenting the video game development process.
Finally, qualitative and quantitative data is expected to be acquired through the participant’s
involvement so that the project’s main finality is met: the development of a list of guidelines that can
assist in the video game development process.
Samuel de Jesus Almeida | sja@ua.pt | Abstract 3
4. Characterization of the Investigation Problem
In an industry where millions of dollars are spent and generated every year, the need for existing and
surfacing video game development companies to secure their portion of the market is greater than ever.
A PricewaterhouseCoopers’ study (Hartig, n.d.) suggests that by the following year, with the arrival of
next‐generation consoles and handsets, worldwide video game sales could climb to over 45 billion
dollars. Acquiring a share of that revenue is crucial for video game development companies if they
intend to remain active in the industry. One of the main factors that can help in attaining this goal is the
development of a well planned video game.
Some videogames and companies (e.g. Electronic Arts1) have established their proper status over the
past years. However, for new coming companies and game titles, quality makes all the difference. To
achieve success, a video game must be appealing to the consumer. This is done by delivering quality on
all game aspects such as the game play, the mechanics, the interface (Clanton, 1998), the interaction as
well as the experience.
The demand to deliver successful games, however, results occasionally in the development of weak
games in terms of gameplay, mechanics or interface. This in part is due to the fact that many video
games are developed and released without having been, or improperly evaluated. This represents, in
fact, one of the biggest problems in the video game industry. To overcome the mentioned limitations,
research and other projects are being conducted that propose solutions and strategies for developing
more quality video games.
Nevertheless, much of the existing research focuses on the use of solutions that are traditionally used in
regular product and system assessment. Despite this approach not being inadequate, it is, on some
levels, limited. Video games are products of a specific nature and require specific evaluation and
developmental guidelines. The use of game heuristics, based on existing product and system evaluation
heuristic guidelines, while being useful, do not attend to all existing game usability issues as well as
other video game aspects previously mentioned.
For that motive, it is necessary to look into other technologies and solutions that can help in the
overcoming of the referred limitations. The use of eye tracking as a method to evaluate video games is
one of those options. Eye trackers provide data that can be useful for evaluating certain aspects of video
games. However, despite it being a useful support, it may not be assumed that this technique alone can
solve all usability evaluation problems.
Bearing in mind the questions and concerns mentioned above, the following problem is proposed:
In what way can the analysis of the ocular globe movement contribute to the development
of guidelines that can assist in the evaluation of enjoyment in video games?
The reason for proposing the evaluation of enjoyment rather than usability relates to the specificity of
video games as mentioned above. Due to their nature, games are products developed for leisure
purposes as opposed to practical everyday uses. For that reason, it seems reasonable that the
enjoyment they offer through their gameplay require that their particular mechanics and interface be
assessed. As stated, eye tracking alone cannot solve all the problems and nor present all the solutions
for the mentioned problem. Nonetheless, eye tracking may be suggested as a valid option for
discovering usability and other specific game problems especially if used alongside other usability
evaluation techniques such as the think‐aloud protocol and questionnaires.
1
Electronic Arts: http://www.ea.com
Samuel de Jesus Almeida | sja@ua.pt | Characterization of the Investigation Problem 4
5. With that in mind, the project to be developed will attempt to meet the limitations of some of the
existing methodologies used in game evaluation through the use of an eye tracking system. A case study
will be implemented where participants will be asked to play a game and their eye movements will be
registered. Through the analysis of that data source as well as others, it is expected that a list of
guidelines for augmenting video game development be written and validated.
Samuel de Jesus Almeida | sja@ua.pt | Characterization of the Investigation Problem 5
6. Project Finality and Objectives
This project presents a series of objectives that meet the needs of the project’s problem as mentioned
above. These are:
• To identify the potentialities associated to the use of eye tracking technology as a method of
usability evaluation;
• To identify currently used methods for evaluating usability in a video game context;
• To adapt usability evaluation methods to assess enjoyment in video games;
• To identify and build a study case;
• To evaluate and validate the use of guidelines that can assist in the evaluation of video game
enjoyment.
The project’s overall finality is, as briefly presented in the last point, to develop a list of guidelines that
can be used in video game development as well as usability studies.
Additionally, and projecting future developments, one of the finalities is to establish a basis for the
development of a methodology that can be used for video game development.
Samuel de Jesus Almeida | sja@ua.pt | Project Finality and Objectives 6
7. State of the Art
Human Visual System
Extracting light from the world, filtering it and transforming it into something understandable is the goal
of the visual system. The human visual system is a complex structure that is composed of various
elements such as the cornea, the iris, the pupil as well as the retina. Of the mentioned, it is the retina
and its components that complete the vision process. The retina is composed of three main layers
(Hubel, 1995, p. 36) and it is at the innermost layer where we find the rods and cones (Bianco, 2000).
When light hits the retina, a complex sequence of chemical reactions occur producing the chemical
activated rhodopsin that in turn generates electrical impulses that are transmitted to the brain and
interpreted as light (Bianco, 2000). The eye is also capable of various eye movements. However, these
are only possible through the work of six extraocular muscles that work in pairs: medial and lateral recti
(sideways movements); superior and inferior recti (vertical movement) and the superior and inferior
obliques (responsible for twist) (Hubel, 1995). As stated, the eye is capable of executing the following
eye movements: saccadic, smooth pursuit, vergence, vestibular and fixations.
Eye Tracking Systems
In the past few decades, the technological community has witnessed the rise of a new system used in
the most various fields: the eye tracker.
The eye tracker is presently the most common of the devices used for determining eye movement. Two
types of eye movement techniques are generally considered: the technique that measures eye’s
position relative to the head and second, the technique that measures the orientation of the eye in
space, also known as the point of regard (Young & Sheena, 1975 apud. (Duchowski, 2007, p. 51)). The
eye tracking system presents both limitations and potentialities. Jacob and Karn’s (Jacob & Karn, 2003)
research presents three possible reasons for the technology’s slow start in usability research: technical
problems with eye tracking in usability studies, labor‐intensive data extraction and difficulties in data
interpretation. In what concerns technical issues, despite the existence of table top and head mounted
systems, both impose some limitations in terms of comfort and movement restrictions. As for data
extraction, if not previously decided what data to analyze, the quantity of data eye trackers produce can
become large (Jacob & Karn, 2003, p. 579). Finally, despite the previous two limitations not always
posing a threat, interpreting the data collected still is one of eye tracking’s most problematic barriers
(Jacob & Karn, 2003). Nevertheless, eye tracking does have its strengths. Keith Karn (2006) defends the
use of eye tracking in studies of human‐computer interaction where its benefits have been previously
demonstrated. That is the case for visual search, learning, visibility of product features and analysis of
certain tasks where other traditional usability testing methods have found a problem. Furthermore,
Karn presents a list of benefits associated to the use of eye tracking for usability evaluation (Karn, 2006).
The list covers technologically related benefits that centre on the equipment’s cost as well as the
growing development of tools to assist in data analysis.
Usability
There is no one correct definition for usability. However, many of the existing definitions converge to
the idea that usability measures the quality of a user’s experience when interacting with a product or a
system2. The International Organization for Standardization (ISO) norm 9241‐11 defines usability as the
extent to which a product can be used by specified users to achieve specified goals with effectiveness,
2
http://www.usability.gov/basics/whatusa.html#whatdoes
Samuel de Jesus Almeida | sja@ua.pt | State of the Art 7
8. efficiency and satisfaction in a specified context of use3. In this definition, there are three main concepts:
effectiveness, efficiency and satisfaction. A second definition is Jakob Nielsen’s4 approach with a 5
component definition. The 5 components are, as Nielsen (Nielsen, 2003) himself describes: learnability,
efficiency, memorability, errors, and satisfaction.
Just as important as knowing what attributes to measure when developing a usable product, are the
methods used to evaluate a product’s usability once it has been completed. There are various
techniques that can be applied and used (Dix, Finlay, Abowd, & Beale, 1998, p. 406). Dix et al. (1998)
present two styles of evaluation: laboratory studies and field studies. Further on, they separate design
evaluation from implementation evaluation. The authors present the well‐known cognitive walkthrough
and heuristic evaluation as examples of design evaluation and divide implementation evaluation into
three possible methods: the empirical method: experimental evaluation; observational techniques and
query techniques. Shneiderman (Shneiderman, 1998) also presents a grouping of usability evaluation
methods: expert reviews, usability testing and laboratories, surveys, acceptance tests, evaluation during
active use, controlled psychologically oriented experiments, practitioner’s summary and researcher’s
agenda.
Video games and usability are two concepts that are not easily matched. In Melissa Federoff’s (Federoff,
2002) study, she discovers that even for people in the video game industry, the concept of game
usability is unused or even unknown. This problem, nevertheless, has not passed unanswered among
HCI research and studies. Chuck Clanton (Clanton, 1998) divided some of the core game usability issues
into three categories: game interface, game mechanics and game play. Pinelle, Wong and Stach (Pinelle,
Wong, & Stach, 2008) approach this question in similar terms. Their work attempts to counter the lack
of existing methods used to evaluate the usability of video games, specifically, the game interface.
Despite the fact many authors and researchers support the use of a method such as an adapted
heuristic evaluation for evaluating the usability of games; others critique the use of traditional
evaluation methods for the assessment of a product. Cockton and Woolrych (Cockton & Woolrych,
2002) defend that so‐called discount methods (heuristic evaluation included) aren’t safe and that they
can and should be improved. They characterize these methods as ones that cut corners and that are
used in the hope that using some usability method is better than having used no method at all.
Video Games
There are many thousands, if not more, video games. These in turn are classified with a genre that,
according to various authors, is the result of the type of interaction that the game presents. Chris
Crawford (Crawford, 1997), stated that, “It is (….) impossible for us to devise a single, absolute
taxonomy.” Two of the studied taxonomies (Crawford, 1997) & (Wolf, 2000) group genres into distinct
categories. Crawford’s taxonomy divided games into two major categories, Skill‐and‐Action Games and
Strategy Games whereas Wolf lists 42 different types of video game genres that, despite not being
grouped together, are presented as being related to each other.
Two of the most successful of the genres mentioned are First Person Shooter Games and the Role
Playing Game (RPG) genre. Considering FPS as a subgenre of the Shooter genre (Rollings & Adams, 2003)
one of the most distinguishable characteristics of the genre is their direction for violence as a main
game mechanic. However, this doesn’t imply (even though the genre name suggests it) that Shooter
3
http://www.upassoc.org/usability_resources/about_usability/definitions_of_usability.html [Consulted:
January 5, 2009]
4
Jakob Nielsen is a renown usability consultant, holding a Ph.D. in Human‐Computer Interaction from
the Technical University of Denmark.
Samuel de Jesus Almeida | sja@ua.pt | State of the Art 8
9. games involve constant shooting. Role‐playing games, on the other hand, have two core components:
configurable player‐characters that improve with experience and strong storylines. Other features
include the act of exploring as well as combating (Rollings & Adams, 2003). The authors consider these
two last features of secondary nature describing them as a means to an end.
Video Game Evaluation
The need to deliver more enjoyable games has resulted in projects and research (eye tracking and non
eye tracking related) that meet this requirement.
Without Eye Tracking
Melissa Federoff’s Heuristics Study
Melissa Federoff’s study, Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in
Video Games (Federoff, 2002), proposes a list of heuristics and suggestions that could be used by game
developers to better develop and design video games. To do so, she examined implicit and explicit
heuristics and usability evaluation processes utilized by a leading game developer (Federoff, 2002).
Desurvire, Caplan & Toth’s Heurstics Study
Desurvire, Caplan and Toth’s study, Using Heuristics to Evaluate the Playability of Games (Desurvire,
Caplan, & Toth, 2004), focuses, much like Melissa Federoff’s study, on the use of heuristics to evaluate
video, computer and board games. They present Heuristic Evaluation for Playability (HEP), a set of
heuristics used to evaluate video, computer and board games and tested to understand their validity
and real effectiveness compared to standard user testing methodologies.
The Tracking RealTime User Experience Study
Tracking Real‐Time User Experience (TRUE): a comprehensive instrumentation solution for complex
systems (Kim, Gunn, Schuh, Phillips, Pagulayan, & Wixon, 2008) takes video game analysis to a new
level. Their case studies as well as their system, TRUE; demonstrate how they have developed a
methodology to improve the design of video games.
With Eye Tracking
ElNasr and Magy Seif’s Study
Magy Seif El‐Nasr and Su Yan’s study, Visual Attention in 3D Video Games (El‐Nasr & Yan, 2006), defends
the current need to understand visual attention patterns within an interactive 3D game environment as
it is an important area of research to better improve video games (El‐Nasr & Yan, 2006).
Johansen, Norgaard and Rau’s Study
Sune Alstrup Johansen, Mie Norgaard and Janus Rau’s study, Can Eye Tracking boost usability evaluation
of computer games, explores how eye tracking can address three core challenges faced by computer
game producer IO Interactive (IOI) from Denmark in their on‐going work to ensure games that are fun,
usable and challenging. These challenges are: 1. Persuading game designers about the relevance of
usability results; 2. Involving game designers in usability work; 3. identifying methods that provide new
data about user behaviour and experience (Johansen, Nørgaard, & Rau, 2008).
Samuel de Jesus Almeida | sja@ua.pt | State of the Art 9
10. Methodology
The methodology that will be applied during this project will be the exploratory method. The
methodology has already commenced with the elaboration of the first part of the project – state of the
art – that is, the collection of facts and information related to the area of usability and video games. The
state of the art part was elaborated through the use of bibliographical research. Furthermore, the data
collection procedure will be executed through the elaboration of a case study where participants will
play a video game so that eye tracking data may be acquired.
In terms of data collection, two techniques will be applied: inquiry and observation. First, a technique on
its own, an eye tracking system – Tobii T60 Eye Tracker – will be used to collect quantitative data related
to user eye movements.
Second, the inquiry technique will be applied with the use of the questionnaire instrument. The
observation technique will also be taken into consideration so that eventual problems and information
observed during the case study be written for further analysis.
In what concerns the study participants, the study will consist in a minimum of 15 participants. For
typical experimental evaluation studies, the number of participants used should be in enough quantity
that they can stand in as a representation of the general population (Dix, Finlay, Abowd, & Beale, 1998,
p. 416). As the general population of video game players is vast, a minimum number of 15 participants
will be used. Furthermore, video game playing abilities vary from player to player. For that reason,
participants with different aptitudes for playing games will be selected and further divided into three
different groups: novice gamers, regular gamers and hardcore gamers. The process of dividing
participants into three distinct groups will be done through the use of a questionnaire to assess each
participant’s history and comfort with playing games as well as other valuable information.
The next part of the methodology consists of the participants familiarizing themselves with the
technology so that the final study produces more reliable results. To do so, participants will be asked to
play a game with some similarity to the case study video game used. This will allow for them to get
accustomed to the head movement limitations the eye tracker imposes. Once all participants feel they
are comfortable with the eye tracker, the final study will take place.
The study will consist in the participants playing a video game (the game is yet to be determined) during
a predetermined amount of time. The complementary use of the think‐aloud process while the players
are interacting with the game is a possibility. However, it is thought that the use of this technique
simultaneously with the eye tracker influences the participant’s interaction and cognitive workload
(Johansen, Nørgaard, & Rau, 2008). Bearing that in mind, a retrospective think aloud method will be
used to obtain information on what the participant thought of his or her participation. Additionally, it is
plausible that a questionnaire be used at the end of the participation to collect other pertinent
information related to the user’s participation in the study: thoughts, feelings and opinions as well as
personal suggestions.
The next step in the project’s methodology consists in collecting all the data and analyzing it.
Quantitative eye tracking metrics such as fixations and saccades (Jacob & Karn, 2003) will be questioned
to understand in what context they took place. In addition, qualitative results acquired through the
post‐study questionnaires and retrospective think‐aloud instrumentation will be thoroughly looked at to
understand in what way both quantitative and qualitative data are linked.
The final and most challenging element to be completed in the methodology is the development of
guidelines that can assist in augmenting video game development. Making sense of all the data – both
Samuel de Jesus Almeida | sja@ua.pt | Methodology 10
12. Chronogram
September October November December January February March April May / June
14/09 ‐ 4/10 05 to 18 19 to 01 02 to 15 16 to 29 30 to 13 14 to 27 04 to 17 18 to 31 01 to 14 15 to 28 01 to 14 15 to 04 05 to 18 19 to 02 03 to 16 17 to 3 31
General
Reunions with Supervisors
First contact with eye tracker
Phase 1
Research
Writing of the State of the Art
Phase 2
Preperation for First Teste Study
Selection of the participants
Execution of the First Teste Study
Analysis of First Test Study Results
Phase 3
Preperation for Final Study
Execution of Final Study
Analysis of Final Test Study Results
Phase 4
Elaboration of proposed Methodology
Phase 5
Writing of remaining diss. Sections
Phase 6
Preperation for Final Hand in
Hand ins / Presentations
The months of September to mid‐January will be dedicated exclusively to research and the elaboration of the State of the Art. The remaining months will be used to
conduct the Case Study as well as update the State of the Art.
Samuel de Jesus Almeida | sja@ua.pt | Chronogram 12
13.
Expected Results
Considering the project’s objectives and finality, the project can culminate in a set of results that will be
discussed in the following lines.
It is expected that potentialities of using an eye tracking system confirm that in fact the technology,
solely or in parallel with other methodologies, can be used as a method of evaluating usability.
Furthermore, it is expected that current research, methods and projects that evaluate usability in a
video game context be found and explored.
Additionally, in what regards the participant’s contribution to the study, it is expected that the
categorization of the participants into three different categories produce results that suggest that
different gaming experiences result in distinct quantitative data when using the eye tracker. Based on
the possible hypothesis that gamers with different gaming experience play video games in distinct forms
(namely in terms of their eye movements), it is expected that the quantitative data the eye tracker
generates demonstrates that there are similarities in how fixations and saccades are registered between
players with similar experience and differences between players with distinct experience.
A greater expected result is one that fulfills the project’s core finality: a list of guidelines that can assist
in the evaluation of video games. For this to be possible, the participants’ qualitative and quantitative
results must show some consistency related to the existence of a certain problem in the case study
video game. If a significant number of the participants’ results indicate an issue related to the game then
it is plausible that there is a usability or game problem that could be fixed.
Also, what is expected is that the case study proves that the use of eye tracking is a valid method for the
evaluation of video games based on the data that it provides and that can be used for augmenting video
game development. This is to say that it is awaited that eye tracking be recognized as a valid instrument
in the assessment of video games, whether its contribution to video game evaluation or development
be of greater or less weight.
Finally, if the collected data proves to be of great value, it can also be expected that with further
experimental studies and research, a game development methodology can be developed and validated
for the video game development community and industry using, for that purpose, the guidelines to be
proposed.
Samuel de Jesus Almeida | sja@ua.pt | Expected Results 13
14. References
Bianco, C. (2000, April 01). How Vision Works. Retrieved December 26, 2008, from HowStuffWorks.com:
http://www.howstuffworks.com/eye2.htm
Clanton, C. (1998). An Interpreted Demonstration of Computer Game Design. Retrieved January 9, 2009,
from ACM Portal: http://portal.acm.org/citation.cfm?doid=286498.286499
Cockton, G., & Woolrych, A. (2002, September). Sale must end: should discount methods be cleared off
HCI's shelves? Retrieved January 09, 2009, from ACM Portal:
http://portal.acm.org/citation.cfm?id=566981.566990
Crawford, C. (1997). A Taxonomy of Computer Games. Retrieved November 11, 2008, from The Art of
Computer Game Design by Chris Crawford: http://www.vancouver.wsu.edu/fac/peabody/game‐
book/Chapter3.html
Desurvire, H., Caplan, M., & Toth, J. A. (2004). Using heuristics to evaluate the playability of games.
Retrieved December 12, 2008, from ACM Portal:
http://portal.acm.org/citation.cfm?id=985921.986102
Dix, A., Finlay, J., Abowd, G., & Beale, R. (1998). Human‐Computer Interaction (Second ed.). Prentice
Hall.
Duchowski, A. T. (2007). Eye Tracking Methodoloy: Theory and Practice (Second ed.). Springer.
El‐Nasr, M. S., & Yan, S. (2006). Visual attention in 3D video games. Retrieved December 23, 2008, from
ACM Portal:
http://portal.acm.org/citation.cfm?id=1178823.1178849&coll=GUIDE&dl=GUIDE&CFID=17728904&
CFTOKEN=70883066
Federoff, M. (2002). Heuristics and Usability Guidelines for the Creation and Evaluation of Fun in Video
Games. Retrieved Novembro 20, 2008, from Melissa Federoff: thesis:
http://melissafederoff.com/heuristics_usability_games.html
Hartig, K. (n.d.). Next‐generation consoles drive global game revenues to more than $45 billion by 2010.
Retrieved November 27, 2008, from PricewaterhouseCoopers:
http://www.pwc.com/extweb/pwcpublications.nsf/docid/C987CEB2D179131F852572090083B4B3/$
FILE/VideoGames_KH_ls.pdf
Hubel, D. H. (1995). Eye, Brain and Vision. New York: Scientific American Library.
Jacob, R. J., & Karn, K. S. (2003). Eye Tracking in Human–Computer Interaction and Usability Research:
Ready to Deliver the Promises. In R. Radach, J. Hyona, & H. Deubel, he Mind's Eye: Cognitive and
Applied Aspects of Eye Movement Research (pp. 573‐605). Boston: North Holland.
Johansen, S. A., Nørgaard, M., & Rau, J. (2008). Can eye tracking boost usability evaluation of Computer
Games? Retrieved October 27, 2008, from IT ‐ University of Copenhagen:
http://www.itu.dk/docadm/detail.php?DocID=1706
Karn, K. (2006). Eye Tracking for Usability Testing, You’ve Got to Know Your Strengths and Weaknesses.
Retrieved January 13, 2009, from Amberlight ‐ Usability, User‐centred Research and Design Services:
http://www.amber‐light.co.uk/index.shtml
Samuel de Jesus Almeida | sja@ua.pt | References 14
15. Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R., & Wixon, D. (2008). Tracking real‐time user
experience (TRUE): a comprehensive instrumentation solution for complex systems. Retrieved
December 12, 2008, from ACM Portal: http://portal.acm.org/citation.cfm?id=1357126
Nacke, L., & Lindley, C. A. (2008, November). Flow and Immersion in First‐Person Shooters: Measuring
the player's gameplay experience. Toronto, Ontario, Canada.
Nielsen, J. (2003, August 25). Usability 101: Introduction to Usability. Retrieved January 13, 2008, from
useit.com: Jakob Nielsen's Website: http://www.useit.com/alertbox/20030825.html
Pinelle, D., Wong, N., & Stach, T. (2008, April 5‐10). Heuristic evaluation for games: usability principles
for video game design. Retrieved January 8, 2009, from ACM Portal:
http://portal.acm.org/citation.cfm?doid=1357054.1357282
Rollings, A., & Adams, E. (2003). Andrew Rollings and Ernest Adams on Game Design. New Riders
Publishing.
Shneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human‐Computer
Interaction (Third ed.). Addison Wesley Longman.
Wolf, M. (2000). Genre and the Video Game. Retrieved November 28, 2008, from Genre and the Video
Game: http://www.robinlionheart.com/gamedev/genres.xhtml
Samuel de Jesus Almeida | sja@ua.pt | References 15