Experimental prototyping and play testing using iterative game design is discussed. Prototypes allow designers to base decisions on successive iterations of a game as it is tested and refined. Different types of prototypes are described from early paper prototypes to later software prototypes. Various types of play tests are also outlined such as focus tests, expert evaluations, and free-form play sessions. Obtaining useful data from play tests to answer specific research questions is the focus. Methods like surveys, interviews, video recordings, and data mining game interactions are presented.

1. experimental prototyping and play testing using iterative game design 2010 Lecture in the course International Game Production Studies I Mirjam Palosaari Eladhari Gotland University, Sweden

3. Types of Prototypes

4. Types of Play Tests

5. Obtaining Data

6. Trade-offs Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

7. Focus of this talk: experimental prototyping and play testing using iterative game design how to plan for obtaining data from play tests that can yield answers to specific research questions. Read more: Design for Results: Considerations for experimental prototyping and play testing using iterative game design by Mirjam P Eladhari and Elina M I Ollila. Article to be published in a special issue of the Journal Simulation and Gaming focusing on game research methods. (Uploaded to course website) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

8. Expressive AI (2003) Material on this page is from “Expressive AI: Games and Artificial Intelligence” by Michael Mateas, LevelUp Conference, Utrecht 2003 Michael Mateas: “AI-based art and entertainment constitutes a new interdisciplinary agenda linking games studies, design practice, and technical research. “[…] expressive AI provides a language for talking about “readable” behavior, that is, behavior that a player can read meaning into.” Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

9. Prototype A prototype is “played, evaluated, adjusted and played again, allowing the designer or design team to base decisions on the successive iterations or versions of the game. Iterative design is a cyclic process that alternates between prototyping, play-testing, evaluation, and refinement.” Salen and Zimmerman (2001) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

10. Early questions • What types of game play dynamics and game play experiences can a certain, mechanic, feature, approach or method result in?1 • What are the qualities of the play experience, and how can these qualities be documented, analysed, interpreted and explained comprehensively? • How can it be decided if a certain game play feature results in something valuable, such as a new type of experience, a meaningful experience, or a ‘better experience’ in some other way, and if so, compared to what? Researchers may find themselves in situations where innovative results do not necessarily lend themselves to comparisons. • What type of data can be useful to obtain in order to explore the research question? Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

11. Beware Researchers need to, in the design process, take into account what questions they aim to explore and stay focussed on these throughout the design and implementation work. It is easy to fall into a frame of mind where one aims to produce a good game, losing focus of obtaining research material. In the process (especially when implementing) take a step back regularly to think about the research question. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

12. Wicked Problems ‘wicked problem’ is used in social planning to describe problems where every attempt at producing a solution changes the understanding of the problems (Rittel & Webber, 1973) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

13. Wicked problems in game design “For a wicked problem such as game design, exploring design space consists of navigating the complex relationships and constraints among individual design features, while at the same [time] discovering or inventing new features and approaches that expand the design space. All existing games form tiny islands of partially understood regionsof design space; all around these islands lays a vast ocean of unexplored potential design space waiting to be brought into existence through the invention of new features and approaches, and mapped out through the hard empirical work of exploring a variety of designs.” Mateas and Stern (2005) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

14. Build it to understand it “[...] if game studies is limited to analysing existing games and design spaces, it can be problematic to imagine or theorise about potential game features outside of these design spaces. Models about the nature of games and their features run the risk of being incomplete or wrong, simply because certain design spaces have not yet been explored.” (Mateas and Stern 2005) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

16. Types of Play Tests

17. Obtaining DataMirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

18. Types of prototypes

19. Prototyping by acting and showing Early body storming - participants imagine the game and act as though it would exist, in either a real or imaginary place where it could be played. (Burns, Dishman, Verplank, & Lassiter, 1994) polished video scenarios, the purpose of which is to show how a concept would work in its natural settings. FIGURE The player is taking a picture of a person smiling, 2. The smile is mapped to a smiley in a grid that the player needs to complete (i.e., take pictures with people having similar expressions on their faces as the smileys do) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

20. Paper (or physical) prototypes A physical prototype can at an early stage give pointers to whether a designed game mechanic results in the intended game dynamics when played. A physical prototype is often made of paper mock-ups of the intended game, but can also include physical ‘bits’ such as figurines and tokens. game-mastering method, where the player was brought to various game situations with the help of a game master, like in pen-and-paper role-playing games. Middle Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

22. Computationally aided physical prototypes Middle In cases where the game mechanics demand higher degrees of computation of significant values it can be useful to add aids for calculation to a paper prototype. A common and very simple method is to use excel sheets Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

23. The developers of Raptor found that using a table-top was superior for collaborative sketching compared to the interface of traditional PCs. Designers were presented with a sand-box environment where they could shape the geography using hand-gestures. They could also ‘stamp’ game bits into the environment such as cars for a racing game, and attach a ‘camera’ to the object representing the player, giving the user, who had access to a PC client inter- face, their point of view in the geography. Raptor, a tool for sketching and prototyping games using a table-top surface. Raptor: Sketching Games with a Tabletop Computer, by JD Smith, FuturePlay 2010 Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

24. Software Prototypes Software prototypes are quite often thought to be something that is created later in the project, when there is already game design documentation available. However, plenty of fast prototyping tools are available, ranging from general pur- pose tools like Flash to more specialised frameworks. Late Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

25. Production cycle, example Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

26. Types of play tests

27. ad-hoc test An ad-hoc test is a quick informal test which requires minimal organisation. It can be as simple as finding a colleague not working on the same project to quickly test some aspect of the prototype on them. (grab someone in the corridoor- test) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

28. focus test In a focus test a group of potential players are probed about their perceptions, opinions, beliefs and attitudes towards the prototype. Questions are asked in an interactive group setting. (put a group of people from your target group in the same room and ask them your questions) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

29. expert evaluation An expert evaluation is the appraisal of a prototype by someone who has the professional training or experience to make an informed judgement on the design. (Show what you have to an expert, get their advice) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

30. QA Test The functional test, or quality assurance (QA), can be conducted by the production team or by QA-experts. QA is conducted in order capture errors in the functionality of the prototype as well as for balancing game play according to the intended game play experience (Find the bugs and fix them) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

31. Participatory design Participatory design workshops are conducted by the production team and a small number of invited guests, potentially experts in the field or potential players of the game. The workshops are often intended to aid in balancing the game and to eliminating dysfunctional elements or features. They can also be useful for gathering new ideas for further iterations or for enhancing the quality the prototype according to the goals by specific investigations. (Have people your trust go through the test with you, and get their advice on your design as you go along) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

32. Guided scenarios In guided scenarios a test-leader may use the Wizard of Oz - method to simulate user-interaction. The scenarios designed for the particular prototype are played individually by participants (Wizard of Oz experiment is a research experiment in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being.) (Game master what you haven’t implemented. If it doesn’t work you save lots of time knowing that.) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

33. Example (4 roles) The Player, the Game master, the Engine, and the Manual (taking the pictures) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

34. game-mastered play sessions In game-mastered play sessions several participants may interact with a prototype, or scenarios designed for the particular prototype, guided by a game master. Similar to guided scenarios, but later, when the prototype is functional. Ie, no wizard-of-ozzing for functionality. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

35. free-form play sessions In free-form play sessions participants are interacting with the prototype unaided by guiding test-leaders or game-masters. (common end-phase play test. Put the user in front of the screen and see what happens. ...Or send it off to a testing consultant along with your testing script.) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

36. Practical Considerations Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

37. Obtaining data

38. Surveys? Surveys can be used to capture players’ attitudes, get ideas for development and can also be used for a ‘memory check’ - in those cases a player remembers or have forgotten something about the design The effort required for this method is less than for many other methods for obtaining data. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

39. Example – Surveys as stand-alone tool Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

40. Interviews? Interviews can allow for obtaining data that might not arise in surveys. In some cases it may be useful to add a survey in addition to the interview in order to verify the results and to see if some results were received for ‘pleasing’ the interviewer, that is, give answers based on their interpretation of what they think the test leader wants to hear Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

41. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

42. Teach back tasks ‘teach-back tasks’ – players are asked to describe the system they tested to someone who has no prior knowledge of it. The player ‘does the job’ Can be used both in surveys and interviews. Teach-back tasks are often used in order to assess users’ mental models of systems (Veer, Wijk, & Felt, 1990; Puerta-Melguizo, Chisalita,& Veer, 2002). For example, in a test of a prototype where the players’ avatars had different ‘moods’ given by a psychological model, players were asked to explain to a friend how the mood affected what they could do or not do in the game world prototype (Eladhari, 2009). Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

43. Video tape it? Is it is useful to videotape the interaction? Possible to later on make verbatim transcriptions of what players said in the test. Allows for detailed study of body-language, actions and utterances. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

44. Bio feedback? Is it useful to obtain physical data from the participants of the test, and if so, what data is most likely to be useful? Potential physical data include: eye-tracking, heart-rate variability, galvanism and temperature in fingers, electroencephalography (EEG) or f MRI, posture (posture sensors may be placed on participants’ chairs, see (Plass, Perlin, Nordlinger, & Isbister, 2010) Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

45. Data mining? If the test involves a software prototype, it can be useful to also develop or use a back end-system for capturing events in the form of player-input to the system. Consider: to what extent it is useful to log the player's interactions with the system, which types of interactions to log. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

46. Data mining example The team at Microsoft Labs who conducted the testing of Halo 3 maps showing recurring events mapped to the geography of the game. This was useful for the designers of the game when they iteratively redesigned the environment and the placement of items in it relevant to the game play. Map used at Microsoft Game Labs showing a top down view of the fourth mission of Halo 3. The dots show the locations of player deaths. (thanks to John Hopson and MS game labs for permission to use picture) Thompson, C. (2007). Halo 3: How Microsoft Labs invented a new science of play. Wired Magazine (15.09). Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

47. Process the data Is it useful to make a rough coding scheme, for example counting certain types of utterances, behaviours, and/ or expressions or reactions? Is it useful to make verbatim transcriptions of the videotaped play tests? If so, is it useful to not only transcribe the utterances but also (or in- stead) make notes of facial expression, body language, laugher etc. to the transcripts? Is it useful to trace and describe the actions a player performs in relation to the paper prototype? For example it can be useful to observe and describe in what order players do something, and if there are patterns in how they approach different elements of the prototype. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

48. Obtaining Data Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

49. Designer Role Fallman (2003) discusses the role of the designer in HCI field and argues that there are three points of views to this: conservative, romantic, and pragmatic account. The conservative account believes in methods and sees the designer as a ‘glass-box’ whose actions can be rationalized and modelled with scientific methods. The romantic point of view is that the design originates from the designers unique artistic talents and genius, and sees the designer as a ‘black- box’ and that the design process is something that cannot be explained. The pragmatic account views the design as a dialogue between the designer and the environment and the result of the design as something that is integrated in the world. According to Fallman, all these aspects should be taken into account and be in balance with each other. Furthermore, he argues that the process of design cannot be seen as linear, even with iteration, but more as a continuous dialogue between analysis, synthesis, and evaluation.

50. The trade-off of research freedom vs resources Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

51. Navigation 1. Question. 2. What kind of prototype can answer it? 3. What data is needed to proove/refute? What tests to make? How process the data? Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

52. Mirjam P. Eladhari, mirjam.eladhari@hgo.se Gotland University, Department of Game Design, Technology and learning

53. Ending words . The idea is not to try to solve how the game should work and what are the important research questions to answer first and then start building the prototype, but rather having a rough idea of this first and then be flexible to change the approach later if, and often when, needed. In our view, prototypes constitute important thinking tools for researchers and designers alike, aiding us in exploring the wicked problem space of game design.

54. Thank you for listening