In the design process we follow, once we have defined the conceptual direction and content strategy for a given design and refined our approach through user research and iterative usability testing, we start applying visual design. Generally, we take a key screen whose structure and functionality we have finalized—for example, a layout for a home page or a dashboard page—and explore three alternatives for visual style. These three alternative visual designs, or comps, include the same content, but reflect different choices for color palette and imagery. The idea is to present business owners and stakeholders with different visual design options from which they can choose. Sometimes there is a clear favorite among stakeholders or an option that makes the most sense from a brand perspective. However, there can often be disagreements among the members of a project team on which direction to choose. If we’ve done our job right, there are rationales for our various design decisions in the different comps, but even so, there may be disagreement about which rationale is most appropriate for the situation. As practitioners of user-centered design, it is natural for us to turn to user research to help inform and guide the process of choosing a visual design. But traditional usability testing and related methods don’t seem particularly well suited for assessing visual design for two reasons: 1. When we reach out to users for feedback on visual design options, stakeholders are generally looking for large sample sizes—larger than are typical for a qualitative usability study. 2. The response we are looking for from users is more emotional—that is, less about users’ ability to accomplish tasks and more about their affective response to a given design. With this in mind, we were very interested in articles we saw on Desirability Testing. In one article, the author posits desirability testing as a mix of quantitative and qualitative methods that allow you to assess users’ attitudes toward aesthetics and visual appeal. Inspired by his overview, we researched desirability studies a bit further and tried a modified version of the techniques on one of our projects. This presentation reviews the variants of desirability testing that we considered and the lessons we learned from a desirability study on visual design options for one of our projects. Interestingly, we found that while desirability testing did help us better understand participant’s self reported emotional response to a visual design, it also helped us identify other key areas of the experience that could be improved.

1. boston upa
Rapid Desirability Testing
ANALYZING EMOTIONAL RESPONSE TO A DESIGN (ON A
BUDGET)
Prepared by:
Michael Hawley – VP Experience Design
Megan Grocki – Senior Experience Designer
June 9, 2009

2. boston upa
Agenda
• Introduction
• The Situation
• Desirability Testing Overview
• Methods Considered
• Our Selected Process
• Case Study
• Lessons Learned
2

3. boston upa
About Mad*Pow
3

4. boston upa
The Situation
4

5. boston upa
The Situation
Visual Designs Applied to Wireframe
5

6. boston upa
The Situation
Visual Designs Applied to Wireframe
6

7. boston upa
Desirability Testing Overview
7

8. boston upa
What Is Desirability Testing?
A collection of research methods intended to assess target audience s
emotional response to a design or stimulus.
What It Is What It Is Not
• Measure of how closely a • Measure of how much people like
stimulus achieves the desired something
emotional response
• Figuring out which is the best
8

9. boston upa
Positioning Desirability Studies
http://www.xdstrategy.com/2008/10/28/desirability_studies/
9

10. boston upa
Why Is It Important
First impressions of a design to impact a product s or application s perceived
utility, usability, and credibility.
Functionality
Usability Aesthetics
10

11. boston upa
Methods Considered
11

12. boston upa
Triading
Definition
Present three different concepts or ideas to participants, and ask them to
identify how two of them are different from the third and why.
12

13. boston upa
Quantitative Questionnaires
Definition
Broad, experience-based questionnaires,
that also include questions relating to
visual appeal and aesthetics
• SUS (System Usability Scale),
• QUIS (Questionnaire for User Interface
Satisfaction)
• WAMMI (Website Analysis and
Measurement Inventory)
13

14. boston upa
Quick Exposure Memory Tests
Show participants a user interface
for a very brief moment, then take it Attention designers:
away. Participants recall their first
impression, then moderator probes
for meaning You have
50 milliseconds
• Helpful for layout decisions,
prominence of content, even labels
to make a good
first impression
• www.fivesecondtest.com
14

15. boston upa
Physiological and Neurological Measurements
Definition
• Sensors track participants physiological
measurements to particular designs. Changes in
suggest a particular emotional response.
• Paired with attitudinal and self-reporting surveys
measurements give a multifaceted view of
emotional reactions to a design
• Electroencephalography (EEG): Brain activity
• Electromyography (EMG): Muscles and Excitement
• Electrodermal Activity (EDA): Sweat, Excitement
• Blood Volume Pressure (BVP): Arousal
• Pupil Dilation: Arousal and Mental Workload
• Respiration: Negative Valence or Arousal
15

16. boston upa
PrEmo Emotional Measurement
http://www.premo-online.com
Dr. Pieter Desmet,Technical University of Delft
16

17. boston upa
Product Reaction Cards (Our Selected Approach)
http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc
17

18. boston upa
Product Reaction Cards Method
18

19. boston upa
Before You Begin
Determine intended brand attributes (and their opposites)
1. Leverage existing marketing/brand
materials
2. Alternatively, stakeholder brainstorm to
identify key brand attributes/descriptors
using full list of product reaction cards
as a start
3. Tip: If the brand was a person, how
would it speak to your customers?
19

20. boston upa
Process - Conducting
Methodology
1. Include 60/40 split of positive and negative words
2. Target 60 words, optimized to test brand
3. Simple question: Which of the following words do you feel best describe the
site/design/product (please select 5):
4. One comp per participant, or multiple comps per participant (no more than 3)
Participants
1. Qualitative: Paired with usability testing
2. Quantitative: Target minimum of 30 per option if possible
20

21. boston upa
Process - Analyzing
1. Calculate percentage of positive
and negative attributes per design
68% Positive
32% Negative
2. Visualize overall sentiment of
feedback using word
clouds (see wordle.net)
Tip: Use word list spreadsheet available at http://
www.userfocus.co.uk/articles/satisfaction.html
21

22. boston upa
Case Study: Greenwich Hospital Website Redesign
22

23. boston upa
Case Study: Greenwich Hospital Website Redesign
Background and Goals
• Align the website with the character of Greenwich Hospital
• luxurious, approachable, friendly, capable, multi-cultural/inclusive, established
• Update the site after nearly 10 years
• Counter impressions that Greenwich Hospital is more than just about maternity
and elder care, without damaging those notions
• Communicate that they are long-standing members of the community
23

24. boston upa
Case Study: Greenwich Hospital Website Redesign
Methodology
• 3 visually designed comps
• 50 people reacted to each comp (quantitative) via survey
• Additional feedback obtained via participant interviews (qualitative)
Survey Questions
Hello, I am requesting feedback on a website I am working on.
Your answers let me know if the site is conveying the right feel.
1. What are your initial reactions to the web site?
2. Which of the following words best do you feel best describe the site (please
select 5):
24

25. boston upa
Three Different Visual Designs
25

26. boston upa
Results: Concept 1
My initial reaction to this web site is that it seems kind of
plain. There is not much going on in the page, and the 12% Negative
colors seem kind of drab.
This is a nice looking website. It is well designed, well
laid out, and is appealing to look at. It makes me want to 88% Positive
continue to navigate the site to learn more.
26

27. boston upa
Results: Concept 2
Men don`t really go with children… where`s a
baby, there must be a mother.
87% Positive
My initial reaction to the website is that it 13% Negative
seems very clean and modern. I like the layout,
it looks like its easy to find information.
27

28. boston upa
Results: Concept 3
I felt love. I saw a mother holding a child..
that`s pretty touchy. The site looks good, and it
makes the hospital trustworthy. 5% Negative
My initial reaction was that the Hospital is
represented by a caring, warm and friendly 95% Positive
website.
28

29. boston upa
Lessons Learned
29

30. boston upa
Lessons Learned
Methodology
• Mix of qualitative and quantitative is key. Qualitative helps provide color to the
results, quantitative resonates with stakeholders and executives
• Position results as one form of input to decision-making process, not declaring
a winner
• Simple, cost-efficient way to assess audience s emotional response to a design
30

31. boston upa
Key Take Aways
The Challenge:
• Measuring emotional responses to a design important, but complex.
Experiences of a visual design are multifaceted, and a number of design
aspects can impact their response to a product.
• There are a number of alternatives available to measure emotional response
Our Experience:
• Leveraging Product Reaction Cards provides a low-cost, low-effort means to
help us align aesthetics and general feel with desired brand attributes
31

32. boston upa
Thank You
Documentation Have a question?
Case Study results and full presentation slides: Michael Hawley
http://www.madpow.net mhawley@madpow.net
@hawleymichael
Megan Grocki
mgrocki@madpow.net
@megangrocki
603-436-7177
32

33. boston upa
Additional Reading
Benedek, Joey and Trish Miner. Measuring Desirability: New Methods for
Evaluating Desirability in a Usability Lab Setting. Proceedings of UPA 2002
Conference, Orlando, FL, July 8–12, 2002.
http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc
Lindgaard, Gitte, Gary Fernandes, Cathy Dudek, and J. Brown. "Attention Web
Designers: You Have 50 Milliseconds to Make a Good First Impression!" Behaviour
and Information Technology, 2006.
http://www.imagescape.com/library/whitepapers/first-impression.pdf
Rohrer, Christian. Desirability Studies: Measuring Aesthetic Response to Visual
Designs. xdStrategy.com, October 28, 2008. Retrieved February 10, 2010.
http://www.xdstrategy.com/2008/10/28/desirability_studies
33

34. boston upa
Additional Reading
User Focus. "Measuring satisfaction: Beyond the Usability Questionnaire." Retrieved
February 10, 2010.
http://www.userfocus.co.uk/articles/satisfaction.html
UserEffect. "Guide to Low-Cost Usability Tools." Retrieved May 12, 2010.
http://www.usereffect.com/topic/guide-to-low-cost-usability-tools
Tullis, Thomas and Jacqueline Stetson. A Comparison of Questionnaires for
Assessing Website Usability. Usability Professionals Association Conference,
2004.
home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf
Westerman, S. J., E. Sutherland, L. Robinson, H. Powell, and G. Tuck. A Multi-
method Approach to the Assessment of Web Page Designs. Proceedings of the 2nd
international conference on Affective Computing and Intelligent Interaction, 2007.
http:// portal.acm.org/citation.cfm?id=1422200
34

35. boston upa
Additional Tools
Five Second Test
http://fivesecondtest.com/
Feedback Army
http://www.feedbackarmy.com
Wordle
http://www.wordle.net
PrEmo
http://www.premo-online.com
35