Do you use surveys or questionnaires in your work? If so, it is important to be sure you are collecting the data you need. Too often, we write surveys without much attention to how participants will understand and respond to our questions. Just like forms, instructions, websites, or any other product, users often have their own interpretations of our content. We can improve our survey questions with a method called cognitive interviewing. This method was developed to improve questionnaires for large survey companies and government organizations. The method is similar to usability testing, as it evaluates how a user experiences a survey, but there are also some critical differences between the methods. In this session, we will describe cognitive interviewing, and show how it is similar to and different from usability testing. We will discuss how to conduct cognitive interviews, what data to collect, and how to analyze the results. Presented by Jean Fox, Jennifer Edgar and Scott S. Fricker

1. 1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Improving Your Surveys and
Questionnaires with Cognitive
Interviewing
Jean E. Fox
Scott S. Fricker
Jennifer Edgar
U.S. Bureau of Labor Statistics
UXPA 2018
June 28, 2018
All views expressed in this paper are those of the authors and do not reflect the views or policies of BLS.

2. 2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox
Thank you
Scott Fricker
Jennifer Edgar

3. 3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Agenda
 Surveys in UX work
 Why we need Cognitive Interviewing (CI)
 Background on CI
 How CI is similar to and different from
Usability Testing (UT)
So you can get started with CI
 More ways that CI can help!

4. 4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
What is a Survey?
 A series of questions given to respondents to
collect information from them
“Questionnaires” and “polls” have same purpose
 Systematic
Ask the same thing to all participants
Can get data from many people easily
 Self- or Interviewer-administered

5. 5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Surveys in UX Work
 We can use surveys
On their own
With other methods
– Usability Testing
– Card Sorting
– Field Observations

6. 6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Standard UX Surveys
 Known UX surveys that have been tested
Reliable
Valid
Sensitive
 Examples
System Usability Scale (SUS) (Brooke)
SUPR-Q (Sauro)
SUMI, WAMMI (Kirakowski)
QUIS (Norman)

7. 7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
BUT…
 If standard surveys can’t address our issues
We write our own.

8. 8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Home-Grown Surveys
 Benefits—
Target the information we want
Easy to write (right??)
 BUT…
Are you sure you are measuring what you want?
How do you know?
 AND
You know how to design interfaces, but you still
run usability tests
And you still find problems.

9. 9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox
Cognitive Interviewing
Make sure you are measuring
what you want to measure

10. 10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
So, What is Cognitive Interviewing?
 It’s kind of like Usability Testing for surveys.
Participants answer survey questions
You talk to them about their response process
Identify problems with the survey
 But there are some important differences.

11. 11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Background on Cognitive Interviewing
 Developed in the field of Survey Methodology
 Goal is to learn how respondents:
Understand the question
Retrieve the information
Decide what answer to give
Respond to the question
 Problems can occur in any of these stages

12. 12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Types of Problems CI Can Uncover
 Complicated Instructions
 Confusing or vague question wording
 Jargon
 Inappropriate assumptions (e.g., the question
doesn’t apply to your respondent)
 Unexpectedly sensitive topics
 Questions that are difficult to answer
 Answers that are difficult to recall

13. 13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Just to Give You an Idea…
 How many UX professionals are in your group?
What is your answer?
How did you determine your answer?
Who did you include as a “UX Professional”?
Exclude?
How did you define your “group”?
What does it mean to be “in” your group?
What time frame did you include?

14. 14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Testing Overview – Both UT and CI
 Define research goals, create materials, etc.
 Recruit participants
 Conduct study
Informed consent
Complete task
Debriefing
Incentives
 Analyze data
 Make recommendations

15. 15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Testing Goals
 UT
Uncover usability problems and potential
solutions
 CI
Ensure the question/survey is capturing the
intended information

16. 16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Moderators
 UT & CI: Similar skills and training
Work comfortably with all participants
Interact in an unbiased manner
React and adjust the test session as needed
Analyze and interpret data

17. 17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
When to Test
 UT & CI
Ideally, start early and continue through
development stages
Goals are different at different stages
– Early – Test wireframe / concepts
– Midway – Test features / questions
– Late – Test overall product / survey

18. 18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Participants
 UT and CI
Target specific populations, determined by
research goals
Typically use convenience samples
 UT
Often 5-10 (per user group)
Or more for a quantitative study
 CI
Typically 20-30

19. 19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Moderating Sessions
 UT
In-person or remote
Moderated or unmoderated
 CI
In-person or over the phone
Almost always moderated

20. 20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Tasks
 Type of Tasks
UT: Complete tasks using the product/website
CI: Answer survey questions
 Scenarios/vignettes to provide context
UT: You are a school counselor, and you want to
know the education requirements for accountants.
CI: Pretend this is your high school transcript, and
answer these questions about your courses.

21. 21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thinking Aloud
 UT and CI
Information about participant’s processes,
thoughts, and reactions
Pros and cons, decide based on research goals

22. 22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Debriefing Questions / Probes
 To learn about participants’ thoughts and
reactions
Helpful for UT
Essence of CI

23. 23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Debriefing Questions / Probes
 Can be done
During the task
– Get thoughts as participants are working
– But this can interrupt their work
Post-Task / Post-question(s)
– Get feedback on specific tasks/questions
Post-Test / Post-survey
– Get overall impressions

24. 24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Goals of Questions / Probes
During the Task
 UT and CI
Explore unexpected behaviors, responses, and
reactions
Understand why participants are struggling

25. 25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Sample Questions / Probes
During the Task
 UT and CI
I noticed you hesitated, can you tell me what you
are you thinking?
What are you looking for?

26. 26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Goals of Post-Task
Debriefing Questions / Probes
 UT
Why participants did what they did during an individual
task
How they felt about their experience
Scripted or spontaneous
 CI
What participants were thinking as they answered a
specific question or group of questions
How they decided what to do
Scripted or spontaneous

27. 27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Sample Post-Task
Debriefing Questions / Probes
 UT
How easy or difficult was the task?
How confident are you that you completed the
task?
 CI
In your own words, what was the question
asking?
What does <term or phrase> mean to you?
How did you decide on your response?

28. 28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Goals of Post-Test
Debriefing Questions / Probes
 UT and CI
Overall impressions about the product / survey
What participants liked and didn’t like
Any recommendations they have
Scripted or spontaneous questions

29. 29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox
Data Collected
Data Usability Testing
Cognitive
Interviewing
Qualitative
• Observations
• Comments
• Debriefing responses
Quantitative
• Task times
• Success rates
• Satisfaction ratings
• Etc
• Actual responses to
survey questions, in
context of the
debriefing
information

30. 30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Analysis for Both UT and CI
 Qualitative data – look for patterns and
themes across participants
 Quantitative data
UT: look at means by user group, use statistics as
appropriate
CI: review responses in context of the answers to
the probes

31. 31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Outcome of Analysis
 UT
Identify usability problems
– Effectiveness
• Tasks that are difficult to complete
• Features that are hard to use
– Efficiency
• Activities that took too long to complete
– Satisfaction
Make recommendations
– Ways to fix the usability problems

32. 32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Outcome of Analysis
 CI
Identify problems with
– Understanding the questions
– Retrieving the information
– Deciding what answer to give
– Responding to the question
Make recommendations
– Rewording of questions
– Rework skip patterns
– Provide context for interpreting survey results.

33. 33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Getting Started with CI
 Whenever you are asking users questions,
consider testing the questions first.
For usability testing, use Cognitive Interviewing to
evaluate your questions during the pilot test.
For other surveys, consider starting with a review
by your UX colleagues or actual respondents
You want to know
– Do they interpret the questions the way you want?
– Can they answer the questions?
– Do the response options make sense?

34. 34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
CI Can Do Even More for UX work!
 Evaluate web content, instructions, or other
text products
Can users understand it?
Does it mean what you think it means?
 Evaluate other types of forms
Do people know what goes in each field?

35. 35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Conclusions
 It’s important to be sure that our surveys are
asking what we want them to ask.
 CI can help you do that
 What you know about UT already may make it
easy for you to incorporate CI into your
research
 To learn more about CI, see:
Willis, G. B. (2004). Cognitive Interviewing: A Tool for
Improving Questionnaire Design. Sage Publications.

36. Contact Information
36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Jean E. Fox
Bureau of Labor Statistics
Fox.Jean@bls.gov

37. 37 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox
Usability Testing Cognitive Interviewing
What are the objectives
of the methods?
Identify usability problems
and potential solutions
Understand how
respondents interpret
survey questions and the
challenges they face
completing a survey
When in development
would you use the
method?
Anytime in development
Who conducts the tests?
Typically User Experience
professionals, possibly with
a formal coursework related
to usability testing
Typically Survey
Methodologists, often with
informal training in cognitive
interviewing
Who participates in the
tests?
Participants are from the target population(s)
Smaller sample sizes for
qualitative tests, larger for
quantitative tests
Generally aim for 20-30
participants

38. 38 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox
Usability Testing Cognitive Interviewing
What methods are used?
Participants complete tasks (answer survey questions)
while their experience is observed in some way
Participants complete tasks
assigned to them, often with
scenarios providing context
for the task
Participants complete
surveys with their own
responses
In-person or remote testing Usually in-person testing
Sometimes moderated Always moderated
Can be done in lab or at respondent’s location
May use think aloud
What data are collected?
Quantitative and/or
qualitative data
Generally qualitative data
only
Focus is on performance
and user satisfaction
Focus is on cognitive
processes while completing
the survey