Presentation on what you can learn from usability testing and projects in ballot and elections. Dana Chisnell and Whitney Quesenbery at EVN 2012

1. Need a little usability?
What you can learn from usability testing
Whitney Quesenbery
Dana Chisnell
Usability in Civic Life

2. 2
What questions do you have?

3. 3
Usability testing answers questions, so we have to
understand the question we are asking
If we want to know:
Which ballot design helps voters be more accurate?
We are asking a question that is
Comparative, quantitative (errors, time)
(and why they made those mistakes)
If we want to know:
Do voters understand how to mark their ballot?
We are asking a question that is abut
Mental models, variability, qualitative insights
(and how prevalent each model is)

4. 4
Helpful models for thinking about usability
A’s Ability, aptitude, attitude
(what voters bring to the election)
E’s Efficient, effective, engaging, error-tolerant, easy to learn
(dimensions of usability)
Use Useful, usable (operable), desirable
(how we define success)

5. 5
The recipe for planning a usability test
Location and context: formal informal
Recruiting: defined opportunistic
Activities: instructed tasks free tasks
Questions: structured unstructured
Data collection: observation task/data only
Results quantitative qualitative

6. 6
A few examples

7. 7
Which instructions work better?
Traditional ballot instructions vs. plain language version
About this project
 45 participants
 3 lab setting, 8 days
 2 moderators
 Instructions for voting
 Careful observation
 Satisfaction survey
We learned about
 Voter accuracy
 Order effects
 Voter preference
Results
 Statistical analysis
 Participant preference

8. 8
Which layout works better?
Left aligned vs. right aligned names?
About this project
 100 participants
 2 locations, 1 day
 4 moderators + 6 others
 Instructions for voting
 Minimal observation
We learned about
 Voter preference
 Time to vote
 Number of errors
Results
 Statistical analysis of marked
ballots

9. 9
What do voters do with unusual options?
How do voters react to “double-vote” options?
About this project
 200 participants
 5 locations, 1 day
 12 moderators + 5 others
 Self-defined tasks
 Minimal observation
We learned about
 Voter awareness of party
 Types of marks and errors
 Need for voter ed.
Results
 Inspection of ballots for
marking patterns

10. 10
What problems might voters have with a new ballot?
What problems might voters have with a new ballot design?
About this project
 1 of 5 similar tests
 10 participants
 1 locations, 1 day
 2 moderators + 1 others
 Self-defined tasks
 Observation
 Post-task questionnaire
We learned about
 Navigation on the ballot
 Instructions
Results
 Found pattern of errors for
one interaction
 Time on task

11. 11
What do voters think about different systems?
Which accessible voting system do voters prefer?
About this project
 496 surveys; multi-disabilit
 11 systems
 Multiday “voting fair”
They learned about
 Voter assessment of usability
 Voter comments about
accessibility
 Differences by disability
Results
 Statistical analysis survey
results
 Qualitative analysis of
comments

12. 12
Do voters understand how their votes are counted?
Can voters explain and act on that understanding?
About this project
 20 participants
 4 locations, 1 day
 8 moderators
 Self-defined tasks
 Minimal observation
We learned about
 Navigation on the ballot
 Instructions
 Mental models of counting
 Whether there’s a problem
Results
 Qualitative insights

13. 13
Do voters understand a new voting system?
What is their reaction to new procedures?
About this project
 Focus group
 Mock election
 1 moderators
They learned about
 Attitudes and reactions
Results
 Qualitative insights

14. 14
Things to watch out for when
planning a usability test

15. 15
Treating participants with respect
Language and cultural issues
Managing context
Participant protections and informed consent

16. 16
Avoiding bias
Leading questions
Closed vs. open questions
Sampling bias
Moderator influence
Observer (or moderator reactions)

17. 17
Thoughts?

18. 18
Whitney Quesenbery
whitney@wqusability.com
Dana Chisnell
dana@usabilityworks.net
Field guides to ensuring voter intent
http://tinyurl.com/ensuringvoterintent
Usability in Civic Life
http://usabilityinciviclife.org
Civic Design
http://civicdesigning.org
LEO Usability Testing Kit
http://www.usabilityinciviclife.org/voting/leo-testing-kit/