Usability of any product, whether it is software application, a website, or technical gadget is an important consideration. Sherry Marcy, Science and Technology Market Strategies will talk, from a business perspective, on how Pfizer improved its software tools for chemistry. She will share important lessons about design considerations, and user testing that will be relevant for businesses managers (start-up and established), entrepreneurs and product development teams.

1. Usability Testing: A Story
LA2M
12 May 2010
Sherry Marcy
Science & Technology Market Strategies LLC
sherry.marcy@gmail.com
On linkedin, twitter, facebook
734-368-7806

2. Case Study – Situation at Start
• Multiple brilliant scientific computing
tools
• None of them used by chemists
• No understanding of usability in Pfizer
Informatics

3. Usability Testing and the
Technology Transfer Process
1. Design
consistent “look and feel”
2. Production Environment
no development glitches
3. Testing
a) alpha b) beta c) usability testing
4. Training
WBT, viewlets, business-led courses…
5. Support
context-sensitive help, cheat sheets
6. Communications
flash sessions, invitations, balloons!

5. User Experience Engineering
Contextual Design by Beyer & Holtzblatt
Building Software that Works for Users
User
Task
Groups
Examples
Usability Testing Software
Paper Prototyping
Storyboarding Design
Visioning
Brainstorming Vision
Consolidating
Work Modeling Understanding
Contextual Inquiry

6. WebDock / Docking Study
• Chemists Participating in WebDock Phone Interviews
id Ed At Pfizer WebDock WebDock Project
Training? Use?
U1 PhD 2 years No No PI3-K
U2 BS 5 years Yes Yes FXa
U3 BS 4.5 years No Yes PI3-K
U4 MS 2 years Yes No CDK4
U5 PhD 4 years Yes Yes FXa
U6 MS 2 months Yes No CDK4
U7 PhD 3 years Yes Yes CDK4, …

7. WebDock Usability Testing
• Chemists You areThe Modeler Chemist on the Protein Factor Xa project at
Pfizer.
a Medicinal
assigned to your project has created a 3D
id Ed At WebDock Recent Project Use Test Observers
•
model of the protein and set up a WebDock job to allow you to try
Pfizer Training? Real*
Task out docking your own ligands. WebDock
Use
U1 PhD 2 years No Never PI3-K 11/7/01 2:00 Sherry Marcy, David
• Facility For your series, you have several ideas for variations of the lead Wild, Rob Goulet,
David Dunn
you want to try out to see if they will improve the binding. You
U3 BS 4.5 No 2 months PI3-K 11/8/01 9:00 David Wild, Rob
years ago (just
want to use WebDock to dock your ideas to the protein in the 3D Goulet, David Dunn
•
“to try”)
Facilitator model created by the modeler.
U4 MS 2 years Yes Never CDK4 11/8/01 1:00 David Wild, Rob
Goulet, David Dunn
U5 PhD 4 years Yes 1 day ago FXa 11/7/01 8:00 Rob Goulet
Your specific task is:
•
U7 PhD 3 years Yes 5 months CDK4, 11/7/01 10:30 Rob Goulet
Equipment Use WebDock to see how your ideas (found in Figures 1 &
ago …
2) might bind to the protein Factor Xa.
• Observers

8. Usability Testing Output
• Stakeholder’s Direct Observation of
Users’ Experience
• Assessment of Relative Quality of Use
• Identification of Obstacles and Insight
into Responses

9. Quality of Use
• Quality of Use
– Ease of Use
– Ease of Learning
– Reliability
– Validity
– Performance
• Other Aspects of Software Quality
– Maintainability, Extensibility,…

10. Quality of Use Metric
Determination
Hard |--------------------------------------------------| Easy
Frustrated |--------------------------------------------------| Satisfied
Failure |--------------------------------------------------| Success
Time:
More |--------------------------------------------------| Less
than As than
Expected Expected Expected
Task
Task
11 2
2 3
3 4 All
4 All
User
User 1
1 73 65 95 70
2
2 94 85 78 82
3
3 81 96 80 85
All
All 82

11. Quality of Use Metric
Interpretation
100
users will consistently accomplish their tasks, will sometimes exceed users' expectations
90
most users will easily accomplish their tasks
80
users will accomplish their tasks, but many rough edges will impact users
70
many users will accomplish their tasks, but many will not and will reject
60
most users will not accomplish their tasks and will reject
0

12. Quality of Use Metric
results we’ve seen
(before & after)
100
90
80
70
The full range
60
Dramatic improvements
0

13. Is there room How do I
for a hydroxyl Quality of Use find the distance
group in correlates with other indicators two
the between
pocket? atoms?
How much of the time
• users are thinking about their work rather than
the tool
• instructors are describing how to leverage the
tool rather than how to operate it

14. Quality of Use vs Use
Quality of Use predicts Use Over Time
100
90
80
70
60
0

15. WebDock QofU
100
90
80 Most chemists will usually be able to dock
70 WebDock their ligands with WebDock, but many rough
edges will impact them, causing some
November 2001 chemists to not use it
60
•Success
•Obstacles: Opportunities
0

16. Chemists’ Experience with
WebDock
Where’s WebDock?

17. People don’t read…

18. Chemists’ Experience with
WebDock

19. Chemists’ Experience with
WebDock

20. Chemists’ Experience with
WebDock
This ChemDraw
link does nothing!
Scroll!

21. Chemists’ Experience with
WebDock
Enter every time!
Why a filename?
How many?
Backspace makes
the page go away!
I can’t reuse my Ctrl-z for Undo How is
previous sketch! doesn’t work! stereochemistry
handled?

22. Chemists’ Experience with
WebDock

23. Chemists’ Experience with
WebDock
It’s a pain to
switch off all the
bindings!
Are the check
boxes on or off?
Those little boxes
are hard to hit!
How do I zoom
Things come back and translate?
on when I don’t
expect! (Protein
on causes H’s on)

24. Chemists’ Experience with
WebDock

25. Metrics of WebDock Use
(As of January 5, 2003) 100
90
80
70
60
50
40
30
20
10
0
WebDock Launch
(September 13, 2001)
Holiday Season Lithium Launch
(Dec, 2001 – Jan 2, 2002) Holiday Season
(January 29, 2002)
(Dec, 2002 – Jan 2, 2003)

26. Metrics of WebPK Use
Original Usability Testing Score of 85
90
80
70
60
50
40
30
20
10
0
WebPK Launch
(October 9, 2001) Holiday Season
(Dec, 2001 – Jan 2, 2000)

27. What’s Quality of Use Improvement
Worth?
Value proposition:
• Save development costs
100 • Save development time
90
• Reduce maintenance costs
80
• Save redesign costs
70
• Attract usage (increase appeal)
60
• Retain users (frequency of use)
• Decrease training time and costs
• Decrease support costs
• Increase trust in systems
• Drive productivity!
0

28. Better Decisions
Sooner
Tools to Think!
Sherry Marcy