by Robert Schumacher, Ph.D. Presentation given on 21 May to the GCC HIMSS group in Chicago with ~50 people present. www.usercentric.com The point was to provide some background on usability (a gentle introduction to some of the science), some case studies, and introduce the measurement AND design components of user centered design. Note because of all the animations, some pages do not view properly. Please contact me if you would like more information: bob at usercentric.com

1. Usability: Introduction and Application to HIT
GCC HIMSS Meeting
Robert M. Schumacher, Ph.D.
Managing Director
User Centric, Inc.
Oakbrook Terrace, Illinois
rschumacher@usercentric.com
May 21, 2009

2. Who has the highest risk of accident?
(1) Conversing while (2) Conversing with hands
holding a phone free phone
(3) Conversing with (4) Drunk
person in car

3. Cognitive Impairment While Driving with Cell Phone
Meta-analysis by Horrey & Wickens (2006) found:
- There are significant costs to driver reactions to external hazards
or events associated with cell phone use, and
- Hands-free cell phones do not eliminate or substantially reduce
these costs
- Talking on cell phone when driving (hands-free or handed)
shows performance similar to driving drunk (Strayer and
colleagues)
Using a cell phone impairs a driver’s cognitive function
Drews, F. A., Pasupathi, M. & Strayer, D.L. (2008). Passenger and Cell Phone Conversations in Simulated Driving. Journal of Experimental
Psychology: Applied. 14 (4), 392–400
Horrey, W., & Wickens, C. (Spring 2006). Examining the Impact of Cell Phone Conversations on Driving Using Meta-Analytic
Techniques, Human Factors, 38 (1): 196–205.
Strayer D. L. & William J. A. (2001). Driven to distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular
Telephone.quot;. Psychological Science, 6 (12): 462–466.
3

4. Which Phone Is Best for Text Entry?
Speed
Speed
Accuracy
Accuracy
(1) iPhone (3) Blackberry
Full QWERTY QWERTY
(Touchscreen) (Hard Key)
(2) Razr
Numeric
(Hard Key)

5. Motor Differences When Typing on Cell Phones
Average Time to Enter Messages Average Number of Errors
 QWERTY fast and accurate
 iPhone fast but inaccurate
 Numeric slow but accurate
Much of performance can be
characterized by the Speed
Accuracy Tradeoff Lew, G. et al. (2007). User performance with iPhone. Unpublished
research.

6. Intended Keypresses vs. Actual Keypresses
Presented Letter (Intended Key Press)
Actual Key Pressed
Correct key presses > 10 incorrect > 20 incorrect > 40 incorrect
6

7. Implications for Design
7

8. Imagine You Are Shown This Picture: Hokusai's Wave
8

9. Perceptual Processes
9

10. Eye tracking Reveals Perceptual Processing
Quantitative Measures Meaning
Informativeness of an area / user
# fixations on an area
interest in the area
Info clarity / info density / info
Fixation length
processing demands
# fixations before target
Layout effectiveness / search
Time to 1st target fixation
demands
Scanpath complexity
% users fixating on an area
Prominence / perceived importance
Order of 1st fixation
of an area
# visits to area
Cognitive processing demands /
Pupil diameter
user mental workload
e.g., Bojko, A. (2009). Informative or misleading? Heatmaps deconstructed. Proceedings of the 13th International
Conference on Human-Computer Interaction (HCII). San Diego, CA.
10

11. Usability Derives from Behavioral Science
Motor Cognitive
Perceptual

12. Plan
 Usability Defined
 Some Examples and Implications
 Usability in Healthcare Technology
12

13. Usability Defined
Usability is not…
So what is usability?
13

14. Usability Defined
What Usability is Not
User
Acceptance
Testing
Aesthetic Functional
Attitude & Opinion
(Not behavior)
14

15. Usability Goes by Many Names
 Human factors
 Engineering psychology
 Ergonomics
 User experience
 User research
 User-centered design
15

16. Usability Defined
yü-zə-ˈbí-lə-tē
Usability is the effectiveness, efficiency, and satisfaction
with which the intended users can achieve their tasks in
the intended context of product use.
Adapted from National Institute of Standards & Technology
Outcome Often Get Confused
Process
Or
© May 28, 2009 – Proprietary and Confidential
Attribute
16

17. Some Context
 Usability derives from applied Key Components of Usability:
behavioral sciences – Measurement (Diagnosis)
– Psychology – Intervention (Design)
• Cognitive
• Experimental
• Biological
– Neuroscience
– Anthropology
– Industrial engineering
– Human-computer interaction
– Computer science
– Linguistics
– Market research
 From these we borrow:
– Research base
– Experimental methods
17

18. Usability Defined
A Brief History of Usability
18

19. A Few that Embrace Usability and User Research
Each of these organizations
has more than 100
usability staff members
19

20. A Few Ways to Measure User Performance
 Reaction times  Error
– Stimulus onset to response – Number / type / severity
 Task Completion Times  Efficiency
– Complex sequence of actions – Ratio of Actual behavior to
Optimal behavior
– E.g., Steps, keystrokes, clicks
 Attention
– Eye tracking
 Subjective Measures
– Ratings of Satisfaction / Usability /
 Subjective time on task duration
Usefulness
– Difficult tasks overestimate time
 Learnability
– Measure trials to criterion
20

21. Usability as a Process
Existing Context Mental Iterative Validation
interface for use model design & testing
Diagnose Explore Identify key user Design, review, validate design, repeat
context for use goals and approach
• Navigation and
• User • Goals
screen flow
• Evidence- research analysis
based interviews • Interaction
evaluation/ • Task storyboards
Cognitive • Design
• Field analysis
reviews
walkthrough observations
• Interaction • Low-fidelity
• User surveys • Site visits models wireframing
• Validation
interviews, lo
• Content/ • Card sorting w-fidelity UT
feature • High-fidelity
mapping • Site prototyping, inc
Mapping luding • Usability
• Eye tracking iconography testing
and visual
aesthetics • A/B testing

22. User Research (Measure & Design)
 Observe users in their “native” Pharmacy Visit
environment
 Identify features most needed and
relevant to users
 Identify user interface and
functional areas that users see as
most lacking
 Determine metrics that can be
used to measure return on
investment
Hospital Visit
On-site in an airplane simulator
22

23. Lab-Based Usability Testing (Measure & Design)
 Representative users perform
representative tasks in a controlled
setting
 Collect performance and feedback
data
Usability Testing
 Outcome:
Diagnose
problems,
discover
remedies for
repair and /or
collect base-
line performance metrics for future
comparisons
Observation Using One-way Mirrors and Video
23

24. Eye Tracking (Measure)
 Captures the location of the eyes while
looking at a stimulus
 Offers insight into the perceptual and
cognitive processes involved in user
interaction
 Can be used with computer
interfaces, physical products, and Eye tracking study for a financial Web site
printed material
Eye tracking quot;heat mapquot; report
24

25. Evidence-Based Evaluation & Cognitive Walkthrough
(Diagnose)
 A detailed assessment of the artifact:
– Grounded in applied research
– Tempered by experience
– Guided by best practice and Evaluating a mobile application
population stereotypes
– Scenario-based simulation of
common and
critical tasks
based on
user
personas
Actionable Recommendations in a Report
25

26. Information Architecture & Interaction Design (Design)
Information architecture (IA) is the Interaction design defines the user
blueprint for a website or interface. interactions with the interface.
 Structure
 Organization
 Navigation
 Labeling
Good interaction design makes the user’s
interactions as simple and efficient as
possible.
26

27. Experience Design (Design)
User-Centered (Clinician-Centered) Design
 Discovery
– Measurement
– Observation
– Interview
– Survey
 Information architecture
 Wireframe mockups Card sorting
 Prototyping
 Specifications
 Visual design
Wireframe mockups
Graphic design
27

28. Some Examples and Implications
Typography
Language and labeling
Visual search
Form design
28

29. Impacts of Typography on User Performance
29

30. Evaluation: Simple example
Ficticious Patient
Let’s look at something as the fonts:
Button labels: ALL CAPS, color coded,
Button labels: ALL CAPS, color
12 pt, Arial, Bold
coded, 14 pt, Arial, Bold
Field labels: Mixed case, 12 pt, Arial,
Field labels: Mixed case, 12 pt, Arial,
regular
regular regular
pt, Arial, © May 28, 2009 – Proprietary and Confidential
Patient info1: Mixed case, 32 pt, Arial,
Patient info1: Mixed case, 32 pt,
blue, bold
Arial, blue, bold
pt, Arial, blue, bold
Patient info2: Mixed case, 9 pt, Arial,
Patient info2: Mixed case, 9 pt, Arial,
underlined, regular
underlined, regular
pt, Arial, underlined, regular
30

31. Typography for Labels – Which is best?
Baseline ALL CAPS Underline Bold Highlighted
Arial 12 Arial 12 Arial 12 Arial 9 Arial 14 Times 12 Arial 12 Arial 12
Common COMMON Common Common Common Common Common
Common
Department DEPARTMENT Department Department Department Department Department
Department
Meds MEDS Meds Meds Meds Meds Meds
Meds
OrderSets ORDERSETS OrderSets OrderSets OrderSets OrderSets OrderSets
OrderSets
Fluids FLUIDS Fluids Fluids Fluids Fluids Fluids
Fluids
Drips DRIPS Drips Drips Drips Drips Drips
Drips
Insulin INSULIN Insulin Insulin Insulin Insulin Insulin
Insulin
TPN TPN TPN TPN TPN TPN TPN
TPN
Unsent UNSENT Unsent Unsent Unsent Unsent Unsent
Unsent
Existing EXISTING Existing Existing Existing Existing Existing
Existing

32. Line Length & Case
It was the best of times, it was the worst of times, it was the age of wisdom, it was the
age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the
season of Light
Line lengths of 4 (+/- 1.2) inches
are easiest to read. Lines greater
It was the best of times, it was the worst than 5 inches strain the eye as it back
of times, it was the age of wisdom, it scans, making it easier to jump to the
was the age of foolishness, it was the
wrong next line. Lines less than 2.5
epoch of belief, it was the epoch of
incredulity, it was the season of Light
inches slow reading due to the large
number of eye movements
IT WAS THE BEST OF TIMES, IT WAS ALL CAPS can reduce reading
THE WORST OF TIMES, IT WAS THE speeds by as much as 20%
AGE OF WISDOM, IT WAS THE AGE
OF FOOLISHNESS, IT WAS THE
EPOCH OF BELIEF, IT WAS THE
EPOCH OF INCREDULITY, IT WAS
THE SEASON OF LIGHT
32

33. What Do We know?
 Reading speed:12% slower on screen than paper
 Typographic characteristics also negatively affect reading speed:
ALL CAPS,Bold, underline, font type and fon t s ze
i
 Reading comprehension: Often worse on screen than paper
 Reading speed and comprehension all affected by:
Font color, background color, contrast, alignment, and line length
Implications? So What?
 Reading charts on line is slower with lower comprehension than
paper
 With fatigue, typography has cumulative effects
 Use visual design to improve (and even beat) status quo

34. So What Should You Do? Typography Recommendations
 Stay within one font family (usually sans serif online)
 Keep text sizes readable (12 pt optimal, >9 pt, < 14 pt)
 Use high contrast ratios darker letters on a lighter background
 Use mixed case
 Avoid use of italics and underlining as emphasis tools
 Use left aligned and ragged right text (Not centered or right aligned)
 Use only a few unique margins on page (Use indentations sparingly)
 Use “Title Casequot; or bold when emphasis is needed
 Avoid color coding text
34

35. Implications of Language
35

36. Language Affects Speed And Comprehension
False
True statement,
statement, Negat
Affirmative (Yes)
ive (No)
Response
Response
The circle isis above the star
The circle not above the star
Yes or No?
36

37. Findings
Statement Statement is Response is
(1) The circle is above the star. True Affirmative
(2) The star is above the circle. False Affirmative
(3) The star is not above the circle. True Negative
(4) The circle is not above the star. False Negative
 True statements evoked faster responses than false statements
 Positive (affirmative) responses much faster than negative (~ 500 msec)
Carpenter, P. A., and Just, M. A. (1975). Sentence comprehension: A
Clark, H. H., & Chase, W. G. (1972). On the process of comparing psycholinguistic processing model of verification. Psychological Review,
sentences against pictures. Cognitive Psychology, 3, 472-517. 82 (1), 45-73.
37

38. So What? Think About Dialog Boxes in CPOE…
Affirmative Negative
Non Destructive
What just
Happened?
Different phrasings are used when you want different responses
Statement Type Use When
 True Affirmative Fast, easy, low-cost-to-user outcome, confirmation only
 False Affirmative Need user to think about the response, high cost to user
 True Negative Should almost never use
 False Negative Never use, unless trying to deceive user
e.g., Opt out response [ ] Do not send me the newsletter
38

39. Visual Search
Finding needed data
39

40. Visual Design and Signal Detection
 Users have goals, e.g.,
– Review lab reports for abnormalities
– Ensure no allergies to certain medications
 We use perceptual and cognitive processes to
review screens
– Goal is “target” or “signal”
– All else is “noise”
 Good design has high signal:noise ratios
– Anticipates important signals
– Enables location of signals
• Proximity
• Coding
• Redundant queing
 To illustrate this point…we need a little help
40

41. Signal Detection: Find Waldo

42. End of a Long Frustrating Day…
Based on patient
information would you
prescribe Amoxicillin?

43. Ficticious Patient
Expert Evaluations: CPOE example
© May 28, 2009 – Proprietary and Confidential 43

44. End of a Long Frustrating Day…
How did the patient’s
white cell count change?

45. Improper
Incorrect
Alignment
use of
Of fields
color
Incorrect
use of Poor use
Incorrect
color of coding
use of
data
color

46. What Should You Look for in Evaluations & Walkthroughs?
Use the published research, knowledge of best practices, population stereotypes, and
expert opinion to analyze the user interface. For example we would go deep into the
following areas:
• Page Layout • Screen-Based Controls
• Navigation • User experience (e.g., response
• Scrolling & Paging times)
• Headings, Titles & Labels • Graphics, Images and Multimedia
• Links • Content Organization & Style
• Text Appearance • Search
• Lists • Accessibility
Additionally, we simulate common and critical use cases and scenarios based on
personas
46

47. Navigation & Mental Models
47

48. User Research: Defining Process Flows
 Stocking Errors: Drug is placed where it is not supposed to be
 Selection Errors: Wrong drug is selected from among other drugs
 Verification Errors: Incorrectly selected drug is confirmed as correct.
 Data entry Errors: Drug administering device is incorrectly programmed.
48

49. Examples: User Testing, Eye Tracking & Form Design
May 28, 2009 49

50. What Do We Know (or Think We Know) About Form Layout?
Ficticious Patient
50

51. “Left” Layout
 Advantages
– Because eyes are naturally drawn to
the hard edge on the left:
• It is easy to find the beginning of
the next label.
• Labels are easy to scan (in
previewing an unfamiliar form or
finding required fields).
 Disadvantages
– Large distances between labels and
fields can make the label-field
association difficult.
– Cannot easily accommodate labels of
very different lengths.
51

52. “Right” Layout
 Advantage
– Labels and fields are easy to associate
because of their proximity.
 Disadvantages
– Ragged left edge makes it more
difficult to:
• Scan of all labels quickly in order
to preview the form.
• Find the beginning of the next
label when completing the form.
52

53. “Top” Layout
 Advantages
– Can accommodate long labels
and fields.
– Provides more flexibility for
localization.
 Disadvantages
– Page length increases and the
form may require scrolling.
– Labels may “get lost” between
the fields.
53

54. “In-Field” Layout
 Advantages
– Requires the least amount of space.
– Does not interfere with other elements
on the page as much as the other
layouts do.
 Disadvantages
– Sometimes it is not obvious that the
fields are editable.
– Fields must be at least as long as their
labels, which can lead to unnecessarily
long fields.
– Interaction can be unclear, frustrating,
and cumbersome:
• If the label disappears on focus,
users might miss it.
• If the label does not disappear on
focus, users have to delete it.
54

55. “Flow” Layout
 Advantages
– Labels and fields are easy to associate
because of their proximity.
– Labels are easy to scan.
 Disadvantages
– Form can appear cluttered and
unprofessional, which may decrease the
site’s credibility.
– Quick scan of user inputs is difficult due
to the lack of left alignment of the
fields.
55

56. The Five Layouts, Again
FORM 1 FORM 2 FORM 3 FORM 4 FORM 5
“left” layout “right” layout “top” layout “in-field” layout “flow” layout
 We asked for familiar information that everyone should know:
– Name, address, phone number(s), email address(es), and date of birth
56

57. Lab Setup
Moderator’s station to view Face Tobii 1750 remote eye-tracking
eye gaze in real time and camera system integrated into a 17”
control eye tracking software monitor (set to 1024 x 768 px)
One-way
mirror
Moderator Participant
57

58. Participants and Procedure
 33 adults
– 16 males and 17 females
– Ages 22 – 61 (M = 29)
– From the Chicago area
 Individual, 10-minute long sessions
 Within-subjects experimental design
– Every participant completed all five forms.
– Layouts were presented in a counterbalanced order.
 At the end, participants viewed all 5 layouts and rated them in terms of:
– Ease of filling out
– Visual appeal
58

59. Findings Outline
 Performance
– Form completion time
– Errors
• Corrected
• Uncorrected
 Eye movements
– Number and distribution of fixations
– Average fixation duration
 User satisfaction
– Ratings of ease of filling out
– Ratings of visual appeal
59

60. Form Completion Time
Time measured from the
form appearance on the
screen to its submission.
Form Completion Time (s)
80
60
Participants
Form 1 (left)
spent less than a
minute per form. Form 2 (right)
40 Form 3 (top)
Form 4 (in-field)
Form 5 (flow)
20
54.2 55.0 58.8 59.3 54.8
0
There were no differences
between the form layouts in
any of the error categories All layouts were
(p > .05). equally fast (p > .05)
60

61. Number and Distribution of Fixations
FORM 1 FORM 2 FORM 3 FORM 4 FORM 5
“left” layout “right” layout “top” layout “in-field” layout “flow” layout
61

62. Number and Distribution of Fixations
Indicates scanning
efficiency.
Number of Fixations
100
On
average, partici
pants made 77 80
– 94 fixations
(min 40 s) per Form 1 (left)
form. 60 Form 2 (right)
Form 3 (top)
40 Form 4 (in-field)
Form 5 (flow)
20
92 94 77 90 91
0
There were no
Form 3 had the least fixations of differences between
all forms (p < .05). Forms 1, 2, 4, and 5.
62

63. Number and Distribution of Fixations
 To understand why Form 3 had fewer
fixations than the other forms, we
decided to focus on fixations on the
fields and labels as opposed to
fixations on the entire form.
63

64. Number and Distribution of Fixations
Form 3 had fewer fixations on white space
than the other four forms (p < .005).
Number of Fixations
100 Form 1 (left)
Form 2 (right)
Form 3 (top)
80
Form 4 (in-field)
Form 5 (flow)
60
40
20
92 94 77 90 91 48 52 51 51 50 42 41 25 36 38
0
Entire Form Only Labels & Fields White Space
64

65. Number and Distribution of Fixations
…can be explained by…
Number of Fixations
100 Form 1 (left)
Form 2 (right)
Form 3 (top)
80
Form 4 (in-field)
Form 5 (flow)
60
40
20
92 94 77 90 91 48 52 51 51 50 42 41 25 36 38
0
Entire Form Only Labels & Fields White Space
Form 3 had the least fixations because participants did not look at white space as much as they did
on the other forms.
65

66. Number and Distribution of Fixations
 Form 3 had a focused top-down scan pattern, and in terms of
scannability, was the most efficient.
 Fewer fixations were “wasted” on white space
66

67. Average Fixation Duration
Indicates difficulty in
extracting and processing
information (due to higher
information
Average Fixation Duration (ms) density, ambiguity, or
350
complexity).
Average fixation
duration was
~one quarter of 300
a second.
250 Form 1 (left)
Form 2 (right)
200
Form 3 (top)
150 Form 4 (in-field)
Form 5 (flow)
100
50
236 263 294 298 262
0
Forms 3 & 4 had longer fixations than
the other forms (p < .05).
67

68. User Ratings: Ease of Filling Out & Visual Appeal
 Using a scale from 1 to 10, please rate each of the forms you filled out based on how
easy they were to fill out:
 Using a scale from 1 to 10, please rate each of the forms you filled out based on how
nice they look / how visually pleasing they are:
1 2 3 4 5 6 7 8 9 10
Very Very
Difficult Easy
FORM 1 FORM 2 FORM 3 FORM 4 FORM 5
“left” layout “right” layout “top” layout “in-field” layout “flow” layout
68

69. User Ratings: Ease of Filling Out & Visual Appeal
User Ratings
10
The two scales have
a different loser.
7
Form 1 (left)
Form 2 (right)
Form 3 (top)
4 Form 4 (in-field)
Form 5 (flow)
8.5 8.9 7.4 4.9 6.4 8.3 8.6 6.1 5.1 4.2
1 Form 5 was considered the
least visually appealing
Ease of Filling Out Visual Appeal
(likely due to the lack of
field alignment).
Form 4 received the
lowest ratings for ease
of use (likely due to its
label impermanence).
69

70. Summary of Findings
PERFORMANCE: All EYE MOVEMENTS: Form EYE MOVEMENTS:
forms were completed 3 had fewer fixations than Forms 3 & 4 had the
in the same amount of the other layouts because longest fixations due to
time and with similar fewer fixations were higher information
accuracy. “wasted” on white space. density per fixated area.
FORM 1 FORM 2 FORM 3 FORM 4 FORM 5
“left” layout “right” layout “top” layout “in-field” layout “flow” layout
SATISFACTION: Forms 1 & 2 were SATISFACTION: SATISFACTION:
rated as the easiest to fill out and the Form 4 was rated as Form 5 was rated as
most visually pleasing. the most difficult. the least appealing.
70

71. Illustration of…
 Pure user testing methods Ficticious Patient
– No differential findings
 Eye tracking
– Teased out form 3 – “Top”
 Preference / satisfaction data
– Visual appeal ruled out Form 5 - Flow
– Satisfaction eliminated Form 4 – Inline
 Pragmatics / style allow selection of one of three styles (left, right, top)
© May 28, 2009 – Proprietary and Confidential 71

72. Design Can Be Fraught With Usability Issues
Bad screen
Layout
Wrong
Mental Model
Poor use of
Language
Inefficient
Navigation
72

73. Usability in Healthcare Technology
Tangible Financial Benefits of Usability
User Centric repeatedly sees and
the data support
 Reduced development costs
 Increased user productivity
 Decreased user errors
 Increased market acceptance
 Increased satisfaction
 Decreased training costs
 Decreased user support costs Roger Pressman, Software Engineering:
A Practitioner’s Approach, McGraw-Hill
Support calls reduced by one third for
canterburyofnz.com
73

74. So How Does Usability Fit in HIT?
74

75. What Can Usability Do For HIT?
 Measure  Design
– Where is current performance? – Design user interface architecture
• Baseline key activities – Develop UI workflows
• Understand errors
• Measure satisfaction
– Wire frame screens
– Where can automation improve on
current practices?  Iteratively Test
– Test early with real users
– Test often
 Define User Needs – Redesign
– Interview key users
– Translate those to user interface  Commitment to Performance
requirements Objectives
75

76. Representative Design / Test Cycle
76

77. STEEP and Usability Benefits
 Improved Safety – Reduce errors
 Timeliness – Improve comprehension and decision making
 Effectiveness – better decision making, reduced errors, improved adoption
 Efficiency – Faster comprehension and decision making, reduced learning
time
 Equitable – Improved adoption…
 Patient – Centered – Better decision making, reduced errors
77

78. What Would Make a Successful Project Team
Clinician-Centered Design Practice Involves:
 Clinical Staff (Users)
– Representatives from all impacted groups
 Information Technology Team
– Business analysts
– Programmers
 Usability Team
– Researchers
– Designers
 Visual designer
78

79. Improving Usability in Healthcare Technology
Understand the business objectives and constraints, content, and users.
Business/Context
 Understand the business objectives
and constraints, technological
constraints, resources, culture and
politics
IT Professionals
Content
 Understand the quantity and the
quality of the content
Users Healthcare Usability Experts
 An effective information architecture Professionals
must reflect the way users think.
Expand the “sweet spot”
79

80. Barriers to implementing Electronic Health Records (EHRs)
81

81. Implementing EHRs: At what cost?
82

82. How does this happen? Content analysis of RFPs
 Analyzed 47 RFPs and Selection Guidelines for
EHRs
 32 out of 47 RFPs made no mention of usability
 Of the 15, where usability is mentioned,
descriptions are often vague:
– “Ease of use” is mentioned in the selection
guideline
– “Demonstrations are evaluated on
intuitiveness and usability.”
– “Ease of Use (minimizes typing, is intuitive,
simple layout)” is given a suggested 8%
weight in the vendor selection criteria
– “General usability”
83

83. Evaluating the usability of EHRs during the procurement process
How to evaluate the usability of EHR systems:
1. Identify the key user groups
2. Identify critical and frequent tasks
3. Benchmark key task efficiency
4. Estimate usability
5. Measure usability
84

84. Key Process Element #1: Identify the key user groups
 Users who will be doing the
critical tasks
– Physician
– Nurses
– Administrative staff
85

85. Measuring usability of EHR systems
User Groups: Physicians | Nurses | Administrative
Measuring Usability Relative to Goals
Effectiveness Efficiency Satisfaction
Unassisted task Maximum user time (not Post-task ratings on 5-
Task completion rate of: machine time) of: point usability scale:
Task A
Task B
Task C
86

86. Key Process Element #2: Identify critical and frequent tasks
 Critical tasks and frequent
tasks will help prioritize
EHR features.
 ROI is maximized by:
– Task efficiency
– Effective training
87

87. Measuring usability of EHR systems
User Groups: Physicians | Nurses | Administrative
Measuring Usability Relative to Goals
Effectiveness Efficiency Satisfaction
Task Unassisted task Maximum user time (not Post-task ratings on 5-
Create an appointment
for a new patient
Check patient eligibility
for Medicare
Enter patient vitals
88

88. Key Process Element #3: Benchmark current task efficiency
 Benchmark current critical and
frequent tasks in order to:
– Show that the EHR system
produces better task efficiency than
paper methods
– Show improved efficiency over an
existing electronic system
– Project ROI for a given period
 Benchmark by:
– Directly measurement
– Expert estimation
89

89. Measuring usability of EHR systems
User Groups: Physicians | Nurses | Administrative
Measuring Usability Relative to Goals
Effectiveness Efficiency Satisfaction
Task Unassisted task Maximum user time (not Post-task ratings on 5-
Goal: 100% Goal: 2 mins Goal: 4.00
Create an appointment
for a new patient
Goal: 100% Goal: 1 min Goal: 4.00
Check patient eligibility
for Medicare
Goal: 100% Goal: 30 secs Goal: 4.00
Enter patient vitals
90

90. Key Process Element #4: Do First Pass Assessment
 Estimating usability, as a first
pass, helps to narrow the field of
available EHR systems.
 Estimation can be obtained via:
– Cognitive walkthrough
– Expert testing
– User Performance Modeling
GOMS
91

91. Measuring usability of EHR systems
User Groups: Physicians | Nurses | Administrative
Measuring Usability Relative to Goals
Effectiveness Efficiency Satisfaction
Task Unassisted task Maximum user time (not Post-task ratings on 5-
Goal: 100% Goal: 2 mins Goal: 4.00
Estimated Estimated Estimated
Create an appointment EHR A: 90% EHR A: 4 mins EHR A: 3.00
for a new patient EHR B: 100% EHR B: 1.5 mins EHR B: 4.50
Goal: 100% Goal: 1 min Goal: 4.00
Estimated Estimated Estimated
Check patient eligibility EHR A: 95% EHR A: 2.5 mins EHR A: 4.00
for Medicare EHR B: 100% EHR B: 1 min EHR B: 5.00
Goal: 100% Goal: 30 secs Goal: 4.00
Estimated Estimated Estimated
EHR A: 90% EHR A: 45 secs EHR A: 3.25
Enter patient vitals
EHR B: 100% EHR B: 30 secs EHR B: 4.75
92

92. Key Process Element #5: Measure usability
 Measure the usability of the
remaining EHR candidates.
 Measuring usability can be
obtained via:
– Usability testing
– Log files
– Observation
– Questionnaires
 Experts are required to measure
the usability of EHR systems.
93

93. Measuring usability of EHR systems
User Groups: Physicians | Nurses | Administrative
Measuring Usability Relative to Goals
Effectiveness Efficiency Satisfaction
Task Unassisted task Maximum user time (not Post-task ratings on 5-
Goal: 100% Goal: 2 mins Goal: 4.00
Estimated Estimated Estimated
Create an appointment EHR A: 90% EHR A: 4 mins EHR A: 3.00
for a new patient EHR B: 100% EHR B: 1.5 mins EHR B: 4.50
Actual Actual Actual
EHR B: 100% EHR B: 2 mins EHR B: 5.00
Goal: 100% Goal: 1 min Goal: 4.00
Estimated Estimated Estimated
Check patient eligibility EHR A: 95% EHR A: 2.5 mins EHR A: 4.00
for Medicare EHR B: 100% EHR B: 1 min EHR B: 5.00
Actual Actual Actual
EHR B: 100% EHR B: 45 secs EHR B: 5.00
Goal: 100% Goal: 30 secs Goal: 4.00
Estimated Estimated Estimated
EHR A: 90% EHR A: 45 secs EHR A: 3.25
Enter patient vitals
EHR B: 100% EHR B: 30 secs EHR B: 4.75
Actual Actual Actual
EHR B: 100% EHR B: 30 secs EHR B: 5.00
94

94. User Centric, Inc.
 User experience consulting firm specializing in user research and user-centered
design.
 Experience in the healthcare industry:
– Medical and pharmaceutical devices and applications
– Patient and professional reference materials
– Drug dispensing devices
– Prescription and medical labeling
– Patient-oriented and physician-oriented websites
– Medical packaging
– Personal Health Records
– Electronic Health Records
2 Trans Am Plaza Drive, Ste 100
Oakbrook Terrace, IL 60181
+1 630.320.3900
For papers and presentations: www.UserCentric.com
© May 28, 2009 – Proprietary and Confidential 95