1. UPA 2012
Usability Testing Medical Devices Las Vegas, NV
Dan Berlin Chris Hass
Experience Research Director SVP of Experience Design
@banderlin @chrishass
dberlin@madpow.net chass@madpow.net

2. Usability activities are NOT clinical Trials
Generally the process will be less stringent:
• More loosely defined, faster
• Will likely involve a relatively small number of people per activity
• Initial studies stress simulation, with no medicinal agents or clinical
impact.
• Dovetail with interaction design, visual design, and marketing activities
Usability
Tes,ng
Medical
Devices
2

3. It’s going to feel like a clinical trial
• Extremely detailed “Usability Specification” documents
• “Usability Validation” (Risk assessment and amelioration)
• “Usability engineering file” / “Design Efficiency File”
• Know Federal and International regulations:
• FDA Quality System Regulation
• HE: 75 2009
• ISO/IEC 62366:2007
• Knowledge of (and inspiration for)
emerging standards
Usability
Tes,ng
Medical
Devices
3

4. Medical Device Manufacturers may be unfamiliar
with your practices and lack of regulations
Usability
Tes,ng
Medical
Devices
4

5. Safeguarding participants may be more challenging
• Your safety protocols must be exacting, fully defined,
piloted, and effective
• Even simulated product use may put participants at risk
• Target audiences may be physically or cognitively
vulnerable
• Activity environments may be unusual
• Work with client and medical care specialists
Usability
Tes,ng
Medical
Devices
5

6. Reporting responsibilities, guidelines, formats, and
requirements will be more complex
• Reporting guidelines, formats, and requirements will likely
be extensive and complex
• Industry reporting standards (CIF)
• Specialized reporting formats
• Raw data may be reviewed again and again and again. . .
Usability
Tes,ng
Medical
Devices
6

7. Research locales may be more challenging
Device use environments may be:
• Unfamiliar
• Sterile
• Stressful
• Germ-laden
• Emotionally distressing
• Gross
• Secure
• Rarely observable
• Complex
• or otherwise unusual to non-medical personnel
• (And fainting can be a factor!)
Usability
Tes,ng
Medical
Devices
7

8. Client Relationship
Source: http://robot6.comicbookresources.com/2010/07/take-a-look-at-the-avengers-assemble-painting-by-alex-ross/
• A robust set of client team members is essential –
as is knowing their roles:
– Legal
– Medical advisor
– Safety advisor
– Compliance
– Adverse Event Specialist
– Physician is helpful, but not enough
– The usual suspects
• Clients may ask for proof of insurance
– But too much insurance could be a bad thing
Usability
Tes,ng
Medical
Devices
8

9. FDA Relationship
Source: http://news.petpardons.com/fda-denies-rumors-over-
possible-discovery-of-chicken-jerky-toxin/
• If the study is for FDA approval – learn about
the client’s relationship with the FDA:
– What are the relevant regulations?
– Who is the client’s liaison to the FDA?
– What is the client’s relationship with the FDA?
– Has the client met with the FDA?
– Have they received comments from the FDA?
– Does the client understand usability testing?
– Are we allowed to talk to the FDA?
Usability
Tes,ng
Medical
Devices
9

10. Working with an IRB
• IRB = Institutional Review Board Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/
who_is_watching_the_irb_watchdogs_asks_nature
• For FDA approval, typically work with an independent IRB
• Their job: to ensure that research conducted on human subjects is carried
out correctly and humanely
• Not as scary as you may think
– We are NOT evaluating medical efficacy or safety
– But we DO have to be conscientious
• Have your ducks in a row and it will be a smooth process
Usability
Tes,ng
Medical
Devices
10

11. Working with an IRB
• Adds about $2000-$5000 to the cost of the study
Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/
who_is_watching_the_irb_watchdogs_asks_nature
• Adds a day or two to the project for dealing with
the IRB
• Be cognizant of any non-disclosure agreements
• Prior to the study, signature pages should be
signed-off, circulated, and returned in a timely
manner
• Everyone on the team will take this very
seriously!
Usability
Tes,ng
Medical
Devices
11

12. IRB Documents - Forms
Source: http://blog.pennysaverusa.com/tag/tax-tips/
• Indemnification Agreement: So the IRB doesn’t get sued
• Financial Disclosure: The researchers don’t have conflicts of interest
• Investigator Statement: The researcher code of conduct
• Satellite Site Application: May be needed if the study involves multiple
locations
• Shipping & Invoice: Where documents are sent
• Study Application: Detailed information about the study
• Curriculum Vitae for all researchers
• Other forms that your IRB requires
Usability
Tes,ng
Medical
Devices
12

13. IRB Documents – Study and Device Info
• Recruitment Screener: Be very precise Source: http://healthcareit.etisbew.com/
• Recruitment Flyer: To hang in doctor’s office to aid with recruiting
• Moderator’s Guide: A second guide may be needed if there are child
participants
• Consent and asset forms: For adults and children, respectively
• Safety Concerns: Outline potential risks
• Device Background: Detailed explanation of the device including
indication and new features
• Device Instructions for Use: If they are also included in the study
Usability
Tes,ng
Medical
Devices
13

14. IRB Documents – Validation Plan
• A detailed explanation of how the study will
validate that the device is usable
– If you’re used to winging it, you can’t do that
here
– Lay out every question and how it will be
interpreted
– Lay out a rationale for every metric
– Break down the pieces of the tasks and what
will be observed
– A detailed explanation of how the will be
analyzed and presented
• Quantitative measures (asked and observed)
– Rating scales
– Yes/No observations
Source:
hDp://www.vnet5.org/pub/approach/uva.html
Usability
Tes,ng
Medical
Devices
14

15. Chart
Color
Quantitative Measures
%
of
chart
that
is
red
%
of
chart
that
is
black
• Rating Scales
– Asked – Observed
• Ease of use • Usability
• Instructions for • Help needed
Use clarity • Number of
• Design errors
• Ergonomics
• Yes/No observations
– Did the participant do this, that, AND the other thing?
Ra,ng
No
Help
Needed
LiDle
Difficulty
Some
Help
Much
Help
Task
Failure
Explana,on
Completed
with
no
help
1-­‐2
small
errors
3-­‐5
errors
Moderator
gets
involved
Failure
Small
errors:
Doesn't
press
the
green
buDon
Doesn't
remove
cap
properly
Etc.
Usability
Tes,ng
Medical
Devices
15

16. Recruiting
• Give your recruiter a heads up LONG before the
study starts
– And start recruiting early
• Make a flyer to be hung in doctor’s offices and
distributed electronically which includes:
– The purpose of the study
– Why they would want to participate
– Compensation Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php
– Recruiter contact information
• During the project, any screener changes need to be
re-approved by the IRB
• Many institutions can’t “promote” a study that is
under their auspices
Usability
Tes,ng
Medical
Devices
16

17. Sample Flyer
Usability
Tes,ng
Medical
Devices
17

18. Recruiting
• Don’t be shy – makes friends with local hospitals
and clinics
– Nurses and admins are the gateway
– Have the sanitized IRB approval in-hand
• Find groups that can be a gateway to your desired
audience
Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php
– Healthcare institutions
– Local and national support groups
– Federal agencies
– Community support groups
– Craigslist & Social Media
• Plan for surprises – like extensions to the timeline
Usability
Tes,ng
Medical
Devices
18

19. Working with Children
• From a legal standpoint, a child is someone under 18
years old
Source: http://evilstaring.com/2011/06/08/amazingly-evil-child-3/
• Children cannot consent to participate in the study,
they can only assent
– An assent form explains the details of the study, in the
first person
• You will treat a 17 year old different than an 8 year old
– Be cognizant of the protocol to ensure you are
consistent in your data capture
• The parent conundrum: should they participate?
– At what point does it become co-discovery and is that Source: http://cautionmagnet.blogspot.com/2011/05/confused-kids.html
okay?
Usability
Tes,ng
Medical
Devices
19

20. Working with Children
• Put the child at ease:
– Explain the lab (to a point)
– Ask if s/he would like a break Source: http://lovefunnypictures.com/funny-pic-bush_funny_face.php
– Give reminders that s/he is doing a great job in the study
– Structure the study so that it isn’t drudgery
– Don’t make them remember rating scales
– Not all children like to be spoken to like a child
• Be extra vigilant for signs that the participant is telling you want you
want to hear
– As well as signs of fatigue and confusion
• You don’t need to be their friend, but don’t scare them away
Usability
Tes,ng
Medical
Devices
20

21. Moderator’s Guide
• Be precise with a tightly written protocol Source: http://tips.tntdental.com/?tag=website-navigation
• Must be well-defined in advance and piloted
– No changes to the protocol without revisiting the IRB, the client, the FDA
approval minutes, etc.
• Clients may be used to clinical trials and view usability methods as “thin”
– We’re not running a clinical trials, but that’s no excuse for not adequately
preparing
• Adhere to the protocol!
– Any changes require IRB approval
Usability
Tes,ng
Medical
Devices
21

22. Moderator’s Guide
• The moderator’s guide should capture: Source: http://tips.tntdental.com/?tag=website-navigation
– The tasks
– Steps in the tasks
– What should be observed
– The questions to be asked of the participants
– Observer rating scales
• Basically, a shortened version of the Validation Plan
• Be precise in the metrics and methods
– They cannot vary
– Data reporting requirements will be significant
– Gaps in the reporting will be very visible
Usability
Tes,ng
Medical
Devices
22

23. Notes Grid
• We are using “micro-tasks” and want to
capture precise information
• The notes grid should facilitate the analysis
Source: http://beta.armagetronad.net/design/tron.html
yet be easy to use
– Every step and question on its own row
– Have clear study sections (introduction, etc.)
– Grey out areas that can be ignored
• Team note taking: moderator concentrates on
certain items, note taker does the rest
Usability
Tes,ng
Medical
Devices
23

24. Notes Grid
Usability
Tes,ng
Medical
Devices
24

25. Lab Preparation
• Capture the participants from two
different angles
– A close-up view of the device interaction
(over-the-shoulder)
– Straight-on to capture the entire field
• Have all device supplies at the ready
– Alcohol wipes, extra needles, batteries, Source: http://www.ebaumsworld.com/pictures/view/218095/
etc.
– You’ll use more supplies than you expect
• Get a first aid kit and nitrile gloves
– Have a first aid plan and practice
Usability
Tes,ng
Medical
Devices
25

26. Participant Safety
• As with any usability, participant comfort and safety is the
top priority Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg
• Must be mindful of participants with health conditions
– Will they need accessibility accommodations?
– Will they have an aide/caretaker?
– How will the condition affect their participation?
• What you can do to ensure participant comfort and safety:
– Have a robust first aid kit (and possibly medical personnel)
– Provide water, juice, sugary, and non-sugary snacks
– Ensure they know that they will be working with a placebo
– Accessible food
Usability
Tes,ng
Medical
Devices
26

27. Participant Safety
• Warn participants of safety hazards while using the device, if it’s
part of the protocol Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg
• Have a plan of action if someone is injured
• Establish a path of notification if a participant is injured
– Reporting may have many phases: reporting it internally, then to the
client, to the FDA, and the IRB
• Educate yourself about the condition
• Ensure the protocol does not exacerbate a disease or condition
• Be vigilant about fatigue, confusion, disorientation, dehydration, &
stress
Usability
Tes,ng
Medical
Devices
27

28. Adverse Effects
• An adverse effect (AE) event is when a participant
tells you about a side effect or other experience
while using a device or drug
• This is critical information for the client and must
be reported, typically immediately
Source: http://www.cartoonstock.com/directory/d/Din.asp
• Some clients may require you to go through AE
training
• Before the study begins, determine to whom AEs
are reported When in doubt,
report!
• AEs can be complex to identify, but are VERY
important
Usability
Tes,ng
Medical
Devices
28

29. Adverse Effects
• In the course of a usability study for a medication website, a participant
mentions that she stopped taking the drug. She explains that the blister
packs and pills sometimes arrived crushed.
• While testing an infusion pump, the participant mentions that the tube
always falls out of his previously used pump, which is the client’s
product.
• During a study, a participant mentions that her arms started to itch when
she stopped taking the client’s medication.
• A male participant mentions that his wife got pregnant two weeks after
he started taking the client’s medication, even though she used the birth
control pill.
Usability
Tes,ng
Medical
Devices
29

30. Analysis
• The setup from the validation plan, moderator’s
guide, and notes grid should make analysis easy
Source: http://www.webseoanalytics.com/blog/how-to-conduct-market-
research-competitor-analysis/
• Dealing with gaps in the data can be tricky
– Did the question get asked?
– Was the behavior observed?
– Do we have to go back to the recordings?
• Make sure you HAVE the recordings!
• Rely on quantitative measures
– What percentage of participants passed the task?
– What ease of use ratings did participants give?
– How many participants committed the most
common errors?
Usability
Tes,ng
Medical
Devices
30

31. Analysis
• Remember to think on a micro-task scale – every little
step of using the device should be observed
– It’s difficult to watch and report on every step of a
complex interaction and document it precisely, Source: http://www.webseoanalytics.com/blog/how-to-conduct-market-
research-competitor-analysis/
especially live
– Tallying needs to be specific, yet practical
• Be mindful of capturing and organizing the important
qualitative data
– This can be especially tough when dealing with many
participants
– Summarize qualitative data directly after the session –
don’t wait
• Know what you’re reporting – there may be data to
help marketing
• Sometimes the goal of these studies is to identify
marketing claims, which can raise concerns
Usability
Tes,ng
Medical
Devices
31

32. Thank You!
• Conducting a usability study for a medical
device is similar to a typical study, except
with much stricter protocols Questions/Comments?
• Your safety and the safety & comfort of the Dan Berlin
participants are paramount dberlin@madpow.net
@banderlin
• Use quantitative measures and report
percentages of participants that did or Chris Hass
didn’t perform an action chass@madpow.net
@chrishass
• It’s all about proper planning!
Usability
Tes,ng
Medical
Devices
32