1.
BUT DOES IT
WORK?
THE CRITICAL ROLE OF
EVALUATION
IN DIGITAL HEALTH
DUSTIN DITOMMASO | @DU5TB1N
SVP, BEHAVIOR CHANGE DESIGN

2.
| @DU5TB1N

3.
| @DU5TB1N

4.
| @DU5TB1N

5.
| @DU5TB1N

6.
| @DU5TB1N

7.
by Andrei Mironov
400,000

8.
by Andrei Mironov
2%

9.
by Andrei Mironov
%
+

10.
Larsen, M.E., Huckvale, K., Nicholas, J. et al. Using science to sell apps: Evaluation of mental health app store quality claims. npj Digit. Med. 2, 18 (2019).
DIGITAL
MENTAL
HEALTH
64%
CLAIMED EFFECTIVENESS AT
DIAGNOSING MENTAL
HEALTH CONDITION OR
IMPROVING SYMPTOMS

11.
Larsen, M.E., Huckvale, K., Nicholas, J. et al. Using science to sell apps: Evaluation of mental health app store quality claims. npj Digit. Med. 2, 18 (2019).
DIGITAL
MENTAL
HEALTH
3%
COULD BACK IT UP WITH
EVIDENCE

12.
by Bianca Williamson

13.
400,000
HOW DO WE
DECIDE?
COST?
AESTHETICS?
REVIEWS OR PERSONAL
RECOMMENDATIONS?
DATA-PRIVACY
& SECURITY?
REPUTATION, TRUST, &
CREDIBILITY OF APP
PROVIDER?
IF WE THINK IT WILL
WORK?
HOW MUCH EFFORT?

14.
IMPROVE
IN SOME WAY
SOMETHING

15.
MEET A
PERSONAL GOAL

16.
NEGATIVE EFFECTS
WARD OFF
OR SYMPTOMS

17.
RETAIN HEALTH

18.
FEEL BETTER

19.
WORK
MAKE THE DAY TO DAY
OF MANAGING A
CHRONIC CONDITION

20.
COGNITIVELY
EMOTIONALLY
PHYSICALLY

21.
EASIER

22.
THIS IS
OUR
PROMISE

23.
EFFECTIVENESS
IS THE MOST IMPORTANT
DIFFERENTIATOR
BETWEEN DIGITAL HEALTH
OFFERINGS

24.
DIGITAL HEALTH
DIGITAL MEDICINE
DIGITAL THERAPEUTICS
Digital therapeutic (DTx) products
deliver clinically evaluated,
evidence-based therapeutic
interventions to treat, manage, or
precent a broad spectrum of
diseases and disorders.
Digital medicine includes
evidence-based software
and/or hardware products
that measure and/or
intervene in the service of
human health.
Digital health technologies,
platforms and systems that
engage consumers for
lifestyle, wellness, and
health-related purposes;
capture store or transmit
health data; and/or
support life science and
clinical operations.
Clinical evidence is
required for all digital
medicine products
Clinical evidence and real-world
outcomes are required for all
digital medicine products.
Clinical evidence is
not required.

25.
DIGITAL THERAPEUTICS

26.
STANDARDS, FRAMEWORKS, & REGULATIONS
TO BE INCLUDED IN A HEALTHCARE SYSTEM FORMULARY OR PRESCRIBED BY AN MD,
YOUR APP MUST:
• DEMONSTRATE CLINICAL EFFICACY (OR PATIENT-RELATED IMPROVEMENTS) AND SAFETY;
• BE USABLE, ACCESSIBLE, AND INTEROPERABLE WITH HEALTH SYSTEM INFRASTRUCTURE;
• HOLD A HIGH STANDARD FOR DATA SECURITY AND PRIVACY.

27.
STANDARDS, FRAMEWORKS, & REGULATIONS
• A NUMBER OF APP RATING FRAMEWORKS HAVE EMERGED
• PAYORS/PROVIDERS CREATING FORMULARIES OF EFFECTIVE,
SAVE, AND TRUSTWORTHY DIGITAL HEALTH APPS AND SERVICES

28.
WHAT COULD WE
BE DOING?
CONSIDERING EVALUATION
AN INTEGRAL, DE FACTO
PART OF YOUR DESIGN
PROCESS
INCLUDE EVALUATION
ACTIVITIES THROUGHOUT THE
ENTIRE DEVELOPMENT CYCLE
FROM THE EARLIEST
CONCEPTING ONWARDS
START BY:

29.
QUALITATIVE EVALUATIONS:
MORE THAN RCTS
TO UNDERSTAND HOW PEOPLE EXPERIENCE OR FEEL ABOUT YOUR PRODUCT OR SERVICE.
GATHERING RICH DATA THROUGH SMALL SAMPLES. *YOU ARE PROBABLY ALREADY DOING THIS.
[ACCEPTABILITY, USABILITY]
DESCRIPTIVE EVALUATIONS:
TO UNDERSTAND WHAT’S GOING ON. HOW MANY PEOPLE ARE USING YOUR SERVICE, HOW OFTEN DO
THEY USE IT? HOW LONG DO THEY STAY ENGAGED, WHAT DO THEY USE? [PROCESS, ENGAGEMENT]
ECONOMIC EVALUATIONS:
TO ASSESS COST-BENEFIT ANALYSIS FOR YOUR PRODUCT COMPARED TO AVAILABLE ALTERNATIVES. OR
LIKELY COST-IMPLICATIONS OF ROLLING OUT/ADOPTING YOUR PRODUCT. [ECONOMIC EVALUATIONS]
COMPARATIVE EVALUATIONS:
BUT ALSO RCTS
TO UNDERSTAND WHETHER OR NOT YOUR PRODUCT IS WORKING AS INTENDED.
[EFFICACY, EFFECTIVENESS]

30.
CONCEPTING /
PRE-PROTOTYPE
PROTOTYPE ALPHA BETA LIVE
ACCEPTABILITY
FEASIBILITY
ENGAGEMENT
PROCESS
EFFECTIVENESS
OPTIMIZATION
USABILITY
EFFICACY
ECONOMIC

31.
HOW MIGHT WE
GET STARTED?
START BY CREATING A MODEL OF YOUR PRODUCT OR SERVICE
OUTCOMES
BEHAVIORS
BEHAVIORAL
DETERMINANTS
BEHAVIOR
CHANGE
TECHNIQUES
PRODUCT /
SERVICE
FEATURE
(CONTENT)

32.
REDUCED HIV
VIRAL LOAD
REDUCE
ALCOHOL
USE
ART / PREP
ADHERENCE
INTENTIONS
(TO REDUCE)
INTENTIONS
(TO MAINTAIN)
HEALTH BELIEFS
MODELING THE
BEHAVIOR
SOCIAL
SUPPORT
MENTAL HEALTH
STATUS
BEHAVIORAL
SKILLS
COMPONENT 1
MOTIVATIONAL
INTERVIEWING
COMPONENT 2
PEER
MENTORNING
COMPONENT 3
TEXT MSSGING
COMPONENT 4
MINDFUL
MEDITATION
COMPONENT 5
BEHAVIORAL
SKILLS
TRAINING
OUTCOME
BEHAVIOR
BEHAVIORAL DETERMINANT
PRODUCT COMPONENT

33.
IN SUMMARY
1. WE ARE RAPIDLY MOVING TOWARDS EFFECTIVENESS AS THE KEY
DIFFERENTIATOR IN DIGITAL HEALTH.
2. DIGITAL HEALTH PRODUCTS AND SERVICES NEED TO BE EVALUATED
THROUGHOUT THE DESIGN PROCESS (FOR A HOST OF REASONS).
3. THERE ARE A NUMBER OF EVALUATION OPTIONS FOR GATHERING
EVIDENCE ABOUT IF, HOW, AND WHY YOUR PRODUCT WORKS.
4. THERE ARE THINGS YOU CAN AND SHOULD DO NOW LIKE CREATING A
MODEL THAT WILL HELP YOU BE MORE EFFECTIVE IN YOUR ABILITIES TO BE
MORE EFFECTIVE.

34.
THANK YOU!
DUSTIN DITOMMASO | @DU5TB1N
SVP, BEHAVIOR CHANGE DESIGN