Evaluating Biometric Wearables: From Academics to Industry.

1. Evaluating Biometric
Wearables: From Academics
to Industry
Jennifer Bunn, PhD Chris Eschbach, PhD
Campbell University Valencell Inc.

2. The Plan
• Current wearables and biometrics
• Academic perspective
• What is driving industry
• Standards in evaluating biometric device
• Validating biometrics

3. Wearables

4. Many Form Factors
Earbuds Armbands Wrist Watches

5. Biometrics
• What the numbers mean and
why they are important
• Activity
• Sleep
• Health
• Stress
• Movement context
• Heart Rate
• Oxygen saturation
• Heart Rate Variability
• Breath Rate
• Galvanic skin response
• Blood Pressure
• Electroencephalography
• Calories
• Oxygen consumption (including
VO2max)

6. The Problem

7. The industry [and academics] lacks a clear method
of benchmarking sensor accuracy…damaging
reputation of the industry and allowing poor quality
sensors to flood the market and erode prices.
If industry awareness of what constitutes a good fitness
tracker does not improve, the fitness tracking industry
will be a low-quality, low-value industry
ABI Research. (February 19, 2015). Hot Tech Innovators. Retrieved from https://www.abiresearch.com/whitepapers/Hot-Tech-Innovators/

8. Academic perspective

9. A Few Notes about the Literature
• Selection of products to test
• Respecting the industry
• Updated hardware and software
• Proper wear of the device under
test (DUT)

10. Research Potpourri
• Benchmark comparison
• Steps – real-time (Sears et al., An et al., Modave et al., Fokkema et al.), pedometer,
video monitoring (Chen et al., Tudor-Locke et al.), none (Wen et al.)
• Heart rate – ECG (Gillinov et al., Wallen et al., Jo et al.), chest strap (Stahl et al.)
• Energy expenditure – metabolic analyzer (Wallen et al., Woodman et al.
Chowdhury et al.)
• Sampling rate of the metric
• Specific time points (Wallen et al., Gillinov et al.), specific steps (Modave et al.),
completion of activity (An et al, Sears et al.), continuous (Jo et al.)

11. Research Potpourri
• Lab vs. free-living
• Lab (Gillinov et al, Wallen et al., Sears et al.) lacks external validity
• Free-living (An et al., Jo et al.) – difficult to test, what is a step?
• Acceptable accuracy
• 5% (Feito et al.)
• 10% (CTA Physical Activity Standard)
• ±3-5 bpm (Terbizan et al.)
• Appropriate statistics

12. Research potpourri
Study Correlation MAPE Test of
differences
Test of
Equivalence
Bland-
Altman
An et al.
Chen et al.
Chowdhury et al.
Fokkema et al.
Gillinov et al.
Jo et al.
Nelson et al.
Stahl et al.
Wallen et al.
Woodman et al.

13. Industry drivers

14. Base: Online U.S. adults who are physically active (n=932); Online U.S. adults who own a fitness tracker and who are physically active (n=496)
Q5. For which of the following reasons, if any, do you exercise?
Consumer Electronics Association. (January 6, 2015). Wearable Activity Trackers: Engaging Consumers to Monitor Their Health. Retrieved from http://www.ce.org/research.aspx
The needs
4%
10%
9%
11%
5%
20%
14%
30%
41%
36%
42%
43%
50%
51%
58%
68%
10%
13%
14%
18%
18%
24%
27%
51%
56%
65%
66%
67%
69%
71%
79%
85%
Part of a rehabilitation program
Manage diabetes
Manage a chronic disease (other than heart disease…
Manage heart disease
Train for a race or competition
Your doctor or physician recommended/prescribed…
To save money on medical costs
Prevent disease
Maintain current weight
Increase endurance
Enjoyment
Build muscle or increase strength
Lose weight or reduce body fat
Reduce stress
Look better
Improve overall health
Consumer Motivation to Exercise
Fitness tracker owners
Online U.S. adults

15. The Needs
• What consumers want
• Health/Stress Reduction/Body Composition* (performance?)
• What industry wants
• Consumers
• What everyone needs
• ACCURACY
• Health
• Physical
• Workload context – energy expenditure
• Stress
• Workload context- exercise and non exercise
• Performance
• Workload context – training effect
• Do we actually have the above?
• Depends on who you are asking and what your measuring with

16. Standardization

17. Standardization efforts
• The Health and Fitness Technology Division of CTA strives to raise
awareness of how consumer technologies can help improve health and
fitness.
• CTA’s Health and Fitness Technology Subcommittee (R6.4) develops
standards, recommended practices, and related documentation for
consumer health and fitness technology, including fixed, portable and
wearable health and fitness devices.
• R6.4 WG 1 – Sleep Monitors
• R6.4 WG 2 – Physical Activity Monitoring Standards
• R6.4 WG 3 – Consumer EEG Data (No Report – On Hiatus)
• R6.4 WG 4 - Consumer Stress Monitoring Technologies
• R6.4 WG 5 – Mobile Health Applications Others

18. Standardization efforts
• Steps
• Physical Activity Monitoring for Fitness Wearables - Step Counting ANSI/CTA-2056
• Blood pressure
• Non-invasive sphygmomanometers — Part 2: Clinical investigation of automated measurement type
ANSI/AAMI/ISO 81060-2:2013
• IEEE Standard for Wearable, Cuffless Blood Pressure Measuring Devices IEEE Std 1708-2014
• Heart rate
• Physical Activity Monitoring for Heart Rate ANSI/CTA-2065
• ECG
• Medical electrical equipment — Part 2-27: Particular requirements for the basic safety and essential
performance of electrocardiographic monitoring equipment ANSI/AAMI/IEC 60601-2-27:2011
• Sleep
• Definitions and Characteristics forWearable Sleep Monitors CTA-2052.1
• Methodology of Measurements for Features in SleepTracking CTA-2052.2
• In progress
• Sleep- ANSI/CTA/NSF-2052.3, Performance Criteria andTesting Protocols for Features inSleepTracking
ConsumerTechnology Devices and Applications.
• Intensity-ANSI/CTA-2074, Intensity Metrics: Physical Activity Monitoring
• Stress- ANSI/CTA-2068, Definitions andCharacteristics of ConsumerStress MonitoringTechnologies
• Mobile health-CTA-2073, Guiding Principles of Practice andTransparency for Mobile Health Solutions

19. Validation

20. Why Validate
• Because ACCURACY is required
• During everyday life
• Determine moderate and vigorous intensity
• Quantify training load
• Determine caloric measures for energy balance
• Monitor stress levels
• Detect changes in “health”
• During periods of poor health (clinical or first responder)
• Accuracy is a matter of life and death
• During gaming
• To deliver ideal conditions (fun/stress/emotions)

21. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

22. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

23. Validating Biometrics
• Know your sensors- context matters
WORKLOAD
CONTEXT

24. Validating Biometrics
• Know your sensors- context matters
WORKLOAD
CONTEXT

25. Stress Testing the Biometric Sensor
• Know your sensors
• What is the use case

26. Stress Testing the Biometric Sensor
• Know your sensors
• Underlying science
• PPG vs ECG

27. Stress Testing the Biometric Sensor
• Know your sensors
• Points of failure
• How it fails
• Where it fails

28. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

29. Stress Testing the Biometric Sensor
• Set-up
• Data retrieval from
devices

30. Stress Testing the Biometric Sensor
• Set-up
• Data retrieval from the
DUT

31. Stress Testing the Biometric Sensor
• Proper methodology
• Set-up
• Data retrieval from the
device under test (DUT)

32. Stress Testing the Biometric Sensor
• Planning
• Documentation
• DUT - ABCv.w.x.y#z
• ABC-> indicates project
• v) Modifications to the mechanics of the core structure (i.e. external
id, stalk, sensor angle, ear tip)
• w) Additions to the core (gel designs, straps, fin modifications)
• x) Modifications to the sensor, sensor components or firmware
• y) Indicates sizing
• #z) Actual unit number (i.e ABC 5.4.3.M#1 should be the same as ABC
5.4.3.M#5)

33. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

34. Stress Testing the Biometric Sensor
• Suitable benchmarks
• Heart rate
• Heart rate variability
• Accelerometery
• Distance
• Breathing rate
• Oxygen consumption including VO2max (Calories)
• Blood Pressure

35. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

36. Stress Testing the Biometric Sensor
• Correct pool of participants
• Type (age, bmi, gender, skin type)
• Number

37. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

38. Lifestyle In-Session Health Monitoring
Lifestyle In-Session Health Monitoring
Wearability 24/7 comfort; visible Stable during target activities;
visible or invisible
24/7; invisible
Accuracy “Good enough” for
assessments
Real-time accuracy critical Real-time accuracy critical
Battery Life ≥ 3 days ≥ 3 hours ≥ 1 month
Engagement Daily, weekly, & monthly Daily, weekly, & monthly Clinician-dependent
Stress Testing the Biometric Sensor
• Proper methodology

39. Stress Testing the Biometric Sensor
• Proper methodology
• Protocols for assessment should include the following (regardless of use
case):
• Rest
• Steady
• Dynamic
• Mode (running, cycling, strength, lifestyle)
• Environmental conditions

40. Stress Testing the Biometric Sensor
• Proper methodology
• Protocols for assessment
• Environmental conditions
• Heat, cold, sunlight

41. Validating Biometrics
• Why validate
• Stress testing the biometric sensor
• Know your sensors
• Plan for assessment
• Suitable benchmarks
• Correct pool of participants
• Proper methodology
• Analysis

42. Stress Testing the Biometric Sensor
• Considerations
• All “benchmark units” have potential problems.
• Synchronize the data start time (because of latency or lack of sync)

43. Stress Testing the Biometric Sensor
• Sound and consistent evaluation techniques
• Qualitative and quantitative measures
• Subjective scoring
• Quantitative scoring
• Mean absolute percent error
• Distribution data
• Correlation
• Equivalence testing
• Mean bias
• Data availability
• Latency

44. Best practices
• Standardization
• Participants
• Settings
• Use cases
• Benchmark devices
• Data dropout