How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective

© CSS Corp | Confidential | www.csscorp.com 1 Customer Engagement Reimagined
How accurate are the wearable fitness trackers for step
counting(10000 steps a day): A tester’s perspective
James Mathew, Karthikeyan Swaminathan and Dr. Kiran Marri
CSS Corp Solutions

Disclaimer
This research work, data and results is not intended to substitute for informed medical advice or fitness regime
One must not use this information to check, confirm, diagnose or treat any health problem or fitness conditions.
Always check with your doctor before altering fitness regime or starting any new fitness routine
Several popular brands are tested in this research work, and thereby these brand names are intentionally not
mentioned in the paper

Table of Contents
Background - wearables, types and fitness band
Story of 10000 steps
Learnings, Conclusions
Thinking like a tester - Scenarios, Design and Wearable Test Framework
Key results
Automation strategies

71% in age group
16 to 24 want
“wearable tech,”
- Global Web Index
 https://www.grandviewresearch.c
om/industry-analysis/wearable-
technology-market
 https://www.statista.com/chart/33
70/wearable-device-forecast/
Rapid growth in wearables market Wrist wear - most popular wearable
2011: Fitbit Ultra (2011)
2012: Nike FuelBand, Fitbit One, Sony SmartWatch
2013: Pebble, Fitbit Flex, Sony 2, Samsung Gear, Nike FuelBand SE, MI Band
2014: Fitbit Force, Samsung Gear 2, LG Watch, Motorola watch, Garmin
2015: Apple watch
Popular products in the wrist wear category
Landscape of wearables
Market size | Types of wearables | Popular products | Sensors
 UV sensor
 GPS | Gyro
 Ambient light sensor
 3 axis accelerometer
 Skin temperature sensor
 Galvanic skin sensor
 Haptic vibration motor
Wrist wear: Microsoft Band 2 sensors
With 300% growth, wrist wear (band, watch) is the most popular device in the market
3 4
1 2

Wrist (fitness) band
Purpose | Measurements | Experience | Why 10000 steps
32 Experiences of wearable like a Fitbit,
Garmin watch, or Apple watch
Parameters measuredWhy fitness band?1
 Activity tracking
 Track progress on different days
 Focus on training/goal-oriented
 Habit information
 Brand-style quotient
 Weight loss programs
 Calorie burn counter
 Step counter/tracker
 Time trends in different segments
 Distance tracker
 Speed of pace in segments
 Infographics
 Personalized activity tracking is the most common use of wrist band
 Burning 300 calories is recommended by fitness community to lose weight, and this translates to 10000 steps (30% at
brisk pace, 2500 steps is 1 mile approximately)
10000 steps a day in urban group, and over 90% enjoy using wrist bands

Tester’s perspective:
Fitness band measurement | Hypothesis driven approach
Is the data displayed by the fitness band accurate?1
Representative Test Hypothesis
Influence of arm swing on the band readings
2
Effect of high activity vs low activity vs stationary position
3 Indoor (treadmill) vs external (outdoor) for different step count
4
Influence of person’s height on the step count
5
Problem statement: How accurate is the band, functionally?
High end wrist band vs low cost wrist band vs smart watch
6
Type of walking (regular vs running vs brisk walking)
7
Functional |
Performance | Security |
Usability | Localization |
Network connectivity |
Rigidity | Battery | Storage
capacity | Luminous | Data
integrity | Reports |
Compatibility | Environment
Coverage
GROUP A
GROUP B
GROUP D
GROUP C
Test Scenario Design : Activities with Steps | Stationary | Individuality | Fitness brand internals

Test considerations:
 Demography of the wearables: Typical user groups identification such as runners, swimmers etc. across geos
 Pre-condition: Definition of ordinary and extra-ordinary usage of the wearable
 Post-condition: Defects could relate to the hardware (such as rigidity), software (such as algorithm fine-tuning) or configuration (such as calorie re-calibration)
 Scope of testing: device black box functional QA.
Test Design Phase
Scenario design
& define
Hypothesis
Analysis
(factors,
attributes
Planning Reviews Servo
automation
Manual
Requirements
Analysis
Wearables Test
Strategy
Test Design Test Execution
Story
Demo/Acceptance
Test Execution Phase
Test
reports
Release
Retrospection
Proposed elements of wearable testing methodology driven by Hypothesis test approach
Independent
accelerometer
verification
Image
detection
Test Scenario Design : Activities with Steps | Stationary | Individuality | Fitness brand internals

1 STEP
Heel Off Toe OffHeel Strike Flat foot Mid stance Mid swing
Definition of a step:
A typical step and arm swing

Scope Measurement Automation Approaches/Tools
Outdoor walking + Automation Accelerometer
Indoor walking + Automation Video/ Object tracking
Arm swings + Automation Video/ Object tracking
Stationary (Non activity) Manual -
Stationary (In motion) Manual -
Height influence + Automation Video/ Object tracking
Distance walking + Automation Simulation
Time bound walking + Automation Simulation
Environment/ Surface Manual -
Test Scenario Design
Automation strategies
Is the data displayed from the fitness ….1
Influence of arm swing on the band …..
2
Effect of high activity vs stationary ….
3 Indoor (treadmill) vs external (outdoor) ….
4
Influence of person height on the step ….
5
High end wrist band vs low cost wrist ……
6
Type of walking ……..
7

Study Details: Subjects | Fitness Bands | Distances | Steps
Parameter Details and information Specifics for the results
# of participants 20-30 adults over a period of 12 months 22 healthy adults#
Gender All 3 Females | 19 Males
Age group Tested on Geriatrics, Gen X, Gen Y 23 to 50
Height of subjects 153 to 181cm
# of brand tested 6 brands, discarded 2 (too few samples or errors) 4 popular brands
Concurrency test 1 brand x 3 1 brand
Fixed steps test Factors: Practical, Manageable 40 to 500 steps
Fixed distance test 100m to 5000m 100m to 400m
Arm swing Normal, and up to elbow levels elbow levels benchmark
Speed of walk | run Individual capability None
# Results carried from Mar 2018 is included in the study

Steps Walked
Counting the steps walked Accelerometer variation during walking
 Manual counting of steps during testing
can be cumbersome and error prone
 A mobile app which measures the
accelerometer values is used
 Z axis indicates direction of move
 X axis indicates the left to right
movement
 A combination of readings from X and Z
axis accelerometer yields an accurate
count of steps
-6
-5
-4
-3
-2
-1
0
1
2
3
4
5
1
8
15
22
29
36
43
50
57
64
71
78
85
92
99
106
113
120
127
134
141
148
155
162
169
176
183
190
197
Accelerometer variation
X axis Z axis
Automation : Motion tracking of steps for indoor & outdoor
An mobile app was build to log the recordings, and # of steps derived

Reference: https://physlets.org/tracker/
Gait | Speed | Arm length | Swing
Motion Video
Feature analysis
Feature selection
Feature vectors
Model
Frames
Feature
extraction
Step
detection
Feature detection: Machine learning Object Tracking Software
Automation : Motion tracking of indoor events
Object tracking software is simple and easy but can be used only for treadmill with reference

Simulation
 Changing gear while driving a car:
With the vertical arm orientation,
testing for a person changing gear of
car can be done
 Writing activity: With a horizontal
arm orientation, testing for a person
writing can be done
 Activities such as steering wheel
simulation, writing, physical exercise
can be simulated
Simulation
 Short and long arm influence can be
tested by changing the arm length
 Fast and slow swing variations in gait
can be replicated
Servo
motor
Device
Arm
Servo
motor
Device
Arm
Configuration: The device and arm are
hanging from the servo motor
Configuration: The device and arm are
held up from the servo motor
Hand movement | Arm length | Swing
Prototype model is developed for testing durability*
*Results not included in the paper
Automation : Simulation of arm swing

Results

Hypothesis: Fixed distance walk results in similar variations for the band (if any)
Result 1. Consistency check of the Fitness band
Subjects are requested to walk casually for a fixed distance | Two brands are experimented | Outdoors
%variations
Unique variations
1. Data spread observed in both positive and negative variation (-16% to +8%) for normal walking conditions
2. Negative variations are stronger in both the brands
3. Indoor (on the treadmill) showed fewer variations as compared to outdoor
KEY OBSERVATIONS AND ANALYSIS
Result: Fail | Fitness bands tend to have inconsistent performance over 10+ walks
TEST SCENARIO
TESTDATA&PLOTS

Hypothesis: Speed of walking must notinfluence the fitness band output
Result 2. Effect of speed (brisk vs normal walking)
Subjects are requested to walk at normal and brisk speed for a fixed distance | Activity conducted both at outdoors and indoors
1. Fast walking had more spread as compared to slow walking
2. Spikes of inconsistency observed in normal walking and not reproducible
3. Statistically the hypothesis can be rejected as p-value is 0.06 (for 0.05) but due to F and F-crit value shows otherwise
Result: Fail | Fitness band outcome may have strong influence on the speed of walking
ANOVA
Source of Variation SS df MS F p-value F crit
Between Groups 90.38 1 90.38 3.78 0.06 4.08
Within Groups 957.24 40 23.93
Total 1047.62 41
Reference: http://www.statisticshowto.com/probability-and-statistics/f-statistic-value-test/
FAST
NORMAL
TEST SCENARIO
TESTDATA&PLOTS

Hypothesis: With increase or decrease in steps, consistency is maintained
Result 3. Effect of multiple steps
Result: Fail | Fitness band accuracy reduces with steps, and marginally more in indoors
Outdoor
Indoor
TEST SCENARIO
TESTDATA&PLOTS
Subjects are requested to walk fixed steps or distance | Natural speed | outdoors and indoors
1. Trends show that the error deviation increases with step count
2. The analysis is performed for over 100 steps
3. No statistical significance between outdoors and indoor data, with indoor showing higher COV

Hypothesis: In Idle state, steps do not add up
Result 4. Stationary position performance
Subjects in stationary position for fixed time, and requested to perform movements such as writing, eating, typing | Multiple brands | IndoorsTEST SCENARIO
1. Most of the brands gave at least 30% of the data accurately, with the highest being 70%
2. The average deviation ranged from 2% to 17%, and standing and sitting stationary movements, gave similar results
3. Accuracy of the steps is specific to the brand
Result: Fail | Error specific to Fitness band, but accurate at least 30% of times
TESTDATA&PLOTS
In motion (driving car| commute)
1. Error in the range of 10% to 100%
2. Data tested on different brands with fewer
samples, and hence results are non-conclusive

Hypothesis: Arm swing must not have any influence on Fitness band accuracy
Result 5. Effect of arm swing
Subjects are requested to walk with arm swing, normal and forcefully high for 100 steps | Activity conducted both outdoors and indoorsTEST SCENARIO
1. Few brands had minimum influence; certain brands had statistically significant difference with arm swing
2. Maximum and min variation between 2% and 14%
3. Overall analysis did NOT show any statistical difference, and hence this issue is very specific to brand
Result: Fail | Fitness band accuracy has influence on arm swing and very specific to the brand
SHORT
LONG
TESTDATA&PLOTS

Hypothesis: Fitness band when not calibrated for height results in error
Result 6. Influence of Height (UNCALIBRATED)TESTDATA&PLOTS
1. Height of the individual is important in the step count and concurred with Industry observations#
2. Actual steps matched closer with industry data for taller people as compared to shorter people
3. Fitness band accuracy is better for taller people (175+ cm) as compared to shorter people, due to difference in strides
Result: Pass | Fitness bands appear to be influenced by person height
# Reference: https://www.verywellfit.com/how-many-walking-steps-are-in-a-mile-3435916
#ofSteps
Height (cm)
Subjects are requested to walk casually for a fixed distance | Calibrated for 175cm | Activity conducted outdoorsTEST SCENARIO

1. The main objective of this study was to bring out the factors that influence fitness devices (wearables)
2. Wearable devices testing framework designed using Hypothesis testing, and 3 automation methods
have been proposed, and tested on 5 popular brands
3. The study brings out the importance of “individual parameters” (height, stride style, gait and arm
swings) that can alter the accuracy of the step count
4. Dominant activities such as walking are prone to lesser errors, but many stationary activities involving
hand movement lead to 50% or more issues
5. The study has a significant importance in rehabilitation and geriatrics, where the day long “steps” are
taken into account. This testing approach can be applied to validate other wearables in the field of
medical research, sports mechanics and animal management
6. The study proves that principles of testing can be enhanced using analytics, IOT, imaging and
automation
Conclusions
Summary | Learnings | Future plans

How accurate are wearables as step-counters?(target of 10000 steps a day)
Results show that dependency factors can skew the step count (Non-walk/run activities:
Stationary Cycling, Car driving, Commute, hand movements involved during eating, writing and more !!)
Recommendations
1. Decide when to wear (Wear fitness tracker only during active walk/ running)
2. Calibrate before use(brands that we tested have similar issues on different scales)
3. All day users (6AM to 10PM) to reset target as 12000-14000 steps a day, to meet
the goal
Conclusions
Inferences| Recommendations

Q & A
Comments:
Dr. Kiran Marri : Kiran.Marri@csscorp.com
James Mathew : James.Mathew@csscorp.com

Appendix

Results reporting
QA focus
Test activities considered
• Mobile activities
• Cycling
• Driving a car
• Travelling in a bus
• Immobile activities
• Typing/writing
• Exercise such as dumb bells
• Idling activities
• Sitting in a chair
• Sleeping in a bed
• Travelling in a car
• Typical walking
• Outdoor walking (slow or fast)
• Indoor walking (treadmill based)
Test subjects
• 16 people
• 3 brands of fitness wearables
• Both genders (3 females, 13 males)
• Age group: 23 to 50
• Testing done over 1 year (data
taken in last 5 months are
represented
• Over week-days and week-ends
Factors considered
• Arm length variations
• Arm swing angle and speed
• Fitness tracker on dominant vs.
non-dominant hand
• Mobile vs. immobile activity
• Wearing it in hand vs. ankle

Results reporting …continued
QA observations
Row Labels
Average of
Deviation (%)
In motion with no physical
activity 35%
In motion with physical
activity -1%
In stationary, with dominant
hand active 55%
In stationary, with non-
dominant hand active 0%
Date
Tester
ID
Activity
ID Test condition Test steps
Expected
results
(steps)
Actual
results
Observations
-Deviation
(%)
27-May-18 ABC01 A02
Human: Subject is
involved in slow/fast
walking activity
Tester walks 100 steps exactly. The number of
steps before and after the travel are read from the
fitness tracker 100 99 1%
27-May-18 ABC01 A02
Human: Subject is
walking activity
steps before and after the travel are read from the
fitness tracker 100 105 -5%
27-May-18 ABC01 A03
Human: Subject is
involved in performing a
regular activity such as
eating, typing
Tester sits in a chair and performs a regular activity
such as eating, typing, after wearing the fitness
tracker in the dominant hand. The number of steps
before and after performing the activity are noted 0 0 0%
01-Jul-17 ABC02 A01
Human: Subject is
involved in indoor-
cycling
Tester wears the fitness tracker in ankle. Uses an
indoor cycling exercise machine. Makes 100 times
of circular motion. Stops the cycle. The number of
steps before and after the exercise are noted
0 100 100%
01-Jul-17 ABC02 A04
Human: Subject is just
being idle
Tester sits in a chair for 30 minutes in a easy
position, with causal movements. The number of
steps before and after this positioning are noted 0 0 0%
25-May-18 ABC03 A02
Human: Subject is
walking activity
steps before and after the travel are read from
the fitness tracker 100 112 -12%
28-May-18 ABC04 A02
Human: Subject is
walking activity
steps before and after the travel are read from
the fitness tracker 100 99 1%
-10%
0%
10%
20%
30%
40%
50%
60%
In motion with
no physical
activity
In motion with
physical
activity
In stationary,
with dominant
hand active
In stationary,
with non-
dominant
hand active
Step count deviation chart

RoI analysis of test harness
Objective: Measure step count on fitness tracker with indoor slow/fast walking using different subjects
Inference: Measuring step count on fitness tracker with indoor walking, is optimal to get tested in a servo based test harness rather
than using different personnel subjects, as we get 14% from the 1st test iteration
Cost (USD) Savings (USD) Remarks
Infrastructure, Lab
Equipment(s)
2,500 Cycle time reduction 2,500 (10 USD per person x 250 person)
Early defects 11,750
(1000 USD per complex defect, 750 USD per medium
defect to 500 USD per simple defect; Complex - 5;
Effort (Research, Training,
POC)
10,000
Total 12,500 Total 14,250
ROI = 14%

How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective

Similaire à How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective (20)

Plus de STePINForum

Plus de STePINForum (13)

Dernier

Dernier (20)

How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective