Visual presentation of the preprint:
Petteri Teikari and Aleksandra Pietrusz (2021)
“Precision Strength Training: Data-driven Artificial
Intelligence Approach to Strength and Conditioning.”
SportRxiv. May 20. https://doi.org/10.31236/osf.io/w734a
Alternative download link:
https://www.dropbox.com/scl/fi/47nqp579t1b4m1zs0irhw/precision_strength_training.pdf?rlkey=05mzzw2ep8id71mq86936hvfi&dl=0
Call Girls Service Surat Samaira ❤️🍑 8250192130 👄 Independent Escort Service ...
Precision strength training: The future of strength training with data-driven deep learning
1. Petteri Teikari, PhD
https://www.linkedin.com/in/petteriteikari/
Version “Fri 15 September 2023“
Precisionstrength
training:
Data-driven artificial intelligence
approach to strength and
conditioning
see the article:
10.31236/osf.io/w734a
PetteriTeikariandAleksandraPietrusz(2021)
“PrecisionStrengthTraining:Data-drivenArtificial
IntelligenceApproachtoStrengthandConditioning.”
SportRxiv.May20.doi:10.31236/osf.io/w734a
2. Scope
Data-driven “precision strength training” can be used by “recreational gymgoers”,
professional athletes, general population to increase their “strength dose” (see the map
of Europe), elderly people to fight sarcopenia, physiotherapists as part of their rehab
practices (i.e. not just for athletes, while the focus here is mostly for sports)
Bennieetal.(2020): “"Theepidemiologyofmuscle-strengtheningexercisein
Europe: A 28-countrycomparison including280,605adults"”
https://www.csp.org.uk/professional-clinical/digital-physio
4. Population
Sell the same template
online for all
Subpopulation
Cluster athletes based on some
heuristics, years of training, simple
quantitative test
Individualization
Precision strength training.
Continuous monitoring and
adjustment of training via
quantitative testing
Mammogram
suspicious
Malignant
biopsy
Chemo
yes
yes
no
no
Subtyping
breast
cancer
patients
Individual Health trajectory
Population ‘one-size-fits-all’
Precisionmedicineapproach
how it could look like in strength training context
PRECISION MEDICINE PRECISION STRENGTH TRAINING
5. Personalizedtrainingatthemoment
Internal vs. External Load
Impellizzeri et al.(2020): "TrainingLoadandItsRoleinInjuryPrevention,Part I:BacktotheFuture"
EXTERNAL
“What coach / program
prescribes you to do?”
INTERNAL
How well executed by athlete,
environmental stressors, etc.
OUTCOME
Modulated by internal and
external load
(“biopsychosocial model”)
7. Autoregulation Fitness-fatigue model
Dose (e.g. 5 sets of 5 reps of squats) you would like to lead to improved performance while
controlling fatigue levels without getting into overtraining (or simply optimal adaptation to
the dose).
- but how would you quantify individual fatigue/fitness levels?
- when to reduce/increase (external) training load?
- howto do “optimal tapering”?
Greigetal.(2020): "Autoregulation inResistanceTraining:AddressingtheInconsistencies" ChrisBeardsley(2019)
8. Respondersvs.Non-Responders
Could you now in advance how the individual athlete responds to the
programming?
Drummond et al. (2012):“Targetinganabolicimpairment in response to
resistance exercise in older adultswith mobilityimpairments: potential
mechanismsand rehabilitationapproaches”
Padillaetal. (2021):
“Identifyingrespondersvs.
non-responders:
Incorporation of controls
are required for sound
statistical inference”
Rosset al. (2019):
“Precision exercise
medicine:
understanding
exercise response
variability"”
10. Frommanuallaborto(partial)AI automation
Leading hopefully to better quality training programs as well
JoeLemire(2020) "EvaluatingNBATalent: EveryoneKnowsTheseThingsCanBeDoneSmarter’"
ʻ
Marcus Elliott’s Peak Performance Project (P3) was first invited to set up a portable lab at the NBA Draft Combine in
2014, and he’s been at the Chicago-based summer event every year since (except this year’s, which was cancelled due
to the pandemic). Prior to the 2019 draft, the NBA hired Fusion Sport to white-label its Smartabase athlete
managementsystem and rebrand it as the NBA’s Combine HQ app, where it collected a multitude of data and offered a
standardized player interview. The league also partnered with Synergy Sports, a centralized video and advanced data
clearinghouse, and installed a Noah Basketball shot tracker in Chicago for that year’s combine.
https://www.p3.md/expertise/athlete-assessment
/talent-identification
12. Wearable sportssensors
A ton of products and research papers exist, mainly for endurance sports, with not so much sensor use among
strength sports “prosumers”
Gambhiretal.(2018):
“Towardachieving
precision health”
Wang etal.(2021): “AdvancesforIndoorFitnessTracking,Coaching,andMotivation: A ReviewofExisting
TechnologicalAdvances”
13. Consumerwearables
some work in strength training too beyond endurance sports
https://barbend.com/heart-rate-variability-strength-training/
”Heartratevariability(HRV)as aproxymeasureforstresslevels,i.e.
readinesstotrain andneedofadeload”
Gilgen-Amman etal.(2019)
medilog® AR12plus Holtermonitor and thePolar H10 heartrate monitor
Velmovitskyetal. (2022)validatingAppleWatchforHRVmeasurements
Kinnunen etal. (2020): “Objective: TovalidatetheaccuracyoftheOuraring in thequantification of
resting heartrate(HR)and heartratevariability(HRV).
14. VBTSystems non-video add-ons for barbells
Filzoneetal.(2019): "LiftSenseBarbell
Collars"DIY ”Arduinocollars”
https://twitter.com/trainwithPUSH
https://twitter.com/GymAware-
https://gymaware.com/
(GymAware”sAPIenablesAthlete
ManagementSystemstoseamlessly
integratestrengthtrainingdata.)
https://simplifaster.com/articles/choose-right-vbt-system-athletes/
VBT–Velocity-based training
https://vitruve.fit/blog/a-powerlifters-guide-to-velocity-based-training/
15. Strength trainingmetrology
Obviously not so feasible in practice, even for elite-level athletes, to measure everything
Electromyography activation
Force plate GRF
Smart insole plantar loadingand CoP
Ultrasound SWE
muscle activation, fatigue, and stiffness
EEG motor control and mental load fNIRS brain oxygenation
BiochemicalSensing
Ammonia, Lactate
Muscle fNIRS
Muscle
oxygenation
Hormones
Testosterone
Motion Capture
kinematic datafor
movement quality
assessment
Subjectivemeasures
RPE / RIR
Thermaland Circadian
21. TrainingFoundationmodels
with self-supervised learning (SSL), dubbed “the dark matter of intelligence”, with various different
approaches (“families”) and their variants
FoundationModel Ops(FMOps)asacombination oftraditional Ops(MLOps)
(deployment, optimization, monitoring), datamanagement and model alignment
https://blog.ml6.eu/developing-ai-systems-in-the-foundation-model-age-c540dcfa239a
RandallBalestrieroetal.(2023)https://arxiv.org/abs/2304.12210
1)TheDeepMetricLearningfamily[e.g.SimCLR]
2)TheSelf-Distillationfamily[e.g.DINOandMoCo]
3)TheCanonicalCorrelationAnalysis(CCA) family[e.g.SwAV,VicReg]
4)MaskedImageModeling(MIM)family[e.g.MaskedAutoencoder,MAE]
“RETFoundwith different
contrastive SSL strategies
showeddecent performance in
downstream tasks.” -
Zhouet al. (2023)
22. Foundation modelsinmovement studies
Figure from
Sárándi et al. (2022)
FOUNDATION
MODEL
Uselarge-scale
(open-source)general
skeleton movement
datasets
1)Train thefoundation model
yourself
2)Downloadsomepublished
model,and startfinetuning it
“SPORTS
MODEL”
Usethisthenforyourscientificstudy,
Inyourproduct,publishforthe
researchercommunity,etc.
Sarraf et al. (2021):“Estimating 3D posefor athlete tracking
using 2D videos and AmazonSageMaker Studio”
SSL
Finetune
e.g. onlytheclassifier
layerofSSL
LARGE DATASET
SMALLDATASET
Domain-specific
Domain-specificmodel
23. Foundation modelsinmovement studies
How useful are actually the generic datasets for movement research if
requirements for joint location accuracies are higher? Open question atm?
“Seethapathietal.(2019)
highlighted in theirstudythat
for movement science
research, the off-the-shelf
deep learning modelssuchas
OpenPose (Caoetal.2021)
trained on genericmovement
datasetssuchasNTU-RGB
120 (Lietal.2019)might not
provide good enoughspatial
jointlocationshighlighting the
need forcustom dataset
acquisition“ Seethapathi et al. (2019): “Movement science needs different pose tracking algorithms”
27. Ahujaand Morency(2019): "Language2Pose: NaturalLanguageGroundedPoseForecasting"
Generative AIbuzzwordstogym?
Nice to be able to generate animations for 3D content production, would it
flow to sports science too
28. GPT-4 forExercise
Would automated textual summaries of the exercise videos help coaches and athletes?
Could you integrate language feedback to the systems (voice text text2motion)
→ →
GPT-4https://twitter.com/DrJimFan/status/1634244545360609289
29. FLAG3D Dataset with fitness-language pairs
Tanget al.(2023): "FLAG3D: A 3D FitnessActivityDatasetwithLanguage
Instruction"
30. Creatingyour
owncustom
datasets
If you are a sports scientist or a
startup trying to validate the efficacy
of your product, you are designing
your experiments and would like to
get the most out of them in Machine
Learning-sense
31. Example of“machine learning” thinking
Pretrain on generic datasets (A) and finetune models (B) on your custom dataset (C)
Teikariand Pietrusz (2021): “Precision Strength Training:
Data-driven Artificial Intelligence Approach toStrength and Conditioning.”
32. Deploymentmatters#1
Record ground truth (e.g. Vicon) for some pilot study simultaneously with some low-cost system (such as WIFI
pose estimation) if your use case for example does not need fine resolution for joint positions, but you would
still like to quantify your error when using the low-cost system.
Jiang etal.(2020): "Towards3D human poseconstruction using wifi"
33. Deploymentmatters#2: non-sports example
When the ease of use (especially in home environments) and low-cost matter more than high accuracy
“Itcanbe ahassle togettothe doctor’soffice. And the task canbe especially
challenging for parentsof children with motor disorderssuchascerebral palsy,
asa clinician mustevaluate the child in person on a regular basis,oftenfor
an hour ata time. Making ittothese frequentevaluationscan be expensive,
time-consuming, and emotionally taxing.
MIT engineershope toalleviate some of thatstresswith anewmethod that
remotely evaluatespatients’ motor function. “
https://blog.ml6.eu/developing-ai-systems-in-the-foundation-model-age-c540dcfa239a
34. Deploymentmatters#3
Similar for smartphone deployment. You would like to know whether the smartphone quality is good enough of
an approximation of high-quality systems
Basiclow-cost
RGBcameraphone
Dual RGBcamera
Phone
Multi-RGBcamera
with depth camera
OptiTrack
Optical Motion Capture
~15,000USD
Affordable Consumer Tech Expensive Professional Systems
Vicon “GoldStandard”
Optical Motion Capture
~200,000USD
COST
“Deep
Target”
Level 2
“Deep
Target”
Level 3
“Deep
Target”
Level 4
1
2
3
4
5
QUALITY
35. Deploymentmatters#4
How much more sensors you can add to the simultaneous acquisition (especially nice if you would like publish
an open-source dataset)? e.g. how well do single IMU and IMU suits match optical motion capture (MoCap)?
1
2
3
4
5
Multiquality
Optical
Motion
capture
v
Deep Full-BodyMotion Network fora SoftWearableMotion Sensing Suit
https://doi.org/10.1109/TMECH.2018.2874647
1
2
Single Inertial MeasurementUnit (IMU)
+ faster to setup and easierto use,withlowercost
-not asaccurate asmultisensorsuit
http://doi.org/10.1136/bmjopen-2018-026326
+
36. Deploymentmatters#5: Modalities
Can you squeeze in some non-kinematic measures to the same experimental design, again with multiple
quality levels (from insoles to force plates?)
1
2
3
4
5
v
1
2
+
1
2
3
Low-cost insole
High-end insole
Forceplate
”Goldstandard”
IMUvs.
Force
Plate?
Laboratory motion and force plate data capture overlay.
The eight labeled markers used are shown artificially colored and
enlarged, and visible through the body. The force plate is
highlighted blue, and the ground reaction forces and moments
depicted. Photo credit Jodie Schulz from the Australian women’s
fieldhockey team, and Dr. GillianWeir.
Johnson etal.(2019): “PredictingAthleteGround
Reaction ForcesandMomentsFromSpatio-
TemporalDriven CNNModels”
37. Deploymentmatters#6: Mitigating artifacts
Could you for example add some infrared reflectors and measure non-sports measures in order to get higher-
quality ground truths?
1
2
3
4
5
v
1
2
+
Occlusions
Morecameras?
Deep learning?
Shinysurfaces
Polarization measurement?
Background/
Foregroundseparation
(“image matting”)
Optimize sensor
and illumination placement?
More suitable for industrial robotics
applications though
SoftTissueArtifacts
Algorithmiccompensation
More rigid suits?
Innovationsin the materials?
39. PoseEstimationGlitches
optical mocap: self-occlusion, cluttered background, poor illumination,
etc. influence joint data quality
KinectSDK 2.0
jointnoise
FromKimoredataset:
https://github.com/petteriTeikari/KiMoRe_wrapper/issues/2
Stenum etal. (2021):“Applicationsof PoseEstimation in HumanHealth and Performance acrossthe Lifespan”
40. PoseEstimationGlitches
IMUs/markers: human body is soft and the sensors do not stay in place
Peterset al.(2010): “Quantificationofsofttissueartifact in lowerlimbhumanmotion
analysis: Asystematic review”
Schalligetal.(2021): “Theinfluenceofsofttissueartifactson multi-segment
footkinematics”
42. From2D/2.5Dvideofeed(s)to 3Dskeleton
This is called “pose estimation” in computer vision literature. The quality of your pose estimation
obviously influences the quality of your 3D skeleton
Mehtaet al.(2017):“Vnect:Real-time3d human poseestimationwithasinglergb camera” Cited by 1007
estimatethe
location of the
human subject
track the
location of the
subject in the
frame
estimatethe
“human
skeleton” from
2D keypoints
track the
→
skeleton as
graph
“Segment” the
subject from
background
and estimate
pose from 2D
image
43. Example ofsegmentation+tracking
for knowing where to get the skeleton estimates
SegmentAnything+DINO for visual tracking of anyobject
(evennovel objectsnot found in the dataset, thus the open-setdetection)
AlaaMaaloufetal.(2023):“FollowAnything:Open-setdetection,tracking,andfollowinginreal-time”|explainervideo|
https://github.com/alaamaalouf/FollowAnything
45. 3DSkeleton
“Joint Spaces”
When pretraining on pooled existing
datasets, finetuning to your own data,
or when annotating your own
recordings, you might have noticed
that the joints are not exactly
standardized
49. AzureKinectthe mostrecent i.e. improvement from v2
Albertetal.(2020): “Evaluation ofthePoseTrackingPerformanceoftheAzureKinectandKinect
v2forGaitAnalysisin Comparison withaGoldStandard: A PilotStudy”
Colombeletal.(2021):“Markerless3DHuman PoseTrackingin theWild
withFusion ofMultipleDepthCameras: ComparativeExperimental
StudywithKinect 2and3(AzureKinect)”
50. Hand(finger,wrist) Joints typically not recorded
If you would need wrist data in curls, arthritis screening, or for monitoring interactions with objects?
https://youtu.be/HhygSSknY9s?t=190
Johansson et al.(1999): “
Controlofgraspstability
duringpronation and
supination movements”
Herrera-Lunaetal.(2019):
“SensorFusion Usedin
ApplicationsforHand
Rehabilitation: A Systematic
Review”
Sundarametal.(2019):"Learning
thesignaturesofthehuman
graspusingascalabletactile
glove"
51. FeetJoints impossible optically through shoes obviously
But maybe system with smart socks and smart insoles in the future?
https://me.me/i/your-foot-type-under-pronation-neu
tral-mild-severe-supination-pronation-13238362
https://www.justrunlah.com/2015/08/01/do-yo
u-know-whats-your-pronation-type/
Sensors2019,19(11),2641;
https://doi.org/10.3390/s19112641
DevelopmentofaBendable
OutsoleBiaxialGroundReaction
ForceMeasurementSystem
Mostofthedatacoacheswantissimplyonhowthefootis
functioningduring walking,running,sprinting,andjumping.
Other sport-specificactionscanalsobeevaluated,liking
kickingandlateralagilitymovements.
AnIn-DepthBuyer’sGuidetoPressure
MappinginSportinthelasttwoyears,there
hasbeen agrowingshift in sportstechnology
frommeasuringgeneralbodymotion to
measuringfootaction. Withaquarterof the
boneslocatedbelowtheankle,thefoot
complexisavery difficultjoint systemto
evaluateand an even hardersystemto
manage.In thenextfiveyears,pressure
mapping isgoing toexplode,withseveral
companiesallfightingoverthedatato
monetizethecaptureand analysisofathlete
gait.Ofallthebuyer’s guidesfromSimpliFaster,
thisisthemostdemandingsubjectareayet,
andperhapsthemost importantpartofhelping
athleticperformance.
https://simplifaster.com/articles/depth-buyers-
guide-pressure-mapping-sport/
52. Harmonizingjointspacestosamejointspace
Sárándi et al. (2022):“Learning3D Human Pose Estimation from
Dozensof Datasetsusinga Geometry-Aware Autoencoderto Bridge
Between Skeleton Formats”
Soyoucouldpool datasetswithdifferent joint
spacesandusethemalltopretraina
“foundational movementmodel”
53. MovementScience
DeepLearning
Working on the 3D skeletons
The most papers and Github
repositories you will find on
movement analysis (in contrast to
EMG deep learning for example)
54. Action Recognition
The most common task for movement data, i.e. what the detected people are doing in the video feed
Task: Skeleton-based Action Recognition
NTU RGB+D 60 Dataset commonly used to benchmark developed models
You couldautomatically segmentrecordedvideosbased
ondetectedactions(phases)ofasportsaction.
Lietal. (2020):“Temporal GraphModeling for Skeleton-based Action Recognition”
Andyou coulddetectautomatically the typeofexercise,is
the persondoing squatsorlunges. Thiscouldbethenused
forautomaticgymdiarylogging
Khan et al. (2022): “HumanActivityRecognition viaHybrid Deep LearningBased Model”
55. Whenskeletonisnotenough context matters
In exercise scenarios, you might easily start having elastic bands, yoga mats, and some other accessories in the scene
Context-awareHumanMotionPrediction
EnricCorona,AlbertPumarola,GuillemAlenyà,
FrancescMoreno-Noguer 12 Apr2019
https://arxiv.org/abs/1904.03419-Citedby101
In this work we present motion concepts, a novel
multimodal representation of human actions
in a household environment. A motion
concept encompasses a probabilistic
description of the kinematics of the action
along with its contextual background,
namely the location and the objects held during
the performance. Furthermore, we present
Online Motion Concept Learning (OMCL), anew
algorithm which learns novel motion concepts
from action demonstrations and recognizes
previouslylearnedmotionconcepts.
We plan to develop further work on the
representation of actions performed by multiple
agents and the extension of the motion concept
representation for task learning. We believe that
action representations are fundamental tools for
attaining a profound understanding of human
behavior in an environment and, ultimately, for
the widespread use of artificial agents in
householdenvironments.
57. Khurana et al. (2019): “GymCam: Detecting,
Recognizing and Tracking Simultaneous Exercises in
Unconstrained Scenes”
Build“gymsystems”
Imagine for example a S&C session with a NBA team in which you would like log
automatically each athlete’s exercises to an Athlete Management System (AMS)
without too much manual hassle
https://youtu.be/33kXm8-SSJY?si=xt5w4CP4ehlJ-WT3
59. Rep Segmentation (repcounting)#1
Segmenting in time from video, knowing which frames correspond to
which repetition
r
Milankoetal.(2020):"DesigningJust-in-TimeDetectionfor GamifiedFitnessFrameworks"
Dwibedi etal.(2020,GoogleResearch, DeepMind): “CountingOutTime:
ClassAgnosticVideoRepetition Countingin theWild”RepNet
61. Formcorrection#1
Coarse movement correction possible for some time, but could you do fine-resolution
correction (or even tracking) for elite-level athletes? And what would even the “best
technique” be for an elite-level athlete(think of powerlifting technique variations at IPF meets)
ChenandYang(2018):PoseTrainer: CorrectingExercisePostureusing Pose
EstimationwithOpenPose https://github.com/CMU-Perceptual-Computing-Lab/openpose
62. Formcorrection not limited to gymbro(ette)s
MilkaTrajkova(2020)
”Designing AI-Based
Feedback forBalletLearning”
“Artificialintelligence (AI)-based
videotoolscould represent an
affordable andnon-invasive
alternative: they would allow
dancers andteachers to
quantitatively self-assess aswell
asenable skilledballetteachersto
connect witha wider audience.
In my dissertation research, Istudy
howtodesignand evaluate AI-
basedtools for ballet dancersand
teachers toquantify performance
andfacilitate learning.”
63. FormCorrection
AI Replacing humans eventually? Or at least have nice systems to help the job
of powerlifting/S&C coaches and athletes
Kinetic AdvantageConsultinghttps://www.kineticadvantage.ca
MeganBryanton-Jones,PhD
64. SkillAssessment beyond just sports
needed for example for surgery skills assessment
Parmar and Morrisas(2019):“What and How Well You
Performed? AMultitaskLearningApproach toAction Quality
Assessment”
Lametal.(2022):“Machinelearningfortechnicalskillassessment insurgery:asystematic review”
68. Predictmissing modalities
ifyou have multimodaldatasetsasgroundtruth,youcouldtrainamachine learningmodel topredict thehigh-
cost measure.Ifyouforexamplewant todeploysomethingon smartphone andcannotdogroundforce
measurements,butwouldliketohaveestimatesof thegroundforces
Johnson et al. (2019): “Multidimensional ground reaction forces
and moments from wearable sensor accelerations via deep
learning” https://github.com/johnsonwr/digitalathlete
69. And when youhaveallthemodalitiesrecorded,
youcandostudieson theindividualcontributionofmodalitiestoyourtask
Softrobotperceptionusingembeddedsoft
sensorsandrecurrentneuralnetworks
ThomasGeorgeetal.(2019):
10.1126/scirobotics.aav1488
SensorDataAcquisitionandMultimodal
SensorFusionforHumanActivity
RecognitionUsingDeepLearning
2019https://doi.org/10.3390/s19071716
We adopt a two-level ensemble model to
combine class-probabilities of multiple
sensor modalities, and demonstrate that a
classifier-level sensor fusion technique can
improve the classification performance. By
analyzing the accuracy of each sensor on
different types of activity, we elaborate
custom weights for multimodal sensor
fusion that reflect the characteristic of
individualactivities
DeepLearningfor Musculoskeletal
ForcePrediction
https://doi.org/10.1007/s10439-018-02190-0
Department ofBioengineering, ImperialCollege London,
LondonUK
"The dataset comprised synchronously captured kinematic (lower
limb marker trajectories obtained by optoelectronic capture—Vicon MX
system, Vicon Motion Systems Ltd, Oxford, UK), force plate (ground
reaction force and centre of pressure—Kistler Instrumente AG,
Winterthur, Switzerland) and EMG (Trigno Wireless EMG system,
Delsys, USA) data from 156 subjects during multiple trials of level
walking"
72. Providingreal-timefeedbackfrom‘precisionmodel’
Conceptual example of a non-visual feedback for with knee valgus: 1) not feasible and dangerous for the
athlete to follow some screen haptic feedback, 2) another question if a knee valgus is somethingthat
→
requires haptic cues, or just more strength for example in adductors
Incorrect
execution
Hapticfeedback
on inner thighs
Precision
Model
Physiological Inputs
kinematic, kinetic,
activation, and
viscoelastic
parameters
FeedbackCorrectionsignal
Exercise
Recommender
Engine
Video from “Facebook 2020
Research: Photorealistic Avatars &
Full Body Tracking”
https://youtu.be/Q-gse_hFkJM
Integration of game
engines and
neuromuscular models
73. Real-time Feedback
vibratory insoles for that tripod foot support
Elvitigalaetal. (2019): "GymSoles: ImprovingSquatsandDead-Lifts byVisualizingtheUser'sCenterofPressure"
https://www.gymguider.com/not-getting-the-
results-you-want-from-the-king-of-leg-exerci
ses-the-squats-here-are-some-likely-culprits
/
Luetal.(2022): “Effectof
HeelLiftInsoleson Lower
ExtremityMuscleActivation
andJointWork during
BarbellSquats”
74. Offline visualizations#1
How to communicate the subtle changes in form over weeks the most effectively?
https://news.mit.edu/2018/creating-3-d-printed-motion-sculptures-from-2-d-videos-mit-csail-0919
https://youtu.be/i0WpFwBuXvI
https://twitter.com/serena_ivaldi/status/1043449589448232960
Prediction ofHuman Whole-Body Movements with AE-ProMP hal-01895148
⟨ ⟩
Elkholyet al. (2019):“Efficientand Robust Skeleton-Based QualityAssessmentand AbnormalityDetection in Human Action Performance”
76. “QualityControl”
How to get the coach or the
athlete self-correct the logs?
And how much correction is
even feasible?
Good UX for the human-in-the-
loop data validation needed
77. Interactive MachineLearning
Machines with the human help
“Theideaofincludinghumansintheloop,thuschanging theworking
methodology,continuesin Porter etal.(2013).Thisworkproposesthathumans
andcomputersshouldworktogetheronthesametaskdoingwhat
eachof themdoesbestatanyspecificmoment.”
Mosqueira-Reyetal.(2022): "Human-in-the-loop
machinelearning: astateoftheart"
79. Howtoquantifyadherenceandengagement?
VerificationofaPortableMotionTrackingSystemforRemote
ManagementofPhysicalRehabilitationoftheKneeSensors2019, 19(5),
1021; https://doi.org/10.3390/s19051021
(Thisarticle belongsto theSpecial Issue GyroscopesandAccelerometers)
“We developed a remote rehabilitation management
system combining two wireless inertial measurement units
(IMUs) with an interactive mobile application and a web-based
clinician portal (interACTION). However, in order to translate
interACTION into the clinical setting, it was first necessary to
verify the efficacy of measuring knee motion during rehabilitation
exercises for physical therapy and determine if visual
feedbacksignificantlyimprovesthe participant’s ability
toperformthe exercisescorrectly.
Exercises were recorded simultaneously by the IMU motion
tracking sensors and a video-based motion tracking
system (OptiTrack, running the Motive: Tracker software was
utilized as the “gold standard [Thewlis et al.2013, Carse et al. 2014]
). Validation
showed moderate to good agreement between the two systems
for all exercises andaccuracy was within three degrees. Basedon
custom usability survey results, interACTION was well
received. Overall, this study demonstrated the potential of
interACTION to measure range of motion during rehabilitation
exercises for physical therapy and visual feedback
significantly improved the participant’s ability to
performthe exercisescorrectly.
(A) Yost Lab’s two 3-Space Bluetooth sensors is a 3D printed case designed to align the sensors during
alignment, (B) Padded elastic straps secured on the thigh and shank, Cary, (C) Screenshot of the mobile
application screen thatprovidesthe participant with visual feedback.
80. Adherencedependsalotonthe engagement andrehabsystemUX
Computer vision and hardware might work perfectly, but if the UX is garbage, your system is useless
Adherencemonitoringofrehabilitation
exercisewithinertialsensors:Aclinical
validationstudysLuckshman Bavana, Karl Surmacz,
David Beard, Stephen Mellon, Jonathan Rees(Nuffield
Department ofOrthopaedics, Oxford)
https://doi.org/10.1016/j.gaitpost.2019.03.008
“Aims to evaluate the feasibility of using a single
inertial sensor (MetaMotionR, MbientLab,) to
recognise and classify shoulder rehabilitation
activity using supervised machine learning
techniques.”
PatientInvolvementWithHome-BasedExercisePrograms:CanConnectedHealth
InterventionsInfluenceAdherence?sRob Argent et al., Beacon Hospital, University College Dublin Beacon
Academ https://doi.org/10.2196/mhealth.8518
“Adherence to home exercise in rehabilitation is a significant problem, with estimates of
nonadherence as high as 50%, potentially having a detrimental effect on clinical outcomes. In this
viewpoint, we discuss the many reasons why patients may not adhere to a prescribed exercise
program and explore how connected health technologies have the ability to offer numerous interventions
to enhance adherence; however, it is hard to judge the efficacy of these interventions without a
robustmeasurementtool.”
“It is widely accepted that at present, there is no gold standard for the measurement of adherence to
unsupervised home-based exercise, as the significant proportion of outcome measures used in the
literature rely on patient self-report and are therefore susceptible to bias [Bollenetal.2014]. In a
systematic review of 61 different self-reported outcome measures for adherence to home-based
rehabilitation, only two measures scored positively for a single psychometric property of validation [
Bollenetal.2014]. Furthermore, the outcome of any research studies using paper diaries or retrospective
recall has been called into question as it is highly prone to recall and self-serving bias [
Stoneetal.2003]. Equally, these measures make no allowance for the quality of performance, as
highlightedintheabovementioneddefinition.”
“Sensing platforms such as the use of IMUs or motion capture camera are rapidly advancing and
couldbeanopportunitytomakeamoreobjectiveassessmentofadherence,continuouslytrackingmotion
data obtained from an individual [Rizk etal.2013; Oeschetal.2017]. However, the use of these devices to
measure adherence is questionable as they arguably influence/enhance adherence itself by means of
the user knowing that they are being recorded.In thisway the end pointis influenced greatly bythe
measurement strategy, leading to questionable results as the patient no longer has the choice on whether
to adhere [Bollenetal.2014].Regardless of the challengeswith accurately measuring adherence, it isclear
thatthereareproblemswithadherencetoprescribedexerciseinthehomesetting.”
81. insightsfrom BehavioralMedicine
Are all the extra gadgets just a hindrance or could the athlete actually enjoy usingthem routinely in their
training? e.g. whatever you can integrate onto a standard swimsuit should be easy for swimmers?
Arigoet al.(2019):“Thehistory andfutureofdigitalhealthin thefieldofbehavioralmedicine"
https://www.iasp-pain.org/resources/fact-sheets/innovations-in-physiotherapy-and-digital-health/
Fritz etal.(2019):
“Implementation ofa
behavioralmedicine
approachin physiotherapy:
aprocess evaluation of
facilitation methods”
83. Gamifyexercise#1
When VirtualReality MeetsInternetofThings
intheGym:EnablingImmersiveInteractive
MachineExercises
FazlayRabbi, Taiwoo Park, Biyi Fang,MiZhang,Youngki Lee(2018)
Michigan State University/ SingaporeManagement University
https://doi.org/10.1145/3214281
Toward this vision, we present JARVIS, a virtual exercise assistant that is able to
provide an immersive and interactive gym exercise experience to a user.
JARVIS is enabled by the synergy between Internet of Things (IoT) and
immersive VR. JARVIS employs miniature IoT sensing devices removably
attachable to exercise machines to track a multitude of exercise information
including exercise types, repetition counts, and progress within each repetition in real
time.
Based on the tracked exercise information, JARVIS shows the user the proper
way of doing the exercise in the virtual exercise environment, thereby helping the
user to better focus on the target muscle group. This machine-attachable approach
not only equips exercise machines with sensing capabilities without being
instrumented but also turns JARVIS into a mobile system that allows a user
toenjoyimmersiveVRexerciseexperienceanywhere.
86. BiomechanicsmeetRoboticsandcontroltheory
Bersani etal.(2023):“ModelingHuman SuboptimalControl: A Review”
Davide Scaramuzza:
Why can
ReinforcementLearning
(RL) achieve results
beyond OptimalControl
(OC) in many real-world
robotics control tasks?
We investigate this
question in our paper
Song et al. (2023): “Reaching the
limit in autonomous racing:
Optimal control versus
reinforcement learning”
https://youtu.be/HGULBBAo5lA?
si=_oNAGWdoyfWcuu3U
Ritzmann etal. (2022):
“Neuromechanicsin
Movementand DiseaseWith
Physiological and
Pathophysiological
Implications: From
Fundamental Experimentsto
Bio-InspiredTechnologies”
87. Reinforcement LearninginMedicine too
Sepsis and its treatment as the low-hanging fruit
Finale Doshi-Velez @NeurIPSMachineLearning forHealh 2018 (ML4H)
Associate Professor ofComputer Science, Harvard PaulsonSchool of Engineeringand Applied Sciences(SEAS)
88. Reinforcement LearningIssues #1
How would you even now what is the optimal training programming for someone wanting torehab their
injury, to be the strongest possible at the main contest of the year, or maintaing their strength levels?
Somewhat fortunately this is not an unique problem to have in medicine
SupervisedReinforcementLearningwithRecurrent
NeuralNetworkforDynamicTreatment
Recommendation
LuWang,WeiZhang,Xiaofeng He,HongyuanZha
KDD'18Proceedingsofthe24thACMSIGKDDInternationalConferenceon
KnowledgeDiscovery&DataMining https://doi.org/10.1145/3219819.3219961
The data-driven research on treatment recommendation involves two main
branches: supervised learning (SL) and reinforcement learning (RL) for
prescription. SL based prescription tries to minimize the difference between the
recommended prescriptions and indicator signal which denotes doctor
prescriptions. Several pattern-based methods generate recommendations by utilizing
the similarity of patients [Huetal.2016, Sun etal.2016]
, but they are challenging to directly learn
the relation between patients and medications. Recently, some deep models achieve
significant improvements by learning a nonlinear mapping from multiple diseases
to multiple drug categories [BajorandLasko2017, Wangetal.2018, Wangetal.2017
. Unfortunately, a key
concern for these SL based models still remains unresolved, i.e, the ground truth of
“good” treatment strategy being unclear in the medical literature [Marik 2015].
More importantly, the original goal of clinical decision also considers the outcome of
patientsinstead of onlymatching theindicatorsignal.
!
The above issues can be addressed by reinforcement learning for dynamic
treatment regime (DTR) [Murphy2003, Robins1986]. DTR is a sequence of tailored
treatments according to the dynamic states of patients, which conforms to the clinical
practice. As a real example shown in Figure 1, treatments for the patient vary
dynamically over time with the accruing observations. The optimal DTR is determined
by maximizing the evaluation signal which indicates the long-term outcome of patients,
due to the delayed effect of the current treatment and the influence of future treatment
choices [Chakrabortyand Moodie2013]. With the desired properties of dealing with
delayed reward and inferring optimal policy based on non-optimal prescription
behaviors, a set of reinforcement learning methods have been adapted to generate
optimal DTR for life-threatening diseases, such as schizophrenia, non-small cell lung
cancer, and sepsis [e.g. Nematietal.2016]. Recently, some studies employ deep RL to
solve the DTR problem based on large scale EHRs [Pengetal.2019, Raghuetal.2017, Wengetal.2016
.
Nevertheless, these methods may recommend treatments that are obviously different
from doctors’ prescriptions due to the lack of the supervision from doctors, which
may cause high risk [Shen et al.2013] in clinical practice. In addition, the existing methods
arechallengingforanalyzing multiplediseasesandthecomplex medication space.
89. Reinforcement LearningIssues #2
Could you introduce some constraints to the process? Like athlete A would prefer to do Smith squats
instead of normal squats? Doonly 2 training sessions a week instead of 4 days per week.
Diversity-InducingPolicyGradient:UsingMaximum
MeanDiscrepancy toFindaSetofDiversePolicies
MuhammadA.Masood, FinaleDoshi-VelezHarvardUniversity
(Submitted on31 May2019)https://arxiv.org/abs/1906.00088
Standard reinforcement learning methods aim to master one way of solving a task whereas
there may exist multiple near-optimal policies. Being able to identify this collection of
near-optimal policies can allow a domain expert to efficiently explore the space of
reasonable solutions. Unfortunately, existing approaches that quantify uncertainty over
policies are not ultimately relevant to finding policies with qualitatively distinct behaviors.
In this work, we formalize the difference between policies as a difference between the
distribution of trajectories induced by each policy, which encourages diversity with
respect to both state visitation and action choices. We derive a gradient-based optimization
technique that can be combined with existing policy gradient methods to now identify diverse
collections of well-performing policies. We demonstrate our approach on benchmarks and a
healthcaretask.
Liu et al. (2017) characterize the uncertainty over policies via computing a
posterior over policy parameters, but differences in policy parameters may not result
in qualitatively different behavior (especially in overparameterized architectures).
Haarnoja et al. (2017) encourage diversity via encouraging high entropy
distributions over actions (given states), which may result in sub-optimal behavior.
Fard and Pineau (2011) seek a single non-deterministic policy that may make
multiple decisions at any state, which may be overly restrictive if action choices
across statesmust becorrelatedtoachievenear-optimalperformance.
We argue that differences in trajectories (state visits and action choices) better
capture the kinds of distinct behavior we are seeking. For example, does oneprefer
a policy that achieves wellness through a surgery, or via prolonged
therapy? More formally, stochasticity in the environment dynamics and the policy
will induce a distribution over trajectories. We use the maximum mean
discrepancy(MMD)metrictocomparethesedistributions overtrajectoriesunder
different policies. As noted in Sriperumbudur et al. (2010), the MMD metric has a
closed form solution (unlike the Wasserstein and Dudley metrics) and exhibits better
convergencebehaviorthanφ-divergencessuchasKullback-Leibler(KL).
90. TheHardPart
So what is the “correct exercise form” for a given patient. Reinforcement Learning -based
solution seems the best as it is getting popular in prescriptive modeling anyway (prescribe
a treatment, and maybe patients do not fully comply, the control system
can adapt itself to this non-adherence?). As in Control meets
ReinforcementLearning+Human-in-the-loopSupervision(ifneeded)
Tang et al. (2018) Deep Progressive
Reinforcement Learning for Skeleton-based
Action Recognition
http://openaccess.thecvf.com/content_cvpr_2018
/papers/Tang_Deep_Progressive_Reinforcement
_CVPR_2018_paper.pdf
Sun, Jinming, "Dynamic Modeling of Human Gait Using a
Model Predictive Control (MPC) Approach" (2015) Paper 526.
http://epublications.marquette.edu/dissertations_mu/526
meets +
Standard reinforcement learning methods aim to master one way
of solving a task whereas there may exist multiple near-
optimal policies. Being able to identify this collection of near-
optimal policies can allow a domain expert to efficiently
explore the space of reasonable solutions. Unfortunately,
existing approaches that quantify uncertainty over policies
are not ultimately relevant to finding policies with qualitatively
distinct behaviors. We argue that differences in
trajectories (state visits and action choices) better capture the
kinds of distinct behavior we are seeking. For example, does one
prefer a policy that achieves wellness through a surgery,
or via prolonged therapy? More formally, stochasticity in the
environment dynamics and the policy will induce a distribution
over trajectories. We use the maximum mean discrepancy
(MMD) metric to compare these distributions over trajectories
under differentpolicies.
Masood and Doshi-Velez (2019) Diversity-Inducing Policy Gradient:
Using Maximum Mean Discrepancy to Find a Set of Diverse Policies
https://arxiv.org/abs/1906.00088
Control RLonGCN Clinician/PTSupervision
91. Apps
Not all this is sci-fi still
Existing “training AI apps” though do
not do kinematic or any other
modality for feedback.
Prescribe dose (external load), and check the
outcome (lifted sets, reps @ what weight?).
Can include subjective assessment of effort
(e.g. RPE or RIR)
92. “AIApps” Prosumer market
Heavy DeadliftSession withKristenDunsmore|JuggernautAI.Exampleof usingRPE
(RIR)withloggingyourtraining.https://youtu.be/VVBWzo83ueE?t=505
https://youtu.be/GZEx-ZJETv0?si=j8btRzqfaPL4932M
Juggernaut AI forpowerlifting Hypertrophy app forbodybuilding
https://www.reddit.com/r/StrongerByScience/comments/12mja6x/thoughts_on_rp_hypertrophy_app/
https://www.reddit.com/r/powerlifting/comments/z5o87x/program_review_juggernaut_ai_powerbuilding_14week/
93. “AIApps” Professional market
You could develop integrations to existing Athlete Management Systems (AMS)
and provide the “magic AI” part that way
Firstbeat is delighted to announce its API
integration partnership with Opteamal, a
European-based data integration platform
for team sports thattransforms data into
actionable information.
the integration of Smartabase & Kinduct Athlete
Management Systems. Any data, such as raw or
processed motion capture (kinematic, kinetics) EMG,
force, etc. can be automatically uploaded into your
preferred Athlete Management System. This is an
exciting addition for our sports &clinical clients as it
streamlines data collection, analysis and reporting –
facebook post
https://externlabs.com/blogs/ai-and-ml-in-sports/
AdamMattiussi,StrengthandConditioningCoachat TheRoyalBallet
andTheRoyalBalletSchoolcoverstheuseoftheSmartabaseto
assistin monitoring jumpingtrainingloadsofRoyalBalletdancers,to
reducetheriskofinjury-https://youtu.be/Ez4na4XSgzM?si=1L39GDWoOtJgCKqE
95. FIGUR8
Body movement as a biomarker
https://www.media.mit.edu/posts/figur8-2/ https://www.sportsbusinessjournal.com/Articles/2023/08/08/figur8-funding-series-a1.aspx
98. Tempo the “workout mirror”
https://www.goodhousekeeping.com/health-products/a36804688/tempo-studio-review/
note! The side glance while squatting
https://techcrunch.com/2020/02/26/tempo-weight-lifting-screen/
“Tempo wants to be the Peloton of barbells”
99. ‘Yogasystems’ the smart garment form factor
TuringSense, the developer behind PivotYoga
https://www.theverge.com/2018/11/28/18116831/smart-yoga-pants-clothing-pose-built-in-sensors
Nadi Xhttps://www.wearablex.com/collections/nadi-x-smart-yoga-pants