xAPI-Enabled Mobile Health System with Context Awareness Recommendation Engine for Patients
1. XAPI-enabled Mobile Health System with Context-
Awareness & Recommendation Engine for Patients
JESSIE CHUANG
CLASSROOM AID INC.
VISCA ANALYTICS
XAPI CHINESE COP
4. Background and Motivation (1/2)
Fact Mobile Health(mHealth)flourished. (Saxon, 2016.)
150
Health
60%
Other
40%
Apple Research Kit & Care Kit
Google Fit SDK
5. Background and Motivation (1/2)
Related
Monitoring
physiological value
Health Information
(Text, Picture, Video)
Fact Mobile Health(mHealth)flourished. (Saxon, 2016.)
Mostly focused on providing information and
monitoring physiological value. (Vashist, et al, 2014. Saxon,
2016.)
6. Background and Motivation (1/2)
1. We need to monitor patients daily activities and
motions of upper limb. (Subbarao, et al, 1995.)
Gap
92% 50 - 70%
Fact
Related
Mostly focused on providing information
and monitoring physiological correlation value.
(Vashist, et al, 2014. Saxon, 2016.)
Mobile Health(mHealth)flourished. (Saxon, 2016.)
7. Background and Motivation (1/2)
2. Applications provided for Spinal cord injury patients
only taught health information, but lack of monitoring
and management.
Solution 1. Develop an intelligent assistance system - SmartChair APP
2. Proposed Context Awareness Suggestion Engine (iCASE)
Fact Mobile Health(mHealth)flourished. (Saxon, 2016.)
Mostly focused on providing information
and monitoring physiological correlation value in Health APPs.
(Vashist, et al, 2014. Saxon, 2016.)
Gap
Related
1. The need for patients daily activities and rehabilitation
of upper limb motion should be monitored. (Subbarao, et al,
1995.)
8. Background and Motivation (2/2)
SmartChair APP
(Dept Engineering Science, NTU)
Motor power wheelchair
(Dept Mechanical Engineering, NTU)
Physician, occupational therapist
(Dept Occupational Therapy, NTU)
Spinal cord injury
(Taoyuan Potential Development Center, National Taiwan University Hospital)
Data
Prescription
9. Objective
Identify problems, dynamic correction,
improve the system.
System
Developer
User
Related
Staff
Implement a mHealth APP for
Patients with SCI.
Purpose
10. Objective
Click:1 Click:2 Click:3 Click:4
System
Developer
User
Related
Staff
Implement a mHealth APP for
Patients with SCI.
Issue Frequent clicks will result in chronic
injuries.
Purpose
11. Objective
Click:1 Click:2
System
Developer
User
Related
Staff
Implement a mHealth APP for
Patients with SCI.
Issue Frequent click will result in chronic
injuries.
Purpose
Solution
Machine builds context-awareness
from user history, prescription, &
other contexts and prompts
recommendations dynamically.
12. Objective
Solution Using Experience API (xAPI) , data can
be transferred between different
services
System
Developer
User
Related
Staff
Implement a mHealth APP for
Patients with SCI.
Purpose
IssueIssue Collect data from different services
14. Objective
Solution
Add "Recommended Interface” to
the current architecture to get
feedback. (A/B testing)
System
Developer
User
Related
Staff
Implement a mHealth APP for
Patients with SCI.
Purpose
System rebuilding is usually
inefficient and time-consuming.
Issue
16. Methodology
Technique
Server
Level
Client
Level
Access easily to information,
but can not collect JavaScript
event. (Srivastava, 2000.)
Server Log
Direct Access
Intermediary
Server-Side
There will exist the problem of
grammar incompatibility in
migration. (Corbi and Burgos, 2014.)
Leverage Escrow Services to
avoid grammar migration
issue.
Software / APIs
Collection
Analysis
17. Methodology
Technique
Server
Level
Client
Level
Server Log
Direct Access
Intermediary
Server-Side
xAPI
Since the Experience API (xAPI) is an open
standard, so it is used as Intermediary Server.
Collection
Analysis
Access easily to information, but
can not collect JavaScript event.
(Srivastava, 2000.)
There will exist the problem of
grammar incompatibility in
migration. (Corbi and Burgos, 2014.)
Leverage Escrow Services to
avoid grammar migration issue.
18. Methodology
Technique
xAPI
Client
Level
Through Escrow Services to
avoid incompatibility problem
with the grammar migration.
Intermediary
Server-Side
xAPI
Cross-
platform
Use the Activity Streams to record
user experience.
Actor
(Who)
Verb
(How)
Object
(What)
Collect and transfer data between
heterogeneous platforms through
Learning Record Store (LRS).
Context
description
Parameters for different situations
can be recorded as context data.
Collection
Analysis
Data
integrity
19. Methodology
Context-
Awareness
Collect user behavior through xAPI
Definition
Categories
Dynamic
a person, place or object. (Dey,
et al. 2001.)Static
Actor behaviors (Actor, Verb,
Object) (G. Chen and D. Kotz, 2000.)
i.e. user profile, location(G. Chen and D. Kotz, 2000.)
Collection
Analysis
Computing
context
User
context
Physical
context
Time
context
i.e. network connectivity
i.e. time of a day, week
i.e. lighting, noise level
User
context
Time
context
20. Methodology
Methods
Statistics
Sequential
Pattern
Statistical inference (frequency,
average, etc.) is the most popular.
Investigate the probability that
when an event appears, another
event also appears.
Classify old data, and then predict
the future data.
Cluster the data by property
similarity.
Analyze data pattern on timeline.
Statistics
Sequential
Pattern
Context-
Awareness
Collect user behavior through xAPICollection
Analysis
Association
Rule
Clustering
Classification
24. Context-Awareness Model
xAPI
Behavior Model
xAPI format
Mapping
Interface
Segmentation
Behavior
Library
(LRS)
23
Behavior Model: Collecting User Data through xAPI
Discomfort
Record
Activity
Record
Route
Record
Exercise
Exercise
Time
Block
Naming
29. Behavior Model
xAPI
Recommended
Interface
Sort Results
Rule filtering
engine
Filter Model
Context-Awareness
Model
Filter Model + Expert Advice
# System functions
Therapist
prescription
135347 Route Distance 5 - 10 km
116170
Frequency of
Wheelchair Repair
At least once
a month
99242 Rehabilitation
Three times a
week
Hash Table
Expert
Knowledge
Therapist
30. # System functions
Therapist
prescription
135347 Route Distance 5 - 10 km
116170
Frequency of
Wheelchair Repair
At least once a
month
99242 Rehabilitation
Three times a
week
Sort Results
Rule filtering
engine
Filter Model
Filter Model
Adjust the weight to improve the bad habits.
Time
Frequency
Therapist prescription
(Threshold)
Forcibly removed
Time
Frequency
Therapist prescription
(Upper limit)
Forcibly removed
Therapist prescription
(Lower limit)
1) User habits
2) Increase the weight
Reduce the weight
31. Expert
Knowledge
Therapist
Behavior Model
xAPI
Sort Results
Rule filtering
engine
Filter Model
Context-Awareness
Model
L0.5
L1
L0
Recommend
L2
L3
Login
functio
n1
Index
functio
n1.1
functio
n1.3
functio
n2
functio
n3
functio
n2.1
functio
n2.2
functio
n3.1
functio
n3.2
functio
n3.3
functio
n1.2
Ln:Level n;n: clicks required to access the function
1) Sequentially outputs.
2) Until satisfy the size of
the recommended list.
Recommendation Interface
Recommended
Interface
32. xAPI
Behavior Model
xAPI format
Mapping
Interface
Segmentation
Block
Naming
Behavior
Library
(LRS)
Future works (1/2)
Join other devices to increase data integrity.
Thoracic
Vertebrae
Cervical
Vertebrae
Lumbar
Vertebrae
Sacral
Vertebrae
User
Injuries
Level
Feature
1 C4
No upper limb movements.
Use the chin to operate APP.
2 C5 Only the elbows bent, not straight action.
3 C6 The elbow can straighten and wrist cock.
4 C6 The elbow can straighten and wrist cock.
5 C7 The fingers weakness.
6 T11 The upper limb is fine.
34. Related work – Food Control for Cancer Patients
LRS
Behavior
Prescription
E-health brain
xAPI E-Health
System
Human-in-the-loop Machine Learning : machine is human’s collaborator.
35. Physiological sensors : blood
pressure; blood glucose level;
temperature; blood oxygen level;
and the signals related to ECG, EEG,
and EMG.
Biokinetic sensors : to measure the
acceleration and the angular rate of
rotation that results from body
movements.
Ambient sensors : to measure
environmental factors such as
temperature, humidity, light, and
the sound pressure level.
Self-reporting : alarm, habit,
discomfort recording, survey, check-
list, request help.
Patient-centered “Sensor Network”
XAPI records rich CONTEXT information,
which is crucial for medical data.
36. Serve Humanity ASAP
XAPI data are highly structured
in a pre-designed way, can be
integrated meaningfully as
soon as collected, data can be
put to use right away.
for human to read,
for machine to compute &
respond (less guess),
services can talk to each
other & work together in
real time !! (If … then … )
If data talk in different languages, we can
NOT make sense out of them or use them
UNTIL the time and computing power are
committed to integrate and interpret them.
37. What if…
It takes you 3 days to “see”
the woman standing in
front of you…
Will your response be way
too late…?
39. Takeaways
XAPI is a very effective tool in enabling Apps to
serve humanity ASAP, because it connects
heterogenous data immediately.
XAPI is about people working together.
XAPI is about connecting current technologies,
instead of re-inventing wheels.(API’s power)
@classroomaidinc
Jessie@classroomaid.org
Editor's Notes
My first question : How many have watched this movie “Big Hero 6”? In case you don’t know, it’s a fun Disney movie, and Baymax is a personal health companion – a huggable robot that will take care of you, always. Second question: If possible, do you want to give your family member who’s sick one personal health companion? My boss of my first job taught me “you must tell the conclusion at the beginning of your report”, so here is my conclusion – xAPI is a very effective tool to help us build a Baymax! Of course I’ll tell you why later.
A little about us…in short, Classroom Aid is our consulting company, I have been working on xAPI project for 3 years, Visca is our effort to give real-time dataviz for xAPI data, we believe returning data to learners and instructors immediately is crucial. We even provide API for you to embed dataviz inside your own App or dashboard. I am also the lead of xAPI Chinese CoP, Here are our major CoP members. We set up our AcrossX profile, recipes and vocabulary for our CoP. BTW, profile and recipe are to define how we record specific experiences. xAPI is totally different with SCORM. xAPI spec. only guarantees data structure interoperability, but it’s highly extensible and flexible, this is great, but semantic interoperability must rely on people working together to decide how to record a behavior, that’s why xAPI depends on CoP and DISC. This extensibility and community driven vocabulary/profile/recipe is also why xAPI is so powerful.
Here comes the 2nd conclusion: successful xAPI implementation is about “people working together”, among the lessons learned along the way trying to solve all difficulties in implementing xAPI, this is the no. 1 factor.
Today’s story is from ICAN Lab of National Taiwan University.
According to research, average Americans check their mobile phones 150 times a day. 60% is for health purpose, like tracking fitness goal, they said they will use health App more in the next 5 years…. we all agree mobile phone are now our most intimate friend. Apple offers ResearchKit and CareKit for healthcare support and research, and Google has Google Fit SDK, too. Now hospitals also use mobile portal to engage and serve patients.
2/3 of mHealth App only provides health information, 1/3 is to monitor physiological values, it’s not enough for special need patients.
92% of spinal cord injury patients need to use wheelchairs,within 10-15 years, 50-70% have upper limb overuse problem, the probability of chronic illness increases significantly.
The solution is to develop Smartchair App that provides instructional videos of rehab. For patients to follow along, medical knowledge for this type of patients. It also include social media platform for patients to connect with peers. From the data collected, the system behind it, we called Context Awareness Suggestion Engine(iCASE), builds context-awareness and then provides recommendations dynamically to support patients.
.
At the center is Smartchart App, data from the wheelchair and the prescription from physician or therapist are sent here to integrate with data collected by Smartchair. This App is now offered to spinal cord injury patients in these 2 organizations.
Let’s identify problems to be solved regarding these 3 aspects. Notice that this is how we should use xAPI – pinpoint problems first, not sending whatever xAPI data we can send without purposes in mind.
For user’s need, frequent clicks will result in chronic injuries.
The solution is to build context-awareness from user’s history records, the prescription and other context data to prompt the recommended action for user. This reduces clicks and provides guidance at the same time.
The 2nd issue is we need to communicate between different services – for example, the site for therapists, and wheelchair vendor’s system, in order to solve this, xAPI is leveraged for data transfer and integration.
And, system developers need a more efficient way to collect user behavior to feedback to improve the system. Also after revision, users also need effort to re-learn newly-designed interface. Is there a more flexible and agile way?
The solution is a recommendation engine to prompt recommended action for supporting users as well as collecting feedback, developers will update the system design only after the hit rate of recommendation is higher than a threshold.
Smartchair is a hybrid App with its UI based on web language, here we discuss how to collect interaction data.
XAPI is a user-centered data format, use Actor-Verb-Object along with other rich contextual and result data, the experience can be captured with necessary details. You might know better than me that for medical data, context information is crucial. This way allows collecting and transferring data in between heterogeneous platforms through intermediary server – the Learning Record Store, LRS.
Now let’s look at context-awareness modeling. Contexts could be static or dynamic, and belong to many categories that matter to our analytics. You can imagine sensors are like our 5 senses, and computing is like our brain. Only with xAPI, we can record heterogenous data in a coherent way, so those 5 senses data can be put together immediately for computing brain to build context-awareness. For example, it’s like Baymax observes when, how long, with whom, the patient did exercises.
So how does computer model data? it’s like Baymax can reason that the patient walk longer when it’s morning than afternoon, or exercise longer when exercising together with peers. There are many well-developed data science / machine learning methods and even open-source tools to let machine do reasoning and predictions. For 1st phase, statistics and sequential pattern are used to model the behavior data here.
This is the conceptual system architecture, mostly you can have an idea about how computer “brain” works to process data to support users. xAPI collects data from functionalities interface into building behavior model, which will be sent to context-awareness model for calculation, and then combine with expert knowledge, these data will be processed by filter model to sort out suggested next steps and offer to the patient. The cost-benefit analysis is to analyze if this recommendation engine really improve user experience and solve the issue.
This block “behavior model” is where xAPI works. So let’s go into more details here.
This is a way to model interaction data from this mobile App – first, interface segmentation.
2nd, block naming, it’s suggested to make it meaningful for both humans and machine to read.
3rd, map and design xAPI statements
A simple example is like this : John recorded discomfort record. That’s the most basic sentence of an xAPI statement, along with it context information can be recorded as well – such as timestamp (when did this happen), location (where did this happen), result (what is the discomfort level), or even the environment temperature.
Then user’s behavior records will be processed along with context awareness model, using those mentioned machine learning methods to model data, also taking into consideration of contexts such as current time and location.
This filter model is to filter and sort the recommended next steps for the user.
Usually this kind of recommendation engine only factors in user’s habits, but for a patient, we need to put therapist’s prescription into filter model. All recommended actions are referring to the prescription list, the priority is adjusted according to user’s history and context.(for example, time)
It adjusts the weight for recommending actions according to user’s history data, for behaviors with lower frequency, the weight will be increased, for behaviors with higher frequency, the weight will be reduced, or totally remove an action if the frequency exceeds the threshold.
The recommendation engine prompts recommended action at 0.5 layer, on top of the current hierarchy. As I mentioned, this reduces clicks, and provides guidance as well.
Future work: we can add voice recognition and voice control, and wearables for monitoring purpose, for example, measuring quantified data of patient’s movement, or monitoring eyeballs, brainwave…. Depending on the pinpointed purposes.
If we have more sensors, those data might help us get more context information, then we can answer more questions. But some context data will be collected by designing the functionalities such as enabling interactions between patients and caregivers, physicians, and we can analyze the impact of such social context.
Based on the same system architecture, this related work is to help cancer patients choose the right foods and balanced between preferences and nutrition requirements.
Actually another related work undergoing is in learning and training domain, it uses the same system architecture too. The strategy here is the collaboration relation between machine and experts (therapist or teacher). First the machine builds recommendations from expert’s prescription and user’s behavior history to balance both dynamically, the output of machine learning is under monitoring of expert, the expert can modify the recommendation if needed, and then machine will adjust its modeling to fit to that label every time. That means, machine learns from its human collaborator and real data continuously.
The benefit for experts to use this system is that collected data can give them a better picture about their patients or students, they can interact with them whenever needed; and with enough training, machine’s recommendation can be very helpful and share the expert’s work loading.
Now we can have tons of medical devices and sensors to collect data around a patient, even swallow a pill and take pictures inside the body. Data, data, data everywhere, 2 things we care here. As mentioned before, for medical data the related contexts of a data point is very important, and xAPI can record that in a standard way.
2nd, now all these data talk in different languages…
Can you imagine? What if…
But each data from different sources are like a piece of the puzzle, if without xAPI, we can not see the real-time whole picture.
Here we go. Dataviz can help you see the whole picture of integrated xAPI data IN REAL TIME, zoom in to see a tree or zoom out to see a forest, for learners, instructors, patients, family, and therapists.
3 takeaways:
…
… Usually driven by people who care about patient-centered or learner-centered service. Then talk to an xAPI expert about your questions needed to be answered, work together to design xAPI implementation. To reduce technological barrier, xAPI wrappers and connectors are available.
… maybe you’ve heard of API economy, XAPI is API for experience data, if with proper xAPI profile/recipe design, it can be plugged with machine learning APIs or our iCASE brain as introduced in this presentation. Think of the fact that we have such “personalized advertisements” from Google and Facebook etc., can we bring that “personalization” technology to learners and patients? Up to all of us.