Personal informatics systems help people collect and reflect on behavioral information to better understand their own behavior. Because most systems only show one type of behavioral information, finding factors that affect one’s behavior is difficult. Supporting exploration of multiple types of contextual and behavioral information in a single interface may help.
To explore this, I developed prototypes of IMPACT, which supports reflection on physical activity and multiple types of contextual information. I conducted field studies of the prototypes, which showed that such a system can increase people’s awareness of opportunities for physical activity. However, several limitations affected the usage and value of these prototypes. To improve support for such systems, I conducted a series of interviews and field studies. First, I interviewed people about their experiences using personal informatics systems resulting in the Stage-Based Model of Personal Informatics Systems, which describes the different stages that systems need to support, and a list of problems that people experience in each of the stages. Second, I identified the kinds of questions people ask about their personal data and found that the importance of these questions differed between two phases: Discovery and Maintenance. Third, I evaluated different visualization features to improve support for reflection on multiple kinds of data. Finally, based on this evaluation, I developed a system called Innertube to help people reflect on multiple kinds of data in a single interface using a visualization integration approach that makes it easier to build such tools compared to the more common data integration approach.
Thesis Defense - Personal Informatics and Context: Using Context to Reveal Factors that Affect Behavior
1. P E R S O N A L
I N F O R M A T I C S
& C O N T E X T
USING CONTEXT TO REVEAL FACTORS
THAT AFFECT BEHAVIOR
IAN LI
ANIND DEY JODI FORLIZZI NIKI KITTUR JOHN STASKO
Co-Chair Co-Chair HCII Georgia Tech
14. Opportunity!
34% of U.S. adults are obese
(National Health and Examination
Survey, 2010)
27% of adult internet users
have tracked health data
online (Pew Internet Report, The
Social Life of Health Information, 2011)
14
15. Thesis
A personal informatics system
that allows users to associate
context with behavioral information
can better
reveal factors that affect behavior
compared to systems that only show
behavioral information.
15
16. Model of Personal Informatics
Created a model to guide the design of
personal informatics systems.
16
17. Model of Personal Informatics
Field Studies
Showed evidence in field studies that context
can reveal factors that affect behavior.
17
18. Model of Personal Informatics
Field Studies
Visualization Support
Explored what kinds of visualization support
personal informatics systems should provide.
18
19. Model of Personal Informatics
Field Studies
Visualization Support
Personal Informatics Dashboard
Developed a personal informatics dashboard
that makes it easier for users to associate
different kinds of data in a single interface.
19
20. Model of Personal Informatics
Field Studies
Visualization Support
Personal Informatics Dashboard
20
21. Goal
Create a model as a guide in designing
personal informatics systems.
21
22. Survey and Interviews
Recruited 68 people who use personal
informatics tools
Asked participants what tools they use and
problems they’ve encountered.
22
23. Sample Questions
• How difficult is it to collect this personal
information?
• How do you explore this collected personal
information?
• What patterns have you found?
Transcript of the survey is at:
http://personalinformatics.org/lab/survey
23
24. Analysis
Identified problems that people experienced.
Affinity diagrams to identify themes.
Derived a model composed of:
• 5 stages
24
27. PREPARATION COLLECTION INTEGRATION REFLECTION ACTION
Alice
Wanted to become active
Decided to track her
physical activity
Chose to track step counts
using a pedometer
27
31. PREPARATION COLLECTION INTEGRATION REFLECTION ACTION
The stage when people
choose what they are going to
do with their new-found
understanding of themselves.
31
32. Properties of the Stages
1. Problems cascade.
2. Stages are iterative.
3. User- vs. System-driven
4. Uni- vs. Multi-faceted
32
33. Properties of the Stages
1. Problems cascade.
2. Uni- vs. Multi-faceted.
3. Stages are iterative.
4. User- vs. System-driven
33
37. 1. Problems cascade.
Problems in the earlier stages can affect the
later stages.
Consider all the stages when building
personal informatics tools.
37
38. 2. Uni- vs. Multi-faceted
Users expressed desire to see associations
between different facets of their lives.
“If it were easily collected, information on
food intake, calories, fat, etc., would make an
interesting starting point for analysis.”
User who tracks medication intake
38
39. 2. Uni- vs. Multi-faceted
Most personal informatics are uni-faceted.
Some personal informatics tools
have multi-faceted collection,
but only support uni-faceted reflection.
39
40. 2. Uni- vs. Multi-faceted
Active
Inactive Inactive
M! T! W! Th! F! Sa! Su! M! T!
40
42. 2. Uni- vs. Multi-faceted
Most personal informatics are uni-faceted.
Explore support for collecting data
on multiple facets of one’s life.
42
43. Benefits of the Model
Identified the problems with existing tools.
Highlights the many challenges of building
effective personal informatics tools.
A common framework for describing,
comparing, and evaluating personal
informatics tools.
43
47. Physical Activity
Lack of physical activity is a common problem
that leads to obesity, diabetes, and high
blood pressure.
Lack of awareness of physical activity is one
reason why people are not active.
47
48. Sedentary People & Walking
Research suggests that they are less aware of
their physical activity and how to become
active. (Sallis & Hovell 1990)
Encourage walking because it is easier to
integrate into daily life. (Norman & Mills 2004)
48
49. application. This is shown in Fig. 2c and d. network. The network inputs are the sum of signal strength
fluctuation across all monitored cells, and the number of
3.1 Sensing activity distinct cells monitored over a given time interval. The
network consists of a single layer of eight hidden neurons;
The current activity of the user is inferred using patterns of weights are learnt using back propagation. The network
fluctuation in GSM signal strength and changes to the IDs outputs the currently sensed activity for the given input
of detected cells. This method has been demonstrated as a values. The network is trained by repeatedly presenting data
Physical Activity Awareness
reliable and unobtrusive way of sensing current activity [2],
and has the advantage over the more traditional approach of
using an accelerometer in that it does not require additional
sensor hardware as in Sensay [17] and the multimodal
collected during each method of movement.
The current activity of the user is conditionally depen-
dent upon their previous activity. In order to provide instant
feedback to the user interface, the neural network deliber-
sensor board of [11]. Similarly, while the processing of ately does not model this behaviour. Instead, when deter-
physiological and biometric data could complement our mining if any additional minutes have been earned, we
approach, the benefits of encapsulating the system within a apply task knowledge based upon the output from the
mobile phone would be lost. An alternative approach would neural network over the previous two and a half minutes.
be to utilise the positioning information available from This enables noise to be filtered out and a more accurate
some mobile phone networks, however this approach representation of the users’ activities achieved. For exam-
Products frequently involves prohibitive cost, as well as depending
upon much of the same technology as our client based
monitoring.
ple, periods of low signal strength fluctuation such as
stopping at traffic lights whilst driving can be ignored when
placed between periods of high fluctuation where many
Rather like a traditional accelerometer, the levels of distinct neighbouring cells were monitored. It could be
signal strength fluctuation change when a mobile phone is argued that activity would be more accurately inferred if a
moved. For example, Fig. 3 shows the total signal strength longer rolling filter had been applied to the GSM data.
fluctuation across all monitored cells during successive 30-s Introducing longer filters would have increased the likeli-
time periods whilst walking, remaining still and travelling hood of active minutes ‘disappearing’ from the users’
Fish’n’Steps: Encouraging Physical Activity with an Interactive Computer Game
1 2
Research 3
4
!!
Fig. 1. One participant’s display after approximately two weeks into the trial in the Fish'n
team-condition, also the public kiosk and pedometer platform, which rotated through e
the team fish-tanks. The components of the personal display include: 1) Fish Tank - Th
tank contains the virtual pets belong to the participant and his/her team members, 2) Virtu
Figure 2 The phone interface. Images a and b show screens for examining relative – The participant’s own fish in alevels:view on the right side next to the fish tank, 3) Ca
and individual activity frontal compare Daily Activity and
This Week’s Activity Images. c and d show two of the screens showing the estimated current activity level: Stationary and progress bar, personal and team ra
tions and feedback - improvement, burned calories, Walking
UbiFit Shakra Fish ‘n Steps
etc., 4) Chat window for communicating with team members.
To evaluate the effect of Fish’n’Steps, we recruited 19 participants from the
Consolvo et al. ’08 Maitland et al. ‘06 Lin et al. ‘06
of Siemens Corporate Research to participate in a 14-week study. Two experim
conditions were designed to separately assess the impact of the virtual pet an
social influences. Application of the TTM to assess behavior that changed durin
study demonstrated that Fish’n’Steps was a catalyst of a positive change for 14 o
19 participants. This effect was evident in either an increase in their daily step
49
(for 4 participants), a change in their attitudes towards physical activity (for 3 pa
pants) or a combination of the two (for 7 participants). The greatest change in
50. Research on Factors
Physical activity is affected by lack of time,
choice of activities, the environment, and
social influence. (Sallis & Hovell 1990)
CDC suggests understanding of factors to
circumvent barriers to physical activity.
50
51. Research on Factors
Diabetes awareness of blood sugar level and
food consumption (Frost & Smith ’03)
• Images of food associated with blood sugar
level.
• Used in a class where people discussed
their images and blood sugar level.
• Made a prototype, but only tested with one
person.
51
52. Research on Factors
Asthma patients videotaping daily routines
found that they are in the presence of harmful
allergens more often than they realized
(Rich et al. ‘00)
• Users videotaped daily routines, but a
trained observer looked at the video for
assessment.
52
56. Goal
Before building a prototype,
explore what people would do when they
have access to both physical activity and
contextual information.
56
57. SenseWear Pedometer Booklet
Date:
Time How active were you? What? Where? With whom? Time How active were you? What? Where? With whom?
6a: 1p:
: :
: :
: :
7a: 2p:
: :
: :
: :
8a: 3p:
: :
: :
: :
9a: 4p:
: :
: :
: :
10a: 5p:
: :
: :
: :
11a: 6p:
: :
: :
: :
12p: 7p:
: :
: :
: :
Continue to the next page.
57
58. Takeaways
“It was nice to see that I walked more than I
did. There was one day when I was
babysitting. I walked so much with the baby.
I walked all over campus.” A1
Activity Location Person
58
59. Takeaways
“Housework and walking to the bus stop can
contribute, really. I mean, I take that for
granted in terms of energy expenditure.” A4
Activity Location
59
60. Matching SenseWear graphs with booklet
FRI DEC 8, 2:03 ... Start Time
- Fri Dec 8, 2006 05:14 AM
End Time
entries. End
Session end - Fri Dec 8, 2006 02:03 PM
2:03 PM
, 2:16 ... FRI DEC 8, 2:03 ... Start Time
- Fri Dec 8, 2006 05:14 AM
End Time
Session end - Fri Dec 8, 2006 02:03 PM
Start End
5:14 AM 2:03 PM
Active ... Physical Activity (2.5 ME... Step Count Lying Do... Sleep Duration of Vi...
off-body
of 4 cal
438 cal 1 hr 43 m... 11346 Not detect... Not detect... 8 hrs 49 m...
60
61. Summary
Participants made associations between their
physical activity and contextual information
helping them become aware of factors that
affected their physical activity.
Can a prototype support this
in a field study with more people?
61
65. Plus-Context
e
Day with
context labels
f
Table and chart
of steps and
context
g
Steps by hour
and by period
of day
Figure 3. a) Interface for recording steps. Steps-Only additions.
65
b) One day of steps. c) Week of steps by day. d) Week of steps for
66. Pedometer Booklet Dashboard
Steps
Baseline
1 week
Visualization of Steps
Steps
Steps-Only
3 weeks
Visualization
of Steps & Context
Steps
IMPACT 1.0 Activity
Location
3 weeks People
66
67. Participants
30 participants (B1-B30)
• Sedentary. Pre-screened using Stages of
Exercise Behavior Change (Marcus et al. 1998)
Questionnaires at the end of each phase
67
68. Mentioned Context
“It helped me realize which activities were
more important. For example, I didn’t
understand the importance of walking home
versus taking the bus.” B8
“It turns out I get the most walking done
to and from work, which I can't say I wasn't
expecting, but I also had no idea that
walking around Squirrel Hill for just an hour
or two made such a difference.” B24
68
69. Of the 30 participants…
Mentioned Context
(Activities, Location, People)
IMPACT 1.0 13 participants
Visualization
of Steps and Context
Steps-Only 7 participants
Visualization of Steps
Baseline 6 participants
No Visualizations
69
70. IMPACT supports
reflection on context
“The [visualization] I used the most was the
one asking who I was with; I hadn’t realized
that I was so sedentary most of the time I
spent with my friends.” B1
70
72. Possible Improvements
“IMPACT gave a lot of cool information, but
having to input all the various factors was a
hassle.” B4
90% reported they would continue using
IMPACT if collection of context was
automated.
72
73. Possible Improvements
“IMPACT gave a lot of cool information, but
having to input all the various factors was a
hassle.” B4
90% reported they would continue using
IMPACT if collection of context was
automated.
Next: IMPACT 2.0
73
79. Mobile Phone Dashboard
Collected
Baseline Steps Only
Visualization of Steps
Collected
Steps-Only Steps Only
Collected Steps, Activity, Visualization
Location, and People of Steps & Context
IMPACT 2.0
79
81. Participants
35 participants (C1-C35)
• Sedentary. Pre-screened using Stages of
Exercise Behavior Change (Marcus et al. 1998)
Questionnaires at the end of each phase
81
82. Results
No complaints about inputting data.
But people complained about carrying
multiple devices.
• “I would not like carrying two devices (GPS
and phone), that was too much.” C30
82
83. Awareness of factors increased for
all groups between the phases
32,./2*" 4.)5(67,*8" -9:;3<"=#>"
$"
!"#$%&%''()*(+#,-)$'(
%#$"
%"
!#$"
!"
&'()*+,)" -,.)/0),12,"
F[2,32] = 3.98, p = .0547
83
84. Mentioned Context
Mentioned Context
(Activities, Location, People)
IMPACT 2.0 6 of 11 participants
Visualization
of Steps and Context
Steps-Only 3 of 12 participants
Visualization of Steps
Control 5 of 12 participants
No Visualizations
84
85. Short-Term Benefits/Problems
Short-term
IMPACT 1.0 Harder to collect,
Manual Collection but more engaged
IMPACT 2.0 Easier to collect,
Automated Collection but less engaged
85
86. Long-term reflection
What is the value of contextual information in
the long-term?
6-months later when they were more likely to
have forgotten the data
86
87. Follow-Up Interviews
Expressed interest in comparing over long
periods of time.
Curious about the peaks in physical activity.
But only those who had visualizations of
contextual information had reminders of what
happened during those peaks.
87
88. Long-Term Benefits/Problems
Short-term Long Term
IMPACT 1.0 Harder to collect, No reflection
Manual Collection but more engaged opportunity
IMPACT 2.0 Easier to collect, Has reflection
Automated Collection but less engaged opportunity
88
89. Overall Summary
Provided some evidence that a system that
shows context can reveal factors that affect
behavior.
But the value of the data is highly
dependent on the type of support.
89
96. Analysis
Identified the kinds of questions people asked
about their data.
Affinity diagrams to identify themes.
Derived 6 kinds of questions.
96
97. Six Kinds of Questions
Status What is my current status?
History
Goals
Discrepancy
Details
Factors
97
98. Six Kinds of Questions
Status
History What happened in the past?
Goals
Discrepancy
Details
Factors
98
99. Six Kinds of Questions
Status
History
Goals What goals should I pursue?
Discrepancy
Details
Factors
99
100. Six Kinds of Questions
Status
History
Goals
Discrepancy How does my behavior compare
Details to my goals?
Factors
100
101. Six Kinds of Questions
Status
History
Goals
Discrepancy
Details What other things happened
Factors during a particular point in time?
101
102. Six Kinds of Questions
Status
History
Goals
Discrepancy
Details
Factors What influences my behavior
over a long period of time?
102
103. Importance of the Questions
Not all questions are important all the time.
Some questions are more important than
others as people’s information needs change.
103
104. Importance of the Questions
Not all questions are important all the time.
Some questions are more important than
others as people’s information needs change.
Maintenance Discovery
&
Phase Phase
104
105. Maintenance Phase
Participants already know how different
factors affect their behavior, so they just want
to know what their current status is.
Participants have already identified their
goals. They are only concerned with whether
they are meeting their goal.
105
106. Maintenance Phase
Current Status
P13 just tracks the minutes that he spends on
Facebook, Twitter, and other social media
sites, because he already know how these
affects his productivity.
106
107. Maintenance Phase
Discrepancy
P1 uses Mint to keep track of her
expenditures to see whether she is meeting
the budget that she had set for herself.
107
109. Discovery Phase
Participants collect several types of
information to find out what factors affect
their behavior.
Participants are trying to figure out
what their goals are.
109
110. Discovery Phase
Finding factors that affect their behavior.
P3 has diabetes and she tracked her blood
glucose levels and her food consumption to
find out their interaction.
110
111. Discovery Phase
Figuring out goals
P8 tracks the quality of her sleep so that she is
better rested. She explores her sleep data to
“spot trends for which I can take corrective
action.”
111
112. Discovery Phase
These kinds of questions were the most
important:
• History
• Goals
• Details
• Factors
112
113. Discovery Phase
These kinds of questions were the most
important:
• History
• Goals
• Details
• Factors} Contextual information
& Multiple types of data
113
114. Discovery Phase
These kinds of questions were the most
important:
}
• History Next Question
• Goals What visualization features
would help users find answers
• Details
to these kinds of questions?
• Factors
114
116. Results
History: Looking back in time.
Participants generally agreed that the timeline
sketches were the most appropriate for the
Discovery phase.
116
117. Results Goal
Goals: Seeing goals information.
“I like having the goal line...I always like being
able to see what my baseline should be and
if I am above or below.” P5
117
118. Results
Details: Seeing details to reason what
happened.
“When looking at exercise there are a couple
of times where I really didn’t meet my goal,
so it will be really nice to be able to say ‘why
didn’t I meet my goal then?’” P4
118
119. Results
Factors: Comparison of different
kinds of data.
“The most interesting thing here is the ability
to compare two different time frames
because I’m really interested in the
relationship between data.” P6
119
120. Summary
Contextual information and multiple types of
data is important during the Discovery phase.
Described visualization features to help
people answer questions during the Discovery
phase.
120
121. Summary
Contextual information and multiple types of
data is important during the Discovery phase.
Described visualization features to help
people answer questions during the Discovery
phase.
Next: Built a personal informatics
dashboard with the visualization features.
121
122. Model of Personal Informatics
Field Studies
Visualization Support
Personal Informatics Dashboard
122
123. Goal
Build a personal informatics dashboard that
allows users to see multiple kinds of data
together.
Develop an approach that makes it easier to
build.
123
127. Problems with Data Integration
Dashboard has to:
Access Data
Parse Data
Visualize Data
127
128. Problems with Data Integration
Dashboard has to:
Access Data Managing many data
sources w/ different APIs.
Parse Data
The data source loses
Visualize Data control of the data.
128
129. Problems with Data Integration
Dashboard has to:
Access Data No standard format for
the different types of data
Parse Data that users collect.
Visualize Data Dashboard has to create
parsers for each format.
129
130. Problems with Data Integration
Dashboard has to:
Access Data Dashboard has to create
visualizations for each
Parse Data type of data.
Visualize Data Duplicates creation of the
visualizations.
130
135. Benefits of Viz Integration
Dashboard has to:
Accessing Data Provide an API that
data sources can use.
Parsing Data
Manage the
Visualizing Data communication
between widgets.
135
136. Benefits of Viz Integration
For the perspective of data sources:
Maintain control of the data.
They can choose how the data is visualized.
Create a widget and it can be used with
widgets that others have made.
136
139. Innertube API
Data sources create visualization widgets
using static images, Javascript, and/or Flash.
Data sources use the API to communicate
with the dashboard and vice versa.
139
140. Innertube API
Get the date and range of visualizations to
display.
Get the currently highlighted data point.
Change the appearance of the widget.
• Set height of the widget.
• Reload the widget.
140
146. Data Collection for 1 week
Automatically Collected Manually Collected
Step Counts using Fitbit Mood
GPS Location Amount of Sleep
Weather Information Busyness
Energy Level
People
Notes
146
147. Returned to the Lab
Used Innertube while thinking-aloud.
• What they were looking for
• What they were finding
• What problems they encountered
Answered questionnaires about Innertube.
147
148. Results
13 of the 15 participants agreed that
Innertube was useful.
148
149. Results
“It allowed me to factor in location, times,
and activity in order for me to assess where I
may be able to increase physical activity.”
P12
“Seeing the temperatures of the times I went
on my runs and knowing how well I did on
them would allow me to determine the best
condition for me to run in.” P14
149
150. Results
“It gave me concrete contexts, in space and
time, by which I could measure and evaluate
my own physical activity. Interacting with that
data gave me the opportunity to
hypothesize about what factors influenced
my own physical activity, and what
specifically motivated me or discouraged me
from exercising.” P9
150
151. Results
“I thought certain widgets [factors] were less
useful before I used the PI dashboard, and
then I changed my mind after using it,
because their usefulness became apparent to
me.” P9
151
152. Future Work
Improve the usability of the Innertube
Dashboard.
Make the Innertube API available to
developers. Coming soon!
Create a directory of Innertube Widgets, so
people can find widgets easily.
152
153. Summary
Described visualization integration, an easier
approach to building personal informatics
dashboards.
Implemented Innertube, an example of
visualization integration.
153
155. Contributions
Created a model to guide the design of
personal informatics systems.
Showed evidence that contextual information
can reveal factors that affect behavior.
155
156. Contributions
Explored what kinds of visualization support
personal informatics systems should provide.
Developed an easier way to build
personal informatics dashboards to help
users associate different kinds of data in a
single interface.
156
157. Future Work
Deploy longer field studies.
Conduct studies in other behavior domains.
Explore how to convert awareness of factors
to changes in behavior (Action stage).
157
158. Thank you!
To my committee, Anind Dey, Jodi Forlizzi, Niki Kittur, and John Stasko.
To the many who have helped along the way: Gary Hsieh, Erin Walker,
Karen Tang, Scott Davidoff, Amy Ogan, Ruth Wylie, Moira Burke, Queenie
Kravitz, Gabi Marcu, Rebecca Gulotta, Matt Lee, Turadg Aleahmad, Aruna
Balakrishnan, Min Kyung Lee, Tawanna Dillahunt, Sunyoung Kim, Chloe Fan,
Jenn Marlow, Jason Wiese, Stephen Oney, Chris Harrison, Julia Schwarz,
Eliane Stampfer, Samantha Finkelstein, Aubrey Shick, Matt Easterday, Bilge
Mutlu, Andy Ko, Johnny Lee, Ido Roll, Jeff Nichols, Jeff Wong, Jennie Park,
Sara Kiesler, Laura Dabbish, Scott Hudson, Tessa Lau, Fernanda Viegas,
Jaime Teevan, Alexandra Carmichael, Gary Wolf.
To HCII, QoLT, the Ubicomp Lab, and the Quantified Self.
To my family, Papa, Mama, Robin, and Cassandra.
This work is based on research supported by the National Science Foundation under Grant No.
IIS-0325351 and EEEC-0540865.
158