The document discusses the early years of lifelogging and how the speaker has been using a wearable camera for 12 years to capture billions of data points, exploring use cases for analyzing and retrieving information from this large collection of personal data over time. It outlines challenges in indexing and querying diverse types of lifelog data from sensors and multimedia sources and the need for multidisciplinary approaches involving areas like human-computer interaction, multimedia analytics, ethics, and information retrieval to address open questions around privacy and new applications.

1. “Lifelogging - The Early Years”
Associate Professor & Asst HoS at the School of Computing, Dublin City University, Ireland
Principal co-Investigator at the Insight Centre for Data Analytics
(@cathal - cathal@gmail.com)
Cathal Gurrin
Researcher
Personal Data
Analytics and e-health
Educator
Information Retrieval
and Data Analytics
Lifelogger
A decade of visual
lifelogging experience
University of Kyoto - 15th June 2018
Progress with Emerging Forms of Personal Data

2. In 2006 I put on a wearable camera. 12 years and billions of data
points later, I am still doing it….. WHY?

3. Guided by the idea that you can never
capture what has already passed

4. 1928
Bucky-
Fuller
1980s
Steve
Mann
2004
Williams
(Sensecam)
2015
Early
Products
2006+
Memory
Studies
2004
G. Bell
(MyLifeBits)
1946
Vannevar
Bush

6. Human Ledger
Life Experience
The individual will have a human ledger &
personal search engine for all life
experience… activities, experiences,
behaviours, information, biometrics… huge
volumes of data captured passively.

7. OK.. so why should I care?
MMM
UMAP
DATA
ICMR
SIGIR
ECIR
MM

8. What types of data?
Driven by low-cost sensors, massive volumes of data are being created
and years of data can fit on a $100 hard disk
New data types that we are not used to working with…

9. QS
Quantified Self

10. QS
Quantified Self

11. Rich Sensing

12. Google Glass, Snapchat Spectacles,
ion SnapCam, Narrative Clip, GoPro,
Sony Xperia Eye, etc… capturing
experiences not just biometrics…
Visual Sensing

13. Writing Content
Reading Content
Device Interactions
Loggerman: Privacy-
aware
HCI and information
logging
www.loggerman.org
Z. Hinbarji, R. Albatal, N. O’Connor and C. Gurrin (2016) LoggerMan,
a comprehensive logging and visualisation tool to capture computer
usage. In: 22st International Conference on MultiMedia Modelling
(MMM 2016), 4-6 Jan, 2016, Miami, FL
Information Sensing

14. Using Sony Digital Paper or a digital pen
(EchoPen, LiveScribe), you can create and
annotate digital PDFs
Writing Content
Reading Content
Information Sensing

15. 72 Heart Beats
12 GPS locations
12 Physical Activity Logs
2 images
450 keystrokes
0.07 Glucose readings
And so on…
Captured at different frequencies, with different error rates, and in a
huge number of different modalities…
00:0200:0100:00 00:0500:0400:03 00:0800:0700:06 00:1100:1000:09

16. So… What do first generation human ledgers look like?

17. Visual Diary (DCU - 2006)
A Doherty, C Ó Conaire, M Blighe,A.F. Smeaton, N.E. O’Connor (2008) Combining image descriptors to effectively retrieve events from visual lifelogs. In: MIR
2008 - ACM International Conference on Multimedia Information Retrieval, 30-31 October,Vancouver, Canada.

18. Life Abstraction
(objects, people, products)

19. Linking Multiple Data Sources - images to stress levels

20. A simple search engine enhanced finding important events by 200% and made it 10 x faster for
healthy individuals, when compared to an event-based browsing interface…
A Doherty, K Pauly-Takacs, N Caprani, C Gurrin, C Moulin, N O'Connor and A.F. Smeaton (2012) Experiences of aiding
autobiographical memory Using the SenseCam. Human–Computer Interaction, 27 (1-2). pp. 151-174. ISSN 0737-0024

21. KidsCam (Univ. Otago & DCU) extracted knowledge for
ethnographic study (2015)…
Wearable cameras on 200 school children - to understand exposure to fast-food marketing.
ISBNPA 2017 Publication of the Year: "Children’s everyday exposure to food marketing: an objective analysis using wearable
cameras", L. N. Signal, et al. International Journal of Behavioral Nutrition and Physical Activity. 201714:137

22. The Unanswered Questions
What is a document?
What are the use cases?
How to index the content?
What type of user queries will be make?
How to know what is a good approach?
…. and a lot more ….

23. 1
2 3
4
5
6
7
8
Aggregation / ViewSummary
Human Activity
Moment / short time
period
Single Reading
Event / Scene
Logical Sequence of
Events
Day / Experience
Prior Work
Data Retrieval
Static Event
Segmentation
Summarisation

24. Extract Caffe Concepts
Features
1000 low-level features/image
Extract Microsoft’s Computer
Vision API features
- color
- description
- tagging
- adult score
for each image……
251 275 310 387 1184 1205
Minute as base Unit of retrieval : select first image of every minute
Euclidean distance vector to make event boundaries
with large distance between features of adjacent images of every minute.
Segmenting images into events
Raw Image Stream of One Day
A Basic (not-flexible) Approach to Event
Segmentation… we need dynamic approaches
Approaches for event segmentation of visual lifelog data. R Gupta &
C Gurrin. Proc of MMM2018, Bangkok, Thailand, Feb 2018.
Automatically Segmenting LifeLog
Data into Events. A. R. Doherty &
A. F. Smeaton. 2008 Ninth
International Workshop on Image
Analysis for Multimedia Interactive
Services, Klagenfurt, 2008

25. The Unanswered Questions
What is a document?
What are the use cases?
How to index the content?
What type of user queries will be make?
How to know what is a good approach?
…. and a lot more ….

26. Quantified Self
Personal Insights
Data-driven Health
Augmented
Wellness
Behaviour Change
Enhanced Security
Population-wide
Analytics
Augmented
Community
Augmenting Human
Memory
Nomenclators
Augmented
Memory
Enhanced Productivity
& Education
Enhanced Interactions
Rich Sharing and
Reminiscing
Augmented
Cognition
Many (Individual) Use Cases
Health & Wellbeing Memory & Cognition

27. Quantified-Self
Analysis
Self-discovery
Reflect
Contextual
Reminders
Remind
Sousveillance.
Protection of me
and bystanders
Protect
Find an item from
the digital self
Validate a memory
Contextual support
Answer
Reminiscence
Therapy
Social applications
Reminisce
Digital Agents
acting on our
behalf, during life
and after
Represent
The most interesting aspect (for me) is the potential for memory
support, where the lifelog works in synergy with your own
memory.
Adapted from Abigail J. Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique of lifelogging. Commun. ACM 53,
5 (May 2010), 70-77.

28. Focus on supporting knowledge
acquisition and learning in the
early years.
1. Knowledge Support
From education to the workplace,
providing information and
insights to assist productivity and
fitness.
2. Productivity
Into old age, providing support
for cognition and health to
maintain independence and
activity.
3. Health
CHILD
ADULT
ELDERLY
In reality, we can’t yet imagine the use-cases of human ledgers,
but it could become a permanent companion assisting you
throughout life. Constantly growing in size.

29. Given an information need, find relevant data to
answer the request…
Cross-modal Data Retrieval
Find the moment in which the lifelogger was doing
something of topical interest…
Moment Recall / KIS
Provide abstraction the data to answer a particular
information need (e.g. health topics)
Query-focused Abstraction
Do all of the above across populations, and not just at
the individual level.
Cross-individual analysis & retrieval
Find an event that fully answers an information need of the
individual as a form of recall…
Experience / Dynamic Event Retrieval
Summarise the data to highlight important events
and activities
Summarisation
Examine the
Use-Cases
as IR challenges

30. The Unanswered Questions
What is a document?
What are the use cases?
How to index the content?
What type of user queries will be make?
How to know what is a good approach?
…. and a lot more ….

31. Applying AI (machine/deep learning) can extract value
from multimodal data … to find objects, activities,
environments, brands… even open-source detectors
work well.

32. Microsoft Cognitive Services Example
Google and Microsoft provide online services

33. A lot of opportunity to merge different sensing modalities to develop
human activity models…
Kahneman et al.A survey method for characterizing daily life experience:The day reconstruction
method. Science, 306(5702):1776–1780, 2004.

34. The Unanswered Questions
What is a document?
What are the use cases?
How to index the content?
What type of user queries will be make?
How to know what is a good approach?
…. and a lot more ….

35. TITLE: A380
DESCRIPTION: Find the moment(s) when I was taking a
photo of an A380
NARRATIVE: To be considered relevant, the user must be
seen to be taking a photo of an A380 airplane prior to
boarding or after disembarking from an aircraft
Known-Item Search
Solution: Visual, Location, Activity

36. Real-time Known-Item Search
<Topic duration="180">
<TopicID>LSC01</TopicID>
<TopicType>development</TopicType>
<Descriptions>
<Description timestamp="0">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background.</
Description>
<Description timestamp="30">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background
and a plastic plant on my right side.</Description>
<Description timestamp="60">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background
and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it
took less than two minutes to get there.</Description>
<Description timestamp="90">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background
and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it
took less than two minutes to get there. My colleague in the foreground is wearing a white shirt and drinking coffee from a red paper cup.</
Description>
<Description timestamp="120">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background
and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it
took less than two minutes to get there. My colleague in the foreground is wearing a white shirt and drinking coffee from a red paper cup. Immediately
after having the coffee, I drive to the shop.</Description>
<Description timestamp="150">In a coffee shop with my colleague in the afternoon called the Helix with at least one person in the background
and a plastic plant on my right side. There are keys on the table in front of me and you can see the cafe sign on the left side. I walked to the cafe and it
took less than two minutes to get there. My colleague in the foreground is wearing a white shirt and drinking coffee from a red paper cup. Immediately
after having the coffee, I drive to the shop. It is a Monday.</Description>
</Topic>

37. The Unanswered Questions
What is a document?
What are the use cases?
How to index the content?
What type of user queries will be make?
How to know what is a good approach?
…. and a lot more ….

38. Address the retrieval
challenges…
NTCIR Lifelog
A Lifelog Search Challenge
for Interactive Rertrieval
LSC
Build better annotation tools
for visual and multimodal
content
ImageCLEF
Three Main Vehicles

39. Challenge NTCIR Lifelog ImageClef LSC
Ad-hoc Retrieval 12,13,14
Event
Segmentation
13
Annotation 14 2017, 2018
Abstraction/
Summarisation
12,13,14
Interactive
Retrieval
2018, 2019

40. NTCIR-Lifelog

41. NTCIR12-Lifelog Comparative Benchmarking Competition
Five different teams with five different approaches…

42. NTCIR14-Lifelog3
• Rich lifelog data (60 days, 3 people), fully
anonymised
• Three sub-tasks:
• LSAT - Lifelog Semantic Access Task
• LIT - Lifelog Insights Task
• LADT - Lifelog Activity Detection Task

43. MINUTE AS
THE UNIT OF
RETRIEVAL
24x7 heart rate, blood sugar,
calorie burn, steps, skin
temperature. Daily blood
pressure. Weekly cholesterol
measurements.
Human Biometrics
2,000 images per day from
the Autographer wearable
camera. Accompanying
concept annotations. Audio
levels. Manual photos
captured. Music listening
Wearable Multimedia
Physical activities (walking,
running, transport, etc..),
locations visited, food eaten,.
Human Activity
Onscreen reading, keystrokes
on keyboard, mouse
movements, computer
activity, web pages viewed.
Information Access
0201
0403

44. Sub-tasks
• LSAT - Known-item search… think of it like a
Google for the individual.
• LIT - Lifelog Insights Subtask… health related
insights based on biometrics and images.
• LADT - Lifelog Activity Detection Subtask… find
the activities from an ontology of life activities.

45. Lifelog Search Challenge

46. Lifelog Search Challenge
Hopefully an annual task at
ICMR conference.
Interactive ad-hoc retrieval
challenge in front of an
audience.
Six teams in the first
challenge in 2018.

47. 2
51
6
4
3
DCU
Ireland
Klagenfurt
Austria
Charles U.
Czech R.
VNU
Vietnam
U Utrecht
Netherlands
UPC
Spain

48. Winning team was a VR-System with visual and temporal search

49. ImageCLEF Lifelog

50. Lifelog Summarization Task
Analyse the lifelog data and
summarize them according to
specific requirements.
Lifelog Retrieval Task
Analyse the lifelog data and
according to several specific
queries return the correct
answers.
ImageCLEF Lifelog Task (2017/2018)

51. Teams
66 registrations
21 signed copyright forms
19 submitted runs

52. Easy vs. Hard topics
T8. Transporting
Query: Summarize the moments when user u2 using public transportation.
Description: Photos taken inside a car or a taxi are not relevant. Blurred or out of
focus images are not relevant. Images that are covered are not relevant.
0.81

53. Easy vs. Hard topics
T4. Working at home
Query: Find the moment(s) in which user u1 was working at home.
Description: To be consider to relevant, the user should be using computer for
work, reviewing an article or taking some notes at home. Using computer for
entertainment is not relevant.
0.08

54. The Unanswered Questions
What is a document?
What are the use cases?
How to index the content?
What type of user queries will be make?
How to know what is a good approach?
…. and a lot more ….

55. A variety of data, different
timings, different accuracies,
needing different tools.
Data Processing
Scalable & efficient indexing
with contextual querying and
no defined unit of retrieval.
Search & Retrieval
Use-cases need pervasive
access and contextual
querying.
Anywhere, Anytime
Develop fixed and ubiquitous
capture & access methods
for all stakeholders.
User Experience
The ethics of how to use rich
personal data & doing so in a
privacy-aware manner.
Personal Data
HUMAN
COMPUTER
INTERACTION
MULTIMEDIA
ANALYTICS
ETHICS&
PRIVACY
PERVASIVE
COMPUTING
INFORMATION
RETRIEVAL
MEMORY
MEMORY
ETHINOGRAPHY
Multidisciplinary Approach

56. Privacy Awareness - Automated Negative Face Blurring
with real-time Policy-driven Access Restrictions (Ye et al. 2014)
What about Privacy?
The meaning of privacy changes across different jurisdictions
Different demographics have different expectations

57. Storage
Feature
Extraction
Professional Market
Research Applications
Semantic
Enrichment
User Software
Analytics Engine
Insight &
Query Engine
The Emergence of a Privacy-aware Ecosystem for Personal Media
Storage, Analytics and Monetisation

58. In Summary:
Individuals are beginning to capture our own human ledgers.
These will be goal-driven with positive benefits for the individual.
We need to understand what can be done to assist both researchers in
developing tools and the life loggers to gain value from their content.
We need to understand and find ways to…
02
Clean, segment and
enrich the data,
adding value.
Enrich
03
Index the data in
extensible and
flexible indexing
mechanisms
Index
01
Multiple
heterogenous data
sources
Gather
04
Support access for a
wide variety of use-
cases
Use

59. ありがとうございました
Associate Professor at the School of Computing, Dublin City University, Ireland
Principal Investigator at the Insight Centre for Data Analytics
(@cathal - cathal@gmail.com - http://about.me/cgurrin)
Cathal Gurrin
LifeLogging: Personal Big Data
Cathal Gurrin,Alan F. Smeaton,Aiden R. Doherty
Published: 16 June 2014
Do a google search and download the book from the DCU website.