The research presented here explores how to unlock the enormous power of human processing that is still underutilized, in a social, congenial and convenient way. This is still not well explored but is an emerging area of research. In the frame of the thesis I propose several context–aware social computing prototype systems that assist people to find context–sensitive suitable help and guidance from trusted social peers. I also illustrate the help seeking scenario in different ranges from large–scale city environments to people with dementia. Furthermore, I demonstrate the help seeking scenario in a simulated large and dynamic social network such as, a simulated vehicular network. I also propose approaches that can assist users to select required contextual information and assist to find suitable help providers. This presentation was a part of the PhD public defense of Nasim Mahmud. Place Hasselt University, Expertise Center for Digital Media, Aula (Large Auditorium), Diepenbeek, Belgium on April 25, 2012. PhD supervisors: Promoter : Professor Dr. Karin Coninx (Vice-dean, Hasselt University, Belgium) Co-promoter: Professor Dr. Kris Luyten Jury/Committee: Prof. Dr. Frank Van Reeth (Chairperson, Hasselt University), Prof. Dr. Karin Coninx (Advisor, Hasselt University), Prof. Dr. Kris Luyten (Co-advisor, Hasselt University), Prof. dr. Wim Lamotte (Doctoral committee member, Hasselt University), Dr. Jan Van den Bergh (Hasselt University), Prof. Dr. Yolande Berbers (Distrinet, University of Leuven), Dr. Ann Ackaert (IBCN, Ghent University)

1. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments
25 April 2012
Nasim Mahmud
Advisor: Professor Dr. Karin Coninx
Co-advisor: Professor Dr. Kris Luyten

2. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments

3. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments

4. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments

5. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments

6. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments

7. Exploiting Context-awareness and Social Interaction
to Provide Help in Large–scale Environments

8. Why Help?
• Someone is unable to do what he wants to do
• Someone needs some information
• Someone needs guidance

9. Need More and More Precise Information
Is this for…?
On my way to San Sebastian, Spain

10. Need More and More Precise Information
In San Sebastian, Spain
A person is browsing a map
Other people joined the search
Need more reliable information

11. Need More and More Precise Information
A menu along with a dictionary
Need more interactive information

12. Motivation
• People need fine-grained or interactive
information
• People need reliable information
• Problems in asking someone for help:
– Who is willing or eligible to provide help
– People are often hesitant to ask strangers
– Finding someone in the vicinity

13. RQs
• How to find a suitable person who can help?
• How to exchange contextual information?
• How to select relevant contextual information
and potential groups of help providers?
• How can persons with special need benefit
from context-awareness and social
computing?
• How can social and context-awareness
improve data dissemination?

14. Goal

15. Goal

16. Context
What am I
doing?
Who am I
with?
Where am
How is the
What is
possible?
I?
What time
is it?
… ?
How is the
weather?

17. What am I
doing?
Who am I
with?
Where am
How is the
What is
possible?
I?
What time
is it?
… ?
How is the
weather?

18. Social-context

19. Social-context

20. Social-context
Is available…

21. Social-context
Is available…
Is willing to…

22. Social-context
Is available…
Is willing to…
Is knowledgeable…

23. Ubiquitous Help System (UHS)

24. Ubiquitous Help System (UHS)
• People seek
• Precise and fine-grained information
• Often from other person(s)
• From reliable source
– It utilizes
• External context (time, location)
• Internal context (willingness, ability)
• Social Network (FOAF)

25. Profiles and Preferences

26. How
Extract from my foaf profile
Extract from other users’ foaf profile
Contextual variables
Constraints Constraints
Application logic
My preference Other users’ preferences

27. How Does it Work?
…has a question
? ?
…has a question
? ?
Profile and
preference
matched
Profile
matched
Reply

28. How Does it Work?
? ?
…has a question
? ?
? ?
…has a question
? ?
Profile and
preference
matched
Profile
matched
Reply

29. How Does it Work?
? ?
…has a question
? ?
Profile
matched
? ?
…has a question
? ?
Profile and
preference
matched
Profile
matched
Reply

30. How Does it Work?
? ?
…has a question
? ?
Profile
matched
Profile and
preference
matched
Reply
? ?
…has a question
? ?
Profile and
preference
matched
Profile
matched
Reply

31. Client Structure

32. Query Structure

33. Distributed Search
SPARQL search

34. UHS Java Client

35. How to Exchange Information
• How to exchange contextual
information
• How to exchange rich media
What am
I doing?
Who am I
with?
Where
am I?
What is
possible?
What
time is it?
How is
the … ?
How is the
weather?
Do you like this toy?

36. Who Can Help with the Question?
• A friend
• A family member
• A colleague
• A familiar person

37. Related Work
• Search by using social networks
Facebook, Facebook questions, Quora, Twitter etc.
• Mobile social Q&A
} Mobile
Social
• Photo-based question and answer Search

38. Photo-based Question Answer
Tom Yeh et al.(MM 2008)
Community based

39. VizWiz
Crowdsourcing based
Jeffrey P. Bigham et al. (UIST 2010)

40. Limitations of Existing Solutions
• Limited context-awareness
• Lacking social awareness
• Utilizes community and crowdsourcing
– Not suitable for a range of personal questions
– Not suitable where in-situ help is required
– Not interactive enough

41. Ubiquitous Help System-Next (UHS-Next)

42. UHS-Next System
A mobile system that provides
– Context-aware communication
– Media rich communication
– Usage of users’ personal social network

43. • Take a picture
• Ask a question
• Select a group
• Select contextual
information
• Preview the question
• Send

44. • Take a picture
• Ask a question
• Select a group
• Select contextual
information
• Preview the question
• Send

45. • Take a picture
• Ask a question
• Select a group
• Select contextual
information
• Preview the question
• Send

46. • Take a picture
• Ask a question
• Select a group
• Select contextual
information
• Preview the question
• Send

47. • Take a picture
• Ask a question
• Select a group
• Select contextual
information
• Preview the question
• Send

48. • Take a picture
• Ask a question
• Select a group
• Select contextual
information
• Preview the question
• Send

49. User Test

50. User Test 1: Finding Help
Where are you?

51. What does it mean?

54. Results of User Test 1
• UHS-Next is simple to use
• Voice interaction for ‘spoken audio question’
is needed
• Inspiring result

55. User Test 2: Spontaneous Social Interaction
• Free use of UHS-Next in real life by
– Two users
– One actor
• For two days
– In office environment
– In daily life situations

56. Results of User Test 2
• Other use than seeking help
– Spontaneous social interaction
– Sharing cognitive load
– Sharing daily life experiences (Fun moment,
“Whose office is this?”)
• Easily embedded in daily life
– Useful
– Easy to use

57. Remaining Difficulties
• Selecting right context
• Selecting right group of users
To solve these, we propose a mixed-initiative
approach

58. Mixed-initiative Context Filtering
and Group Selection Approach

59. • Our approach selects and prioritizes the
contextual data for a question, based on
the question content
• Helps to select a group of potential help
providers

60. Mixed-initiative Approach
• Human internal context is subtle to measure
by the available technologies
• A fully automated system requires to know all
the variable about human-activity and
external context
• To reflect that the user’s requirements are
satisfied and make sure that the user is in
control

61. Context Selection
• A broad range (e.g., urgency, time, location, weather
conditions)
• Which contextual information is important? (e.g.,
time critical, quality critical)
• How to capture that information? (e.g., urgency,
location, reliability)
• How to convey that information? (e.g., I am here
(where ‘here’ is unknown to the user))

62. Ubiquitous Help System-Selection
(UHS-Selection)

63. From Voice Question and from Sensors
Asking question in a natural way

64. Ubiquitous Help System for Context and
Group Selection (UHS-Selection)
Main screen

65. Voice to Text
Voice to Text conversion, user in the control

66. Parsing the Text

67. Language Processing
• Utilize the WordNet dictionary
– A social network of words
– Synonyms, meaning and relevance
• Utilize Named Entity Recognition (NER)
– Structure data in XML
– Customized for the purpose

68. Result
The UHS-Selection system sets priority to location

69. Group Selection
• Based on the context priority list (output from
the context selection algorithm)
• Current context (e.g., location, heading)
• Current task
• Next task

70. Group Selection: Visualization

71. Limitations and Workaround
• Need to know more information about the
persons who can provide help (e.g., location)
• Social translucence provides the balance
(Erickson et al. (2000))

72. We have applied the framework in particular
application domain, for Persons with Dementia
(PwD)
And in the dynamic social network
Simulated vehicular network

73. We have applied the framework in particular
application domain, for Persons with Dementia
(PwD)
And in the dynamic social network
Simulated vehicular network

74. Sharing awareness information in specific
context of use – Persons with Dementia

75. – In the early stage of dementia, they can
live their lives as usual, they can go:
– Shopping,
– Bird watching,
– Jogging,
– … …
– When dementia syndrome progresses,
they need more attention, and targeted
help/ more social and navigational help

76. Dementia
• Dementia is a term for a syndrome related to
the loss of cognitive functions
• An acquired decline in memory and thinking
(cognition) due to brain disease that results in
significant impairment of personal, social or
occupational function

77. General Needs
A person with dementia needs more
independence in terms of :
– Spatial
– Temporal and
– Social
awareness

78. As the Dementia Syndrome Progresses
• It becomes an important cause
of dependencies
• …Persons with dementia, are
increasingly dependent on their
social environment (likely to be
less autonomous)
• In most of the cases, in the early
form of dementia the caregiver is
a family member
(Schulz et al. 2010)

79. Scenario
(COMuICSer tool . Haesen, M. et al. 2009)

80. Ubiquitous Help System for Persons with
Dementia (UHSd)

81. Relation Between the Models Used to
Develop the System : Part 1/3
Dialog
model
Application
model

82. Relation Between the Models Used to
Develop the System : Part 1/3
Dialog
model
Application
model

83. Relation Between the Models Used to
Develop the System Part 2/3
Dialog
model
Application
model

84. Relation Between the Models Used to
Develop the System Part 2/3
Dialog
model
Application
model

85. Relation Between the Models Used to
Develop the System Part 3/3
Dialog
model
Application
model

86. System Overview of UHSd

87. System Overview of UHSd

88. One Example
To-do (Baker/Buy bread )
– Time (From 10:00 to 11:00)
– Location (Grote Markt Baker)
– Associated contact (Jane, Ilsa, Mark )

89. Resulting System (UHSd)-Navigation Panel

90. Resulting System(UHSd)-Communication
Panel

91. Summary of Ubiquitous Help System for
Persons with Dementia (UHSd)
• UHSd provides memory aid in terms of
– Spatial
– Temporal
– Social
awareness
• Provides context-aware support
– Ensures (partly) gaining users autonomy
– Ensures feeling of connectedness

92. Lessons Learned
• We presented an interactive system and
observed that applications for people with
dementia can be created by explicitly taking
context into account in the design process
• Three types of context variables involved in
the communication (Space, Time and Social
Context)

93. We have applied the framework in particular
application domain, for Persons with Dementia
(PwD)
And in the dynamic social network
Simulated vehicular network

94. We have applied the framework in particular
application domain, for Persons with Dementia
(PwD)
And in the dynamic social network
Simulated vehicular network

95. Geo-Social Interaction for Context-aware
Help in Large-scale Public Spaces

96. • We present
– an approach, how to utilize social and spatio-temporal
context to improve information
dissemination
– Geo-social relevance with a ‘Dynamic view
approach’
– Evaluated using a simulation with real life car data

97. Motivation
People who are `on-the-move' often do not have an
opportunity to spend long time looking for what they need

98. Social-components
• Person in the network
• Person with matched profile
• Person with matched preferences, help type, urgency

99. Geo-components
• User’s location
• Distance between users (Help seeker and Help
provider)
• Direction of movement

100. Finding Balance
(in Geo-social Components)

101. Friends
• Friendship is ‘asymmetric’ relation (like Twitter)
• Dynamically updating list

102. Help Type Matching (Asymmetric)

103. Validation ( by Simulation in KULeuven)
• Using realistic dataset for cars
• In area of 250 km by 260 km
• Logged simulation data for 24 hours
Socializing Cars Vehicular Network

104. Conclusion
Improved relevance back propagation technique for routing
messages in the network shows better results for each evaluated
parameter

105. Conclusion

106. Lessons Learned from
the Dynamic Social Network
• Social networking capabilities and spatio-temporal
context information significantly
improves purposeful interaction between
individuals
• It improves in terms of both the efficiency of
the network data dissemination and the
quality of the delivered information

107. Conclusion

108. Contributions
• Contributions are situated in
– Context-aware computing
– Social computing
• Several approaches and algorithms to support
`aware interaction’
• We have developed number of context-aware
social computing systems
• We have evaluated the systems
• We have studied dynamic social network systems

109. Context-aware Social Computing Systems

110. ‘Aware’ Interaction

111. Future Research Directions
• An evaluation framework for context-aware
and social computing system
• Emergency response
• Assistive Technology

112. Thank you http://research.edm.uhasselt.be/~nmahmud/