Understanding the Impact of the Role Factor in Collaborative Information Retrieval

Understanding the
Impact of the Role
Factor in CIR
Lynda Tamine and Laure Soulier
1

Collaborative
search
in context
333

Collaborative Search
Complex, exploratory or fact-finding tasks
Bibliographic, medical, e-Discovery, academic search…
What?
Requirement or setup need Shared interests
Insufficient knowledge Division of labor
Why?
Groups vs. Communities
Who?
Synchronous vs. Asynchronous
When?
Colocated vs. Remote
Where?
Crowd-sourcing User mediation
Implicit vs. Explicit intent System mediation
How?

Collaborative Search Models
Collaborative Search | Objectives | Design | Results | Implications | Conclusion
Users' activities in social search [Evans and Chi, 2010]
AFTERBEFOREDURING
● Refining the information need
● Structuring the task guidelines
● Sense-making process :
 Querying
 Reading
 Extracting
 Exchanging...
Organizing/Distributing search outcomes
Assessing collective relevance
Collaborative search practices [Morris, 2008]
Communication channels
[Gonzalez-Ibanez et al., 2013]
Other behavioral models [Hyldegärd, 2009 ; Karunakaran et al., 2013 ; Shah, 2012]
5

Collaborative Search Paradigms
Sharing of knowledge
[Foley and Smeaton, 2009]
Division of labor
[Kelly and Payne, 2013]
Awareness
[Dourish and Bellotti, 1992]
Role-based division of labor
Document-based division of labor
Communication and shared workspace
Ranking-based on relevance judgments
Collaborator's actions
Spatio-temporal context
Shared
information
need
Doc2 Yes, good
Doc2 Yes, good
6

Role-based Collaborative Search
Description
Naturalness Collaboration mediation Flexibility level
Exchange Interfaces
Algorithmic
approaches
Role Algorithmic
User-driven
collaboration
Fixed roles [Imazu et al., 2011] = ++ + -- - -
Freely-negociated roles [Morris
et al., 2008] ++ ++ + -- + -
System-mediated
collaboration
Fixed roles leveraged by
algorithmic mediation [Pickens
et al., 2008, Shah et al., 2010]
- = + ++ - +
User-driven system-
mediated collaboration
Fixed roles :
- freely negotiated
- dynamically mined
- leveraged by algorithmic
mediation
[Soulier et al., 2014]
+ = + ++ + +
Role taxonomy [Golovchinsky et al., 2009]
● Prospector/Miner
● Gatherer/Surveyor
● Domain A expert/Domain B expert
● Search expert/Search novice
→ Analyzing the impact of the role factor in collaborative search
7

Research Questions and Hypothesis
Understanding differences in users' behaviors in role-oriented and non-role-
oriented collaborative search sessions:
●
RQ1 : How do user search behaviors differ ?
●
RQ2 : If any, does the coordination between collaborators differ ?
●
RQ3 : Do users with assigned roles respect the guidelines? If no, why ?
●
RQ4 : Which effect on the search effectiveness?
Research Hypothesis:
• Behaviors of collaborators might be complementary w.r.t. division of labor policies
[Spence et al., 1995]
• Differences in collaborators' behaviors represent complementary signals
[Soulier et al., 2014]

Participants
75 pairs of users (undergrad and Ph.D. students between 18 and 30)
●
Already know each other
●
Already worked on collaborative project together
●
Experience in browsing the web and using search engines
Compensation
●
Material compensation ($20)
●
Additional gift ($50) for three most effective groups

System
Open-source Coagmento collaborative search system

Tasks
Exploratory search task
The mayor of your countryside village must choose between building a huge industrial complex or developing a
nature reserve for animal conservation. As forest preservationists, you must raise awareness about the possibility of
wildlife extinction surrounding such an industrial complex. Yet, before warning all citizens, including the mayor, you
must do extensive research and collect all the facts about the matter. Your objective is to create a claim report together,
outlining all the possible outcomes for wildlife should the industrial complex be built. Your focus is on wildlife
extinction. You must investigate the animal species involved, the efforts done by other countries and the
association worldwide to protect them and the reasons we, as humans, must protect our environment in order to
survive. You must identify all relevant documents, facts, and pieces of information by using bookmarks, annotations, or
saving snippets. If one document discusses several pieces of useful information, you must save each piece separately
using snippets. Please assume that this research task is preliminary to your writing, enabling you to provide all relevant
information to support your claims in your report.
3 settings associated to 3 TREC topics:
●
W/oRole (any role guidelines) – topic 408i (tropical storm)
●
PM (prospector/miner) – topic 392i (robotics)
●
GS (gatherer/surveyor) – topic 347i (wildfire extinction)

User Study Workflow and Data
Step 1 : Sign-up questionnaire
●
Users' characteristics
●
Users' habits in search browsing (search level and role assignment)
●
Users' habits in collaborative work
Step 2 : The training step
●
System tutorial
●
Short time-period to test the system
Step 3 : The 30-minute search task
●
Real task in which search logs where collected
Step 4 : Post-task questionnaire
●
Opinion about CIS
●
Difficulty level of the task
npq Average number of visited pages by query
dt Average time spent between two visited pages
nf Average number of relevance feedback (snippets, annotations &
bookmarks)
qn Average number of submitted queries
ql Average number of query tokens
qo Average number of shared tokens among successive queries
nbm Average number of exchanged messages
14

Users’ Behavioral Differences
Modeling collaborators’ search behaviors [Soulier et al., 2014]
Bookmarked
document
…
…
Annotated
document
Annotated
document
q2
q1
q3
q4
Snipped
document
Shared
information
need
●
Avoiding noisy
search actions
●
Behaviors change
Search feature-based representation Temporal-based representation
Su1
(t)
 (wu1, f1
(t)
,...,wu1, fn
(t)
)
Su2
(t)
 (wu2, f1
(t)
,...,wu2, fn
(t)
) Su1
(t)

wu1, f1
(1)
... wu1, fn
(1)
... ... ...
wu1, f1
(t)
... wu1, fn
(t)












Su2
(t)

wu2, f1
(1)
... wu2, fn
(1)
... ... ...
wu2, f1
(t)
... wu2, fn
(t)












16

Users with assigned roles significantly behave differently
Differences between W/Role and W/oRole settings
●
W/oRole spent longer time on pages and submit more queries
●
W/Role annotated/bookmarked/snipped more
Query overlap explained by the role guidelines
Communication seems to be similarly distributed among scenarios
Objective: Analyzing the impact of the role factor on users' behavior
Slow and exhaustive
evaluation style common
among less (skill-based)
experienced users
17

Mostly coordinative and content messages
●
Coordination more important for W/oRole
●
Content more important for W/Role
●
Users in PM scenario need to coordinate more than those in GS one
Objective: Analyzing the impact of the role factor on communication channels
- Bridging the gap
between individual and
collaborative perception
of the task
- Role negotiation for
W/oRole
Some roles might lead to
ambiguity
18

Users’ Division of Labor Strategies
f1
f2
f3
f4
f1
f2
Δf3
f4
*Δf1
Δf2
*Δf3
*Δf4
Difference significance test
(Kolmogorov-Smirnov)
Δf1
f3
Δf4
Δf3
Δf1
Δf4
1 0.3 -0.5
0.3 1 -0.8
-0.5 -0.8 1
✔
Step 1: Identifying search behavior differences ✔
Step 2: Characterizing users’ roles
Correlations on search behavior differences for:
- Highlighting search skill oppositions
- Identifying in which each collaborator is the most
effective
Objective: Analyzing the impact of the role factor on users' search strategies
Investigating and contrasting complementarities of intra-group users' behavior [Soulier et al., 2014]
●
Differences between users' behaviors
●
Negative correlations between users' differences
19

Users' division of labor strategies
Prospector-Miner Gatherer-Surveyor Without Role
Objective: Analyzing the impact of the role factor on users' search strategies
Higher number of significant correlated pairwise features in W/oRole
Exhibiting significant differences and reach convergence
●
PM: 2 of the 4 pairwise features varied between negative/null correlations
●
GS: after 20-25 min
●
W/oRole : after 15 min
Roles seem weakened the
interdependence between
users:
- no coordination
stability/role drift for PM
- encounter cohesion for GS
Higer flexibility for W/oRole
20

Users' division of labor strategies
Task generally perceived as difficult
A trend for easy in the W/oRole scenario, not assessed by the Chi-Square test
Task difficulty mainly related to page content, organization, and role
●
Users in W/oRole feel that organization is more difficult
●
Role guidelines might be difficult to follow
Objective: Analyzing the impact of the role factor on search task perception
21

Search Effectiveness
Formula
[Shah and Gonzalez-Ibanez, 2011]
Precision
Recall
F-measure
Objective: Analyzing the impact of the role factor on the search task effectiveness
P(g)=
RelevantCoverage(g)
Coverage(g)
R(g)=
RelevantCoverage(g)
GroundTruth
F(g)=
2⋅P(g)⋅R(g)
P(g)⋅R(g)
Effectiveness higher for W/oRole only for the precision measure
●
More likely to spend more time on pages
●
Less involved in making assessments
●
Less communicative about the document content
But, users in both settings able to identify relevant pages
More successful in
discarding irrelevant
pages
22

Mining Latent Evolving Roles
Findings
●
Users in W/Role settings found difficult to follow their role guidelines
●
Users in W/oRole setting were able to coordinated even though coordination relied
on intensive coordination
Opened perspectives
●
Enhanced the user-driven system-mediated approach [Soulier et al., 2014] by :

Mining users' unalbelled and latent roles in a « just in time fashion »

Reinjecting these unlabelled and latent roles in CIR algorithms

Providing Task-based Role Template
Findings
●
Task coordination messages are important for both settings
Opened perspectives
●
Designing content-based role template adapted to the search task type

Enhancing functional roles (PM, GS, ...)

Reducing costs of organizing division of labor search strategies

Detecting collaborators' search intent and knowledge differences

Refining users' browsing behaviors by distributing information nuggets among
collaborators

Guiding collaborators in a « step-by-step »-oriented search according to their knowledge
25

Mining Latent Evolving Roles
Findings
●
●
Users in W/oRole setting were able to coordinated even though coordination relied
on intensive coordination
Opened perspectives
●
Enhanced the user-driven system-mediated approach [Soulier et al., 2014] by :

Mining users' unalbelled and latent roles in a « just in time fashion »

Reinjecting these unlabelled and latent roles in CIR algorithms

Enhancing Role-based Awareness
Findings
●
●
Roles seems to reduce coordination-based communication costs
+ Fixed roles might be required for some tasks (e-Discovery, medical, ...)
Implications
●
Improving awareness w.r.t. role drift

Identifying indicators associated to each pair of roles

Designing interfaces detecting rôle drift and monitoring users' support in order to better
follow role specifications
27

Conclusion
User-study of collaborative and exploratory search tasks constrained (or not)
by role guidelines
– Users' search behaviors
– Users' division of labor strategies
– Search effectiveness
Quantitative and qualitative analysis highlighting :
– Users without prior roles adopt a slow and exhaustive search
– Users without prio roles were more likely to broaden their complementarities and structure their role
relatively early
– Roles seems to limit the precision of the search results
Discussion about promising perspective in CIR/CIS
Limitations of the work :
– Collaboration in dyads
– Exploratory search tasks

30
See you on Friday!
Leif Azzopardi Jeremy Pickens Tetsuya Sakai Laure Soulier Lynda Tamine

Thank you for your
attention
Do you have a question?
31

Understanding the Impact of the Role Factor in Collaborative Information Retrieval

Recommandé

Recommandé

Contenu connexe

Similaire à Understanding the Impact of the Role Factor in Collaborative Information Retrieval

Similaire à Understanding the Impact of the Role Factor in Collaborative Information Retrieval (20)

Dernier

Dernier (20)

Understanding the Impact of the Role Factor in Collaborative Information Retrieval