Recent works have shown that search performance and results could be improved with collaboration among searchers. Most of the previous approaches surrounding collaborative information retrieval (CIR) provide either a user-based mediation, in which the system only supports users' collaborative activities, or a system-based mediation, in which the system plays an active part in balancing user roles, reranking results, and distributing them to optimize overall retrieval performance. In this paper, we propose to combine both of these approaches by a role mining methodology that learns from users' actions about the retrieval strategy they adapt. This hybrid method aims at showing how users are different and how to use these differences for suggesting roles. The core of the method is expressed as an algorithm that (1) monitors actions of users in a CIR setting; (2) discovers differences among the collaborators along certain dimensions; and (3) suggests appropriate roles to make the most out of individual skills and optimize overall IR performance. Our approach is empirically evaluated and relies on two diifferent laboratory studies involving 140 users in 70 pairs. Our experiments show promising results that highlight how role mining could optimize the collaboration within a search session. The contributions of this work include a new algorithm for mining user roles in collaborative IR, an evaluation methodology, and a new approach to improve IR performance with the operationalization of user-driven system-mediated collaboration. http://dl.acm.org/citation.cfm?doid=2600428.2609598

1. User-­‐Driven
System-­‐Mediated
Collabora6ve
Informa6on
Retrieval
Laure
Soulier
(UPS-­‐IRIT,
Toulouse
FR)
Chirag
Shah
(SC&I-­‐Rutgers,
New
Brunswick
USA)
Lynda
Tamine
(UPS-­‐IRIT,
Toulouse
FR)

2. • Collabora6on
in
Informa6on
Retrieval
• Related
Work
• Research
Ques6ons
• Role
Mining
Methodology
• Experimental
Evalua6on
• Conclusion
and
Perspec6ves
Overview
2
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion

3. 1.
Collabora6on
in
Informa6on
Retrieval
Collabora6ve
informa6on
retrieval
system
Shared
informa6on
need
.
.
.
Division
of
labor
Sharing
of
knowledge
Awareness
Level
of
media6on
How
to
leverage
collaborators’
complementary
skills
for
enhancing
collabora6ve
informa6on
retrieval?
3
Collabora8on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
Complementary
skills

4. User-­‐driven-­‐based
media.on
• Collaborators
fully
ac6ve
• Collabora6on
supported
by:
! Devices
[Morris
et
al.,
2006]
! Tools
[Shah
and
Gonzalez-­‐Ibanez,
2011]
• Collaborators
par6ally
ac6ve
• Collabora6on
supported
by:
!
Algorithmic
media6on
[Foley
and
Smeaton,
2009;
Morris
et
al.,
2007]
! Role-­‐based
algorithmic
media6on
Prospector/Miner
[Pickens
et
al.,
2008]
Gatherer/Surveyor
[Shah
et
al.,
2010]
Group
of
experts
[Soulier
et
al.,
2013]
Domain
expert/novice
[Soulier
et
al.,
2014]
4
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
2.1
User-­‐Driven
vs.
System-­‐Mediated
Collabora6ve
IR
2.
Related
Work
System-­‐based
media.on

5. • Using
interac6ons
from
social
networks
• PageRank
[Kwak
et
al.,
2010]
• Clustering
[Pal
and
Counts,
2011]
• Gibbs
sampling
[Nowicki
and
Snijders,
2001]
• Mul6-­‐dimensional
matrix
transforma6on
[Henderson
et
al.,
2012]
• Using
similari6es
and
dissimilari6es
among
users’
interac6ons
[McCallum
et
al.,
2007]
• Author-­‐Recipient-­‐Topic
model
(ART)
• Role-­‐Author-­‐Recepient-­‐Topic
model
(RART)
5
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
2.2
Role
Mining
2.
Related
Work

6. 3.
Research
Ques6ons
6
1.
How
different
collaborators
are?
2.
How
do
we
infer
users’
roles
?
3.
How
to
use
these
roles
to
improve
CIR?
Collabora6on
in
IR
|
Related
Work
|
Research
Ques8ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
Role
Mining
Role
Mining
**
**
Shared
informa.on
need
…
…
Reader
Querier
Annotated
document
q2
q4
*
**
***
*
**
***
Annotated
document
q1
q3
*
***
*
***
Bookmarked
document
**
**
Annotated
document
q6
*
***
*
***
Bookmarked
document
q5
Expert
Novice
Annotated
document
q7
*
**
***
User-­‐driven
media6on
System-­‐based
media6on

7. Role
pacern
:
-­‐ Search
feature
kernel
-­‐ Search
feature-­‐based
correla6on
matrix
-­‐ Role
acribu6on
func6on
• Role
pacern
4.
Role
Mining
Methodology
7
Number
of
visited
documents
Number
of
submiced
queries
Nega6ve
correla6on
PR1,2
KR1,2
= { fk ∈ F}
FR1,2
where FR1,2
( fj , fk ) =
+1for positive correlation
0 for independence
−1for negative correlation
Role(u1 ,u2, RR1,2
)
Reader
Querier
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
4.2
Basic
No6ons

8.
2.
Temporal-­‐based
representa6on
1.
Search
feature-­‐based
representa6on
• Collaborators’
search
behaviors
4.
Role
Mining
Methodology
8
Bookmarked
document
…
…
Annotated
document
Annotated
document
q2
q1
q3
q4
Annotated
document
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
Role
mining
Shared
informa.on
need
4.2
Basic
No6ons
Su1
(t)
= (wu1, f1
(t)
,...,wu1, fn
(t)
)
Su2
(t)
= (wu2, f1
(t)
,...,wu2, fn
(t)
)
• Avoiding
noisy
search
ac6ons
• Behaviors
change
Su1
(t)
=
wu1, f1
(1)
... wu1, fn
(1)
... ... ...
wu1, f1
(t)
... wu1, fn
(t)
!
"
#
#
#
#
$
%
&
&
&
&
Su2
(t)
=
wu2, f1
(1)
... wu2, fn
(1)
... ... ...
wu2, f1
(t)
... wu2, fn
(t)
!
"
#
#
#
#
$
%
&
&
&
&

9. 4.
Role
Mining
Methodology
9
f1
f2
f3
f4
f1
f2
Δf3
f4
*Δf1
Δf2
*Δf3
*Δf4
Difference
significance
test
(Kolmogorov-­‐Smirnov)
Δf1
f3
Δf4
Δf3
Δf1
Δf4
1
0.3
-­‐0.5
0.3
1
-­‐0.8
-­‐0.5
-­‐0.8
1
Reader/Querier
Expert/Novice
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
No
role
4.3
Methodology
Step
1:
Iden6fying
search
behavior
differences
Step
2:
Characterizing
users’
roles
Correla6ons
on
search
behavior
differences
for:
• Highligh6ng
search
skill
opposi6ons
• Iden6fying
in
which
each
collaborator
is
the
most
effec6ve
• Avoiding
prior
assignments
of
any
roles
to
users

10. • pool
of
role
pacerns
described
by
a
feature
correla6on
matrix
• correla6on
matrix
of
search
feature
differences
4.
Role
Mining
Methodology
10
PR1,2
FR1,2
argminR1,2 || FR1,2
¬Cu1,u2 ||
subject to
∀( fj , fk ) ∈ KR1,2
;FR1,2
( fj, fk )−Cu1,u2 ( fj, fk ) > −1
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion
Cu1,u2
Role(u1 ,u2, RR1,2
)
Collabora6ve
ranking
model
4.3
Methodology
Δf3
Δf1
Δf4
Δf3
Δf1
Δf4
1
0.3
-­‐0.5
0.3
1
-­‐0.8
-­‐0.5
-­‐0.8
1
Reader/Querier
Expert/Novice
No
role
Step
3:
Iden6fying
users’
roles

11.
2
user-­‐driven
lab
studies
• 60
vs.
10
paid
par6cipants
• Exploratory
search
task
• Between
25
vs.
30
minutes
11
Category! Description! Measurement!
Query-based
features!
Number of queries! Number of submitted queries!
Query length! Average number of tokens within queries!
Query success! Average ratio of successful pages over queries!
Query overlap! Average ratio of shared word number among successive queries!
Page-based features!
!
Number of pages! Number of visited pages!
Number of pages by query! Average number of visited pages by query!
Page dwell time! Average time spent between two visited pages!
Snippet-based
features!
Number of snippets! Number of snippets!
Number of snippets by query! Average number of snippets by submitted query!
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
5.1
Protocol
Design
5.
Experimental
Evalua6on
Document
dataset
(74,844
docs)
• Visited
web
pages
• Top
100
results
from
submiced
queries
Search
features

12. Role
pacerns
•
Gatherer
/
Surveyor
• Prospector
/
Miner
Query
overlap
vs.
Query
success
Query
overlap
vs.
Dwell-­‐6me
Gatherer:
look
for
highly
relevant
documents
Surveyor:
quickly
scan
result
for
diversity
Query
overlap
vs.
Number
of
submiced
queries
Prospector:
formulate
query
for
diversity
Miner:
look
for
relevant
document
12
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
5.1
Protocol
Design
5.
Experimental
Evalua6on

13. • Baselines
• BM25-­‐CIR:
user-­‐driven
session
on
independent
search
engines
• GS-­‐CIR:
system-­‐mediated
session
with
Gatherer/Surveyor
• PM-­‐CIR:
system-­‐mediated
session
with
Prospector/Miner
• Ra-­‐CIR:
random
version
of
user-­‐driven
system-­‐mediated
session
• Metrics
13
Metric! Formula!
Precision!
Recall!
F-measure! F@R(g) =
1
T(g)
2*Prec@R(g)(t)
*Recall@R(g)(t)
Prec@R(g)(t)
+ Recall@R(g)(t)t∈T (g)
∑
Recall@R(g) =
1
T(g)
RelCov@R(g)(t)
| RelDoc |t∈T (g)
∑
Prec@R(g) =
1
T(g)
RelCov@R(g)(t)
Cov@R(g)(t)t∈T (g)
∑
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
5.1
Protocol
Design
5.
Experimental
Evalua6on

14. 8mestamp
1
2
3
4
5
10
15
20
25
30
US1
QS
AbsDiff
0.53
0.74
0.80
0.83
0.82
1.00
1.02
1.10
1.24
-­‐
P-­‐value
**
**
***
***
***
****
****
****
QN
AbsDiff
1.15
1.33
1.74
1.93
2.33
2.89
3.50
3.98
4.19
-­‐
P-­‐value
*
**
**
***
***
***
***
***
-­‐
QO
AbsDiff
0.30
0.34
0.41
0.42
0.38
0.29
0.30
0.25
0.23
-­‐
P-­‐value
*
*
**
***
***
****
****
****
-­‐
DWP
AbsDiff
52.72
67.16
76.28
87.34
82.10
80.54
72.19
56.76
52.56
-­‐
P-­‐value
**
***
***
***
****
****
****
****
-­‐
US2
QS
AbsDiff
0.33
0.46
0.54
0.69
0.66
0.81
0.98
0.57
0.40
0.10
P-­‐value
**
**
***
***
***
***
***
***
***
QN
AbsDiff
1.20
2.20
2.10
2.50
2.70
3.10
5.80
6.50
7.00
0.50
P-­‐value
*
**
**
**
***
***
****
***
***
QO
AbsDiff
0.27
0.38
0.35
0.31
0.38
0.23
0.22
0.15
0.13
0.14
P-­‐value
*
**
***
***
****
****
****
****
****
DWP
AbsDiff
36.55
34.28
28.11
23.63
17.38
13.43
13.98
13.06
12.55
12.77
P-­‐value
**
**
***
***
***
***
***
***
***
5.2
Result
*:
at
least
one
group;
**:
>50%
of
groups;
***:
>75%
of
groups;
****:
all
the
groups
From
75%
to
100%
of
the
collabora6ve
groups
significantly
different
aper
5
minutes
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
• Differences
between
collaborators
5.
Experimental
Evalua6on
14
At
least
50%
of
groups
At
least
75%
of
groups

15. 0,062
0,064
0,066
0,068
0,07
0,072
0,074
1
2
3
4
5
F-­‐measure
Time
window
0,016
0,0162
0,0164
0,0166
0,0168
0,017
1
2
3
4
5
F-­‐measure
Time
window
US1
US2
***
15
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
• Impact
of
6mewindow
on
the
retrieval
effec6veness
5.2
Result
5.
Experimental
Evalua6on

16. …
…
none
none
PM
PM
GS
PM
GS
GS
none
• Normalized
number
of
role
couples
over
the
session:
!
Search
behaviors
of
the
pairs
stable
US1
US2
• Normalized
number
of
dis6nct
individual
roles
over
the
session:
! Search
behaviors
of
collaborators
stable
Prospector
Prospector
Prospector
Gatherer
Surveyor
Surveyor
Surveyor
Gatherer
Gatherer
Miner
Miner
• Couple-­‐role
ra6o
over
the
session:
!
US1:
global
search
behaviors
more
variant
than
individual
ones
!
US2:
individual
search
behaviors
more
variant
than
global
ones
16
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
0.13
0.09
<1
0.03
5.2
Result
5.
Experimental
Evalua6on
• Analysis
of
mined
roles
Shared
informa.on
need
0.08
0.09
>1
-­‐0.03
• Correla6on
between
couple-­‐role
ra6o
and
F-­‐measure
! The
mining
quality
>>
the
quan6ty
of
dis6nct
mined
roles

17. 17
Prec@20! Recall@20! F@20!
value! %Chg! p! value! %Chg! p! value! %Chg! p!
US1!
BM25-CIR! 0.041! +10.408! *! 0.010! +4.636! *! 0.016! +5.372!
GS-CIR! 0.038! +18.316! ***! 0.008! +25.205! ***! 0.014! +24.521! ***!
PM-CIR! 0.05! -9.482! 0.012! -13.991! 0.019! -13.397!
Ra-CIR! 0.041! +11.484! *! 0.009! +12.895! *! 0.015! +12.777! *!
RB-CIR
0.045
-
0.010
-
0.017
-
US2!
BM25-CIR! 0.075! +3.347! 0.063! +2.586! 0.069! +2.833!
GS-CIR! 0.058! +34.636! 0.040! +63.818! *! 0.046! +52.786! *!
PM-CIR! 0.092! -16.051! 0.078! -16.493! 0.084! -16.317!
Ra-CIR! 0.070! +10.714! 0.056! +16.201! 0.062! +14.324!
RB-CIR
0.077
-
0.065
-
0.071
-
• Individual
scenarios
• Collabora6ve
serng
GS-­‐CIR
with
fixed
predefined
roles
• Collabora6ve
serng
Ra-­‐CIR
with
randomly
assigned
predefined
roles
• Collabora6ve
serngs
PM-­‐CIR
relying
on
users’
ac6ons
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
• Retrieval
effec6veness
at
the
session
level
5.2
Result
5.
Experimental
Evalua6on

18. • Generally
overpass
baseline
curves
–
except
the
PM-­‐CIR
• Seems
to
decrease
at
the
end
of
the
session
(quite
long
session)
18
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua8on
|
Conclusion
• Retrieval
effec6veness
at
the
itera6on
level
5.2
Result
5.
Experimental
Evalua6on
US1
US2

19. • Role
mining
methodology
relying
on:
– Collaborators
search
differently
over
the
session
– Collabora6on
benefits
from
leveraging
complementary
skills
of
collaborators
• Role
mining
methodology
which
suggests
the
evolving
roles
of
collaborators
• An
experimental
evalua6on
on
two
user
studies
with
promising
results:
– Synergic
effect
of
role
mining
(2
>
1+1)
– CIR
model
based
on
fixed
and
predefined
roles
not
always
as
effec6ve
– Randomly
assigning
fixed
roles
is
less
effec6ve
• Considering
knowledge
or
preferences
of
collaborators
• Larger
groups
of
collabora6on
Conclusion
Perspec6ves
19
Collabora6on
in
IR
|
Related
Work
|
Research
Ques6ons
|
Role
Mining
Methodology|
Evalua6on
|
Conclusion

20. Thank
you
for
your
acen6on!
Laure
Soulier
UPS-­‐IRIT
Toulouse,
France
@LaureSoulier
Chirag
Shah
SC&I-­‐Rutgers,
New
Brunswick,
USA
@chirag_shah
Lynda
Tamine
UPS-­‐IRIT
Toulouse,
France
@LyndaTamine
20
Authors
received
the
SIGIR
Travel
Grant.
Many
thanks!