Online providers frequently offer a variety of services, ranging from news to mail. These providers endeavour to keep users accessing and interacting with the sites offering these services, that is to engage users by spending time on these sites and returning regularly to them. The standard approach to evaluate engagement with a site is by measuring engagement metrics of each site separately. However, when assessing engagement within a network of sites, it is crucial to take into account the traffic between sites. This paper proposes a methodology for studying networked used engagement. We represent sites (nodes) and user traffic (edges) between them as a network, and apply complex network metrics to study networked user engagement. We demonstrate the value of these metrics when applied to 728 services offered by Yahoo! with a sample of 2M users and a total of 25M online sessions.

1. Networked
user engagement
user engagement optimization workshop - CIKM 2013
Jane%e
Lehmann
In
collabora*on
with:
Mounia
Lalmas,
Ricardo
Baeza-­‐Yates,
Elad
Yom-­‐Tov

2. outline
How
to
measure
user
engagement?
2. Network
metrics
From
site
to
network
engagement.
3. Measuring
networked
user
engagement
Case
study
based
on
the
network
of
Yahoo
sites.
Lights
on
by
JC*+A!
1. Mo%va%on

3. 182-­‐365+1
by
meaganmakes
How
to
measure
User
Engagement?

4. Measuring user engagement
User
engagement
on
Yahoo!
Popularity
Ac%vity
Loyalty
JaneHe
Lehmann
Web
traffic
reports:
Google
Analy%cs,
Alexa.com,
…
(#Users)
(DwellTime)
(AcJveDays)
MoJvaJon
4

5. Measuring user engagement
However,
Yahoo!
has
many
sites:
maps,
flickr,
weather,
news,
shine,
etc.
How
to
measure
engagement
within
a
network
of
sites?
JaneHe
Lehmann
MoJvaJon
5

6. NETWORK OF SITES
Large
online
providers
(AOL,
Google,
Yahoo!,
etc.)
offer
not
one
site,
but
a
network
of
sites
local
maps
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
omg
shine
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
6

7. NETWORK OF SITES
Each
site
is
usually
op%mized
individually,
with
some
effort
to
direct
users
between
them
local
maps
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
omg
shine
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
7

8. NETWORK OF SITES
local
maps
Online
mul%tasking
Users
switch
between
sites
within
one
online
session
(e.g.
emailing,
reading
news,
social
networking,
jobs
search)
mail
search
weather
news
daJng
frontpage
finance
sports
tv
autos
omg
shine
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
8

9. NETWORK OF SITES
local
maps
Measuring
user
engagement
should
account
for
user
traffic
between
sites...
omg
shine
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
9

10. NETWORK OF SITES
local
maps
…
to
detect
missing
connec9ons
omg
shine
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
10

11. NETWORK OF SITES
local
maps
…
to
detect
sites
that
are
used
together
omg
shine
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
11

12. NETWORK OF SITES
local
maps
…
to
detect
isolated
sites
omg
shine
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
12

13. NETWORK OF SITES
local
maps
…
to
detect
important
sites
omg
shine
mail
search
weather
news
daJng
frontpage
finance
jobs
sports
tv
autos
flickr
shopping
health
JaneHe
Lehmann
games
answer
travel
homes
MoJvaJon
groups
13

14. Danboard
by
sⓘndy°
From
site
to
network
engagement
Looking
beyond
one
own
nose!

15. ENGAGEMENT NETWORKS
Provider
network
Nonprovider
service
Traffic
Provider
service
Provider
service
under
study
Our
approach
State
of
the
art
G=(V,
E,
λ)
V:
n<i>
-­‐
Sites,
services,
funcJonaliJes,
etc.
E:
e<i>
-­‐
Traffic
between
internal
nodes
λ(e):
Traffic
volume
(#Users)
JaneHe
Lehmann
n<e>
-­‐
External
sites
Networked
User
Engagement
15

16. Network-LEVEL METRICS
Network
metrics
Size
9
22
30
90
15
TrafficVolume
Def.:
Sum
of
edge
weights
A
high
value
is
a
sign
of
a
high
networked
user
engagement;
users
navigate
oHen
between
the
sites
of
the
network.
45
78
ConnecJvity
Density
Def.:
ProporJon
of
edges
to
maximum
possible
A
high
value
is
a
sign
of
a
high
networked
user
engagement;
users
navigate
between
many
different
sites.
Reciprocity
Def.:
RaJo
of
number
of
edges
in
both
direcJons
A
high
value
is
a
sign
of
a
high
networked
user
engagement;
users
do
not
only
navigate
from
one
site
to
another,
they
also
tend
to
return
to
previously
visited
sites.
JaneHe
Lehmann
Networked
User
Engagement
16

17. Network-LEVEL METRICS
Network
metrics
Modularity
Modularity
[subNWmod]
Def.:
Captures
the
existence
of
modules
based
on
a
random
walk
approach
A
high
modularity
indicates
that
users
visit
many
sites
of
one
subnetwork,
but
hardly
navigate
to
other
subnetworks.
Global
Clustering
Coefficient
[GCC]
Def.:
Captures
the
existence
of
Jghtly
connected
groups
of
nodes
A
high
value
is
a
sign
of
a
high
networked
user
engagement;
users
access
sites
directly,
instead
of
using
front
pages.
Low GCC
High GCC
JaneHe
Lehmann
Networked
User
Engagement
17

18. ENGAGEMENT NETWORKS
Modularity
[subNWmod]
Yahoo!
Network
with
3
countries
low
modularity
è
low
engagement
JaneHe
Lehmann
high
modularity
è
high
engagement
Networked
User
Engagement
18

19. ENGAGEMENT NETWORKS
Modularity
[subNWmod]
Yahoo!
Network
with
3
countries
Country
low
modularity
è
high
engagement
low
modularity
è
low
engagement
JaneHe
Lehmann
high
modularity
è
high
engagement
Networked
User
Engagement
19

20. lem
by
[
embr
]
Case
Study
Measuring
Networked
user
engagement

21. ENGAGEMENT NETWORKS
Interac%on
data
• July
2011
• 2M
user,
25M
sessions
• 728
Yahoo!
sites
(news,
mails,
search,
etc.)
returning
edge
external
Networks
• 12
weighted
directed
networks,
filtered
by
»
»
»
»
Edge
types
Time
User
loyalty
Country
(internal,
internal
+
returning)
(Wednesday,
Sunday)
(causal,
acJve,
VIP)
(5
EU
countries)
• Applying
metrics
on
networks,
results
are
normalized
using
the
z-­‐score
z-­‐score
above
average
Country1
Country2
Country3
Country4
Country5
average
below
average
trafficVolume
JaneHe
Lehmann
Networked
User
Engagement
21

22. Network-LEVEL METRICS
Edge-­‐based
networks
Size
Connectivity
trafficVolume
Edgeint
Edgeint+ext
density
reciprocity
Timewed
Timesun
Modularity
subNWmod
Usercasual
Usernormal
GCC
UserVIP
Leaving
the
network
does
not
necessarily
entail
less
engagement
• Traffic
increases
significant
when
accounJng
for
returning
traffic
[18.26%]
• ConnecJvity
increases
à
User
use
new
navigaJon
paths
• Modularity
increases
à
Users
stay
in
the
same
part
of
the
network
JaneHe
Lehmann
Networked
User
Engagement
22

23. Network-LEVEL METRICS
Time-­‐based
networks
Size
Connectivity
trafficVolume
Edgeint
Edgeint+ext
density
reciprocity
Timewed
Timesun
Modularity
subNWmod
Usercasual
Usernormal
GCC
UserVIP
Naviga%on
is
more
goal-­‐oriented
during
the
week
• TrafficVolume
and
density
are
lower
on
Sunday
à
Less
acJvity
• Reciprocity,
subNW,
and
GCC
are
higher
on
Sunday
à
User
return
more
omen
to
already
visited
sites
and
visit
more
sites
than
usual
(browsing
with
less
specific
goals)
JaneHe
Lehmann
Networked
User
Engagement
23

24. Network-LEVEL METRICS
User-­‐based
networks
Size
Connectivity
trafficVolume
Edgeint
Edgeint+ext
density
reciprocity
Timewed
Timesun
Modularity
subNWmod
Usercasual
Usernormal
GCC
UserVIP
Loyal
users
have
also
a
higher
networked
user
engagement
• TrafficVolume
is
higher
for
VIP
user
à
More
acJvity
• ConnecJvity
and
modularity
is
higher
for
VIP
user
à
Users
are
interested
in
more
sites
• GCC
is
lower
for
causal
users
à
Users
access
sites
using
the
front
pages
JaneHe
Lehmann
Networked
User
Engagement
24

25. Network-LEVEL METRICS
Country-­‐based
networks
Size
trafficVolume
Country1
Connectivity
density
Modularity
reciprocity
Country2
subNWmod
GCC
Country4
Country5
Country3
Networked
user
engagement
differs
between
countries
• Country1:
Highest
trafficVolume
+
GCC,
high
density
+
reciprocity,
lowest
subNWmod
à
high
engagement
• Country5:
Low
trafficVolume,
but
high
density
+
GCC,
low
subNWmod
à
less
popular,
but
high
engagement
JaneHe
Lehmann
Networked
User
Engagement
25

26. CONCLUSION AND FUTURE
WORK
• Network
analysis
enhances
the
understanding
of
user
engagement.
It
allows:
• To
analyze
users
browsing
behavior
on
a
global
scale
• To
compare
networks
of
sites
(e.g.
of
different
countries)
• To
observe
differences
over
Jme
• Leaving
the
network
does
not
entail
less
engagement
• Some
network
metrics
are
more
useful
than
others
Future
work:
• DefiniJon
of
metrics
that
combine
network
traffic
and
engagement
• Comparing
traffic-­‐
and
hyperlink-­‐networks
• Studying
the
effect
of
changes
in
the
network
JaneHe
Lehmann
Networked
User
Engagement
26

27. Janebe
Lehmann
Universitat
Pompeu
Fabra,
Spain
lehmannj@acm.org
Mounia
Lalmas
Yahoo
Labs,
London,
UK
mounia@acm.org
Ricardo
Baeza-­‐Yates
Yahoo
Labs,
Barcelona,
Spain
rbaeza@acm.org
Elad
Yom-­‐Tov
Microsog
Research,
Herzliya,
Israel
elad@ieee.org