Prior work on user engagement with online news sites identified dwell time as a key engagement metric. Whereas on average, dwell time gives a reasonable estimate of user engagement with a news article, it does not capture user engagement with the news article at sub-document level nor it allows to measure the proportion of article read by the user. In this paper, we analyze online news reading patterns using large-scale viewport data collected from 267,210 page views on 1,971 news articles on a major online news website. We propose four engagement metrics that, unlike dwell time, more accurately reflect how users engage with and attend to the news content. The four metrics capture different levels of engagement, ranging from bounce to complete, providing clear and interpretable characterizations of user engagement with online news. Furthermore, we develop a probabilistic model that combines both an article textual content and level of user engagement information in a joint model. In our experiments we show that our model, called TUNE, is able to predict future level of user engagement based on textual content alone and outperform currently available methods. Slides of our paper presented at The 9th ACM International Conference on Web Search and Data Mining (WSDM 2016), San Francisco, California, USA. February 22-25, 2016.

1. Understanding and Measuring User
Engagement and Attention
in Online News Reading
Dmitry Lagun and Mounia Lalmas
1Thanks to Yahoo Faculty Research and Engagement Program for supporting this work.

2. User Engagement in Online News Reading
2
User engagement:
“emotional, cognitive and behavioral
connection that exists between a user
and a resource” (Attfield et al., 2011)
Stickiness:
concerned with users spending time on a
news site.
User Attention in Online
News Reading
Challenge II:
identifying which aspects of the online interaction
influence user engagement the most.
Challenge I:
attract large shares of online attention by keeping
users engaged.

3. Measuring user engagement with news content
3
Method PROS CONS
Dwell time (click duration)
(Agichtein et al., 2006)
scalable; captures
engagement at coarse level
cannot distinguish time spent
on parts of the article
Eye tracking
(Arapakis et al., 2014)
very detailed small scale; very expensive
Mouse cursor movement
(Huang et al., 2011)
scalable; more fine grained
than dwell time
cursor is often kept still
during article reading, when
no pointing action is required
coarse but more robust instrument to measure user
attention at large scale during news reading
VIEWPORT TRACKING

4. Our Method: Viewport Tracking
4
viewport
time spent at i-th scroll
position
i-th viewport defined by a
rectangle (left, top, width, height)
viewport

5. Research questions
● Where do users spend their time during news article viewing?
● Does media image and video content affect time spent at a vertical
position?
● What are typical patterns of news article reading?
● Can we accurately predict user engagement from textual content?
5
● 1,971 Yahoo news articles
● 267,210 page views on desktopDATASET

6. 6
Overall Pattern of Viewport Time
(proxy for user attention)
Many users spend significantly
smaller amount of time at lower
scroll positions.
Some users find the article
interesting enough to spend
significant amount of time at the
lower part of the article.
Some articles entice users to
deeply engage with their content.

7. Image and Video do matter … for the first screen
7
Video Image

8. How do users browse through the article?
8
comment
header
top
middle
bottom
articlebody
start
top
middle
bottom
comment
leave
Markov States
beginning of a page view
top area occupies most of the viewport
middle area occupies most of the viewport
bottom area occupies most of the viewport
comment area occupies most of the viewport
user leaves the page
V1 V2 ... Vn

9. Mixture of Markov Chains Model
9
Single markov model:
Mixture of K markov models:
probability of starting at state v1
probability of transition from state Vi to V(i-1)
Markov States:
{Start, Top, Middle, Bottom,
Comment, Leave}
weight of k-th mixture
component
K=6 is optimal

10. Patterns of Attention in News Reading
10
Engagement Depth
most probable sequence

11. Engagement depth: Four User Engagement Classes
11
EngagementDepth

12. 12
Engagement depth: Four User Engagement Classes
EngagementDepth

13. 13
Engagement depth: Four User Engagement Classes
EngagementDepth

14. 14
Engagement depth: Four User Engagement Classes
EngagementDepth

15. Distribution of Attention is Different across Engagement Classes
15

16. Modeling of User Engagement from Article Content
16
?
news article
%Bounce
%Shallow
%Deep
%Complete
user engagement profile

17. TUNE: Topics of User Engagement with News
17
TUNE
news article
%Bounce
%Shallow
%Deep
%Complete
user engagement profile
Unlike LDA, in TUNE topic is a combination of
word co-occurrence and similarity of user
engagement profile.
Distribution of user engagement level

18. Experimental Setting
● Task
○ Predict User Engagement Level Profile
● Model
○ Linear regression
● Features
○ Number of words in the article
○ Presence of media content (e.g., image and video)
○ Distribution of article topics with LDA
○ Distribution of article topics with TUNE (our model)
● Evaluation Metric
○ Pearson’s correlation between ground truth and predicted value
○ Ten fold cross-validation
18
%Bounce
%Shallow
%Deep
%Complete

19. Results: Baselines
19
Feature Set %Bounce %Shallow %Deep %Complete
NumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0.071 0.571 0.410 0.185

20. Results: Baselines
20
Feature Set %Bounce %Shallow %Deep %Complete
NumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0.071 0.571 0.410 0.185
NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328
NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379
NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402
NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405

21. Results: Baselines vs. TUNE
21
Feature Set %Bounce %Shallow %Deep %Complete
NumWords 0.063 0.494 0.370 0.017
NumWords + Media (M) 0.071 0.571 0.410 0.185
NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328
NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379
NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402
NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405
NumWords + M + TUNE (T=5) 0.079 0.648 0.544 0.282
NumWords + M + TUNE (T=10) 0.311 0.713 0.660 0.400
NumWords + M + TUNE (T=20) 0.349 0.724 0.682 0.409
NumWords + M + TUNE (T=50)
0.333
(+132%)
0.742
(+18%)
0.697
(+29%)
0.428
(+6%)
NumWords + M + LDA + TUNE 0.334 0.730 0.696 0.442
Dwell 0.392 0.203 0.128 0.351

22. Conclusions
● Unlike in search, user attention in news reading is not constantly decaying
with vertical position (e.g., can be bi-modal)
● Engagement with a news article can be categorized by depth of examination
(Bounce, Shallow, Deep & Complete)
● The proposed engagement metrics go beyond “dwell time” as they capture
user attention and engagement at sub-document level
● We can obtain accurate prediction of article engagement profile purely from
its textual content
22

23. Summary: Viewport time attention as proxy of user engagement
23
Effect of position and content on viewport time at
vertical position
V1 V2 ... Vn
Article examination
can be categorized by
depth of examination
Four engagement classes:
Bounce, Shallow, Deep and Complete
Joint model of article topics and
user engagement classes
improves prediction accuracy:
● Bounce (+140%)
● Shallow (+18%)
● Deep (29%)
● Complete (+9%)

24. Appendix
24

25. User Attention vs. Engagement Classes
25
Metric
Bounce
(N=26542)
Shallow
(N=63982)
Deep
(N=164197)
Complete
(N=12489)
dwell 6.17 (0.02) 63.75 (0.37) 99.02 (0.22) 228.35 (1.48)
header time 2.99 (0.03) 15.39 (0.14) 18.48 (0.08) 17.41 (0.25)
body time 5.06 (0.02) 35.13 (0.21) 86.24 (0.20) 85.00 (0.70)
comment time 0.56 (0.01) 17.27 (0.23) 9.72 (0.07) 110.90 (0.89)
% header time 0.31 (0.00) 0.23 (0.00) 0.17 (0.00) 0.09 (0.00)
% body time 0.62 (0.00) 0.58 (0.00) 0.76 (0.00) 0.40 (0.00)
% comment time 0.07 (0.00) 0.20 (0.00) 0.07 (0.00) 0.51 (0.00)
% article read 0.12 (0.00) 0.23 (0.00) 0.83 (0.00) 0.84 (0.00)
# comment clicks 0.01 (0.00) 0.43 (0.01) 0.00 (0.00 3.14 (0.03)

26. User Engagement Classes and
User Attention
26
Dwell time and viewport time on head, body and comment increase from
Bounce to Complete.
Viewport time on head steadily decreases from Bounce to Complete:
users spend an increasing amount of time reading content deeper in
article.
Percentage of article read steadily increases from Bounce to Complete, as
expected.
Deep and Complete correspond to the situations when the majority (83%)
of the article was read.
Number of comment clicks is highest for Complete and then Shallow:
users may engage with comments even if they do not read a large
proportion of the article.
comment
header
top
middle
bottom
articlebody
header