Women of Silicon Valley 2017

1. Defining true north metrics to quantify
engagement at LinkedIn
BONNIE BARRILLEAUX
STAFF DATA SCIENTIST
CONTENT EXPERIENCE ANALYTICS

2. Outline
Analytics at LinkedIn
Member value metrics
Building a suite of true north and signpost metrics
Example: content experience ecosystem
Measuring throughout the funnel
Complications and tradeoffs to consider

3. Analytics data scientists partner with product teams
• Embed in cross-functional
product teams
• Ensure that our entire product
lifecycle is data-informed
• Create the metrics that we use
to evaluate success

4. Metrics power our world throughout the product life cycle
Are we meeting our
targets?
Performance
management
Where should we
invest resources?
Formulate
strategy
Is this new feature
successful?
Evaluate
experiments
What’s broken?
Investigate site
issues

5. Our goal is to measure “member value”
• Consider the value propositions that bring our
members to LinkedIn:
• Getting a job
• Staying informed
• Building a reputation
• Delivering value today leads to retention and
revenue in the future

6. Properties of an ideal metric
• Metric code lives in a single
source of truth
• Measures something we really
care about
• Simple, actionable definition
• Fast calculation
• Minimal lag time
• Accurate
• Dimensional cuts are available

7. There is no such thing as a perfect metric.

8. The ideal metric’s properties are often in conflict
We want to measure
results today
Short lag time
We care about long-term
changes in user behavior
Measures the thing
we care about

9. Support your true north with a suite of signposts
• Ensure you’re headed in the
right direction
• Shed light on your true north’s
weaknesses
• Create a balanced portfolio to
get a holistic view

10. Metrics for Wikipedia reading experience
• The thing we really want to drive
• Takes a long time to change visitation
behavior
• Difficult for any small initiative to show
significant change
Daily active users
Time spent on site,
Articles read,
Article completion rate
• Generally correlated with the true north
• Faster to measure
• Easier to move
T R U E N O R T H
S I G N P O S T S

11. LinkedIn Content Ecosystem

12. LinkedIn Content Ecosystem
​ MEMBER VALUE PROPOSITIONS
Content Creators
Content Consumers
• Share professional perspectives
• Build a reputation
• Be part of a community
Authors
Readers
• Stay informed about your professional world
• Stay connected with your network

13. From Earth or space
Article authoring
CONTENT ECOSYSTEM

14. Post status updates
CONTENT ECOSYSTEM

15. Start a conversation
CONTENT ECOSYSTEM

16. Editorial curation
CONTENT ECOSYSTEM

17. Content creation
metric development
Members tell us they want to:
• Share professional perspectives
• Build their reputation
• Be part of a community

18. Visit the sharing tool
Write a status update
Post the update
Update gets viewed
Update receives engagement
Creator feels rewarded
Where does the creator receive value?

19. Visit the sharing tool
Write a status update
Post the update
Update gets viewed
Update receives engagement
Creator feels rewarded
Measure throughout the user flow

20. • Metrics near the top of funnel
are often easier to move
• The user may receive more
value near the bottom
• Measure multiple points for a
holistic view
Measure
throughout
the funnel

21. What metric should we monitor for posting updates?
L E T ’ S C O N S I D E R T H E D E T A I L S

22. Which metric should we prefer:
​ POSTS OR UNIQUE POSTERS?
0
5
10
1
2
3
4
5
6
7
8
9
10
MEMBERS
POSTS PER DAY
*Simulated data

23. Which metric should we prefer:
​ POSTS OR UNIQUE POSTERS?
0
5
10
1
2
3
4
5
6
7
8
9
10
MEMBERS
POSTS PER DAY
*Simulated data

24. Long-tailed distributions are everywhere
Annual household income, US ($)
Fracture length in volcanoes (meters)
Length of movie shots (seconds)
Web site visitor engagement (clicks)
US Census Bureau via Wikipedia.org, 2010 L Sonnette et al, J Structural Geology 2010
B. Salt, cinemetrics.lv, 2010 K. Ali and M. Scarr, WWW 2007

25. Yes, it matters!
​ ENGAGED USERS CONTRIBUTE A DISPROPORTIONATE SHARE OF POSTS
50%
Engaged user
90%
Casual users
​ 11 POSTERS
​ 20 POSTS
*Simulated data

26. Now it’s time to inject business logic
​ WHO SHOULD WE PRIORITIZE?
Ten members posting
once a day
Casual users
One member posting ten
times a day
Heavy users

27. • Start with the product vision
• Who are the key users right
now?
Have an opinion

28. Spam: taking out the trash
A D D I N G C O M P L E X I T Y

29. Spam: violates our TOS
• Scam artists
• Inappropriate content
• Abusive language
Spam
detection
Algorithmic
User
reported

30. Let’s exclude spam from the metric
​ WHAT HAPPENS?
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Posts
2016
*Simulated data

31. Let’s exclude spam from the metric
​ WHAT HAPPENS?
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Posts
2016
*Simulated data

32. Let’s exclude spam from the metric
​ WHAT HAPPENS?
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Posts
2016
New improved spam
detection algorithm
rolled out
*Simulated data

33. • Some complexity is worth
adding, if the benefits outweigh
the burdens
• Remember: sometimes it’s good
when the metric decreases
Balance the
complexity
tradeoffs

34. “This magic model will solve all our problems”
T U R N Y O U R S K E P T I C I S M U P T O 1 1

35. We could create a “posting health” model
Posting health
Spam
features
Poster
features
Post
features

36. • Black boxes are dangerous
• Prefer simple definitions
• Educate metric users
Keep it
actionable

37. Takeaways
• Have an opinion
Which users are most important right now?
• Add complexity sparingly
Is the information worth the complication?
• Keep it actionable
If you can’t understand it, you can’t use it.
• Metrics are a tool for humans.
We use them to make decisions.
• Start with member value
Why do people want to use your product?
• Measure throughout the funnel
Support your true north with signposts.

38. Thank you