Lecture for USC Annenberg School for Communication and Journalism PUBD 526 - Public Diplomacy Evaluation/Professor Bob Banks

1. Using analytics to evaluate
digital communication strategies
Dana Chinn
1. Developing measurement models
2.
2 Basic site metrics
3. Some thoughts about external ratings and metrics
4. Social media metrics
PUBD 526 – Public Diplomacy Evaluation
April 2012

2. Improving digital communication channels start
with identifying what needs to change
Start here
not here
2

3. Our site has 5,000 monthly unique visitors.
Last Tuesday that story got 20,000 page
views.
Our iPhone app was downloaded 10 000 times.
10,000 times
We have 2,000 Likes on Facebook.
We have 5,000 Twitter followers
5 000 followers.
3

4. Questions for a e-commerce company
Who came to our site?
e.g., previous vs. new; high vs. low potential
How did they get here?
What did they look at?
Were they successful in getting what they wanted?
A simple e-commerce data story
“Current and potential customers who typed in “t-shirts”
in Google arrived on our t-shirts landing page.
1.5% of them made a purchase.”
4
-- Corey Koberg, Cardinal Path

5. The questions for a content- or mission-based site
are the same…
Who came to our site?
e.g., previous vs. new; high vs. low potential
How did they get here?
What did they look at?
Were they successful in getting what they wanted?
…so why i the typical story usually something lik this?
h is h i l ll hi like hi
Our site has 5,000 monthly unique visitors.
Last Tuesday that story got 20,000 page views.
y yg , p g
The average time spent on our site last week
was 24 minutes.
Our iPhone app was downloaded 10,000 times.
We have 2,000 fans on our Facebook page.
We have 5,000 Twitter followers
5 000 followers.
5

6. Example
Purpose of a community news program
Increase civic engagement
in a community
by bringing stakeholders together
through their shared need for
community news and information
6

7. Using data for decision-making
is dependent first on a clearly defined target audience…
Alhambra Boyle Heights “South LA”
City of Alhambra City of Los Angeles LA Times
…and then on an equally well-defined multichannel strategy
7

8. Each channel reaches different audiences and has
unique functionality, so each needs its own measurement model.
A channel’s model – and metrics – should be developed
based on its role vs. other channels.
Goal: What the org wants the channel to do
Key Performance Indicator: A metric crucial to the org’s survival
Target: The value of the KPI that will indicate success or failure
Segment: A group of visits or visitors, categorized by type and/or behavior,
that is essential to reach the target
8

9. Sample measurement model for a website (eek!)
9

10. 1. Establish program objectives.
What does your org want to do through all of its channels?
10

11. 2. Establish – exactly – how each channel will contribute to
each program objective.
What are the goals of your site? Why does it exist?
11

12. 2. Establish – exactly – how each channel will contribute to
each program objective.
What are the goals of your site? Why does it exist?
Getting a name and e-
mail address is the
first indicator
someone is engaging
with you.
Audience info, obtained with permission, is perhaps the most important function of a site.
12

13. 2. Establish – exactly – how each channel will contribute to
each program objective.
What are the goals of your site? Why does it exist?
13

14. 3. Decide which metrics will be the Key Performance Indicators
for each channel goal. Establish the targets that will indicate
success – or f il
failure.
Analyze audience segments: What
type of audience behavior is
yp
affecting the KPI?
Example:
Site visitors who entered
through search engines visited
an average of once a week.
Is this good?
Yes – Our target was two
times a month
No – Our target was twice a
week! We just added
additional resources to put
new content up daily!
14

15. Maybe th
M b the content you’re putting up on the site
t t ’ tti th it
isn’t the content that people want!
Type of analysis used by J. Paul Getty Trust from “Web analytics success for government websites,”
by Avinash Kaushik, Oct. 12, 2009

16. 3. Decide which metrics will be the Key Performance Indicators
for each channel goal. Establish the targets that will indicate
success – or f il
failure.
16

17. 3. Decide which metrics will be the Key Performance Indicators
for each channel goal. Establish the targets that will indicate
success – or f il
failure.
Nonprofits need KPIs need to be
to decide specific to what
whether the the channel can
number of actually do.
individuals is
more of a Example: A
priority than the $20,000
total amount of sponsorship
money package will be
sold through
nondigital
fundraising
methods.
17

18. Sample measurement model for a website…
…repeat for e-mail, Facebook, Twitter...eek!
18

19. Web analytics is the analysis of data
“to drive a continual improvement of the online experience…
which translates into your desired outcomes.”
y
Just one
part of
web
analytics
19
from Web Analytics 2.0 by Avinash Kaushik

20. Step 1: Understand the clickstream,
or every action relevant to site goals
Behavioral research
What people did
when they came to your site,
h h
as captured by
an action taken on a keyboard or
mouse
20

21. What actions indicate engagement?
Visit
Vi it , regularly
l l
Read/view content, a l
R d/ i lot
Interact,
Interact often
-- rate, print, vote, take a poll, click on an ad
-- share, e-mail, comment, contribute
21

22. Basic site metrics
Unique visitors
U i i it
visit sites
and generate
page views
22

23. Total visits:
One indicator of overall site performance
A visit i counted
i it is o ted
every time
someone comes to a site
Visits: the strongest metric available
An increase in visits is always good.
-- More people are coming to your site.
-- Returning people are coming more often.
A decrease in visits? Always bad.
23

24. Strong vs. weak metrics
g
Strong metrics are useful tools
that give clear indications
of what’s successful or not
c. Kyle Taylor
Weak metrics…
-- are conceptually flawed
“so what?” counts of things
-- are technically flawed
metrics calculated by
web analytics systems c. Kyle Taylor
in ways that give unclear indications
…could be so misleading
they could lead to bad decisions
24

25. Really weak metric #1: Unique visitors
A unique visitor is really a unique computer.
Unique visitors are either over-counted…
…or under-counted.
library,
school,
Internet
I t t
cafe
You’ll never know which or by how much.*
* It doesn’t matter anyway….better to measure outcomes (did people do what you wanted?) than
the number of people who came to your site.
25

26. Really weak metric #2: Page views
An increase in page views can be good - or bad *
bad.*
Bad design navigation site architecture?
design, navigation,
Lots of page views, annoyed users
A redesign improved usability?
?
Fewer page views happier users
views,
Content that should be there but isn’t?
Lots of page views, annoyed users
Dynamic content?
Fewer page views, happier users (probably)
* It doesn’t matter anyway….better to measure outcomes (d d people d what you wanted?) than
d ’ b (did l do h d ) h
the number of pages people went to when they came to your site.
26

27. Really weak metric #3: Time spent on site
An increase in average time spent on site can be good –
or bad.*
Bad design, navigation, site architecture?
g , g ,
?
Lots of time spent, annoyed users
A redesign improved usability?
Less time spent, happier users
p , pp
Technically flawed: Time spent is either over-
counted or undercounted
* It doesn’t matter anyway….better to measure outcomes (d d people d what you wanted?) than
d ’ b (did l do h d ) h
how much time people spent on your site.
27

28. When audiences - new and returning -
Wh di d t i
come, are they staying?
Key Performance Indicator
Bounce rate percent
of the landing page
where most visits start
“I came. I saw. I puked.”
-- Avinash Kaushik on bounce rate
A bounce: a visit with only one page view 28

29. The bounce rate of a landing page is much more actionable than the
bounce rate of the entire site
Start by looking at the top landing page, or the page where most visits start
100%
51%
8,331
Home page bounce rate: 43%
16,304 visits
visits started
on
content
pages
49%
7,973 57% 43% left the site
4,547 3,426 without going
visits went to
started at least to another
on the
one page
other
home page
page
Action: Let’s try
changing the home
page
29

30. Three types of site metrics that can be used
to segment visitors by behavior
1. Visitor acquisition: How did people get to the site? Is your marketing working?
Traffic sources: direct, referrals, search engines, campaigns (e.g., e-mail
newsletters, ads)
2. Site behavior: What did they do once they got to the site?
Bounce rate of a landing page – did people leave after seeing only one page?
Visits that included internal search
Visits that went to a particular type of content
3. Outcomes: Did people take the actions essential to the organization’s success?
Visit frequency and recency
Sign-up for an e-mail newsletter
Buy a benefit dinner ticket
Donate; sign up for membership

31. Internal metrics External metrics
for for
Strategic Planning Marketing, Advertising
• Census data • Panel, toolbar data
100% of all visitors, visits, page Activity from a sample of self-
views, etc. in a site selected people. Usually not
relevant for small sites.
• Analysis, decisions, • Marketing, trending,
actions, evaluation competitive analysis
• Omniture • comScore
Google Analytics Nielsen
WebTrends Compete
etc.
etc etc.
• Digital Analytics • Interactive Advertising
Association Bureau
31

32. Understand all of the implications
Increasing the number and
of mapping metrics to goals percent of students with high
SAT scores…
…leads to a higher ranking by
US News & World Report…
…which helps with donations,
partnerships, faculty
recruitment, etc.
Action: Centralize UG admissions
decisions.
One result: Fewer minority
journalism majors coming from high
school

33. Measurement models need to be reviewed regularly
Charities now need to optimize both “Financial Health” and
“Accountability and Transparency”
y p y

34. Why does
Charity
Navigator have
ratings like
this?
They’re not
helpful to
donors.
donors
More
importantly,
they give no
guidance to
charities on
what they
should
manage.

35. ?
?
?

36. Is Liberty Hill’s site successful?
“No more newsletters
mailed to you at
home….please register on
h l i t
the website, even if
you’ve been a supporter
for years ”
years.
KPI: Percent of print
newsletter subscribers who
register
Target: 100%
Action needed based on
analysis of data:
We need more
[Eastside/media/women] to
register.
Let’s try a [follow-up
postcard/raffle/event].

37. Is Liberty Hill’s site successful?
Look at the site traffic
trends after the flyer is
mailed to each
audience type.
-- home page bounce rate
-- pop-up bounce rate
-- sign-up p
g p process
Maybe you’re getting
people to come to
the site, but the site
is losing them.

38. Is Liberty Hill’s site successful?
Is our site selling the
number of tickets
tickets,
sponsorships, ads,
etc. that we want?
Do other channels
work better for some
items?
Analysis needed (multiple data
sources needed):
-- Total sold last year from all
sources, by time period
, y p
-- Tickets (premier, standard);
sponsorships (by type); ads (by
type)
-- P
Percent of registered people
t f i t d l
(by type) who buy tickets,
sponsorships, ads

39. Is Liberty Hill’s site successful?
“Purchase your sponsorship,
ticket and ads…online. Go to
www.LibertyHill.org/dinner.”
Have different direct mail and e-mail
campaigns for each audience segment;
have a different landing page with a
unique URL (e.g., /campaign1;
/campaign2) for each.
Sending people just to www.libertyhill.org
is a wasted tracking opportunity!
Sample e-mail newsletter metrics
KPI: No. of tickets sold by campaign
Week of Jan 26, 2012
Jan. 26
Delivered/sent: 970/1,269 (76% Actions needed based on e-
delivery rate) mail KPIs and ticket sales:
Target: 100% (indicates list quality)
Clicks/delivered: 36/970 (3.7% click- We need to clean up our list.
to-delivery rate)
Indicates the relevancy of the e-mail content)
Let’s try a different message
E-mail newsletter bounce rate: 78%* for [environmentalists/past
*Estimated - links need to be tagged to track all
traffic from a newsletter to the site program advertisers/individual
seats].

40. Is Liberty Hill’s site successful?
Where are we losing
people in the
purchase process?
KPI: Percent and no. of
visits that started with the
dinner overview page and
completed the four-step
p y
payment process
p

41. Social
media
Not only are the technologies new,
but the metrics are as well.
--Online Media and Marketing Association Metrics and Measurement program, June 2009
41

42. We know about “spray and pray” business models…
The social media ‘provide and pray’ model
Not having a purpose for social media efforts “….often leads to a worst
practice we call ‘provide and pray.’
Leaders and managers provide access to a social technology, and
then pray that a community forms
and that community interactions
somehow lead to business value.
In most cases, adoption never really
materializes; communities may
form, but their activity is not
considered valuable to the
organization.”
“Social Media Success Is About Purpose (Not Technology),” by Anthony J. Bradley and Mark P. McDonald, Harvard Business Review,
Nov. 1, 2011

43. Why should
organizations…
…have a Facebook page?
…tweet?
“Effectively measuring social media,” Susan Etlinger, Altimeter Group webinar, April 2011
How important are either in achieving their goals?
Are either of them essential, given
--a large part of the target audience may not be
on Facebook and/or Twitter?
--extremely limited resources?
Social media metrics are just as important as site metrics.

44. Typical (and rarely explicitly
expressed) social media objectives
d) i l di bj ti
for a news or nonprofit organization
1. Drive traffic to the website
Social
S i l media as b
di broadcast advertising mechanism
d t d ti i h i
2.
2 Engage
-- current audiences
-- new audiences it s not getting elsewhere
it’s
Is this realistic?
Using a measurement model is even more important for
social media!

45. “What matters is everything that happens after you post / tweet /
participate….The ‘so what’ matters!”
1. Conversation: “Social means
talk and listen and discuss. So
why not measure that?”
2. Amplification: “The rate at
which your followers take your
co te t a d s a e t t oug t e
content and share it through their
network.”
3.
3 Applause: “What does your audience like?
What like?”
“Best Social Media Metrics,” by Avinash Kaushik, Oct. 10, 2011. Chart designed by Erik Ohlen

46. Understand how to measure Twitter,
and you’ll understand how to measure social media
Content
Followers
not demographics or other typical mass
media audience metrics 46

47. Twitalyzer – user centric TweetReach –
tweet-centric
Overall impact of an account
Influence Looks at tweets for
Reach – potential keywords
Reach – effective
Total reach =
Number of unique people
following people who
Number of: generated one or more
--Followers tweets that contained the
--Followers of followers keyword
y
--References to the account by followers
f h b f ll
--Lists that include the account Total exposure = Number
tweets of people (duplicated)
--Times the account is replied to
--Times the account is retweeted
Times Number of
Tweets with the keyword
Contributors
List of users who
generated the highest
number of impressions
(calculated with tweets,
retweets, followers of the
people who tweeted and
followers of the people
who retweeted
“Twitalyzer and TweetReach – A Symbiotic Pairing for Twitter Analysis,” by Tim Wilson, March 8, 2011

48. Measurable tweets have
have…
1. A call to action
Go here…look…tell me
2.
2 A link that you track with link (e g bit y)
(e.g., bit.y)
and web analytics tools
RT - retweet
MT – modified tweet
3. #Hashtags and/or keywords Via or HT – heard through
Favorite
Lists
4. Topic or person-specific handles
…120 or fewer characters, not 140!
120 characters
48

49. Follower metrics – Avinash Kaushik
“Retention is the key to growth.”
DRAFT
Metric Tool Time period Analysis
Follower growth Twitter Counter Over last three Compare rate against
rate months two others.
Relevant follower Twitalyzer, others Over last three Estimate; sample
growth rate months
No. of unique, GraphEdge (paid) Over last three Compare against two
legitimate months others.
followers; percent
of overlap
Churn ate
Ch n rate: Gain GraphEdge; others
G hEd th 30-day churn rate
30 d h t Also
Al analyze number
l b
in followers vs. no. of relevant followers
of unfollows
Follower retweet Klout (velocity) Over last three
percent months
Unique senders
q Klout (engagement)
( g g )
Chart derived from “Actionable Twitter Analytics” Market Motive video

50. Interaction metrics – Avinash Kaushik
DRAFT
Metric Tool Time period Analysis
Click percent, by URL shorteners – Weekly Put in context vs.
topic; top bit.ly, tr.im news cycle.
referrers Analyze tweet
volume.
No. of retweets, Twitalyzer, Weekly
by topic Retweetrank, Klout
(reach), others
No. of retweets Twitalyzer, Weekly Compare against
per thousand Retweetrank, others two others.
followers, by
topic
Replies sent per Twitter-friends Daily Are you having
day conversations?
Replies received Twitter-friends Daily Are people
per day communicating with
you?
Inbound metrics
I b d t i Klout (engagement)
Kl t ( t)
per outbound
message
Chart derived from “Actionable Twitter Analytics” Market Motive video

51. Outcome metrics – Avinash Kaushik
DRAFT
Metric Tool Time period Analysis
Visits and other Kl.am; Google Need to determine
site by Analytics campaign campaign
“campaign” tracking parameters. Set
up custom GA
report.
t
Sentiment or TweetPsych 23 dimensions such
Emotion – What as learning,
is your Twitter thinking, positive.
account saying Is your account
about you? what you want for
your brand and
what you are trying
to do?
Sentiment or StatsIt
Emotion – What
are other people
saying about
you??
Chart derived from “Actionable Twitter Analytics” Market Motive video

52. Twitter metrics investment levels – Avinash Kaushik
From “Actionable Twitter Analytics” Market Motive video

53. Facebook Insights
Analyze trends in
• Posts
• People are Talking
About This
• Weekly Total
Reach

54. “…it is worse to post something that people do not
react to, than to post nothing at all.”
“…a completely flat level of weekly reach”
“Each post is not really making any difference one way or
another. The
another Th number of people who are talking about this brand is dropping.
b f l h lki b hi b d i d i
“This is an indication that you are boring. This
brand is likely doing the same few things over and over again, and
people are getting bored with it….
“You are slowly turning yourself into a commodity. It
is just something people can follow every day, but you are not motivating
j gp p y y, y g
your audience to act. You are not changing anything.”
“Beyond Facebook Analytics,” by Thomas Baekdal, Nov. 8, 2011

55. Be honest with metrics
Do 538 people
REALLY “Like”
this?
Or do h
O d they jjust
want another
sweepstakes
entry?
55

56. Audiences and actions differ by channel…
…so there are completely different metrics for each!
And you need to report them all separately – you can’t add
them up to get a total audience number
SOCIAL
SITES MEDIA MOBILE
Totals
1. Who? How many?
In target audience? ? ? ? ? ? ?
2. No. of visits?
2 f i i ?
How often? ? ? ? ? ? ?
3. What did they see? ? ? ? ? ? ?
Did they get want
they wanted?
4. Did they interact?
y ? ? ? ? ? ?
What did they do?
How much?
56

57. Evaluating effectiveness
1. Define a measurable audience.
2. Set specific goals across all channels;
Measure
map metrics to goals. Segment.
Optimize
Act Analytics Report
3. Set up each channel to
measure specific actions
that indicate engagement
and lead to outcomes
Analyze
Don’t forget about Voice of
Customer
57