At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled “Is My Digital Ad Working?” Panelists discuss which metrics are the best indicators of campaign performance? Panelists include, Sibel Satiroglu, WW Digital Insights for HP & Suzanne Leighton, EMEA Insights for HP. Panel was moderated by Kate Sirkin, EVP, Global Research of StarcomMediaVest.

1. “IS MY DIGITAL AD
WORKING?”
Moderator
Kate Sirkin
EVP, Global Research
Starcom MediaVest

3. Coming Together of the
Digital Pieces
POV: Measuring Awareness Building
Digital Banner Campaign Performance
Sibel Satiroglu Suzanne Leighton
WW Digital Insights EMEA Insights
HP HP

4. Situation Overview
Move to digital
• HP’s advertising is increasingly digital
• Attitudinal ad and brand surveys used to measure
digital campaigns
– Tracked in some countries only
• Perception that digital is more measurable, given the
abundance of metrics
– Yet silos of metrics which need to be triangulated
• CTR is an easily available, well established, often cited
campaign metric – but what does it really measure?
Ad and brand surveys: control-exposed web intercept methodology

5. CTR Examples
CTR differs vastly between countries and campaigns
without indicating campaign performance
Country Impressions CTR Site Type Impressions Clicks CTR
LEADERBOARD &
UK 26,959,639 0.09% chip.de SKYSCRAPER 788,810 3,781 0.48%
France 32,211,486 0.40% chip.de MPU 138,052 116 0.08%
Germany 26,186,369 0.15% LEADERBOARD &
computerbild.de SKYSCRAPER 100,390 606 0.60%
Netherlands 14,490,373 0.44%
Ligatus IMAGE & TEXT 201,754,920 14,640 0.01%
Italy 24,207,302 0.13%
LEADERBOARD &
Business Ad SKYSCRAPER 599,043 4,499 0.75%
Poland 541,247,145 0.04%
EMEA 665,302,313 0.08% Germany 203,381,215 23,642 0.01%
Or 0.23% excluding Or 0.55% excluding
Poland because of Ligatus as
different media different media
strategy strategy

6. Key Questions
1. What is the relationship between CTR,
attitudinal and site behavioral metrics?
2. Which metrics are the best indicators of
campaign performance?
3. What is the role of CTR in assessing digital
campaign performance?

7. Relationship Between Metrics
Not explicitly understood, tends to be assumed
Media
Behavioral
? Impressions ?
Reach/Frequency
CTR/CPC
Online Ad Breakthrough Interaction Rate/CPI
Visits
Message Breakthrough Site Bounce Rate
Brand Awareness Attitudinal
Behavioral Engagement Rate
Brand Consideration
Brand Perceptions ? Time Spent/Deep Clicks
Which metric or combination of metrics do we use to answer the questions:
Is my digital advertising effective? How can I optimize it?

8. Methods
• Campaign metrics collected across 3 groups
– Data matched by campaign, creative and site identifiers
• Data set included
– 5 brand/awareness campaigns between Oct ‘10-Jan ‘11
– US, UK, France and Germany
– Consumer and Small Business segments
– 1500M impressions, 1M site visits
– 75% of creative units standard flash banners;
25% takeover high impact units (non-video)
• Spearman rank correlation used to analyze relationships
between metrics
Spearman Rank: Dunn-Bonferroni correction applied to account for multiple comparisons.

9. Results
No correlation between CTR and attitudinal metrics
However, strong negative correlation between CTR
and engagement rate on the site
rs P Critical P Significant?
CTR & Unaided Awareness 0.06 0.71 0.01 NO
CTR & Aided Awareness 0.26 0.10 0.01 NO
CTR & Ad Breakthrough -0.07 0.66 0.01 NO
CTR & Message Breakthrough -0.04 0.80 0.01 NO
CTR & Purchase Consideration 0.13 0.41 0.01 NO
CTR & Bounce Rate (Site) 0.28 0.08 0.025 NO
CTR & Engagement Rate (Site) -0.69 <0.0001 0.025 YES
n range = 39-42
Critical P: Dunn-Bonferroni correction

10. Results
What does negative correlation between CTR
and engagement rate mean?
Does NOT mean low CTR results in high engagement.
Simply, as CTR , the proportion of unengaged visits , RELATIVE to engaged visits.
n = 151

11. Key Conclusions
1. What is the relationship between CTR,
attitudinal and site behavioral metrics?
• A No relationship
CTR is not a surrogate for
Media
campaign performance
A Behavioral
B
No
relationship
Inverse
relationship
• B Inverse Relationship
CTR Balance creative/placement CTR
optimization with site engagement
? Site – Linking back to objectives
Attitudinal
Insufficient Behavioral – Site design is important! Ensure
data to analyze
relevance of site to creative.
Brand and Ad Metrics Engagement Rate A relevant site increases opportunity
to increase engagement by meeting
visitor expectations
See also: 1) Mallon K. and Bruner R. 2009: “Can Rich Media Metrics Predict Brand Impact?”, Dynamic Logic and Google, ARF Rethink;
2) McMarthy K and Fulgoni G. 2009: “The Click Remains Irrelevant: Natural Born Clickers Return”, comScore and Starcom;
3) Lotame. 2010: “Online Brand Optimization Metrics That Matter And How To Use Them To Effectively Change Campaign Outcomes.

12. Key Conclusions
2. Which metrics are best indicators of campaign
performance?
Require assessment over the longer term but we must have
short-term metrics as indicators and for optimization
To Measure Metrics Countries Assessment
Campaign
Attitudinal metrics to assess brand impact Select Long-term
Overall
Ad metrics: Branding, stopping power, message
Creative Select
breakthrough
Site engagement coupled with media metrics
• Impressions, reach, frequency and clicks – Short-term
All
Media and cost thereof indicators
• Site visits and engagement for quality of the and for
media response optimization
Site behavioral metrics indicate site performance:
All
Site • Site visits, bounce, engagement, as defined by
site objectives

13. Key Conclusions
3. What is the role of CTR in assessing digital
campaign performance?
• CTR is not a “dashboard metric”
– In aggregate, CTR can become meaningless and result in
incorrect decisions
– It is a measure of creative and media performance for
tactical optimization
• Why are we still having difficulty putting CTR in its place?
– Is it the dilemma of CTR having some value but the real danger
of it being misused or misinterpreted?
– Is it that too much focus on increasing CTR compounds
its inadequacies?
– Is it the apparent contradiction that the website & visits matter
but that CTR doesn’t?
– Are we adequately measuring digital advertising - perhaps not having
to click means the ad has communicated the required information?

14. Thank You