This document discusses how Disney-ABC Digital Media brought A/B testing to their entertainment platforms. It provides background on their business and digital products. It then describes how they set up A/B testing in their organization and why it makes sense for entertainment. The document outlines a case study where they tested different variations of their full episode homepage and found that displaying the top current shows increased video starts by 4.94% compared to the control. They projected this small change could result in over $600k additional annual revenue.

1. Bringing A/B Testing to Entertainment
Disney-ABC Digital Media
The Optimizely LA Experience
8.19.14
Proprietary. Do not distribute without permission (Khai Tran).
Khai Tran
Sr. Mgr, Analytics & Customer Insights
Disney-ABC Digital Media

2. Building A/B Testing within Disney-ABC Digital Media
Background on our business
A case study
Final thoughts
2Proprietary. Do not distribute without permission (Khai Tran).
Why A/B test?

3. 3
Disney-ABC Television – WATCH Products & Platforms
Proprietary. Do not distribute without permission (Khai Tran).
Own, operate, and stream full-episodes, short-form, live video content and
games. Serve as a marketing channel for our shows and brands.
What’s our
Business?
Supported
Platforms
Supported
Brands

4. 4Proprietary. Do not distribute without permission (Khai Tran).

5. 5Proprietary. Do not distribute without permission (Khai Tran).

6. 6Proprietary. Do not distribute without permission (Khai Tran).

7. 7Proprietary. Do not distribute without permission (Khai Tran).

8. 8
Where A/B Testing Lives in our Org
WATCH Products & Platforms
Product Engineering Video
Programming MarketingAnalytics &
Customer Insights PMO
Data
"
Data Design
Requirements
Tagging
Data Integrity
Analytics
"
A/B Testing
Path Analysis
Case Studies
KPI Reporting
Customer Insights
"
Support Agents
Customer
Feedback
Escalation Mgmt
Voice of Customer
Proprietary. Do not distribute without permission (Khai Tran).

9. Why A/B Testing Makes Sense in Entertainment
9
1. Controls for external factors"
• Traffic is randomly split across test variations during
time period of test"
• Based on real users, not focus groups "
2. Launch and iterate faster"
• Eliminates guesswork in evaluating different options"
• Reduces time and resources "
3. Drive innovation and growth"
• Whether the test wins or loses, it enables to learn more
about your users"
• Speeds up innovation by minimizing risk and fear in
trying out new ideas
Controls for
External Factors
"
Originals vs. Repeats
Show Popularity
Seasonality
Windowing
Competitive Offerings
Macro Factors
Proprietary. Do not distribute without permission (Khai Tran).

10. 10
Driving Actionable Insights – Testing in the Larger Picture
Track Key
Success
Drivers
Analyze
User
Interactions
Segment for
Deeper
Insight
Test &
Optimize
Drive
Actionable
Insights
1. Track key success drivers
• Keep pulse of the business
• See if we’ve made a difference
"
2. Analyze user interactions
• Heat mapping, path analysis,
etc.
• Identify areas for improvement
"
3. Segment for deeper insight
• Sub-groups behave differently
• Solve for key segments
"
4. Test and Optimize
• Form hypotheses and tests based
on insights
• Validate different variations
based on cause-and-effect
Proprietary. Do not distribute without permission (Khai Tran).

11. 11
Full-Episodes Main Page"
"• Top page of our user path"
• High traffic page"
• High potential impact"
• Bounce rate: 20-35%
* Omniture: Jan – Jun 2011, FEP visits only (excluded iPad)
40%
35%
0.2% 1% 1%
0.5%
1%
2% 4%
7%
4%
3%
What’s Interesting
"35% of users clicked “All Shows”
in bottom nav. Why?
"
Insight: Users want to get
directly to the show that they’re
interested in.
ABC.com Case Study – What and How We Tested

12. 12Proprietary. Do not distribute without permission (Khai Tran).
Variation A: "
Control Version"
"
Full-Episode Home Page "
www.abc.com/watch

13. 13
Variation:
Display all shows with logos
"
Hypothesis:
"Will increase video starts because it:
" • Reduces click to all shows
• Easier to access non-recent content
"
Potential Concerns:
• Requires lots of scrolling.
• Choice overload may lead users to leave.
• Increases page load time.
Proprietary. Do not distribute without permission (Khai Tran).
Variation B: "
All Shows with Show Logos

14. 14
Variation:
Display all shows text
"
Hypothesis:
"Will increase video starts because it:
" • Is easier to scan and find shows
• Loads faster, requires less scrolling
"
Potential Concerns:
• Choice overload
• Too much text. Unattractive.
Proprietary. Do not distribute without permission (Khai Tran).
Variation C: "
All Shows with Text

15. 15
Variation:
Display top 8 noncurrent shows with link to “All
Shows”
"
Hypothesis:
"Will increase video starts because it:
" • Highlights most popular Fall shows
• Extends long tail for non-recently aired content
• Distinct from current shows displayed in the
slideshow
Proprietary. Do not distribute without permission (Khai Tran).
Variation D: "
Top Shows Fall Season "
(Noncurrent Shows)

16. 16
Variation:
Display top 8 shows that are in season at time of test
"
Hypothesis:
"Will increase video starts because:
" • These shows account for 80% of current video
views
• Easy to find and access these shows
Proprietary. Do not distribute without permission (Khai Tran).
Variation D: "
Top Shows Summer Season "
(Current shows)

17. 17
4.24% 3.74% 3.93%Video Starts 4.94%
Chance to beat 99.3% 98.4% 97.8% 97.9%
So Which One Won vs. the Control?
A
Control
B
All Shows Logos
C
All Shows Text
D
Top Shows
Noncurrent
E
Top Shows
Current

18. 18
0.5 Days
"• Define goals,
success metrics,
scope, etc.
"
"
2 Days
"• Assess dev needs
• Build test versions
0.5 Day
"• Configure
campaign in
Optimizely
Planning Design
Developm
ent
Campaign
Configura
tion
0.5 Day
"• QA
• Launch test
1 Day
"• Hypothesize test
variations and
design comps
Setup time – 4.5 days
7 Days
"• Advised by
T&T to run at
least 1 week
"
"
Ru
n
A/B
Tes
t
An
aly
sis
2 Days
"• Analyzed real-
time results
throughout
Run + Analysis time – 9 Days
A/B Test Details "
"
• Ran for one week (June 28 – July 5) "
• 660k+ unique visitors"
• 20% traffic allocation "
• Test groups maintained across sessions
A/B Testing Steps and Actual Time Spent

19. 19
"
Even with a slight page tweak, which appears below the fold, we can
produce a sizable revenue lift for the business.
Video Starts
Driven from
Homepage Revenue
146 MM $13.7 MM
+ 7.2 MM + $677 K
Lift from
Test
+ 4.94%
Projected
Contribution
Connect A/B Testing Results to Revenue Impact
Actual numbers have been masked, but results are in the ballpark. This test ran in 2011.
x $0.0938 =
x $0.0938 =
Avg. Ad Revenue
per Start
Before Test

20. 20
Image from Entrepreneur magazine (Dec 2012). http://www.entrepreneur.com/article/224967
Final Thoughts