From ideation, to test execution, to interpreting results – your experimentation program requires a variety of sophisticated skills to achieve impactful results. In many organizations managing an experimentation program comes with the added challenge of having limited resources. Check out these slides and get insight into running an impactful testing program with a lean team. Key Takeaways - Tips for fully utilizing your lean team - Ideas for scaling your program - Ways to create a flywheel of winning tests

1. Speakers
Successful Testing
with a Lean Team
Tim Kotchetov, Data Analysis & Business Intelligence, Seattle Times
Kate Nichol, Customer Success Manager, Optimizely

2. Today’s Speakers
Tim
Kotchetov
Kate
Nichol

3. Housekeeping
• We’re recording!
• Slides and recording will be
emailed to you after the
webinar is complete
• There will be time for
questions at the end

4. Agenda
• Staffing for Experimentation Success
• Experimentation at The Seattle Times
• Real-World Tests
• Overcoming Challenges
• Q&A

5. Staffing for
Experimentation
Success

6. ‘Recommended’ Experimentation
Program
Successful Testing Skills
● HTML and CSS
● jQuery and JavaScript
● SEM and SEO
● Statistics
● Project Management
● UX Design
● Analytics
● Copy
● Communication

7. It’s still possible to achieve
success with a smaller team.

8. Experimentation at
The Seattle Times

9. The growing importance
of SeattleTimes.com
● Launched website in early 2000s with
unlimited access to all online content
● In 2013, SeattleTimes.com introduced a
paywall which allows 7 free articles/month
● 3 upsell products:
○ Digital Only
○ Digital Only + Sunday Print
○ Digital + Daily Print

10. Breaking Through the Paywall
Maximize revenue by
increasing number of
subscribers

11. State of the Experimentation Program
● A/B Testing since March 2016
● Goal to increase conversions, subscriber revenue, or
conversions that indirectly leads to subscriptions (such as
newsletters)
● Run approx. 1 a/b test per month
● Integrate with Google Analytics and Google BigQuery
● Approx. 1.5 persons dedicated to A/B testing program
● Testing ideas sourced from across circulation, marketing
team and across the org

12. Resources & Staff
● The Seattle Times Business Intelligence
team is made up of two analysts and a
manager
● Primary responsibility is to provide any
analytical needs to various departments
they cannot get on their own
● One analyst is responsible for
coordinating implementation of and
tracking of A/B tests and results
Business
Intelligence
Circulation Advertisement Newsroom

13. Real-World Tests

14. #1: Newsletter Subscribe Widget Test
● Test whether wording on widget email submit
button prevented people from subscribing to
newsletter
● Conducted 11-day A/B test of two versions:
○ Original: “Subscribe”
○ Variation: “Sign Up”
● Results: Variation generated +4.3%
more signups
Original
Variation

15. #2: Product Pricing Test
● We offer 3 products:
○ Digital Access only
○ Digital Access along with daily print delivery
○ Digital Access plus print delivery on Sundays (our
most profitable)
● Hypothesis: Would a decrease in price of our most
profitable product increase the number of purchasers
enough to offset this and increase overall revenue?
● Partnered with BI team member and Director of Circulation
to create test model accounting for ad revenue, audience
revenue and retention
● Used model to track test results over 9 weeks
Variation
Original

16. #2: Product Pricing Test - Results
We proved our hypothesis:
● Digital Access only purchases
dropped 10 percentage points
● Digital Access + daily print
purchases stayed consistent
● Digital Access + Sunday print
purchases increased 8%
● Overall revenue increased by
8%

17. Test whether newly designed landing page with same wording and messaging as previous
version would not drive down subscriptions and possibly increase them
Original Variation
#3: Paywall Landing Page Design Test

18. #3: Landing Page Design Test - Results
● The new landing page design was successful!
● Generated 8% more overall subscription starts

19. #4: Yearly Subscription Offer Test
● Hypothesis: Would overall revenue increase if we offered a yearly subscription for
‘Digital Only ‘at a 25% annual discount?
● For 8 weeks, tested this offer side-by-side with original monthly subscription
Original Variant

20. #4: Yearly Subscription Offer Test - Results
● Annual subscription offering is not profitable
○ Overall conversions for variant were 9% lower than original monthly offer
○ Digital conversions for variant were 10% lower
○ Print conversions for variant were 8% lower

21. #5 - Paywall Meter & Landing Page Copy Test
● Users can reach 7 articles per
month before hitting paywall
● Readers are shown periodic
pop-up messages giving them
article count
● Hypothesis: Will different
messaging encourage more
subscriptions?
Variant

22. #5 - Paywall & Landing Page Copy Test - Results
● Wording changes made a difference
○ Overall conversions generated 14% more subscriptions
○ Conversion for Digital Access increased 24%; virtually unchanged for Digital +
Print Sunday/Daily
○ Those clicking on meter pop-up messaging instead of waiting to hit paywall
25% more likely to subscribe
○ Those who hit paywall page 28% more likely to subscribe

23. What’s Next?
● A/B Test the # of monthly articles a user can access before hitting the paywall
● A/B Test placement of section blocks on the home page
● A/B Test position of newsletter signup widgets

24. Overcoming
Testing Challenges

25. Developer Resource Constraints
Challenge
● Many A/B tests required developer involvement/coding
● Developers are stretched thin
Solution
● Developers dedicate 2-3 hours to each A/B test
● Analyst can handle organizational, logistical, and measurement aspects

26. Statistical Significance
Challenge
● Industry online conversion rates are low
● Hard to generate statistically significant results
Solution
● Focus on data consistency
● Calculate weekly rate of conversions
● Throttle number of users to focus on test pages

27. Implementing Changes
Challenge
● Limited developer resources delay website changes
Solution
● Make change in Optimizely as stop-gap measure

28. Tips & Tricks for Testing with a Lean Team
● For your first test, try something simple to get you going
● Source ideas from stakeholders across your organization
● Important to establish relationship with IT/developers
○ Use developers’ time efficiently and regular meet to discuss priorities
● Integrate with Google Analytics to show impact to audience segments that are
important to the business
● Regular communication about performance to ensure test results are top of mind and
having an implementation plan for key findings

29. Questions?

30. Thank You!