Driving Bottom Line Impact Through Mass Scale Experimentation at Lending Tree

1. opticon2017
Driving Bottom
Line Impact
Through Mass-Scale
Experimentation at
LendingTree
Manan Singh
VP Optimization, LendingTree

2. opticon2017
Agenda 1. Optimization as a Strategic Initiative
2. Continuously get Optimization wins
3. Run tests faster through iterative
experimentation
4. Measure success for bottom line
wins
5. Share examples

3. opticon2017
About me
Started career in the Optimization
space with Adobe / Omniture in 2010
Built successful Optimization teams
at eCommerce and Fintech
companies

4. opticon2017
About LendingTree
Leading online loan marketplace
Enable users to shop, compare, and save on
the loans they need
Mortgage, Personal Loans, Auto Loans,
Business Loans, Student Loans, Credit Cards
etc.

5. opticon2017
2017 Strategic Initiative: Funnel
Optimization
Doug Lebda
CEO, LendingTree
Q4 2016 Earnings Call
“As a reminder, these strategic initiatives are: one, to expand into new
categories while still growing market share in our existing categories;
two, to strengthen the consumer relationship through more options and
improved intelligence, particularly around our MyLendingTree product;
three, to re-imagine the consumer experience, particularly in mortgage;
and four, to maniacally optimize our conversion funnel from ad unit
all the way through loan funding, which is something that we believe will
substantially move our company forward. I’m proud to say that we’ve
already made great progress even in the last months and are much
more confident in our ability to execute on all of these initiatives.”
maniacally optimize our conversion funnel

6. opticon2017
Results reflecting in the bottom line
Stock up 160% this year!

7. opticon2017
How to drive bottom line impact
Prioritize
Opportunitie
s
Better ROI Improved Efficiency Increased Profitability
Iterate
Experiments
Identify and prioritize
biggest opportunities for
a better ROI
Use quick iterative
experimentation methodology
to drive big results
360° view of the customer
so experimentation wins
translate to the bottom line
Measur
e
Succes
s

8. opticon2017
Prioritize
Opportunitie
s
Better ROI Improved Efficiency Increased Profitability
Iterate
Experiments
Use quick iterative
experimentation methodology
to drive big results
360° view of the customer
so experimentation wins
translate to the bottom
line
Measur
e
Succes
s
Identify and prioritize
biggest opportunities for
a better ROI
How to drive bottom line impact

9. opticon2017
Start with a Traffic Map

10. opticon2017
Example: Acting on the biggest opportunity
Objective: Reduce Bounce Rate and increase form conversion.
Hypothesis: Showing a generic quote using a calculator and then providing an opportunity
to get a personalized quote will result in better form starts and conversions.
Control Test Version

11. opticon2017
Opportunity
Prioritization
Results
• Double digit lift in conversion rate
• CPL went negative to positive
• Allowed marketing team to bring in additional volume

12. opticon2017
Prioritize
Opportunitie
s
Better ROI Improved Efficiency Increased Profitability
Iterate
Experiments
Use quick iterative
experimentation methodology
to drive big results
360° view of the customer
so experimentation wins
translate to the bottom
line
Measur
e
Succes
s
Identify and prioritize
biggest opportunities for
a better ROI
How to drive bottom line impact

13. opticon2017
“Test Fast, Fail Fast, Adjust Fast.” - Tom Peters
Small changes also result in big improvements
Quickly learn and test different features in phases
1
2
Use Visual Editor to quickly set up tests without
any engg. support3

14. Example: Quick Iterative Testing
Objective: Reduce Bounce Rate and increase form
conversion on one of our highest trafficked SEM
keyword ‘HARP’
Hypothesis: Matching LP copy with the Ad copy will
increase form starts and conversions.
Control Phase 1 Phase 2 Phase 3

15. opticon2017
Opportunity
Prioritization
Results
• Slight drop in Phase 1
• 20%+ lift in Phase 2
• Another 15% lift in Phase 3

16. opticon2017
How to drive bottom line impact
Prioritize
Opportunitie
s
Better ROI Improved Efficiency Increased Profitability
Iterate
Experiments
Identify and prioritize
biggest opportunities for
a better ROI
Use quick iterative
experimentation methodology
to drive big results
360° view of the customer
so experimentation wins
translate to the bottom line
Measur
e
Succes
s

17. opticon2017
No point in failing fast if you
fail to learn from your
mistakes

18. opticon2017
Understand test results completely
Quantitative
Analysis
Calling test a winner just
based on lift in primary
metrics is dangerous.
Answer WHY with
secondary metrics
Qualitative
feedback
Surveys from tools
like Qualtrics
or Medallia
User session recordings
for surveys UserTesting
Combine both for
360° view

19. opticon2017
A great way to analyze tests in more detail!
Pass Optimizely
variation ids to
your DWH!

20. opticon2017
Key Takeaways
Prioritize
Opportunitie
s
Better ROI Improved Efficiency Increased Profitability
Iterate
Experiments
Identify and prioritize
biggest opportunities for
a better ROI
Use quick iterative
experimentation methodology
to drive big results
360° view of the customer
so experimentation wins
translate to the bottom line
Measur
e
Succes
s

21. opticon2017
Q&A

22. opticon2017
THANK YOU

23. opticon2017

24. opticon2017