Talk from SearchLove London 2019 sharing case studies from DistilledODN that reveal on-page best practices that are harming your organic traffic.

1. On-Page Best Practices That Harm
Your Traffic
Lessons from another year of SEO A/B testing
Emily Potter - DistilledODN Consultant
@e_mpotter

2. SEO A/B testing?

3. SEO A/B testing in depth
Case studies to apply understanding
Takeaways

4. SEO A/B testing in depth
Case studies to apply understanding
Takeaways

5. What is SEO A/B Testing?

7. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template

8. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template
www.croabtesting.com/cats www.croabtesting.com/cats

9. Variant UsersControl Users

10. Variant UsersControl Users
50%
of visits
50%
of visits

11. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template
www.croabtesting.com/cats www.croabtesting.com/cats

12. CR control
(conversion rate)
CR variant
(conversion rate)
Positive Test

13. CR variant
(conversion rate)
Negative Test
CR control
(conversion rate)

14. CR control
(conversion rate)
CR variant
(conversion rate)
Neutral Test

15. But there is only one Googlebot

16. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template
www.croabtesting.com/cats www.croabtesting.com/cats

17. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template
www.croabtesting.com/cats www.croabtesting.com/cats

18. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template
www.croabtesting.com/cats www.croabtesting.com/cats

19. Cats
H1
Intro copy
Featured image
Cats
H1
Featured video
Variant TemplateControl Template
www.croabtesting.com/cats www.croabtesting.com/cats

20. We can’t split Googlebot

21. We can’t split Googlebot

22. Googlebot would see
something like this

23. https://www.croabtesting.com › Cats
cro test
All Shopping Images News
About 150,000 results (0.35 seconds)
More
SERP on Experiment Day One
Welcome to our page on cats with nice static text and a beautiful image
of your favorite female friends.
Control Page Title - Template A
Same url

24. https://www.croabtesting.com › Cats
cro test
All Shopping Images News
About 150,000 results (0.35 seconds)
More
SERP on Experiment Day Two
Welcome to our new updated page on cats with a beautiful and
informative video on your favourite feline friends.
Variant Page Title - Template B
Same url

25. https://www.croabtesting.com › Cats
cro test
All Shopping Images News
About 150,000 results (0.35 seconds)
More
SERP on Experiment Day Three
Welcome to our page on cats with nice static text and a beautiful image
of your favorite female friends.
Control Page Title - Template A
Same url

26. So how do we test
for Googlebot?

27. DistilledODN
A/B testing for a scientific approach to SEO
Get a demo

28. Variant UsersControl Users
50%
of visits
50%
of visits

29. Example website with an “animals” category
Animals
Cats Dogs Unicorns Badgers

30. Cats
H1
Intro copy
Featured image
Dogs
H1
Featured video
Variant TemplateControl Template

31. Cats
H1
Intro copy
Featured image
Dogs
H1
Featured video
50% of pages 50% of pages
Variant TemplateControl Template

32. Cats
H1
Intro copy
Featured image
Dogs
H1
Featured video
50% of pages 50% of pages
Control organic traffic Variant organic traffic
Variant TemplateControl Template

33. Now our website looks like this
Animals
Cats Dogs Unicorns Badgers

34. Now our website looks like this
Animals
Cats Dogs Unicorns Badgers

35. Everyone (inc. Googlebot) sees the same pages
Animals
Cats Dogs Unicorns Badgers

36. Organic traffic
Control
Organic traffic
Variant
Positive Test

37. Organic traffic
Control
Organic traffic
Variant
Negative Test

38. Organic traffic
Control
Organic traffic
Variant
Neutral Test

39. What is SEO?

40. 3 levers to increase
organic traffic

41. Users click something
because it’s cool

42. Or because it’s there

43. Look cool in SERP to improve organic CTR

44. Improve rankings so you’re “there” more
Not ranking :(

45. These are our three levers

46. Rank for new keywords so you’re “there” more

47. How do you look more cool in the SERP?

48. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Update title Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Optimise titles

49. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Optimise meta descriptions
Update meta
description

50. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Rating: 4.9 - 1,000,000 votes - Price range: Premium
Add schema for
rich results
Implement schema markup

51. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Rating: 4.9 - 1,000,000 votes - Price range: Premium
Add schema for
breadcrumbs
Implement schema markup

52. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net > About > People > Emily Potter
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Rating: 4.9 - 1,000,000 votes - Price range: Premium
Add schema for
breadcrumbs
Implement schema markup

53. when did emily potter join the ODN team?
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
Rating: 4.9 - 1,000,000 votes - Price range: Premium
In March 2019, Emily joined the DistilledODN team full time as a consultant
and began speaking at conferences, including BrightonSEO, SearchLeeds, and
SeachLove.
Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Win featured
snippets
Win featured snippets

54. CTR changes will change
your appearance in the SERPs

55. How do you get “there” more?
Not ranking :(

56. How do you get “there” more?

57. Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. An American abroad, Emily
graduated from the University of California, Santa
Barbara in 2016 where she studied creative
writing, sociology and also dabbled in mathematics
and economics for fun.
Update H1
Update H1s

58. Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. She now works full time on the
Distilled ODN team, running lots and lots and lots
of split tests.
Update on-page content
Write new
copy

59. Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. She now works full time on the
Distilled ODN team, running lots and lots and lots
of split tests.
Emily has also been seen speaking at conferences
like BrightonSEO, SearchLeeds, and SearchLove.
Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Update on-page content
Add new
content

60. Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. She now works full time on the
Distilled ODN team, running lots and lots and lots
of split tests.
Emily has also been seen speaking at conferences
like BrightonSEO, SearchLeeds, and SearchLove.
Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Add internal links
Add internal
links

61. When results get fuzzy

62. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Update title Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Optimise titles

63. Google CTR history - Advanced Web
Ranking
Full study here

64. CTR might be a ranking factor

67. This change improved organic traffic

68. This change hurt organic traffic

69. This change had no measurable
impact on organic traffic

70. This allows us to...

71. Conversion
Rate
Revenue
Increase
Make business cases
Gained
Sessions

72. Rescue sessions
&
Get a competitive advantage

73. wordpress templates
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Include quantity of products in titles...
https://colorlib.com › Home › Free Themes
9 Sep 2019 - The best free WordPress theme you can currently get on the market, all
responsive, mobile-ready and easy to use.
60+ Best Free Responsive WordPress Themes 2019 - Colorlib
https://themeisle.com › wordpress-themes
30+ WordPress Themes - 2019's Best WordPress Templates
https://athemes.com › collections › free-wordpress-themes
3 May 2019 - So, without further ado, keep reading to learn more about the best free
WordPress themes on the market. Airi. Sydney. Neve. Talon. Zakra. Hestia. Flash.
Astrid.
Best Free Responsive WordPress Themes 2019 - aThemes

74. wordpress templates
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Include quantity of products in titles...
https://colorlib.com › Home › Free Themes
9 Sep 2019 - The best free WordPress theme you can currently get on the market, all
responsive, mobile-ready and easy to use.
60+ Best Free Responsive WordPress Themes 2019 - Colorlib
https://themeisle.com › wordpress-themes
30+ WordPress Themes - 2019's Best WordPress Templates
https://athemes.com › collections › free-wordpress-themes
3 May 2019 - So, without further ado, keep reading to learn more about the best free
WordPress themes on the market. Airi. Sydney. Neve. Talon. Zakra. Hestia. Flash.
Astrid.
70+ Best Free Responsive WordPress Themes 2019 - aThemes

75. wordpress templates
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Include quantity of products in titles...
https://colorlib.com › Home › Free Themes
9 Sep 2019 - The best free WordPress theme you can currently get on the market, all
responsive, mobile-ready and easy to use.
60+ Best Free Responsive WordPress Themes 2019 - Colorlib
https://themeisle.com › wordpress-themes
30+ WordPress Themes - 2019's Best WordPress Templates
https://athemes.com › collections › free-wordpress-themes
3 May 2019 - So, without further ado, keep reading to learn more about the best free
WordPress themes on the market. Airi. Sydney. Neve. Talon. Zakra. Hestia. Flash.
Astrid.
70+ Best Free Responsive WordPress Themes 2019 - aThemes

76. wordpress templates
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Include quantity of products in titles...
https://colorlib.com › Home › Free Themes
9 Sep 2019 - The best free WordPress theme you can currently get on the market, all
responsive, mobile-ready and easy to use.
60+ Best Free Responsive WordPress Themes 2019 - Colorlib
https://themeisle.com › wordpress-themes
30+ WordPress Themes - 2019's Best WordPress Templates
https://athemes.com › collections › free-wordpress-themes
3 May 2019 - So, without further ado, keep reading to learn more about the best free
WordPress themes on the market. Airi. Sydney. Neve. Talon. Zakra. Hestia. Flash.
Astrid.
70+ Best Free Responsive WordPress Themes 2019 - aThemes

77. Most people are satisfied with
these answers

78. Wait...but why?

79. wordpress templates
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Let’s revisit this
https://colorlib.com › Home › Free Themes
9 Sep 2019 - The best free WordPress theme you can currently get on the market, all
responsive, mobile-ready and easy to use.
60+ Best Free Responsive WordPress Themes 2019 - Colorlib
https://themeisle.com › wordpress-themes
30+ WordPress Themes - 2019's Best WordPress Templates
https://athemes.com › collections › free-wordpress-themes
3 May 2019 - So, without further ado, keep reading to learn more about the best free
WordPress themes on the market. Airi. Sydney. Neve. Talon. Zakra. Hestia. Flash.
Astrid.
70+ Best Free Responsive WordPress Themes 2019 - aThemes

80. This change hurt organic traffic

81. Adding the quantity to the title
hurt organic traffic

82. But why?

83. Did click-through-rate go down?

84. :’(
Did rankings drop?

85. Did we stop ranking for keywords
we had been ranking for?

86. wordpress templates
All Shopping Images News
About 150,000 results (0.35 seconds)
More
CTR is the likely cause...but what if it wasn’t?
https://colorlib.com › Home › Free Themes
9 Sep 2019 - The best free WordPress theme you can currently get on the market, all
responsive, mobile-ready and easy to use.
60+ Best Free Responsive WordPress Themes 2019 - Colorlib
https://themeisle.com › wordpress-themes
30+ WordPress Themes - 2019's Best WordPress Templates
https://athemes.com › collections › free-wordpress-themes
3 May 2019 - So, without further ado, keep reading to learn more about the best free
WordPress themes on the market. Airi. Sydney. Neve. Talon. Zakra. Hestia. Flash.
Astrid.
70+ Best Free Responsive WordPress Themes 2019 - aThemes

87. 9.81 m/s^2 *
time = ?

89. SEO testing relieves us
from all that noise

90. Three levers inform our hypothesis

91. This change improved organic traffic
This change hurt organic traffic
This change had no measurable
impact on organic traffic

92. The result either supports
or disproves your hypothesis

93. To see how this works,
we’ll cover four case studies

94. SEO A/B testing in depth
Case studies to apply understanding
Takeaways

95. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net > About > People > Emily Potter
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Rating: 4.9 - 1,000,000 votes - Price range: Premium
Add schema for
breadcrumbs
Case Study #1: Fixing breadcrumbs

96. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Update title Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Case Study #2: Price in titles

97. Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. She now works full time on the
Distilled ODN team, running lots and lots and lots
of split tests.
Emily has also been seen speaking at conferences
like BrightonSEO, SearchLeeds, and SearchLove.
Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Case Study #3: Internal linking
Add internal
links

98. £60
Shop our over 600 different styles of women’s shoes to complete your
wardrobe. We have heels, boots, trainers, and any other type of shoe
you could imagine. All our shoes come with free shipping and handling.
Dresses Jackets Shoes Activewear Accessories
Women's Shoes
654 results
New In!
Red heels
£55
Teal heels Black heels
£65
Case Study #4: New product page template

99. Case study #1:
Overview

100. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Emily Potter - Consultant - Distilled
https://www.distilled.net > About > People > Emily Potter
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Rating: 4.9 - 1,000,000 votes - Price range: Premium
Add schema for
breadcrumbs
Case Study #1: Fixing breadcrumbs

101. Had run this test previously,
overwhelmingly positive

102. Mens Swimwear | Shorts, Beach Towels & Swimming ...
Men's Swimwear: Swim Shorts, Swimming Trunks & Briefs ...
Men's Swimwear | Swim Shorts & Speedos for Men | ASOS
mens swimwear
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.asos.com › Men › Swimwear › Category
Later, breadcrumb schema broke
https://www.next.co.uk/shop/men/clothing/swimwear
Define your beach style with men's swimwear in printed & plain, while shorts, beach
towels & swimming accessories complete your holiday bag. Next day ...
https://www.sportsdirect.com › Swimming › Mens Swimwear
Discover the range of men's swimwear at ASOS. Explore swim shorts, board shorts and
speedos for men in a variety of colours and lengths. Shop now at ASOS.

103. Case study #1:
Hypothesis

104. What are we changing?
● Fixing breadcrumb schema

105. What levers will we pull?
● Breadcrumbs in SERP will increase CTR

106. ● We will measure organic traffic
How will we know?

107. Case study #1:
Control vs. Variant

108. Define your beach style with women's swimwear in printed & plain, while
beach towels & swimming accessories complete your holiday bag. Next da ...
Control Template Variant Template
Womens Swimwear | Shorts, Beach Towels & Swimm ...
https://www.next.co.uk/shop/women/clothing/swimwear
Mens Swimwear | Shorts, Beach Towels & Swimming ...
https://www.next.co.uk › Shop › Men › Clothing
Define your beach style with men's swimwear in printed & plain, while shorts,
beach towels & swimming accessories complete your holiday bag. Next da ...

109. womens swimwear
Womens Swimwear | Shorts, Beach Towels & Swimming ...
Women's Swimwear: Swim Bikinis, Swimming Suits & To ...
Women's Swimwear | Swim Bikinis & Suits for Women ...
All Shopping Images News
About 150,000 results (0.35 seconds)
https://www.asos.com › Women › Swimwear › Category
More
https://www.next.co.uk/shop/women/clothing/swimwear
Define your beach style with women's swimwear in printed & plain, while
shorts, beach towels & swimming accessories complete your holiday bag.
Next day ...
https://www.sportsdirect.com › Swimming › Mens Swimwear
Discover the range of women's swimwear at ASOS. Explore swim shorts,
board shorts and bikinis for women in a variety of colours and lengths.
Shop now at ASOS.
Mens Swimwear | Shorts, Beach Towels & Swimming ...
Men's Swimwear: Swim Shorts, Swimming Trunks & Briefs ...
Men's Swimwear | Swim Shorts & Speedos for Men | ASOS
https://www.asos.com › Women › Swimwear › Category
https://www.next.co.uk › Shop › Men › Clothing
Define your beach style with men's swimwear in printed & plain, while
shorts, beach towels & swimming accessories complete your holiday bag.
Next day ...
https://www.sportsdirect.com › Swimming › Mens Swimwear
Discover the range of men's swimwear at ASOS. Explore swim shorts,
board shorts and speedos for men in a variety of colours and lengths. Shop
now at ASOS.
mens swimwear
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Control Template Variant Template

110. Case study #1:
Results

114. Define your beach style with women's swimwear in printed & plain, while
beach towels & swimming accessories complete your holiday bag. Next da ...
Control Template Variant Template
Womens Swimwear | Shorts, Beach Towels & Swimm ...
https://www.next.co.uk/shop/women/clothing/swimwear
Mens Swimwear | Shorts, Beach Towels & Swimming ...
https://www.next.co.uk › Shop › Men › Clothing
Define your beach style with men's swimwear in printed & plain, while shorts,
beach towels & swimming accessories complete your holiday bag. Next da ...

115. So what the f*** happened?

117. Why did control have
a higher CTR than variant?

118. Define your beach style with women's swimwear in printed & plain, while
beach towels & swimming accessories complete your holiday bag. Next da ...
Control Template Variant Template
Womens Swimwear | Shorts, Beach Towels & Swimm ...
https://www.next.co.uk/shop/women/clothing/swimwear
Mens Swimwear | Shorts, Beach Towels & Swimming ...
https://www.next.co.uk › Shop › Men › Clothing
Define your beach style with men's swimwear in printed & plain, while shorts,
beach towels & swimming accessories complete your holiday bag. Next da ...

119. Mens Swimwear | Shorts, Beach Towels & Swimming ...
https://www.next.co.uk › Shop › Men › Clothing
Define your beach style with men's swimwear in printed & plain, while shorts,
beach towels & swimming accessories complete your holiday bag. Next da ...
Define your beach style with women's swimwear in printed & plain, while
beach towels & swimming accessories complete your holiday bag. Next da ...
Control Template Variant Template
Womens Swimwear | Shorts, Beach Towels & Swimm ...
https://www.next.co.uk/shop/women/clothing/swimwear

120. https://www.next.co.uk › Shop › Men › Clothing
Mens Swimwear | Shorts, Beach Towels & Swimming ...
Men's Swimwear: Swim Shorts, Swimming Trunks & Briefs ...
Men's Swimwear | Swim Shorts & Speedos for Men | ASOS
mens swimwear
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.asos.com › Men › Swimwear › Category
Variant Template
Define your beach style with men's swimwear in printed & plain, while shorts, beach
towels & swimming accessories complete your holiday bag. Next day ...
https://www.sportsdirect.com › Swimming › Mens Swimwear
Discover the range of men's swimwear at ASOS. Explore swim shorts, board shorts and
speedos for men in a variety of colours and lengths. Shop now at ASOS.

121. Conclusion
● Don’t blindly follow “best practice”
approaches

122. Conclusion
● Consider why you’re implementing a change,
and how it gets you there

123. Define your beach style with women's swimwear in printed & plain, while
beach towels & swimming accessories complete your holiday bag. Next da ...
Control Template Variant Template
Womens Swimwear | Shorts, Beach Towels & Swimm ...
https://www.next.co.uk/shop/women/clothing/swimwear
Mens Swimwear | Shorts, Beach Towels & Swimming ...
https://www.next.co.uk › Shop › Men › Clothing
Define your beach style with men's swimwear in printed & plain, while shorts,
beach towels & swimming accessories complete your holiday bag. Next da ...
How will this change improve CTRs?

124. Case study #2:
Overview

125. best consultant at distilled
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Update title Emily Potter - Consultant - Distilled
https://www.distilled.net/about/people/emily-potter/
Emily joined the Distilled team as an Analyst in February of 2018 with a background in copywriting
and PR. An American abroad, Emily graduated from the ...
Case Study #2: Price in titles

126. Titles can pull all three levers

127. hotel from $10 a night
All Shopping Images News More
But this isn’t a high volume search query...

128. cheap hotels
All Shopping Images News More
You’ll more likely see this

129. Therefore, we’re mostly talking about CTR

130. Case study #2:
Hypothesis

131. What are we changing?
● Adding the price to title

132. What levers will we pull?
● Having the price in SERP will increase CTR

133. ● We will measure organic traffic
How will we know?

134. Case study #2:
Control vs. Variant

135. San Francisco Hotels from $30/night | Find & compare deals
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels | Find & compare great deals | Trivago
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
Control Template Variant Template

136. San Francisco Hotels from $30/night | Find & compare deals |Trivago
https://www.trivago.com › USA › California
San Francisco Hotels | Find & compare deals | Trivago
https://www.trivago.com › USA › California
hotels in san francisco
Hotels in San Francisco from $29/night - Search on KAYAK
The 30 best hotels & places to stay in San Francisco, USA ...
All Shopping Images News
About 150,000 results (0.35 seconds)
https://www.booking.com› USA › California › Visit San Francisco
More
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
https://www.kayak.com › Hotels › United States › California
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the
Bay Hotel San Francisco. Hyatt Regency San Francisco. Holiday Inn
Express Hotel….
Hotels in San Francisco from $29/night - Search on KAYAK
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.booking.com› USA › California › Visit San Francisco
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
https://www.kayak.com › Hotels › United States › California
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the
Bay Hotel San Francisco. Hyatt Regency San Francisco. Holiday Inn
Express Hotel….
hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Control Template Variant Template

137. Case study #2:
Results

141. San Francisco Hotels from $30/night | Find & compare gr ...
Control Template Variant Template
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels | Find & compare great deals | Trivago
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.

142. So what the f*** happened?

143. What levers will we pull?
● Having the price in SERP will increase CTR

144. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
Variant Template
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $29/night - Search on KAYAK
Competitor cheaper

145. Conclusion
● Having the price in the title can hurt CTRs if it
makes competitors more appealing

146. Later, we re-run the test
with updated price data

147. Case study #2:
Test two - Hypothesis

148. What are we changing?
● Adding price to the title

149. What levers will we pull?
● Adding price to title will increase CTR

150. ● We will measure organic traffic
How will we know?

151. Case study #2:
Test two - Control vs. Variant

152. San Francisco Hotels from $34/night | Find & compare deals
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels | Find & compare deals | Trivago
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
Control Template Variant Template

153. San Francisco Hotels from $30/night | Find & compare deals |Trivago
https://www.trivago.com › USA › California
San Francisco Hotels | Find & compare deals | Trivago
https://www.trivago.com › USA › California
hotels in san francisco
Hotels in San Francisco from $29/night - Search on KAYAK
The 30 best hotels & places to stay in San Francisco, USA ...
All Shopping Images News
About 150,000 results (0.35 seconds)
https://www.booking.com› USA › California › Visit San Francisco
More
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
https://www.kayak.com › Hotels › United States › California
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the
Bay Hotel San Francisco. Hyatt Regency San Francisco. Holiday Inn
Express Hotel….
Hotels in San Francisco from $29/night - Search on KAYAK
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.booking.com› USA › California › Visit San Francisco
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
https://www.kayak.com › Hotels › United States › California
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the
Bay Hotel San Francisco. Hyatt Regency San Francisco. Holiday Inn
Express Hotel….
hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
Control Template Variant Template

154. Case study #2:
Test two - Results

158. Control Template Variant Template
San Francisco Hotels from $34/night | Find & compare deals
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels | Find & compare deals | Trivago
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate
from millions of accommodation deals and save with trivago.co.uk.

159. Wait so then,
what the f*** happened?!

160. Test #1 and Test #2
had the same hypothesis

161. What lever will we pull?
● Having the price in SERP will increase CTR

162. But there was a difference:
Test #1 had out of date prices

163. Here’s test #1
day to day in SERPs

164. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
Variant Template
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $29/night - Search on KAYAK

165. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $35/night - Search on KAYAK
Variant Template

166. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $31/night - Search on KAYAK
Variant Template

167. Here’s test #2
day to day in SERPs

168. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $34/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $33/night - Search on KAYAK
Variant Template

169. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $35/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $30/night - Search on KAYAK
Variant Template

170. hotels in san francisco
All Shopping Images News
About 150,000 results (0.35 seconds)
More
https://www.booking.com › USA › California › Visit San Francisco
San Francisco's best hotels with breakfast. Seaside Inn. Comfort Inn By the Bay Hotel
San Francisco. Hyatt Regency San Francisco. Holiday Inn Express Hotel….
The 30 best hotels & places to stay in San Francisco, USA ...
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $31/night | Find & compare deals | Trivago
https://www.kayak.com › Hotels › United States › California
Hotels in San Francisco from $34/night - Search on KAYAK
Variant Template

171. What if another lever
was also being pulled?

172. Theory one

173. https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
Are negative user signals a ranking factor?

174. https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
Are negative user signals a ranking factor?

175. Book now from
$40/night
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
www.trivago.com/san-francisco-hotels
Are negative user signals a ranking factor?

176. Book now from
$40/night
https://www.trivago.com › USA › California
Compare the prices of 4233 hotels in San Francisco, USA. Find the ideal rate from
millions of accommodation deals and save with trivago.co.uk.
San Francisco Hotels from $30/night | Find & compare deals | Trivago
www.trivago.com/san-francisco-hotels
Prices don’t match
Are negative user signals a ranking factor?

177. Theory two

178. Can having the incorrect
price directly impact rankings?

179. “
For these reasons, we regularly
check your website to ensure
that the information you provided
matched the information on your
website.
”

180. It’s plausible Google
would look out for this

181. Conclusion
● Having an accurate price in your titles
matters

182. Case study #3:
Overview

183. Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. She now works full time on the
Distilled ODN team, running lots and lots and lots
of split tests.
Emily has also been seen speaking at conferences
like BrightonSEO, SearchLeeds, and SearchLove.
Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Case Study #3: Internal linking
Add internal
links

184. Why internal linking?

185. PageRank

186. How can we measure internal
linking in SEO A/B tests?

187. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

188. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
Main Category Pages

189. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
Sub-category Pages

190. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

191. VariantControl
Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

192. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
This becomes our testing section
VariantControl

193. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
This becomes our first measurement section
VariantControl

194. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
This becomes our second measurement section
VariantControl

195. Emily joined the Distilled team as an Analyst in
February of 2018 with a background in
copywriting and PR. She now works full time on the
Distilled ODN team, running lots and lots and lots
of split tests.
Emily has also been seen speaking at conferences
like BrightonSEO, SearchLeeds, and SearchLove.
Consulting Services Conferences Resources About Us Contact Us
Emily Potter
Consultant
Case Study #3: Internal linking
Add internal
links

196. Two examples of the same
test on different sites

197. Case study #3:
Hypothesis

198. What are we changing?
● Adding sub category links to main category
pages

199. ● More links to sub category pages will
increase link equity
What levers will we pull?

200. ● More links from main category pages will
improve relevancy signals
What levers will we pull?

201. ● More links from main category pages may
decrease link equity
What levers will we pull?

202. How will we know?
● We will measure organic traffic to main
categories, sub categories, & overall impact

203. Case study #3:
Control vs. Variant

204. VariantControl
Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

205. Case study #3:
Results

209. VariantControl
Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

210. What happened to
the different page types?

211. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

212. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

213. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

214. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

215. Conclusion
● Internal links have a positive impact on
rankings for pages receiving links

216. Conclusion
● Internal links have a negative impact on
rankings for pages sending out links (?!?)

217. We get a chance to run this test
again on a different site

218. VariantControl
Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

219. Case study #3:
Test two - Results

223. VariantControl
Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads

224. What happened to
the different page types?

225. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

226. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

227. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

228. Mammals Reptiles AmphibiansBirds
Cats DogsBears Snakes TurtlesLizards Owls DucksFinches Frogs NewtsToads
VariantControl

229. Wait so then,
what the f*** happened?!

230. Test #1 and Test #2
had the same hypothesis

231. So we have to compare
the two websites

232. Website #1:
E-commerce

233. VariantControl
Home
Furniture
Dresses
Nursing &
Feeding
Patio
Furniture
Living
Room
Furniture
Nursery
Furniture
Bedroom
Furniture
Patio
Seats
Patio
Tables
Patio
Heaters
Baby
Formula
Baby
Bibs
Baby
Food
Black
Dresses
Party
Dresses
Work
Dresses

234. Website #2:
Grocery

235. Fish &
Seafood
Cooking
Ingredients
Chips &
Potatoes
Fish Pies
Fish
Fillets
Fish
Fingers
Chips &
Fries
WedgesPotatoes Ham BurgersTurkey
Herbs &
Spices
Oil &
Dressing
Gravy &
Stuffing
Meats &
Deli

236. Theory one

237. Nursing &
Feeding
Patio
Furniture
Patio
Seats
Patio
Tables
Patio
Heaters
Baby
Cutlery
Baby
Weaning
Baby
Bottles
Baby
Formula
Baby
Bibs
Baby
Food
Patio
Decor
Patio
Sets
Patio
Daybeds
E-commerce website included more links

238. Nursing &
Feeding
Patio
Furniture
Patio
Seats
Patio
Tables
Patio
Heaters
Baby
Cutlery
Baby
Weaning
Baby
Bottles
Baby
Formula
Baby
Bibs
Baby
Food
Patio
Decor
Patio
Sets
Patio
Daybeds
E-commerce site stuffed keywords in anchor text

239. Theory two

240. Grocery website had stronger relevancy signals
Fish &
Seafood
Cooking
Ingredients
Chips &
Potatoes
Fish Pies
Fish
Fillets
Fish
Fingers
Chips &
Fries
WedgesPotatoes Ham BurgersTurkey
Herbs &
Spices
Oil &
Dressing
Gravy &
Stuffing
Meats &
Deli

241. Theory three

242. Grocery website created pathways for users
Fish &
Seafood
Cooking
Ingredients
Chips &
Potatoes
Fish Pies
Fish
Fillets
Fish
Fingers
Chips &
Fries
WedgesPotatoes Ham BurgersTurkey
Herbs &
Spices
Oil &
Dressing
Gravy &
Stuffing
Meats &
Deli

243. Could this send
positive user signals?

244. Conclusion
● Internal linking can boost rankings for pages
receiving links

245. Conclusion
● Internal linking can benefit pages sending out
links, but not universally

246. Case study #4:
Overview

247. £60
Shop our over 600 different styles of women’s shoes to complete your
wardrobe. We have heels, boots, trainers, and any other type of shoe
you could imagine. All our shoes come with free shipping and handling.
Dresses Jackets Shoes Activewear Accessories
Women's Shoes
654 results
New In!
Red heels
£55
Teal heels Black heels
£65
Case Study #4: New product page template

248. Client had positive CRO
test with new template

249. They wanted to confirm
it wouldn’t hurt SEO

250. The catch:
This new template used React

251. What is React?
● React is a JavaScript framework that’s
popular with developers

255. Case study #4:
Hypothesis

256. What are we changing?
● New product page template utilising
React framework

257. What levers will we pull?
● Improved performance will increase organic
rankings

258. ● We will measure organic traffic
How will we know?

259. Case study #4:
Control vs. Variant

260. $60
Shop our over 600 different styles of women’s shoes to complete
your wardrobe. We have heels, boots, trainers, and any other
type of shoe you could imagine. All our shoes come with free
shipping and handling.
Dresses Jackets Shoes Accessories
Women's Shoes
654 results
New In!
Red heels
$55
Teal
heels
$75
Black heels
$60
Shop our over 400 different styles of men’s sweaters to
complete your wardrobe. We have sweaters of all colours and
material. All our sweaters come with free shipping and handling.
Dresses Jackets Shoes Accessories
Men’s Sweaters
437 results
New In!
Red sweater
$55
Teal sweater
$75
Green sweater
Control Template Variant Template

261. Case study #4:
Results

265. $60
Shop our over 600 different styles of women’s shoes to complete
your wardrobe. We have heels, boots, trainers, and any other
type of shoe you could imagine. All our shoes come with free
shipping and handling.
Dresses Jackets Shoes Accessories
Women's Shoes
654 results
New In!
Red heels
$55
Teal
heels
$75
Black heels
$60
Shop our over 400 different styles of men’s sweaters to
complete your wardrobe. We have sweaters of all colours and
material. All our sweaters come with free shipping and handling.
Dresses Jackets Shoes Accessories
Men’s Sweaters
437 results
New In!
Red sweater
$55
Teal sweater
$75
Green sweater
Control Template Variant Template

266. What the f*** happened?!

267. We haven’t changed the SERP

268. We haven’t changed the SERP

269. We haven’t added new content

270. We haven’t added new content

271. That means there was a ranking increase

272. How?

273. Two important points

274. $60
Shop our over 600 different styles of women’s shoes to complete
your wardrobe. We have heels, boots, trainers, and any other
type of shoe you could imagine. All our shoes come with free
shipping and handling.
Dresses Jackets Shoes Accessories
Women's Shoes
654 results
New In!
Red heels
$55
Teal
heels
$75
Black heels
$60
Shop our over 400 different styles of men’s sweaters to
complete your wardrobe. We have sweaters of all colours and
material. All our sweaters come with free shipping and handling.
Dresses Jackets Shoes Accessories
Men’s Sweaters
437 results
New In!
Red sweater
$55
Teal sweater
$75
Green sweater
Control Template Variant Template
First, Googlebot could process without JS

275. $60
Shop our over 600 different styles of women’s shoes to complete
your wardrobe. We have heels, boots, trainers, and any other
type of shoe you could imagine. All our shoes come with free
shipping and handling.
Dresses Jackets Shoes Accessories
Women's Shoes
654 results
New In!
Red heels
$55
Teal
heels
$75
Black heels
$60
Shop our over 400 different styles of men’s sweaters to
complete your wardrobe. We have sweaters of all colours and
material. All our sweaters come with free shipping and handling.
Dresses Jackets Shoes Accessories
Men’s Sweaters
437 results
New In!
Red sweater
$55
Teal sweater
$75
Green sweater
Control Template Variant Template
Second, the variant now loads faster

276. ● Increased page speed improves organic
rankings (because it improves user signals?)
Conclusion

277. ● React and other JavaScript frameworks can
be positive if implemented with SEO in mind
Conclusion

278. SEO A/B testing in depth
Case studies to apply understanding
Takeaways

279. Takeaway #1:
SEO A/B testing relieves us from noise
created by interactions between our 3 levers

280. Takeaway #2:
Many SEO “best practices” fail. SEO A/B
testing rescues would-be-lost sessions

281. Don’t be sheep.

282. Don’t be sheep.

283. Takeaway #3:
Even tried and true “best practices” can go
wrong if we forget why we’re doing it

284. Takeaway #4:
Don’t blindly add internal links

285. Takeaway #5:
Page speed impacts rankings. JS frameworks
can help if done with SEO in mind

286. Emily Potter -
Distilled
http://bit.ly/2IQNWhI
@em_potter
www.linkedin.com/in/emargaretpotter/