An SEO crawler is a staple tool in any SEO pro's toolkit, but have you ever considered using such a tool to seek out powerful CRO opportunities? Luke is an out and out ecommerce expert who'll be sharing tips and takeaways on how to leverage custom extraction with an SEO crawler to improve ecommerce conversion rates and enhance UX.

1. POWERFUL TIPSCRO
@MrLukeCarthy | #Searchlove
Using a SEO web crawler

2. @MrLukeCarthy | #Searchlove
Ecommerce Consultant
& Founder
Worked
with:

3. @MrLukeCarthy | #Searchlove

4. As a SEO, I’ve failed…
(strictly speaking)
@MrLukeCarthy | #Searchlove

5. @MrLukeCarthy | #Searchlove
Organic traffic
is virtually flat..

6. @MrLukeCarthy | #Searchlove
However, online sales
are up by 175%

7. @MrLukeCarthy | #Searchlove

8. @MrLukeCarthy | #Searchlove
More traffic ≠ more sales

9. @MrLukeCarthy | #Searchlove
More sales ≠ more traffic

14. M o d e r n M u l t i p u r p o s e s p r e s e n t a t i o n t e m p l a t e
Unique & Creative

15. 236,000 indexed
internal search URLs
@MrLukeCarthy | #Searchlove

16. Approx. 6.8million
visitors every month (AHREFS)
@MrLukeCarthy | #Searchlove

20. ‘Zero results’
internal searches do not…
@MrLukeCarthy | #Searchlove

25. SEARCH URLs
CUSTOM
EXTRACTION
> >
@MrLukeCarthy | #Searchlove

26. I crawled just a 4%sample
of the 236KURLs…
@MrLukeCarthy | #Searchlove

27. …and surfaced 296URLs
returning ‘no results found’
@MrLukeCarthy | #Searchlove

28. That’s 190Kpeople
seeing ‘no results found’ pages
every month
@MrLukeCarthy | #Searchlove

29. That’s a HUGE bunch of missed
CRO opportunities
@MrLukeCarthy | #Searchlove

30. Best Buy done fu$#&d up…
@MrLukeCarthy | #Searchlove

31. Well, a little..
@MrLukeCarthy | #Searchlove

33. Approx. 15,000visits every month
@MrLukeCarthy | #Searchlove

34. Just 1item found
57 inch tv
48 inch smart tv
Solar eclipse glasses
Nvidia shield
Tv remove control replacement
@MrLukeCarthy | #Searchlove

35. How is all this impacting Best Buy?
@MrLukeCarthy | #Searchlove

36. At least 190Kpeople aren’t finding what
they’re looking for
@MrLukeCarthy | #Searchlove

37. Could be 1000sof potential places where
demand is outstripping supply
@MrLukeCarthy | #Searchlove

38. Not providing a seamless experience
for non-commerce search queries
@MrLukeCarthy | #Searchlove

39. Not selling lines that
customers want to buy
@MrLukeCarthy | #Searchlove

40. 30% revenue increase via
Internal search optimisation
@MrLukeCarthy | #Searchlove

41. Ok, now tell us how you got
all this cool shit with
CUSTOM
EXTRACTION
@MrLukeCarthy | #Searchlove

42. “Custom extraction allows you to
scrape any data from the HTML
of web pages when crawling”
https://www.screamingfrog.co.uk/web-
scraping/

48. Good reads on custom extraction
Uproer
https://uproer.com/articles/screaming-frog-custom-extraction-xpath-regex/
LinkedIn Article – Brian Shumway
https://www.linkedin.com/pulse/custom-extraction-screaming-frog-xpath-csspath-
brian-shumway
PMG
https://www.pmg.com/blog/how-to-use-xpath-in-screaming-frog/
Search Engine Journal
https://www.searchenginejournal.com/regex-seo-website-data-extraction/234999/
@MrLukeCarthy | #Searchlove

49. What else can you do with
custom extraction?
@MrLukeCarthy | #Searchlove

50. Extract your
product recommendations
@MrLukeCarthy | #Searchlove

51. “35% of what consumers purchase
on Amazonand 75% of what they
watch on Netflixcome from
product recommendations.”
https://www.mckinsey.com/industries/retail/
our-insights/how-retailers-can-keep-up-with-
consumers

52. Understand what direct ‘bolt on’ products
customers proactively buy at scale
@MrLukeCarthy | #Searchlove

53. The first few recs are often the
most popular / best selling
@MrLukeCarthy | #Searchlove

54. Pay close attention to the product price
vs the first recs price
@MrLukeCarthy | #Searchlove

55. Really powerful when scraping recs
from your direct competitors
@MrLukeCarthy | #Searchlove

56. Even more so when competitors have
multi-dimensional recs
@MrLukeCarthy | #Searchlove

58. 1500 lines crawled…
36%
have no
recommendations
at all
86%
Were all priced the
same or higher
80+
%
Appeared to be
alternatives rather than
complimentary

59. 1500 lines crawled…
100%
of out of stock items
Don’t show
recommendations

60. 1000 lines crawled…
100%
have recommendations
showing
56%
Were all priced the
same or higher

61. 1500 lines crawled…
100%
Appeared to be alternatives
rather than
complimentary

62. 1500 lines crawled…
100%
Appeared to be
complimentary rather than
alternatives

63. 1500 lines crawled…
Shoes = Shoes

64. 1500 lines crawled…
Jeans = Shirts

65. 1500 lines crawled…
Shirts = Jeans

66. 1500 lines crawled…
Shorts = Trainers/Sneakers

67. Identify your worst / best
selling products in every category
@MrLukeCarthy | #Searchlove

68. Best selling products are often the first
items in the category
@MrLukeCarthy | #Searchlove

69. Competitive ecommerce
Intelligence to boost conversion
@MrLukeCarthy | #Searchlove

70. Identify your competitor’s top
(and worst) selling products
@MrLukeCarthy | #Searchlove

71. Work out your competitors
search URL pattern
@MrLukeCarthy | #Searchlove

72. example.com/search?t=[KEYWORD]
@MrLukeCarthy | #Searchlove

73. Scrape results for price, reviews
and stock and compare to your own
@MrLukeCarthy | #Searchlove

75. @MrLukeCarthy | #Searchlove

78. Vet your category navigation URLs
in main menus
@MrLukeCarthy | #Searchlove

83. SEXY CUSTOM
EXTRACTION
@MrLukeCarthy | #Searchlove

84. Log files
Backlink data
XML sitemaps
PageSpeed Insights
@MrLukeCarthy | #Searchlove

85. Log files
Backlink data
XML sitemaps
PageSpeed Insights
@MrLukeCarthy | #Searchlove
Find patterns
amongst
lines that
don’t sell

86. Log files
Backlink data
XML sitemaps
PageSpeed Insights
@MrLukeCarthy | #Searchlove
Identify what
reviews
typically
drive sales

87. @MrLukeCarthy | #Searchlove
It’s not just eCommerce…

88. Sitebulb Screaming Frog DeepCrawl OnCrawl
Custom
extraction
NO ( but soon) YES YES YES
Google Analytics YES YES YES YES
Search Console YES YES YES YES
Majestic SEO NO NO YES YES
AHREFS NO YES NO Coming soon
Logfile Analysis NO YES – separate product YES
Yes – separate
product
PageSpeed
Insights
NO Separate crawl YES (JS injection) NO

89. @MrLukeCarthy | #Searchlove
Lukecarthy.com/searchlove
I wrote a thing:

90. Thank you!
@MrLukeCarthy
/in/lukecarthy
lukecarthy.com