Avoid duplicate content and don’t leave money on the table with unoptimized groups of pages linked by canonical declarations! Particularly in ecommerce, you can increase Google's confidence by making sure your groups of product URLs are perfectly canonicalized and clear to search engines. In this keynote, we will use Python in Google Cloud Functions to reorganize canonical clusters automatically and maximize SEO performance. 1. We will use OnCrawl to find duplicate product cluster by SKU 2. Then we will pull production variant search traffic using SEMrush (color, size, etc) 3. We will use an algorithm to regroup the clusters based on search demand 4. We will automate the whole process using Cloud Functions and Pub/sub queues Join Hamlet Batista to look at ways to automate canonicals for ecommerce sites in order to improve product visibility.

1. Automated Duplicate
Content Consolidation with
Google Cloud Functions

2. Speaking today /
Présenté par
Automating Google
Lighthouse
Hamlet Batista // RankSense
slidehare.net/hamletbatista
@hamletbatista

3. https://jamesclear.com/marginal-gains

4. Agenda
➢Finding marginal but
repeatable success
➢Scaling it with automation

5. Cruiseline.com
Success Story

9. ➢ No www to non-www
redirects
➢ No canonicals
➢ Redundant parameter
URLs
➢ Only 1.40% of
indexed pages with
search clicks (out of
+300k pages)

11. The Google SEO
Scorecard Report

13. ➢ Duplicate content
consolidation can be
executed relatively quickly,
as it requires a small set of
technical changes
➢ You will likely see improved
rankings within weeks after
the corrections are in place
➢ New changes and
improvements to your site
are picked up faster by
Google

14. ➢ Natzir found the
total traffic to
pages ranking
for the same
keyword was
less than when
consolidated
with redirects
➢ Same idea but
from a
keywords’
perspective
https://www.youtube.com/watch?v=zI_jkhSyAew

15. Cruiseline.com
Reverse
Engineering

16. ➢ Finding repeatable success
➢ Searching for a machine
learning model to connect
new visits to technical SEO
changes
➢ We focused on the impact
of links, indexing, and
canonical clustering

18. Our best predictive model
achieved 85% test accuracy
➢ Canonicalization drives
repeatable success
➢ The size of the canonical
cluster turned out to be a
strong predictor

19. One oversimplified way to
think about a machine
learning model is to
picture a linear regression
function in Excel/Sheets.
We predicted new users
(Y) within canonicalized
clusters dependent on the
size of the clusters (X).
Machine Learning 101
https://bit.ly/3lGyeqA

20. To Canonicalize
or Not to
Canonicalize

21. Current canonical clustering is
mostly self-referential (orange)
Every product variant
canonicalizes to itself.

22. Their optimal canonical setup is the
inverse.
Most clusters should canonicalize to one
product “leader”

23. For some products, people
specific the color they want
directly in Google. But, for other
products, they don’t.
They decide the color they want
after seeing the options
available in the site.

24. https://bit.ly/36ZxXel

25. Technical Plan
➢ Build clusters using OnCrawl
➢ Get search demand using SEMrush
➢ Canonicalization algorithm
➢ Experiment on CDN using RankSense
➢ Automate everything using Cloud Functions and
Pub/sub queues

26. Coupled vs
Decoupled
Systems

28. Pub/Sub is an asynchronous
messaging service that
decouples services that
produce events from services
that process events.
It allows us to connect
OnCrawl, SEMrush, and
RankSense asynchronously to
complete a custom workflow.

30. Cloud Scheduler acts as a
single pane of glass, allowing us
to manage all our automation
tasks from one place.
It allows us to trigger our custom
workflow on recurring times as
search demand changes with
seasons.

31. Clustering with
OnCrawl

37. Search Demand
Tracking with
SEMrush

39. ➢ Cloud Scheduler triggers
OnCrawl Cloud Function
which uploads each craw
export to Cloud Storage
➢ Cloud Storage update
triggers SEMrush Cloud
Function which then exports
search demand data to
Cloud Storage

41. Canonicalization
Algorithm

42. ➢ We are going to perform an
intermediate step and force
all product groups to
canonicalize to the “leader”
URL in the group.
➢ The “leader” could be the
URL with most search
traffic, more
internal/external links or
most frequently crawled

44. We end up with one cluster that
we need to update, which
means that David Yurman is
leaving a lot of money on the
table with their current setup
that relies on self-referential
canonicals.

45. Deploying to
Cloudflare’s CDN
with RankSense

47. We are going to use the
RankSense API to publish our
new canonical clusters as
experiments in the Cloudflare
CDN
https://bit.ly/3jWm4JP

48. ➢ We automatically populate
a Google Sheet with the
changes
➢ We submit the Sheet to
RankSense’s
PRODUCTION environment

50. Resources to Learn More
➢ Python code covered in this presentation
https://github.com/ranksense/weloveseo
➢ Advanced Duplicate Content Consolidation with
Python
https://www.searchenginejournal.com/advanced-
duplicate-content-consolidation-python/314471/
➢ Cloud Functions https://cloud.google.com/functions
➢ Google PubSub https://cloud.google.com/pubsub
➢ Introduction to Python for SEO Pros
https://www.searchenginejournal.com/introduction-
to-python-seo-spreadsheets/342779/

51. Thank you!