Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling & Indexation Summit)

Crawling & Indexing for
eCommerce
eCommerce
Sam Taylor, Technical SEO Consultant
@_spamtaylor
#brightonSEO

@_spamtayl
or#brightonSE
O
A bit of background
about me
(it’s my first time speaking!)

@_spamtayl
or#brightonSE
O
8 years of agency
experience

@_spamtayl
or#brightonSE
O
Worked with a lot of brands in
this time

@_spamtayl
or#brightonSE
O
Big fan of eCommerce

@_spamtayl
or#brightonSE
O
Big fan of eCommerce
Personally & Professionally

@_spamtayl
or#brightonSE
O
Amazon driver at my house
for the 4th time this week

@_spamtayl
or#brightonSE
O
Why professionally?

@_spamtayl
or#brightonSE
O
1) eCommerce websites
generally have a lot of “nuts
and bolts” tech work to do

@_spamtayl
or#brightonSE
O
2) This work can often have a
direct impact on SEO

@_spamtayl
or#brightonSE
O
3) Work I do + good
results
= job satisfaction

@_spamtayl
or#brightonSE
O
The Current State of
eCommerce

@_spamtayl
or#brightonSE
O
Pre-COVID, UK eCommerce
industry was expected to
grow by
11% in 2020
- Edge by Ascential

@_spamtayl
or#brightonSE
O
Now expected to
grow by 19%

@_spamtayl
or#brightonSE
O
£73.6 billion

@_spamtayl
or#brightonSE
O
£73.6 billion
£78.9 billion

@_spamtayl
or#brightonSE
O
Estimated to have boosted
eCommerce industry by 4-6 years
- Forbes

@_spamtayl
or#brightonSE
O
Amazon UK estimated an
extra £2 billion in 2020
due to COVID

@_spamtayl
or#brightonSE
O
Tesco estimated an extra
£1.7 billion in 2020 due to
COVID

@_spamtayl
or#brightonSE
O
Sainsbury’s estimated an
extra £0.6 billion in 2020
due to COVID

@_spamtayl
or#brightonSE
O
Why should you care?

@_spamtayl
or#brightonSE
O
You should care if you
work in eCommerce

@_spamtayl
or#brightonSE
O
Digital marketing is going to get
even more competitive

@_spamtayl
or#brightonSE
O
So what can you to get the most
out of your SEO?

@_spamtayl
or#brightonSE
O
In my experience, some the biggest
factors that influence positive
rankings for eCommerce sites
are….

@_spamtayl
or#brightonSE
O
Internal linking and anchor text
in category content.

@_spamtayl
or#brightonSE
O
Proper keyword optimisation in
titles & H1s.

@_spamtayl
or#brightonSE
O
and…

@_spamtayl
or#brightonSE
O
Crawl budget.

@_spamtayl
or#brightonSE
O
What is crawl budget?

@_spamtayl
or#brightonSE
O
Crawl budget is the number of
times Google will crawl your
site on any given day.

@_spamtayl
or#brightonSE
O
It can vary.

@_spamtayl
or#brightonSE
O
It is expensive for Google.

@_spamtayl
or#brightonSE
O
If Google thinks the content
crawled is useful, they will
place it in their index.

@_spamtayl
or#brightonSE
O
Think of the index as a
library of information.

@_spamtayl
or#brightonSE
O
When a query is typed into Google
search, the algorithm will generate a
list of results it thinks is most relevant
from what is stored in the index.

@_spamtayl
or#brightonSE
O
We know this as rankings.

@_spamtayl
or#brightonSE
O
So it’s important to make
sure your content is
indexed. Obviously.

@_spamtayl
or#brightonSE
O
It’s good practice to make
sure your site is crawled
efficiently.

@_spamtayl
or#brightonSE
O
eCommerce sites can be big.

@_spamtayl
or#brightonSE
O
Very big.

@_spamtayl
or#brightonSE
O
• Hundreds of categories

@_spamtayl
or#brightonSE
O
• Thousands of products

@_spamtayl
or#brightonSE
O
• Thousands of products
• Millions of filters

@_spamtayl
or#brightonSE
O
Here’s an example of one of my
current clients

@_spamtayl
or#brightonSE
O
Came to us seeing negative
impact from Google updates

@_spamtayl
or#brightonSE
O
Our goal was to reverse that
trend (and continue it!)

@_spamtayl
or#brightonSE
O
We noticed early that the site
had … a little issue.

@_spamtayl
or#brightonSE
O
36,700,000 excluded
vs.
13,700 valid

@_spamtayl
or#brightonSE
O
There’s an issue with
crawling.

@_spamtayl
or#brightonSE
O
Millions of parameter combinations
& long query strings on category
pages caused by faceted navigation.

@_spamtayl
or#brightonSE
O
A miniscule
fraction of them…

@_spamtayl
or#brightonSE
O
And Google was allowed to
crawl them.

@_spamtayl
or#brightonSE
O
This meant that the pages we
did want indexed weren’t
prioritised.

@_spamtayl
or#brightonSE
O
The URLs contained canonical tags
so were not indexed.

@_spamtayl
or#brightonSE
O
But this didn’t cut it.

@_spamtayl
or#brightonSE
O
How did we fix it?

@_spamtayl
or#brightonSE
O
Since they weren’t indexed and
didn’t need to be, we included
parameters in robots.txt

@_spamtayl
or#brightonSE
O
A very, very quick fix.

@_spamtayl
or#brightonSE
O
Products could still be discovered
onsite through links in paginated
category pages.

@_spamtayl
or#brightonSE
O
We did not block these.

@_spamtayl
or#brightonSE
O
The result?

@_spamtayl
or#brightonSE
O
No longer crawling 1,000,000+
URLs per day

@_spamtayl
or#brightonSE
O
Indexed pages went from
13,000 to 22,000

@_spamtayl
or#brightonSE
O
More pages in the index
= more opportunities to rank

@_spamtayl
or#brightonSE
O
More pages in the index
= more opportunities to rank
Especially for long-tail keywords!

@_spamtayl
or#brightonSE
O
A site crawl with robots.txt exclusions
had ~26,000 URLs
(not all your pages will be indexed, and that’s OK!)

@_spamtayl
or#brightonSE
O
And search visibility doubled
following a core update

@_spamtayl
or#brightonSE
O
Successfully reversing the
negative trend we saw.

@_spamtayl
or#brightonSE
O
Me, circa Jan 2020

@_spamtayl
or#brightonSE
O
Apart from some title and
content changes…

@_spamtayl
or#brightonSE
O
This was probably the biggest
onsite change we did with the
resource we had.

@_spamtayl
or#brightonSE
O
Optimising for crawl budget is
important

@_spamtayl
or#brightonSE
O
It will give Google consistent
indexing signals

@_spamtayl
or#brightonSE
O
It won’t dilute internal link equity.

@_spamtayl
or#brightonSE
O
So how can you apply this to your
site?

@_spamtayl
or#brightonSE
O
Google says a ‘substantial’
segment of site owners should
care about crawl budget...

@_spamtayl
or#brightonSE
O
…but the ‘vast majority’ don’t
need to.

@_spamtayl
or#brightonSE
O
Who doesn’t need to worry?

@_spamtayl
or#brightonSE
O
Small eCommerce site with few
parameter combinations?

@_spamtayl
or#brightonSE
O
You probably don’t need worry
about crawl budget.

@_spamtayl
or#brightonSE
O
You could noindex or canonicalise
these URLs as a solution for
indexing.

@_spamtayl
or#brightonSE
O
This will still use crawl budget.

@_spamtayl
or#brightonSE
O
But it shouldn’t have an impact on
your site performance.

@_spamtayl
or#brightonSE
O
They shouldn’t be indexed.
If you are worried…

@_spamtayl
or#brightonSE
O
Check your log files or check
Google Search Console.

@_spamtayl
or#brightonSE
O
If you notice a large ratio of
excluded to valid (Search Console)

@_spamtayl
or#brightonSE
O
Duplicate, submitted URL not selected as canonical

@_spamtayl
or#brightonSE
O
Crawled - currently not indexed

@_spamtayl
or#brightonSE
O
Crawled - currently not indexed
Duplicate, Google chose different canonical than user

@_spamtayl
or#brightonSE
O
Or a lot of hits from Googlebot to
faceted URLs in log files that are
duplicate or low quality…

@_spamtayl
or#brightonSE
O
Then you may want think about
using robots.txt

@_spamtayl
or#brightonSE
O
eCommerce site that has filters
for users, but it doesn’t change
the URL?

@_spamtayl
or#brightonSE
O
gymshark.com/collections/hoodies-jackets/womens

@_spamtayl
or#brightonSE
O
Don’t worry about crawl budget.

@_spamtayl
or#brightonSE
O
Don’t worry about crawl budget.
(but maybe look at implementing optimised
facets to rank for long-tail keywords!)

@_spamtayl
or#brightonSE
O
Large or Medium eCommerce site
that has lots of URLs & filters?

@_spamtayl
or#brightonSE
O
More than likely, you need to keep
an eye on your crawl budget.

@_spamtayl
or#brightonSE
O
adidas.co.uk/women-black-ozweego-shoes

@_spamtayl
or#brightonSE
O
adidas.co.uk/women-black-ozweego-shoes
Disallow: /*-*-*-*-*
adidas.co.uk/women-black-breathable-ozweego-shoes
Not crawled
Crawled

@_spamtayl
or#brightonSE
O
What not do to.
(a.k.a what I’ve seen some sites do in the past)

@_spamtayl
or#brightonSE
O
1) Don’t use noindex: directive in
robots.txt

@_spamtayl
or#brightonSE
O
1) Don’t use noindex: directive in
robots.txt
It doesn’t work anymore. Use a meta
tag or HTTP header.

@_spamtayl
or#brightonSE
O
2) Don’t use nofollow in internal
links

@_spamtayl
or#brightonSE
O
2) Don’t use nofollow in internal
links
Google have said that they can now use this as a
hint for crawling & indexing. It probably won’t be
effective.

@_spamtayl
or#brightonSE
O
3) Don’t rely on URL Parameters
in Google Search Console

@_spamtayl
or#brightonSE
O
3) Don’t rely on URL Parameters
in Google Search Console
This will probably be depreciated soon enough.
I’ve personally not seen it work well for a while.

@_spamtayl
or#brightonSE
O
4) Remember that something may
works for one site but won’t
always work for your site.

@_spamtayl
or#brightonSE
O
4) Remember that something may
works for one site but won’t
always work for your site.
“It depends”.

@_spamtayl
or#brightonSE
O
I’ve worked on sites in the same
group, same CMS, very similar
but both have acted completely
different.

@_spamtayl
or#brightonSE
O
Try to find what works for your
site.

@_spamtayl
or#brightonSE
O
Test things and apply your
knowledge.

@_spamtayl
or#brightonSE
O
Be open-minded.

@_spamtayl
or#brightonSE
O
Don’t be afraid to experiment.

eCommerce
@_spamtaylor
Thank you
#brightonSEO

Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling & Indexation Summit)

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling & Indexation Summit)

Similar to Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling & Indexation Summit) (20)

Recently uploaded

Recently uploaded (20)

Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling & Indexation Summit)