Talk given at SEOntheBeach 2022, discussing what we did at Envato Elements and what worked for us in terms of crawl capacity management. Basically we wanted to show how you can signal Google into crawling and ingesting more of the site when the crawl capacity doesnt change.

1. SOB 2022
Crawl capacity management
on Envato Elements
Gastón Riera - @gastonriera

2. Heads up..
Slides in English 󰎉
Speako en Español 󰎆
Gastón Riera - @gastonriera

3. At the end I'll share a 90%
discount for Elements! ☺
Gastón Riera - @gastonriera

4. That's how I used to look,
all well dressed and all.
Gastón Riera - @gastonriera
Gastón Riera

5. Everything you need to get your
creative projects done.
Gastón Riera - @gastonriera
The big names:
Other very cool products:

6. Gastón Riera - @gastonriera
The two things I like the most about working at envato:
- Being sustainable and caring about the community
- Fully remote (ANZ/MX) and working from abroad

7. Let's get into SEO 🔎
Gastón Riera - @gastonriera

8. The problem:
A big part of the site was
not being indexed 👎
Gastón Riera - @gastonriera

9. ☹
Gastón Riera - @gastonriera

10. Gastón Riera - @gastonriera
It looked like Google was not recrawling a
large amount of pages!

11. Gastón Riera - @gastonriera
And taking very long to recrawl
some pages

12. Was Google actually
crawling the site?🤔
Gastón Riera - @gastonriera

13. Gastón Riera - @gastonriera
Yes, it was👍

14. But, it wasn't getting to
crawl the entire site 😕
Gastón Riera - @gastonriera

15. And, why was that?󰤈
Gastón Riera - @gastonriera

16. No idea
Gastón Riera - @gastonriera

17. Just kidding 😆
Gastón Riera - @gastonriera

18. We came up with two
theories to work on 😎
Gastón Riera - @gastonriera

19. - Content quality
- Internal linking
Gastón Riera - @gastonriera
We needed to work on

20. Gastón Riera - @gastonriera
We needed to work on
I'll get to them in a bit.
- Content quality
- Internal linking

21. What did we do? 🤔
Gastón Riera - @gastonriera

22. The basics!
Gastón Riera - @gastonriera

23. The basics!
- Noindex
Gastón Riera - @gastonriera

24. The basics!
- Noindex
- Redirects
Gastón Riera - @gastonriera

25. The basics!
- Noindex
- Redirects
- Nofollow
Gastón Riera - @gastonriera

26. The basics!
- Noindex
- Redirects
- Nofollow
- Crawl paths (more/less)
Gastón Riera - @gastonriera

27. Gastón, that's nothing new!
😡
Gastón Riera - @gastonriera

28. I know! 😉
Gastón Riera - @gastonriera

29. How are we using them?
Let's get to it
Gastón Riera - @gastonriera

30. Battle_1:
Content quality
Gastón Riera - @gastonriera

31. Content is not just text on the
page, but everything on it.
Every page is content.
Gastón Riera - @gastonriera

32. Battle_1: Content quality
Gastón Riera - @gastonriera
Two options:
1. Add content focussing on quality over
quantity.
2. Remove content from Google's index.
We already had +9M items!

33. Battle_1: Content quality
Gastón Riera - @gastonriera
Two options:
1. Add content focussing on quality over
quantity. ❌
2. Remove content from Google's index.
We already had +9M items!

34. Battle_1: Content quality
Gastón Riera - @gastonriera
Two options:
1. Add content focussing on quality over
quantity. ❌
2. Remove content from Google's index. ✅
We already had +9M items!

35. Do you know what reduces the
content quality of any site?
Gastón Riera - @gastonriera

36. Do you know what reduces the
content quality of any site?
DUPLICATE CONTENT!
Gastón Riera - @gastonriera

37. Noindex and remove duplicates,
RUTHLESSLY
Gastón Riera - @gastonriera
Noindex a good part of
the items library.
-> Several million less
discoverable pages!
Why we decided to
noindex instead of
a fancier solution?
Ask me later
😉

38. A few tips on how to get what to noindex?
- Use google's crawled not indexed as a proxy
- Check duplicate titles/urls/content description
- Just a different image doesn't make it a different page to
the eyes of Google!
Gastón Riera - @gastonriera
Battle_1: Content quality

39. Noindex and remove duplicates,
RUTHLESSLY
Gastón Riera - @gastonriera
Why the redirected path
had 15% of site's traffic
and 20x the destination.
Ask me later
😉
Merged two translations that ended up being way
more similar that intended
-> A few millions pages removed from Google.

40. Other big things we did
● Turned Tag pages into Search pages
● Search pages are noindex by default
The overall result? Decreased the index size to a half
without impacting organic traffic.
Gastón Riera - @gastonriera
50%
Battle_1: Content quality

41. Battle_2
Internal linking
Gastón Riera - @gastonriera
Reference

42. Battle_2: Internal linking
Gastón Riera - @gastonriera
Out of many tactics:
1. Reduce the number of crawl paths
2. Nofollow on links to low-value pages

43. Basically,
Be intentional and smart
about crawl paths.
Gastón Riera - @gastonriera

44. Gastón Riera - @gastonriera
Link to only valuable pages
Added links between related
search pages
10% Organic traffic!
If it's a useful search page,
it will not have a noindex.
Note that

45. Gastón Riera - @gastonriera
Link to only valuable pages
Remove hreflang when you're uncertain
of the quality on other languages
15% size of index!
hreflang are bidirectional,
remove them on every
language.
Remember 😉

46. What are valuable pages?
Gastón Riera - @gastonriera
In short, pages we want Google to index.

47. Gastón Riera - @gastonriera
Link to only valuable pages
As per nofollow:
● Nofollow on links to noindex pages
● Filters and facets, all nofollow
The overall result? Google re-crawled more pages.
60%

48. So, why crawl capacity
management?
Gastón Riera - @gastonriera

49. Crawl budget stayed the
same.
Gastón Riera - @gastonriera
*On average, over the last 2yrs.

50. BONUS TRACK
and unpopular opinion.
Gastón Riera - @gastonriera

51. BONUS TRACK
and unpopular opinion.
Gastón Riera - @gastonriera

52. BONUS TRACK
and unpopular opinion.
We learnt that
● Sitemaps didn't help indexing AT ALL
󰤃
● Helpful only for debugging 🤓
Gastón Riera - @gastonriera

53. 1st month 1 USD🥳
Go to SOB22.com
Gastón Riera - @gastonriera
Yeah no kidding, I did
register that domain to
share the discount ☺

54. Gastón Riera - @gastonriera
Gracias! - Thank You!