For every website on the internet, Google has a fixed budget for how many pages their bots can and are willing to crawl. The internet is a big place, so Googlebot can only spend so much time crawling and indexing our websites. Crawl budget optimization is the process of ensuring that the right pages of our websites end up in Google’s index and are ultimately shown to searchers.
Google’s recommendations for optimizing crawl budget are rather limited, because Googlebot crawls through most websites without reaching its limit. But enterprise-level and ecommerce sites with thousands of landing pages are at risk of maxing out their budget. A 2018 study even found that Google’s crawlers failed to crawl over half of the webpages of larger sites in the experiment.
Influencing how crawl budget is spent can be a more difficult technical optimization for strategists to implement. But for enterprise-level and ecommerce sites, it’s worth the effort to maximize crawl budget where you can. With a few tweaks, site owners and SEO strategists can guide Googlebot to regularly crawl and index their best-performing pages.
2. Hi, I’m
Manick👋
Founder/CTO
Scalable SEO for agencies +
brands
- 89,000 client link placements
secured
- 25,000 landing pages + blog posts
published and ranked
- 8.8 billion search results analyzed
Creator of SearchAtlas
SEO software suite
Formerly Founder & CEO
Rukkus.com (acquired by Stubhub +
TickPick)
Avid researcher of Google ranking
factors (10+ years)
🌐
🚀
🎟
📊
📈
12. Enterprise-level and ecommerce
sites are at risk for maxing out their
budget
- Ecommerce Brands with 1K+
SKUs: sometimes up to 50% of
products get 0 organic traffic
13. Sitemaps are great for making sure
Google can see all of your pages.
Can also hurt you if they’re filled
with lower-value or under-
performing pages.
14.
15. What Tools can we use to analyze
Google’s Crawl Behavior
30. Step 1:
Use a tool like SearchAtlas to find
sites linking to your competitors but
not linking to you, and outreach
them with Digital PR
31. TIP #1
Create free tools, resources, guides, and
other high value products you can give
away to increase natural link acquisition
This is how I got hundreds free DA 60+
backlinks, including from Neil Patel,
Universities, Wordpress, and Google
32. TIP #2
Reclaim lost PageRank via Broken
Backlinks - use “Broken Links” in
Ahrefs to find backlinks that are
pointing to broken pages on your
site.
33. TIP #3
Use HARO to help/pitch journalists
looking for insightful sources in
your industry
46. Find ways to get those pages more
internal links and send more Page
Rank there
47. Strategy #2 - Wasted PageRank
Focus on the pages that have a lot
of internal links, but don’t get much
traffic, search impressions, and
rank for very few keywords