Crawl budget management, optimization, and expansion tips to take your web site to the next level. It all starts with crawling, so understanding how to conqueror/control your crawl is essential to SEO.

1. Jori Ford | @chicagoseopro | #TechSEOBoost
Take Control of Your Crawl Budget!
Crawl Budget
Conqueror

2. Jori Ford | @chicagoseopro | #TechSEOBoost
Agenda
How Does Google See Crawl Budget?
Key Factors to Managing Crawl Budget
Ways to Quantify Crawl Budget on Your Site
Fine-Tuning & Crawl Controls
• Conservation - Tactics to help you maximize the crawl budget you have
• Optimization - Shifting crawl to places you want
• Expansion - Get More, Why Not!

3. Jori Ford | @chicagoseopro | #TechSEOBoost
How Does Google
See Crawl Budget?

4. Jori Ford | @chicagoseopro | #TechSEOBoost
Deconstructing Crawl Budget
Crawl Budget = Crawl Rate x Crawl Demand

5. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Rate is
Variable
“If the site responds really
quickly for a while, the limit
goes up, meaning more
connections can be used to
crawl. If the site slows down
or responds with server
errors, the limit goes down and
Googlebot crawls less.”
-- Gary Illyes, Google

6. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Rate Breakdown
= {Server Response Time x Time}
Error Rate
Result:
= {ttfb x duration}
%server error

7. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Demand = Popularity x Freshness

8. Jori Ford | @chicagoseopro | #TechSEOBoost
Popular on the Internet =

9. Jori Ford | @chicagoseopro | #TechSEOBoost
Staleness aka
Freshness
In general, we try to do our crawling based
on what we think this page might be
changing or how often it might be changing.
So, if we think that something stays
the same for a longer period of
time, we might not crawl it for a
couple of months.
-- John Mueller, Google

10. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Budget =
{Avg. ttfb x duration/%server error} x {(CTR) x (Avg Time
b/t Updates)}
Key Factors to Managing Crawl Budget:
• Time to First Byte
• Server Error Rate
• Click-Thru Rate (CTR)
• Freshness

11. Jori Ford | @chicagoseopro | #TechSEOBoost
Ways to Quantify Crawl Budget on Your Site

12. Jori Ford | @chicagoseopro | #TechSEOBoost
Quantifying Crawl Budget
on Your Site
Step 1: Determine how many pages you’re
expecting to be crawled.
Step 2: Figure out what pages Google
is crawling?
Step 3: How many are there? And with what
frequency are they crawled?
Step 4: Segment your data by page type.

13. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 1: Determine # of Pages Expected
1. What are your money pages?
2. Group by Page Types or Templates
3. Match to XML Sitemaps (ONLY if
optimized)

14. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 2: What Pages are Being
Crawled?
• Raw Server Crawl Logs
• Log File Analyzer Tools (Botify,
OnCrawl, Screaming Frog, Deep
Crawl)
• Custom Tracking (GTM, Cloud
Service Workers)

15. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 3: # of Crawled Pages &
Frequency
• Excel Using Pivot Tables
• SQL Database Query
• Log File Analyzer Tools (Botify,
OnCrawl, Screaming Frog, Deep
Crawl)

16. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 4: Segment
by Type
% Page Type
% Crawl Allocation
% Active vs. Inactive
% Not Crawled

17. Jori Ford | @chicagoseopro | #TechSEOBoost
Calculate Your Crawl
Budget
= (Avg # of Crawled URLs x Freq)
Time

18. Jori Ford | @chicagoseopro | #TechSEOBoost
Fine-Tuning & Crawl Controls

19. Jori Ford | @chicagoseopro | #TechSEOBoost
Conserve Your
Crawl Budget
• Prune Non-Money Pages
• De-Index Dupes
• Remove pages not in XML Sitemap(s)
• Removal or fixing of Erroring Pages

20. Jori Ford | @chicagoseopro | #TechSEOBoost
Optimize
Crawling
• Improve TTFB (time to first byte)
• Optimize XML Sitemaps
• Manage Facets, Filters, & Search Pages w/
NoIndex
• Handle parameters
• Internal Link Optimization
• Orphaned No More!
• Model Crawled/Active Pages

21. Jori Ford | @chicagoseopro | #TechSEOBoost
Expand Crawl Budget
Containers & Clustering Services
• Heroku Dynos
• AWS EC2
• Docker & Kubernetes
• Microsoft Azure
Increase Google’s crawl by auto-
scaling apps

22. Jori Ford | @chicagoseopro | #TechSEOBoost

23. Jori Ford | @chicagoseopro | #TechSEOBoost
Remember…
Google has limited resources
There are only a few major factors you can manage:
• Time to First Byte
• Server Error Rate
• Click-Thru Rate (CTR)
• Freshness
You can quantify your budget manually or with tools, but knowing it is critical
Conserve. Optimize. Expand

24. Jori Ford | @chicagoseopro | #TechSEOBoost
Thank You
–
@chicagoseopro
/in/chicagoseopro
www.joriford.com