Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Crawl Budget Conqueror - Take Control of Your Crawl Budget

186 vues

Publié le

Jori Ford's slides from TechSEO Boost 2019

Publié dans : Marketing
  • Soyez le premier à commenter

Crawl Budget Conqueror - Take Control of Your Crawl Budget

  1. 1. Jori Ford | @chicagoseopro | #TechSEOBoost #TechSEOBoost | @CatalystSEM THANK YOU TO THIS YEAR’S SPONSORS Crawl Budget Conqueror - Take Control of Your Crawl Budget Jori Ford, G2
  2. 2. Jori Ford | @chicagoseopro | #TechSEOBoost Take Control of Your Crawl Budget! Crawl Budget Conqueror
  3. 3. Jori Ford | @chicagoseopro | #TechSEOBoost Agenda How Does Google See Crawl Budget? Key Factors to Managing Crawl Budget Ways to Quantify Crawl Budget on Your Site Fine-Tuning & Crawl Controls • Conservation - Tactics to help you maximize the crawl budget you have • Optimization - Shifting crawl to places you want • Expansion - Get More, Why Not!
  4. 4. Jori Ford | @chicagoseopro | #TechSEOBoost How Does Google See Crawl Budget?
  5. 5. Jori Ford | @chicagoseopro | #TechSEOBoost Deconstructing Crawl Budget Crawl Budget = Crawl Rate x Crawl Demand
  6. 6. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Rate is Variable “If the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Googlebot crawls less.” -- Gary Illyes, Google
  7. 7. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Rate Breakdown = {Server Response Time x Time} Error Rate Result: = {ttfb x duration} %server error
  8. 8. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Demand = Popularity x Freshness
  9. 9. Jori Ford | @chicagoseopro | #TechSEOBoost Popular on the Internet =
  10. 10. Jori Ford | @chicagoseopro | #TechSEOBoost Staleness aka Freshness In general, we try to do our crawling based on what we think this page might be changing or how often it might be changing. So, if we think that something stays the same for a longer period of time, we might not crawl it for a couple of months. -- John Mueller, Google
  11. 11. Jori Ford | @chicagoseopro | #TechSEOBoost Crawl Budget = {Avg. ttfb x duration/%server error} x {(CTR) x (Avg Time b/t Updates)} Key Factors to Managing Crawl Budget: • Time to First Byte • Server Error Rate • Click-Thru Rate (CTR) • Freshness
  12. 12. Jori Ford | @chicagoseopro | #TechSEOBoost Ways to Quantify Crawl Budget on Your Site
  13. 13. Jori Ford | @chicagoseopro | #TechSEOBoost Quantifying Crawl Budget on Your Site Step 1: Determine how many pages you’re expecting to be crawled. Step 2: Figure out what pages Google is crawling? Step 3: How many are there? And with what frequency are they crawled? Step 4: Segment your data by page type.
  14. 14. Jori Ford | @chicagoseopro | #TechSEOBoost Step 1: Determine # of Pages Expected 1. What are your money pages? 2. Group by Page Types or Templates 3. Match to XML Sitemaps (ONLY if optimized)
  15. 15. Jori Ford | @chicagoseopro | #TechSEOBoost Step 2: What Pages are Being Crawled? • Raw Server Crawl Logs • Log File Analyzer Tools (Botify, OnCrawl, Screaming Frog, Deep Crawl) • Custom Tracking (GTM, Cloud Service Workers)
  16. 16. Jori Ford | @chicagoseopro | #TechSEOBoost Step 3: # of Crawled Pages & Frequency • Excel Using Pivot Tables • SQL Database Query • Log File Analyzer Tools (Botify, OnCrawl, Screaming Frog, Deep Crawl)
  17. 17. Jori Ford | @chicagoseopro | #TechSEOBoost Step 4: Segment by Type % Page Type % Crawl Allocation % Active vs. Inactive % Not Crawled
  18. 18. Jori Ford | @chicagoseopro | #TechSEOBoost Calculate Your Crawl Budget = (Avg # of Crawled URLs x Freq) Time
  19. 19. Jori Ford | @chicagoseopro | #TechSEOBoost Fine-Tuning & Crawl Controls
  20. 20. Jori Ford | @chicagoseopro | #TechSEOBoost Conserve Your Crawl Budget • Prune Non-Money Pages • De-Index Dupes • Remove pages not in XML Sitemap(s) • Removal or fixing of Erroring Pages
  21. 21. Jori Ford | @chicagoseopro | #TechSEOBoost Optimize Crawling • Improve TTFB (time to first byte) • Optimize XML Sitemaps • Manage Facets, Filters, & Search Pages w/ NoIndex • Handle parameters • Internal Link Optimization • Orphaned No More! • Model Crawled/Active Pages
  22. 22. Jori Ford | @chicagoseopro | #TechSEOBoost Expand Crawl Budget Containers & Clustering Services • Heroku Dynos • AWS EC2 • Docker & Kubernetes • Microsoft Azure Increase Google’s crawl by auto- scaling apps
  23. 23. Jori Ford | @chicagoseopro | #TechSEOBoost
  24. 24. Jori Ford | @chicagoseopro | #TechSEOBoost Remember… Google has limited resources There are only a few major factors you can manage: • Time to First Byte • Server Error Rate • Click-Thru Rate (CTR) • Freshness You can quantify your budget manually or with tools, but knowing it is critical Conserve. Optimize. Expand
  25. 25. Jori Ford | @chicagoseopro | #TechSEOBoost Thank You – @chicagoseopro /in/chicagoseopro www.joriford.com
  26. 26. Jori Ford | @chicagoseopro | #TechSEOBoost Thanks for Viewing the Slideshare! – Watch the Recording: https://youtube.com/session-example Or Contact us today to discover how Catalyst can deliver unparalleled SEO results for your business. https://www.catalystdigital.com/

×