In this presentation I gave for the Jane and Robot Developer Summit on October 8, 2009 I discuss some of the common problems sites face with crawlability. I invite developers and SEOs to think like a search engineer to integrate best practices into their every day work.

1. Why It's Hard to Crawl Your Website And What You Can Do To Help Nick Gerner Software Engineer and Web Crawler SEOmoz.org

2. Building a Crawler

4. “Vacation in the Caribbean”

6. Crawling The Web while(morePages) { FetchNextPage ParseXHTML AddNewLinksToCrawl }

7. Crawling The Web while(morePages) //duplicate content? { //200 OK? 3xx? 5xx, 404? politeness? FetchNextPage ParseXHTML //hah! AddNewLinksToCrawl //but which ones? }

9. Avoid Spam

10. Send Searchers to Highest Quality Sites

11. Don't crash websites with crawl

12. Keep content up to date

13. ...iterate for reqs v3.0, 4.0...

14. What Can You Do To Help?

15. Build For Users Not Robots! “ The formula is simple: design for people, be smart about robots, and you will achieve long-lasting success.” --Jane and Robot Philosophy

17. subscribe to services

18. listen to advice

20. Search engines do NOT like websites that lie or confuse them

21. Investing in Usability once pays double!

22. Problems You Can Help to Solve

23. Duplicate Content “ A majority of the web is duplicate content” --Joachim Kupke, Google Engineer on Duplicate Content Foo Foo Foo Foo Foo Foo Foo Foo Foo Foo

26. Sorting Lists

27. Session IDs

28. View State

29. Campaign tracking codes

32. robots.txt stuff out ← don't!

33. Redirect (301)

34. rel=canonical (at least for Google today)

35. sitemap.xml ( www.sitemaps.org )

37. www vs non-www www.searchengineland.com dup dup dup dup dup dup dup dup dup searchengineland.com dup dup dup dup dup dup dup dup dup dup dup

39. ...But choose one!

40. Stick to it throughout your site

41. 301 EVERYTHING to the canonical form

43. if you've screwed it up that gives you less risk

45. Single name for a single piece of content

47. Link-able

48. Memorable http://my.url.says/a/lot?about=me

50. Site Volume and Indexation Holy Crap that's a lot of crawling! Time to prioritize what's important!

54. What's updated

55. Bad Architecture Uh... what is important here?

57. Important Content is Deeply Hidden

58. Hard to discover important content from unimportant

59. Good Architecture Oh, I can understand that!

61. Short Path to Relevant Content

62. HTTP status Codes YourSite.com 200 Not Found Uh... That's not right

65. 302 is temporary!

66. 304 for conditional get is great

67. 404 for does not exist (NOT 200!)

68. 5XX for temporarily down (NOT 200, 404!)

69. Familiarize yourself with RFC 2616

70. Robots Exclusion Protocol & Robots.txt

73. Don't know future search engines

76. Easy to mess up special casing for different crawlers

77. Avoid duplicate blocks

78. Some guy's kid got pissed and de-indexed corp site

82. Better yet, 301 these to a better place

85. http://googlewebmastercentral.blogspot.com/2007/03/using-robots-meta-tag.html

87. Graceful Degradation

88. Accessibility (e.g. for the blind, for the deaf)

89. KISS

91. In Telnet

94. Beginner's Guide to SEO (free)

95. site:seomoz.org Site Architecture (free and paid)

96. Help! Questions?