4. IPULLRANK.COM @ IPULLRANK
First There Were Webmasters
Before the web world splintered into 50 million different jobs, there was just the Webmaster. In 1995 I was even one
of them for Microsoft’s ActiveX team.
5. IPULLRANK.COM @ IPULLRANK
Then Came Search Engines
Before Search Engines we just found things on directories or resource pages.
6. IPULLRANK.COM @ IPULLRANK
SEO Grew out of Webmaster/Info Retrieval World
Basically, it was just a bunch of nerds trying to get money and succeeding.
7. IPULLRANK.COM @ IPULLRANK
For A Long Time, It Was Just About Hacks and Tricks
Remember bolding every single instance of a keyword on the page?
21. IPULLRANK.COM @ IPULLRANK
JavaScript is Not Going Away
Nearly every site on the web is using JavaScript to perform some type of transformation on the page. It’s
obvious that Google needed to solve for this.
22. IPULLRANK.COM @ IPULLRANK
Growth of AngularJS
AngularJS, the single page
application, MVW
framework continues to
grow dramatically in the top
million websites.
23. IPULLRANK.COM @ IPULLRANK
AngularJS is Popular in Most Verticals
It’s likely that you’ll run into an AngularJS site soon enough in your optimization duties
24. IPULLRANK.COM @ IPULLRANK
ReactJS is Growing Too
Similarly, Facebook’s MVW
framework is growing as
well. Although, it has a built
in feature for making sure
that it renders proper HTML,
but it’s still a framework you
should know how to
optimize for.
Check out this post https://builtvisible.com/react-js-seo/
32. IPULLRANK.COM @ IPULLRANK
On What Device, OS and Location?
Different browser and OS configurations will show you different results. So what truly is a mobile
ranking?
+ + = Ranking #3+
Nexus 5 Android NYCChrome
Safari
+ + = Ranking #1+
iPhone 6 iOS NYC
37. IPULLRANK.COM @ IPULLRANK
304 Response Codes
The Not-Modified response code tells clients that they do not need to download a page again if it has already been downloaded.
38. IPULLRANK.COM @ IPULLRANK
Googlebot Respects the 304
What is cloaking when Google accepts 304 response codes and does not revisit that content after
indexation?
39. IPULLRANK.COM @ IPULLRANK
What is Cloaking in the Adaptive/Responsive Era?
Many people show less or more content based on the context of the browser/user, so what is
cloaking now?
42. IPULLRANK.COM @ IPULLRANK
Hreflang HTTP Header
I have not seen a tool that looks for this HTTP header. However, this implementation is preferred over the many lines
of hreflang tags on every page.
44. IPULLRANK.COM @ IPULLRANK
JavaScript Redirects for Mobile
Google allows client side redirects for automation redirection of mobile to desktop and vice versa. Many crawling tools do not
account for this.
46. IPULLRANK.COM @ IPULLRANK
Chrome DevTools Does
Natively in Chrome’s Developer Tools you can see all of the HTTP Request and
Response Headers in the Network tab section.
47. IPULLRANK.COM @ IPULLRANK
Remote Debugging
You can also see of this for a specific mobile devices with the remote debugging features of
Chrome.
49. IPULLRANK.COM @ IPULLRANK
Google is Crawling Headless
Make no mistake that Google has not been crawling with just text driven crawlers for a long time.
50. IPULLRANK.COM @ IPULLRANK
Adam Audette Said It So You KNOW It’s True
Adam ran a few tests a little under a year ago further proving out these capabilities
51. IPULLRANK.COM @ IPULLRANK
Also an Ex-Googler Confirmed It
A former Google developer said the headless crawling was his primary project from 2006 – 2010.
54. IPULLRANK.COM @ IPULLRANK
Log Files Went out of Style?
No, log file analysis has not gone out of style. In fact, it’s more valuable than ever before.
LOOK AT YOUR LOG FILES!
55. IPULLRANK.COM @ IPULLRANK
Log File Analysis
We parse the logs and query them in MySQL, but you can use any number of tools for this.
56. IPULLRANK.COM @ IPULLRANK
Search Console’s Crawl Stats are Mediocre
I personally find what Google gives us to be pretty much useless. Don’t less this take the place of
going through your logs.
57. IPULLRANK.COM @ IPULLRANK
Log Files Help You Spot What Really Happens
I had a client come last year that was unsure whether it was Penguin or changes in their offline media
buying habits had led to the downturn in traffic. Layering the Googlebot logs made it clear.
0
50000000
100000000
150000000
200000000
250000000
0
500000
1000000
1500000
2000000
2500000
Unknown organic organic home organic dyor
organic er organic home,dyor,er organic cushion organic net of home
organic ritani organic diamonds direct home paid brand sessions
tv spend web sessions (as reported by adaptv) Top 800 Monthly SV impressions
Googlebot Visits
Penguin Update
58. IPULLRANK.COM @ IPULLRANK
Social Shares Influence Crawl Patterns
We find that there is a higher correlation with crawling and social shares than crawling and links. Commonly
held SEO knowledge would make you believe this is entirely dictated by links.
0
1000000
2000000
3000000
4000000
5000000
6000000
0
100000
200000
300000
400000
500000
600000
700000
Googlebot Visits Links Social Shares FB Twitter G+
60. IPULLRANK.COM @ IPULLRANK
Prerender Can Cause Problems
However, we’ve seen Prerender setups misidentify Googlebot as a human users and serve the wrong the
AngularJS versions.
0
20000
40000
60000
80000
100000
120000
Googlebot Perceived as Human
61. IPULLRANK.COM @ IPULLRANK
Prerender Setup Serving Non-200
Prepping the prerender cache can cause outages and lead to pages falling in and out of the index.
0
2000
4000
6000
8000
10000
12000
14000
16000
206
301
302
304
403
404
405
410
500
503
504
206
301
302
304
403
404
410
503
504
206
301
302
304
404
405
410
500
503
504
206
301
302
304
401
404
405
410
500
503
504
206
301
302
304
403
404
405
410
502
503
504
1 2 3 4 5
-
hit
miss
pass
62. IPULLRANK.COM @ IPULLRANK
Also Prerendering is Not a Requirement
We got rid of prerender on 2 sections of the site and saw the traffic from Organic Search increase YoY.
Google can index it just fine.
63. IPULLRANK.COM @ IPULLRANK
AJAX Crawling Deprecated
Google has gotten rid of the AJAX crawling scheme because they don’t need it anymore. They have perfected their
ability to crawl headless at reasonable scale. They say don’t block the CSS or JS because it’s a browser.
65. IPULLRANK.COM @ IPULLRANK
Prerender Does Speed Up Crawling
There is certainly value in prerendering for speed, but then that makes me wonder, why not just not
use Angular?
67. IPULLRANK.COM @ IPULLRANK
Most Tools use cURL or Similar
There are many similar libraries, but many SEO tools are built on library called cURL for managing HTTP requests. This
is a text based crawler.
68. This is a fundamental flaw of
many SEO tools because they
are not constructing the DOM
and its CSS/JS
transformations. Search
Engines are.
69. IPULLRANK.COM @ IPULLRANK
PhantomJS – For Scraping the Modern Web
There are many headless browsers out there such as Selenium, but PhantomJS is the go to.
http://www.phantomjs.org
70. IPULLRANK.COM @ IPULLRANK
HorsemanJS – The Easy PhantomJS
I prefer HorsemanJS which is a NodeJS module that is incredibly simple. http://www.horsemanjs.org
73. IPULLRANK.COM @ IPULLRANK
Multi-Page In-browser Scraping
If you want to scrape multiple pages check out ArtooJS https://medialab.github.io/artoo/
74. Content & Linking
You knew I wasn’t going to get up here and not talk
about these things, right?
75. IPULLRANK.COM @ IPULLRANK
Google Looks at Entities First
According to Paul Haahr, Google looks at entities first and it appears that we as an industry have talked
about this in theory, but not truly worked it into the optimization workflow.
77. IPULLRANK.COM @ IPULLRANK
Term Relevance
There’s also been a lot of talk about co-relevance, topic modeling and TF*IDF. This is a great
opportunity for technical elements to influence content creation.
79. IPULLRANK.COM @ IPULLRANK
Moz’s New Tool Will Help
Moz has made the related keywords concept more accessible by adding it to Moz Pro.
Searchmetrics also has a similar solution.
81. IPULLRANK.COM @ IPULLRANK
These Rankings Say He’s Wrong
We had a client with millions for 302s and all we did was change them to 301s
June August September
HyperTraffic (1-3) 1272 1,283 1,473
Traffic (4-10) 979 983 837
Striking Distance (11-20) 167 161 110
Emerging (21-40) 106 103 98
Developmental (41-100) 156 90 87
Unranked 0 60 75
0
200
400
600
800
1000
1200
1400
1600
301 redirects
82. IPULLRANK.COM @ IPULLRANK
This Traffic Improvement Says He’s Wrong
Traffic went up pretty dramatically despite seasonality.
August 27th
301 redirects
11,039,188
10,455,517
11,348,902
12,049,481
11,250,148
12,902,847
11,040,192
13,158,296
11,287,790 12,203,998
12,508,164
13,158,296
14,041,313
15,609,390
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
18,000,000
April May Jun Jul Aug Sep Oct
Organic Visits
2014 2015
83. IPULLRANK.COM @ IPULLRANK
Internal Linking Structures
Internal Linking Structures are still an incredibly powerful asset. I’d like to see more tools like what Portent has built to visualize
them and understand where there is opportunity. https://www.portent.com/blog/seo/seo-force-directed-diagrams.htm
87. IPULLRANK.COM @ IPULLRANK
Google Expects Ludicrously Speedy
Google has an expectation that the above the fold content on the page will load within 1 second. You basically get 400 ms to
make that happen.
89. IPULLRANK.COM @ IPULLRANK
DevTools Is Your Friend
In the Timeline section, DevTools gives you a point by point break down of each operation and how it impacts the site’s load
time. This is how you can determine what exactly is slowing down your pages.
92. IPULLRANK.COM @ IPULLRANK
External Resources Also Slow Down the Page
Chartbeat is typically a huge culprit and using rel-dns-prefetch can speed this up considerably.
94. IPULLRANK.COM @ IPULLRANK
Responsive Sites are
SLOOOOOOOW
One of the key things you
can do, aside from optimizing
for the critical rendering
path, is conditional loading.
95. IPULLRANK.COM @ IPULLRANK
PreBrowsing Directives
Browsers have preloading directives which allow the page to download elements in the background for later
viewing or within the same session.
100. IPULLRANK.COM @ IPULLRANK
Use it With the GA API
Get the pageviews of the
ga:nextPagePath based on
the ga:previousPagePath set
to the current page that
you’re on.
You’ll get a list of the pages
your users are most likely to
visit next. Set one of these as
the rel-prerender.
101. IPULLRANK.COM @ IPULLRANK
Code.
Inject rel-prerender into the
page programmatically.
You could also session the
URLs the user is going to to
ensure you are always
prerendering a new page.
105. IPULLRANK.COM @ IPULLRANK
Rel-Preload for Same Page Preloading
I haven’t played with it yet, but Ian says they’re using it on the Portent site.
https://www.smashingmagazine.com/2016/02/preload-what-is-it-good-for/
106. How to Go Forward
Things you should know and do to be prepared for
the technical renaissance
107. IPULLRANK.COM @ IPULLRANK
Understand the Document Object Model
https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction
113. IPULLRANK.COM @ IPULLRANK
Accelerated Mobile & Instant Pages
Even if you’re not a publisher, I encourage you to watch these projects as I expect that Google and Facebook will grow the specs
dramatically once they’re adopted.