Despite the huge shift to content marketing in recent years, the technical end of SEO has gotten increasingly complex and our tools are not keeping pace. As SEOs, we must develop strong working knowledge of the optimal usage of technology to get implementations accomplished. In this talk, Mike will walk through case studies, the impact of different technical implementations, and how to pull together small solutions when nothing on the shelf works for your needs.
13. IPULLRANK.COM @ IPULLRANK
JavaScript is Not Going Away
Nearly every site on the web is using JavaScript to perform some type of transformation on the page. It’s
obvious that Google needed to solve for this.
14. IPULLRANK.COM @ IPULLRANK
Growth of AngularJS
AngularJS, the single page
application, MVW
framework continues to
grow dramatically in the top
million websites.
15. IPULLRANK.COM @ IPULLRANK
AngularJS is Popular in Most Verticals
It’s likely that you’ll run into an AngularJS site soon enough in your optimization duties
16. IPULLRANK.COM @ IPULLRANK
ReactJS is Growing Too
Similarly, Facebook’s MVW
framework is growing as
well. Although, it has a built
in feature for making sure
that it renders proper HTML,
but it’s still a framework you
should know how to
optimize for.
Check out this post https://builtvisible.com/react-js-seo/
17. IPULLRANK.COM @ IPULLRANK
Ironically, React (Facebook) is SEO-ready, Angular (Google) is Not
Angular 1 did not have anything to make it easier to crawl. React has ReactDOMServer.renderToString to allow React to serve
fully rendered HTML from the server.
25. IPULLRANK.COM @ IPULLRANK
On What Device, OS and Location?
Different browser and OS configurations will show you different results. So what truly is a mobile
ranking?
+ + = Ranking #3+
Nexus 5 Android NYCChrome
Safari
+ + = Ranking #1+
iPhone 6 iOS NYC
30. IPULLRANK.COM @ IPULLRANK
304 Response Codes
The Not-Modified response code tells clients that they do not need to download a page again if it has already been downloaded.
31. IPULLRANK.COM @ IPULLRANK
Googlebot Respects the 304
What is cloaking when Google accepts 304 response codes and does not revisit that content after
indexation?
32. IPULLRANK.COM @ IPULLRANK
What is Cloaking in the Adaptive/Responsive Era?
Many people show less or more content based on the context of the browser/user, so what is
cloaking now?
35. IPULLRANK.COM @ IPULLRANK
Hreflang HTTP Header
I have not seen a tool that looks for this HTTP header. However, this implementation is preferred over the many lines
of hreflang tags on every page.
37. All of these HTTP headers are
better than clogging up the
<head> of the code.
38. IPULLRANK.COM @ IPULLRANK
JavaScript Redirects for Mobile
Google allows client side redirects for automation redirection of mobile to desktop and vice versa. Many crawling tools do not
account for this.
40. IPULLRANK.COM @ IPULLRANK
Chrome DevTools Does
Natively in Chrome’s Developer Tools you can see all of the HTTP Request and
Response Headers in the Network tab section.
41. IPULLRANK.COM @ IPULLRANK
Geolocation Emulation is Built-in to Chrome
If you’re running into issues with finding local rankings now that Google has gotten rid of
nearEquals you can just change your location on the browser level.
42. IPULLRANK.COM @ IPULLRANK
Remote Debugging
You can also see this for a specific mobile devices with the remote debugging features of Chrome.
44. IPULLRANK.COM @ IPULLRANK
Google is Crawling Headless
Make no mistake that Google has not been crawling with just text driven crawlers for a long time.
45. IPULLRANK.COM @ IPULLRANK
Adam Audette Said It So You KNOW It’s True
Adam ran a few tests a little under a year ago further proving out these capabilities
46. IPULLRANK.COM @ IPULLRANK
Also an Ex-Googler Confirmed It
A former Google developer said the headless crawling was his primary project from 2006 – 2010.
50. IPULLRANK.COM @ IPULLRANK
Well I Captured Video of Headless Googlebot
It’s a question of what does Googlebot do on your site. Using LuckyOrange I captured Googlebot sessions from Search Console’s Fetch and
Render and then I submitted it to the index. They load the page and leave. The mouse never moves, the headless browser never came back.
51. IPULLRANK.COM @ IPULLRANK
Get a Free Trial of LuckyOrange and Try it Yourself
This code will only place LuckyOrange on the page if a JS-enabled Googlebot comes to your site.
My test did not show what Googlebot from the wild does when it’s crawling.
52. IPULLRANK.COM @ IPULLRANK
Log Files Went out of Style?
No, log file analysis has not gone out of style. In fact, it’s more valuable than ever before.
LOOK AT YOUR LOG FILES!
53. IPULLRANK.COM @ IPULLRANK
Log File Analysis
We parse the logs and query them in MySQL, but you can use any number of tools for this.
54. IPULLRANK.COM @ IPULLRANK
Screaming Frog Log File Analyzer
The Screaming Frog team rolled out an awesome new tool recently for log file analysis.
55. IPULLRANK.COM @ IPULLRANK
Search Console’s Crawl Stats are Mediocre
I personally find what Google gives us to be pretty much useless. Don’t less this take the place of
going through your logs.
56. IPULLRANK.COM @ IPULLRANK
Log Files Help You Spot What Really Happens
I had a client come last year that was unsure whether it was Penguin or changes in their offline media
buying habits had led to the downturn in traffic. Layering the Googlebot logs made it clear.
0
50000000
100000000
150000000
200000000
250000000
0
500000
1000000
1500000
2000000
2500000
Unknown organic organic home organic dyor
organic er organic home,dyor,er organic cushion organic net of home
organic ritani organic diamonds direct home paid brand sessions
tv spend web sessions (as reported by adaptv) Top 800 Monthly SV impressions
Googlebot Visits
Penguin Update
57. IPULLRANK.COM @ IPULLRANK
Social Shares Influence Crawl Patterns
We find that there is a higher correlation with crawling and social shares than crawling and links. Commonly
held SEO knowledge would make you believe this is entirely dictated by links.
0
1000000
2000000
3000000
4000000
5000000
6000000
0
100000
200000
300000
400000
500000
600000
700000
Googlebot Visits Links Social Shares FB Twitter G+
59. IPULLRANK.COM @ IPULLRANK
Prerender Can Cause Problems
However, we’ve seen Prerender setups misidentify Googlebot as a human users and serve the wrong the
AngularJS versions.
0
20000
40000
60000
80000
100000
120000
Googlebot Perceived as Human
60. IPULLRANK.COM @ IPULLRANK
Prerender Setup Serving Non-200
Prepping the prerender cache can cause outages and lead to pages falling in and out of the index.
0
2000
4000
6000
8000
10000
12000
14000
16000
206
301
302
304
403
404
405
410
500
503
504
206
301
302
304
403
404
410
503
504
206
301
302
304
404
405
410
500
503
504
206
301
302
304
401
404
405
410
500
503
504
206
301
302
304
403
404
405
410
502
503
504
1 2 3 4 5
-
hit
miss
pass
61. IPULLRANK.COM @ IPULLRANK
Also Prerendering is Not a Requirement
We got rid of prerender on 2 sections of the site and saw the traffic from Organic Search increase YoY.
Google can index it just fine.
62. IPULLRANK.COM @ IPULLRANK
AJAX Crawling Deprecated
Google has gotten rid of the AJAX crawling scheme because they don’t need it anymore. They have perfected their
ability to crawl headless at reasonable scale. They say don’t block the CSS or JS because it’s a browser.
64. IPULLRANK.COM @ IPULLRANK
Prerender Does Speed Up Crawling
There is certainly value in prerendering for speed, but then that makes me wonder, why not just not
use Angular?
65. IPULLRANK.COM @ IPULLRANK
A Better Idea Is a Library that Doesn’t Need Prerender
Let’s stop going after the shiny object and go with the things that are built to work, shall we? Check
out MeteorJS – http://www.meteor.com
66. IPULLRANK.COM @ IPULLRANK
Or go with Angular2
Brad Green is the Google engineering product manager for AngularJS. He says Angular2 is better in
every way.
67. IPULLRANK.COM @ IPULLRANK
Don’t Forget Google Wants Progressive Enhancement
Thinking with the IoT in mind, progressive enhancement still makes sense. However, frameworks
like AngularJS are completely contradictory in that they don’t offer graceful degradation.
69. IPULLRANK.COM @ IPULLRANK
Most Tools use cURL or Similar
There are many similar libraries, but many SEO tools are built on library called cURL for managing HTTP requests. This
is a text based crawler.
70. This is a fundamental flaw of
many SEO tools because they
are not constructing the DOM
and its CSS/JS
transformations. Search
Engines are.
71. IPULLRANK.COM @ IPULLRANK
PhantomJS – For Scraping the Modern Web
There are many headless browsers out there such as Selenium, but PhantomJS is the go to.
http://www.phantomjs.org
72. IPULLRANK.COM @ IPULLRANK
HorsemanJS – The Easy PhantomJS
I prefer HorsemanJS which is a NodeJS module that is incredibly simple. http://www.horsemanjs.org
77. IPULLRANK.COM @ IPULLRANK
Or You Can Make Screaming Frog Headless
You can make Screaming Frog headless by placing a proxy with a headless browser in front of it. From my initial findings, there
is not much different aside from the order it identifies URLs and its ability to scrape content from pages.
78. IPULLRANK.COM @ IPULLRANK
Here’s the Code
Here’s the node code for how I did it. HorsemanJS is not the best tool for the proxy, but it’s great for a proof
of concept.
Screaming Frog pings for
robots.txt a lot. Make sure you
handle it with a cURL request
rather than a headless one.
For speed, don’t load images
Make sure you’re setting the
right headers before your return
the HTML. HorsemanJS does not
return headers.
79. IPULLRANK.COM @ IPULLRANK
PHP Wrapper for PhantomJS
For those of you that prefer PHP, the PHP PhantomJS wrapper is pretty decent as well.
http://jonnnnyw.github.io/php-phantomjs/
For speed, don’t load images
80. Content & Linking
You knew I wasn’t going to get up here and not talk
about these things, right?
81. IPULLRANK.COM @ IPULLRANK
Google Looks at Entities First
According to Paul Haahr, Google looks at entities first and it appears that we as an industry have talked
about this in theory, but not truly worked it into the optimization workflow.
83. IPULLRANK.COM @ IPULLRANK
Term Relevance
There’s also been a lot of talk about co-relevance, topic modeling and TF*IDF. This is a great
opportunity for technical elements to influence content creation.
85. IPULLRANK.COM @ IPULLRANK
Moz’s New Tool Will Help
Moz has made the related keywords concept more accessible by adding it to Moz Pro.
Searchmetrics also has a similar solution.
88. IPULLRANK.COM @ IPULLRANK
These Rankings Say He’s Wrong
We had a client with millions for 302s and all we did was change them to 301s
June August September
HyperTraffic (1-3) 1272 1,283 1,473
Traffic (4-10) 979 983 837
Striking Distance (11-20) 167 161 110
Emerging (21-40) 106 103 98
Developmental (41-100) 156 90 87
Unranked 0 60 75
0
200
400
600
800
1000
1200
1400
1600
301 redirects
89. IPULLRANK.COM @ IPULLRANK
This Traffic Improvement Says He’s Wrong
Traffic went up pretty dramatically despite seasonality.
August 27th
301 redirects
11,039,188
10,455,517
11,348,902
12,049,481
11,250,148
12,902,847
11,040,192
13,158,296
11,287,790 12,203,998
12,508,164
13,158,296
14,041,313
15,609,390
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
18,000,000
April May Jun Jul Aug Sep Oct
Organic Visits
2014 2015
90. But. We need to stop
saying ‘just do 301
redirects.’
91. IPULLRANK.COM @ IPULLRANK
Instead Do Optimal 301 Redirects
Pattern matching on the .htaccess level is always faster than a list of redirects and always better than doing
it on the code/page level. Developers will respect that you understand what you’re talking about.
ReWriteRule url_pattern
file_reference [R=301, L]
redirect 301 /relative/path/to/file.php
http://www.yoursite.com/path/to/new/file.php
redirect 301 /relative/path/to/file.php
http://www.yoursite.com/path/to/new/file.php
redirect 301 /relative/path/to/file.php
http://www.yoursite.com/path/to/new/file.php
>
92. IPULLRANK.COM @ IPULLRANK
Internal Linking Structures
Internal Linking Structures are still an incredibly powerful asset. I’d like to see more tools like what Portent has built to visualize
them and understand where there is opportunity. https://www.portent.com/blog/seo/seo-force-directed-diagrams.htm
93. IPULLRANK.COM @ IPULLRANK
Searchmetrics Does a Great Job of This
I wish all the link indices had a stronger focus on internal linking structure and visualizing where there are
opportunities to flow more link equity throughout the site. Searchmetrics’ Site Structure tool does this well.
94. IPULLRANK.COM @ IPULLRANK
Paul’s Method is Good for Rolling Your Own
http://searchengineland.com/improve-internal-linking-calculate-internal-pagerank-r-246883
95. If Possible, Always Update your internal links on the database level
WordPress Example
UPDATE wp_posts SET post_content =
REPLACE (post_content,
'http://www.oldsiteurl.com',
'http://www.newsiteurl.com');
97. IPULLRANK.COM @ IPULLRANK
Schema.org / JSON-LD
Schema.org is a lot more reasonable to implement now that they are allowing JSON-LD. Being in knowledge
boxes and such yields a lot more traffic than it takes away.
99. IPULLRANK.COM @ IPULLRANK
Google Expects Ludicrously Speedy
Google has an expectation that the above the fold content on the page will load within 1 second. You basically get 400 ms to
make that happen.
101. IPULLRANK.COM @ IPULLRANK
DevTools Is Your Friend
In the Timeline section, DevTools gives you a point by point break down of each operation and how it impacts the site’s load
time. This is how you can determine what exactly is slowing down your pages.
104. IPULLRANK.COM @ IPULLRANK
External Resources Also Slow Down the Page
Chartbeat is typically a huge culprit and using rel-dns-prefetch can speed this up considerably.
106. IPULLRANK.COM @ IPULLRANK
Responsive Sites are
SLOOOOOOOW
One of the key things you
can do, aside from optimizing
for the critical rendering
path, is conditional loading.
107. IPULLRANK.COM @ IPULLRANK
PreBrowsing Directives
Browsers have preloading directives which allow the page to download elements in the background for later
viewing or within the same session.
108. IPULLRANK.COM @ IPULLRANK
Rel-Preconnect
Rel-preconnect will resolve the DNS, start the TCP handshake and negotiate the TLS tunnel
beforehand, effectively shaving hundreds of milliseconds to several seconds in page load.
109. IPULLRANK.COM @ IPULLRANK
When to Use Rel-Preconnect
When you see a long time for connections to happen or a lot of idle time, it’s a good idea to use rel-preconnect.
116. IPULLRANK.COM @ IPULLRANK
Use it With the GA API
Get the pageviews of the
ga:pagePath based on the
ga:previousPagePath set to
the current page that you’re
on.
You’ll get a list of the pages
your users are most likely to
visit next. Set one of these as
the rel-prerender.
117. IPULLRANK.COM @ IPULLRANK
Code.
Inject rel-prerender into the
page programmatically.
You could also session the
URLs the user is going to to
ensure you are always
prerendering a new page.
121. IPULLRANK.COM @ IPULLRANK
Rel-Preload for Same Page Preloading
I haven’t played with it yet, but Ian says they’re using it on the Portent site.
https://www.smashingmagazine.com/2016/02/preload-what-is-it-good-for/
122. How to Go Forward
Things you should know and do to be prepared for
the technical renaissance
123. IPULLRANK.COM @ IPULLRANK
Understand the Document Object Model
https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction
129. IPULLRANK.COM @ IPULLRANK
Accelerated Mobile & Instant Pages
Even if you’re not a publisher, I encourage you to watch these projects as I expect that Google and Facebook will grow the specs
dramatically once they’re adopted.