Alexis + Max - We Love SEO 19 - Bot X

@AlexisKSanders
Creating a bot experience as
good as your user experience
Alexis Sanders / Max Prin

@AlexisKSanders
part 1:
improving user & bot relations
améliorer les relations utilisateurs/robots

@AlexisKSanders
you are not a
robot.
(probably)
tu n’es pas un robot. (probablement)

@AlexisKSanders
we can’t do math
calculations as fast,
on ne peut pas calculer aussi vite,

@AlexisKSanders
I eat cereal
I eat numbers for
breakfast.

@AlexisKSanders
we're more likely to miss
something in large datasets,
on est plus susceptible de manquer quelque chose dans beaucoup de données

@AlexisKSanders
yeah, apparently robots are better than 5 year olds…
oui, apparemment les robots sont meilleurs que des enfants de 5 ans…

@AlexisKSanders
we aren’t able to work 24/7,
on ne peut pas travailler 24h sur 24, 7 jours sur 7

@AlexisKSanders
(real talk: my irobot
has done more
cleaning than anyone
in my home…)
real data
mon iRobot a nettoyer plus que n’importe qui d’autre chez moi

@AlexisKSanders
and that’s okay.
et c’est OK.

@AlexisKSanders
it’sokayto
not
bearobot.

@AlexisKSanders
human vs. bot
complex tasks calculations
creativity, imagination, language endless loops
heuristic analysis filtering big data
better with people better with other bots

@AlexisKSanders
all of this is to say humans
and bot are different
tout ça pour dire que les humains et les robots sont different

@AlexisKSanders
diversity is a good thing
diversité est une bonne chose

@AlexisKSanders
we can use both of our
strengths to create better work.
on peut combiner nos forces pour plus d’efficacité

@AlexisKSanders
for today, i have two themes:
1. large sites
2. automated testing

@AlexisKSanders
theme I:
large sites

@AlexisKSanders
>= 100,000 pages

@AlexisKSanders
large sites are
interesting in SEO,
because they have
same issues, but
at a larger scale

@AlexisKSanders
6 SEO challenges (especially for large sites):
crawling crawl efficiency indexing
rendering unique content awareness

@AlexisKSanders
how can we
address these?

@AlexisKSanders
crawling:
logs and log
visualization tools

@AlexisKSanders
logs show a
site’s
interactions
with bots
2019-10-03 00:00:00
0.0.0.7 GET /knock-
knock

@AlexisKSanders
let’s start with exploratory testing
what is
that….
that is job
security.

@AlexisKSanders
what are we looking for (by user-agent)?
• anomalies
• segment by folder
• crawling rates
• http response codes

@AlexisKSanders
we must answer:
1. are bots crawling your site in a way
you’d expect (and want) them to?
2. are your top KPI-driving pages being
crawled?

@AlexisKSanders
resulting changes may include:
• more internal linking
• removing dead internal links
• resolving status codes
• canonicalizing page sets (e.g., pdfs)
• bots crawling non-existent pages
• pages no one knew existed

@AlexisKSanders
action:
break up XML sitemaps in meaningful
ways for a human to later analyze
(and submit those individual in GSC).

@AlexisKSanders
effect of a meaningful XML sitemap
Relative effect of the treatment
showed an increase of +33%.
The 95% confidence interval of
this percentage is
[ 19.0% , 45.0% ].
The probability of this effect
being caused by chance is small
therefore it is statistically
significant.

@AlexisKSanders
crawl efficiency

@AlexisKSanders
crawling efficiency:
o important pages close to root?
o no crawl traps?
o no orphan pages?
o all pages have a purpose?
o duplicate content?
o redirects consolidated?
o canonical tags?
o no useless parameters?

@AlexisKSanders
crawling & indexing:
GSC index coverage reports

@AlexisKSanders
there is a wealth of information
hidden within the “excluded” section
=

@AlexisKSanders
tip:
go through each section of the
“excluded” coverage reports,
identify any themes

@AlexisKSanders
bonus life tip:
have or make a master list of
all URLs on the site
»«

@AlexisKSanders
why a master list of all URLs?
1. site migration
2. auditing
3. knowing/agreeing on what’s priority
4. to identify what is not being crawled & indexed
5. automation

@AlexisKSanders
to make a master URL list:
o crawlers
o XML sitemap
o GSC
o analytics platform
o dev team
o google SERP

@AlexisKSanders
we must answer:
1. JS to load important content?
2. performance data when changes are
implemented?
3. added solutions?
4. (bonus) are images important?

@AlexisKSanders
how to tell if your content is being
rendered?
1. check direct quotes in SERP
2. use google’s mobile-friendly testing tool
3. check the DOM (Inspect > Element)

@AlexisKSanders
chart
by Eric Wu
(@eywu)
on best
solution
for JS
concerns

@AlexisKSanders
content
b/c unique content is hard to do at scale

@AlexisKSanders
1. prioritize (by value to your core users)
2. delegate towards strengths:
• programmed = simple (maybe API) data
input/output
• humans = on people + relationships
tips:

@AlexisKSanders
effect of adding computer-generated text
Relative effect of the treatment
showed an increase of +22%
The 95% confidence interval of this
percentage is
[ 13.0% , 30.0% ].
The probability of this effect being
caused by chance is small therefore
it is statistically significant

@AlexisKSanders
chatbots
Relative effect of the treatment showed an increase of +22%
The probability of this effect being caused by chance is high
therefore it is not statistically significant
Relative effect of the treatment showed an increase of +78%
The probability of this effect being caused by chance is small
therefore it is statistically significant

@AlexisKSanders
+1 robots for scalability +1 humans for
emotional
connection

@AlexisKSanders
theme II:
automation

@AlexisKSanders
primary question:
is the site 24/7 doing what’s
expected?

@AlexisKSanders
what is important to monitor:
• robots.txt
• status codes
• http redirects live
• meta robots (noindex)
• canonical
• XML sitemap
• title tags
• meta description

@AlexisKSanders
solutions:
white/grey box testing – dev/QA team
black box testing – everyone else

@AlexisKSanders
automated black box unit testing for SEO:
leveraging tools or scripts that do basic SEO
checks at preset intervals

@AlexisKSanders
tools:
little warden
custom scripts
visual ping
selenium + jenkins
uptime robot

@AlexisKSanders
one last point,
always check robots with robots.

@AlexisKSanders
robots.txt is
confusing
use technicalseo.com/tools/robots-txt/
it still crawls
robots.txt and the
disallow is ignored.

@AlexisKSanders
to conclude (my part…),

@AlexisKSanders
human + robot working together
=
happier human webmaster and user
the bot doesn’t care … it’s a robot

@AlexisKSanders
Merci beaucoup!
@AlexisKSanders

@AlexisKSanders
APPENDIX
• What’s here? Well, basically a bunch of complaints random thoughts
in a rant constructive format about robots.txt and why everyone I
(personally) find it so confusing intellectually stimulating.

@AlexisKSanders
things I find confusing about robots.txt
• allow versus disallow hierarchy of more specific
• [undefined] verdicts, what does Google even do… then
• how Google ad bot doesn’t follow the rules
• implied * at end of every line
• implied .com at beginning of every line
• how $ and * are in robots.txt, but they’re not the same as regex
• the whole noindex header on robots.txt being accepted, then ignored… why…
• https://www.robotstxt.org/, the whole site
• how we can only use robots.txt if URL structure makes sense
• how disallowing the robots.txt is just ignored (it’s so meta)
• when sites overuse robots.txt
• why Google automatically crawls your blocked pages if the robots.txt goes down
• how robots.txt is case sensitive (it’s so close… and yet… so far)

@AlexisKSanders
Part 2
PWA + AMP = PWAMP
@maxxeight

@maxxeight
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Person",
"name": "Max Prin",
"url": "https://maxprin.com",
"jobTitle": "Head of Technical SEO",
"worksFor": "Merkle",
"sameAs": [
"https://twitter.com/maxxeight",
"https://www.linkedin.com/in/maxprin"
]
}
</script>
@maxxeight

@maxxeight
Une experience utilisateur… que Google peut comprendre.
• Pertinent
• Mobile-Friendly
• Rapide
• Securisé
• Populaire
Un bon référencement naturel est basé sur:
• Contenu
• Web Design
• Vitesse du site
• SSL/HTTPS
• Liens
@maxxeight

@maxxeight
Contenu “caché” <picture> and srcset
Onglets
Accordéons
“Lire la suite”
Quand chargé automatiquement
(vs. clique de l’utilisateur)
Design adapté optimisé pour mobiles
<picture>
<source type="image/svg+xml" srcset="pyramid.svg">
<source type="image/webp" srcset="pyramid.webp">
<img src="pyramid.png" alt="large PNG image...">
</picture>
<img srcset="example-320w.jpg 320w,
example-480w.jpg 480w,
example-800w.jpg 800w"
sizes="(max-width: 320px) 280px,
(max-width: 480px) 440px,
800px"
src="example-800w.jpg" alt="responsive web!">
@maxxeight

@maxxeight
Native apps
Web apps
@maxxeight

@maxxeight
Fiable et rapide Engageant
• Mobile-Friendly
• Rapide*
• Securisé (HTTPS)
Progressive Web Apps
• Icône (home)
• Notifications (push)
@maxxeight

@maxxeight
Exploration et rendu
• Googlebot “à la page”
• Lazy Loading
• Contenu “onclick”
• + liens (mega menu, etc.)
Progressive Web Apps
@maxxeight

@maxxeight
Lazy Loading (Images)
@maxxeight

@maxxeight
Accelerated Mobile Pages
AMP HTML+JS AMP Cache (CDN) Pre-loading
@maxxeight

@maxxeight
L’URL AMP ”Réelle” avec les Échanges Signés
@maxxeight

@maxxeight
Javascript Personnalisé dans AMP avec <amp-script>
Restrictions
• 10 000 octets maximum par <amp-script>
• 150 000 octets maximum pour le total des
<amp-script> dans la page
@maxxeight

@maxxeight
AMP pour le E-Commerce
<amp-sidebar>
- Navigation
<amp-carousel>
<amp-list>
- Organisation des produits
<amp-form>
- Recherche
<amp-bind>
- Filtrage et tri
<amp-access>
- Connection
<amp-accordion>
- Images/details
<amp-form>
<amp-carousel>
- Commentaires/avis
<amp-selector>
- Onglets/Vignettes
<amp-bind>
- Selection couleur/taille
<amp-state>
- Ajouter au panier
@maxxeight

@maxxeight
Achat/Paiements dans AMP
PaymentRequest API
- Seulement avec Chrome
<amp-form>
- Information (nom, addresse, etc.) mais
pas de paiement
Rediriger les visiteurs vers le site…
@maxxeight

@maxxeight
AMP
Français
m.
Français
m.
Anglais
Desktop
Français
Desktop
Anglais
rel="amphtml"
rel="amphtml"
rel="amphtml"
rel="canonical"
rel="amphtml"
rel="canonical"
rel="canonical"
rel="alternate"
rel="canonical"
rel="alternate"
rel="alternate" hreflang=”fr"
rel="alternate" hreflang="en"
AMP
Anglais
@maxxeight

@maxxeight
Desktop
Français
Desktop
Anglais
@maxxeight

@maxxeight
AMP: Acquisition Initiale PWA: Interactivité/Engagement
• SERP
• Chargement instantané
• Fonctionnalités limitées
Comment obtenir le meilleur des deux?
• Fonctionnalités avancées
• Dynamique
• Lent la 1ere visite
@maxxeight

@maxxeight
Combiner AMP et PWA
AMP comme
point d'entrée
dans la PWA
@maxxeight
AMP comme
source de
données pour
la PWA
AMP avec des
fonctionnalités
PWA

@maxxeight
AMP comme point d'entrée dans la PWA
<amp-install-serviceworker>
@maxxeight

@maxxeight
AMP et PWA avec les même URLs?
@maxxeight

@maxxeight
AMP
Français
m.
Français
m.
Anglais
rel="amphtml"
rel="amphtml"
rel="amphtml"
rel="canonical"
rel="amphtml"
rel="canonical"
rel="canonical"
rel="alternate"
rel="canonical"
rel="alternate"
AMP
Anglais
Desktop
Français
Desktop
Anglais
@maxxeight

@maxxeight
self.addEventListener('fetch', event => {
if (event.request.mode === 'navigate') {
event.respondWith(
fetch(event.request.url + '?pwa=true')
);
} else {
event.respondWith(
caches.match(event.request).then(function(response) {
return response || fetch(event.request);
})
);
}
});
service-worker.js
@maxxeight

@maxxeight
RewriteEngine on
RewriteCond %{QUERY_STRING} pwa=true [OR]
RewriteCond %{HTTP_REFERER} ^https://pwamp.site/.* [OR]
RewriteCond %{HTTP_REFERER} ^https://pwamp-site.cdn.ampproject.org/.*
RewriteRule (.*) /pwa.php [L]
.htaccess
@maxxeight

@maxxeight
Les page AMP ne sont pas rendues
• Bots voient seulement <amp-img> (vs. <img>)
• Pas d’access à l’URL dans src=“”
•
• Utilisez <noscript>
Les images AMP ne sont pas indexable
@maxxeight

@maxxeight
“Et pour le SEO?”
Les robots ne crawlent/indexent que la version AMP
Pas de gaspillage des resources (crawler plusieurs URLs avec le meme contenu)
Signaux clairs (pas de sourcis de balise canonical/alternate)
Les pages rapide et pre-chargées dans la SERP (AMP viewer ou “URL Réelle”)
Les pages sont “adaptées aux mobiles”
@maxxeight

@maxxeight
PWAMP - Examples et Resources
https://pwamp.site https://www.howpwampworks.com (by @aleyda)
@maxxeight

@AlexisKSanders
Merci!
@maxxeight
@maxxeight

Alexis + Max - We Love SEO 19 - Bot X

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to Alexis + Max - We Love SEO 19 - Bot X

Similar to Alexis + Max - We Love SEO 19 - Bot X (20)

More from Alexis Sanders

More from Alexis Sanders (13)

Recently uploaded

Recently uploaded (20)

Alexis + Max - We Love SEO 19 - Bot X

Editor's Notes