Ranking highly in SERPs to achieve high volumes of organic traffic is a key goal for many webmasters and digital marketers. In this presentation, DeepCrawl's Tristan Pirouz discusses the topic of indexing and focuses in on two common scenarios: over-indexation and under-indexation. By identifying and understanding the reasons websites fall into these categories, it is possible to achieve 'smart indexing'.

1. Ultimate SEO Audit |
Smart Indexing
@tristan_pirouz
Tristan Pirouz, Head of BD EMEA, DeepCrawl
@tristan_pirouz

2. HOW TO POSITION YOUR YOURSELF AS A WEBMASTER
‘Smart Indexing’
@tristan_pirouz
You want to read more: http://info.deepcrawl.com/science-of-crawlspace
NumberofIndexedPages

3. Indexing
INDEXING IS THE CONTROL, PROCESS OR MEASURE OF WHICH PAGES ARE INDEXED
When showcasing your digital
assets to Google you want your
preferred pages to be seen.
@tristan_pirouz
Source: How to rank yours in Google search results, Google in-Depth Articles
Why we want to be indexed?

4. HOW OFTEN A BOT CRAWLS A SPECIFIC PAGE OR DIRECTORY PER HOUR OR DAY?
If a page hasn’t been crawled recently, it won’t rank well.
If a page hasn’t been crawled within a certain time
period then it receives less authority.
@tristan_pirouz
You want to read more: http://info.deepcrawl.com/science-of-crawlspace
Source: Crawl Optimization, blindfiveyearold

5. INDEXED PAGES ARE THOSE WHICH ARE RETURNED IN SEARCH RESULTS
The method to understand where you are is…
URL Analyze Validate
Universe
@tristan_pirouz

6. INDEXED PAGES ARE THOSE WHICH ARE RETURNED IN SEARCH RESULTS
The method to understand where you are is this…
• Ranking
checking
• Log analysis
• Search console
• Sitemaps
• Organic visits
• Site:
• Info:
• Dedupe
• Crawl with
DeepCrawl
Compare
this to the
indexed
pages count
in search
console
Analyze Validate:URL Universe:
@tristan_pirouz

7. HOW TO ACHIEVE SMART INDEXING?
Achieving ‘Smart Indexing’
@tristan_pirouz
Over-Indexation
Incorrect Canonical tags
Lack of NoIndex
Lack of Canonical tags
Under-Indexation
Thin Content
Undiscovered Pages
Inefficient Architecture
Slow Performance
Made NonIndexable by mistake
Smart Indexing

8. EXAMPLES OF UNDER-INDEXING
@tristan_pirouz

9. PAGES MAY NOT BE DISCOVERED DUE TO INSUFFICIENT LINKING/ HIGH CRAWL DEPTH
Undiscoverable pages
@tristan_pirouz
Googlebot may not access your content reach pages as they are
hidden too far down your website

10. HOW CRAWLABLE IS YOUR SITE?
Crawl efficiency
@tristan_pirouz

11. EXAMPLES OF OVER-INDEXING
@tristan_pirouz

12. A LACK OF NOINDEX CAN RESULTS IN AN INDEX BLOAT CAUSING A DILUTION OF LINK
JUICE AND CONSTRAINED CRAWL BUDGET
Lack of noindex
@tristan_pirouz

13. DO INCLUDE A CANONICAL TAG ON EVERY PAGE, WITHOUT EXCEPTION
Lack of canonical tags
Prevent any possible duplication. Even if there are no
other versions of a page, then that page should still
include a canonical tag that links to itself
Source: Cononical Tags: Easy Dos and Don’ts, DeepCrawl
@tristan_pirouz

14. CONTROLS
@tristan_pirouz

15. UNDERSTANDING HOW TO USE NOINDEX, DISALLOW & NOFOLLOW MEANS YOU CAN
ADVISE SEs AROUND
What to consider when optimizing your crawl efficiency?
You want to read more: https://www.deepcrawl.com/knowledge/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/
You want to read more: https://www.deepcrawl.com/knowledge/best-practice/noindex-disallow-nofollow/
Backlink Authority
CrawlEfficiency
meta canonical
robots.txt disallow
meta noindex
rel nofollow
meta noindex + rel nofollow
robots.txt noindex
@tristan_pirouz
CrawlEfficiency
Backlink Authority

16. DETERMINE YOUR PERFORMANCE RELATIVE TO YOUR COMPETITORS
How am I performing relative to my competitors?
Determine your performance relative to
your competitors
@tristan_pirouz

17. IS THE ABILITY TO CONSTRUCT YOUR WEBSITE ON THE FORESIGHT OF THE MARKET
LEADERS WEBSITE ARCHITECTURE SIZE
Concept of architecture
3 Values:
Market Size: What is the average website
size set by market leaders?
Actual Size: What is the actual website size
of the given site?
Indexed Size: What are the total number of
returned pages in search results?
@tristan_pirouz

18. WHEN YOUR WEBSITE IS BELOW THE EXPECTED MARKET SIZE
Under-architected
When does Under-Architected occur:
1.When there is a lack of inventory,
products, content
2.Lack of relevant categorization and
facets
@tristan_pirouz

19. WHEN YOUR WEBSITE IS ABOVE THE EXPECTED MARKET SIZE
Over-architected
When does Over-Architected occur:
1.Low relevancy or low value pages
without unique content created
2.Uncontrolled categorization
3.Incorrect management of expired content
@tristan_pirouz

20. THEREFORE…
@tristan_pirouz

21. HOW TO ACHIEVE SMART INDEXING?
‘Smart Indexing’ | Checklist
Over-Indexation
Incorrect Canonical tags
Lack of NoIndex
Lack of Canonical tags
Under-Indexation
Thin Content
Undiscovered Pages
Inefficient Architecture
Slow Performance
Made NonIndexable by mistake
Smart Indexing
@tristan_pirouz

22. HOW YOU WANT YOUR WEBSITE TO LOOK LIKE?
‘Smart Indexing’
@tristan_pirouz

23. MULTIPLE SCENARIOS YOU CAN FIND YOURSELF AS A WEBMASTER
‘Super Smart Indexing’
ArchitectureIndexing
UnderUnder OverOver
@tristan_pirouz

24. Your Site Audit with DeepCrawl
Tristan Pirouz
Head of Business
Development EMEA,
DeepCrawl
Tristan@DeepCrawl.com
@tristan_pirouz