If you are facing issues with your web pages not crawled by Googlebot, This Tutorial may help you out. Contains information: Robots.txt Robots.txt Format Robots meta noindex tag Robots meta noindex tag format Robots meta nofollow tag Robots meta nofollow tag format link rel=nofollow Tag link rel=nofollow tag format This complete tutorials will guide you to optimize your website well on Google.

1. Robots.txt vs Rel=Nofollow vs
Meta Robots Nofollow tutorial
www.gsesoftsolutions.com
A Tech Blog
Twitter: @GSESoftSolution
My Facebook Page
@gsesoftsolutions

2. Robots.txt
1. Web site owners use the /robots.txt file to give
instructions about their site to search engines
robots
2. Does not crawled by Googlebot
3. Robots.txt file shows that what content you do
not want indexed and give instructions to Search
Spiders
4. Using Wildcards incorrectly can be very
expensive and does not let your web pages
crawled.

3. • Disallow indexing of everything
User-agent: *
Disallow: /
• Allow indexing of everything
User-agent: *
Disallow:
• Disallow indexing of a specific folder
User-agent: *
Disallow: /folder/
Robots.txt Format

4. Robots meta noindex tag
1. Prevent your content from appearing in Google
Search Results
2. Crawled by Googlebot
3. Links on a noindex page are still crawled by
search spiders even if the page does not appear
in the search results (unless they are used in
conjunction with nofollow on that page).
4. Consumes Page Rank but do not pass it to
Another pages.

5. To prevent all robots from indexing a page on
your site
<meta name="robots" content="noindex">
Preventing only Google's robots from
indexing the page:
<meta name="googlebot" content="noindex">
Robots meta noindex tag format

6. Robots meta nofollow tag
1. Do not allow Page rank to flow out
2. Crawled destination page only by Googlebot
3. Links on a noindex page are still crawled by
search spiders even if the page does not
appear in the search results (unless they are
used in conjunction with nofollow on that
page).
4. Prevent from Multiple content values

7. To prevent : Page rank does not flow out
<META NAME="ROBOTS" CONTENT="NOINDEX,
NOFOLLOW">
Does not pass Page Rank
<META NAME="ROBOTS"
CONTENT="NOFOLLOW">
Robots meta nofollow tag format

8. link rel=nofollow Tag
1. Do not allow link and make it nofollow not to
pass page rank
2. Crawled destination page only by Googlebot
3. Links are not crawled by Google and does not
pass page rank juice
4. Prevent from Bad links or tell search engine
about link character

9. To prevent : Page rank does not flow out
Don't follow links on this page" or "Don't
follow this specific link."
<a href=“login.php" rel="nofollow">sign in</a>
link rel=nofollow tag format

10. Important Tips with Bonus
• If you are facing issues with your web pages
not crawled by Googlebot, This Tutorial may
help you out.
• http://www.seobook.com/robots-txt-vs-rel-
nofollow-vs-meta-robots-nofollow
• http://www.seobook.com/archives/002361.sh
tml(For Wild cards used in Robots.txt)

11. Thanks for Watching 
www.gsesoftsolutions.com
A Tech Blog
Twitter: @GSESoftSolution
My Facebook Page
@gsesoftsolutions