how_first_click_free_can_help_in_SEO

Amaliegade 3-5, 3. sal
1256 Copenhagen K
Tel: 7027 8089
Mail: info@guava.dk

Copenhagen, Wednesday, 25 May 2011

FIRST CLICK FREE IMPLEMENTATION
Client: NE.SE
Website: http://www.ne.se
Report date:
Version:
Author:

th

26 April, 2011
1.0
Mark Edmondson & Kristoffer Ewald (Guava DK)

Introduction
This report outlines a set of recommended updates for the existing website that will enable
Google’s “First Click Free” implementation. (FCF)

Version: 1.0- Created: April, 2011 - Last modified: May, 2011
Nuera Limited is registered in England number 3849708. Registered Office:
70 Conduit Street, London. W1S 2GF. Guava is a subsidiary of Guava A/S.
International Tel: +44 (0)1209 722340, International Fax: +44 (0)1209 717263

1

SEO recommendations
Page
First Click Free Implementation........................................................................................................................................... 1
Introduction ....................................................................................................................................................................... 1
Technical Implementation ...................................................................................................................................................... 3
1. Detecting Referrals .................................................................................................................................................. 4
Verifying Googlebot with reverse DNS lookup .................................................................................................................. 5
2. Meta tag – noarchive................................................................................................................................................ 6
3. Robots.txt ............................................................................................................................................................... 7
4. Cookies for FCF ...................................................................................................................................................... 8
Implementing Flash LSO cookies ........................................................................................................................................ 9
USER ABUSE CONSIDERATIONS .................................................................................................................................... 10
3rd PARTY WEBSITES ....................................................................................................................................................... 12
Other Search Engines ....................................................................................................................................................... 12
Social Media websites ...................................................................................................................................................... 12
SEO considerations with backlinks .................................................................................................................................... 12
Partner Websites / Content Syndication .............................................................................................................................. 12


2

TECHNICAL IMPLEMENTATION
First Click Free for Web Search - http://googlewebmastercentral.blogspot.com/2008/10/first-click-free-for-websearch.html
This first section of the report focuses on the technical requirements to have Google’s First Click Free (FCF) working on
the website.
Google’s guidelines for First Click Free are:
Google’s crawlers must be able to access the content of the site without the need for cookies or filling out forms
Suggested implementation is to configure webservers to not serve the registration pages to the crawler
Allow crawler in robots.txt
Multi-paged articles must all be accessible for the user
Page displayed must be identical to users to that which Google sees
This document will also look at what ways to implement enhancements to this package such as:
Implementing five articles per day with cookies
Complementary meta-tags
Using similar techniques for other traffic sources apart from Google
Blocking user’s attempts to abuse FCF


3

1. Detecting Referrals
The Referrer request-header - RFC: http://tools.ietf.org/html/rfc2616#section-14.36
All implementations of First Click Free need a mechanism to detect where the visitor has arrived from. This is preferably
implemented by using the HTTP referrer in the server headers, rather than on-page scripting such as JavaScript, since
this could be circumnavigated.
For FCF, the referrer is recommended to look for user agents of the form:

*.google.*
Since this will cover the combinations of its search engines around the world (www.google.se , www.google.com, etc.)
A full list of Google User agents is here: http://www.google.com/support/webmasters/bin/answer.py?answer=1061943 –
this may be useful if looking to only give access to Googlebot-Video or Googlebot-Mobile, for example
Using Apache and .htaccess as an example, this would typically be implemented similar to the following:

#Have a 403 error document for Denied users with subscription page
ErrorDocument 403 /subscribe-now.html
#set the first click free variable
SetEnvIfNoCase Referer "*google*" first_click_free=1
#Send first_click_free past registration screen, others get 403
<FilesMatch "(/subscription-content/.*)">
Order Deny, Allow
Allow from env=first_click_free
Deny from all
</FilesMatch>
The above is intended as example only, and it may be preferred to use a server side scripting language such as JSP to
handle the referral requests – however, the principle remains the same: detect if the referral is from *.google.* and
redirect past a subscribe request.


4

Verifying Googlebot with reverse DNS lookup
Google Help files: http://www.google.com/support/webmasters/bin/answer.py?answer=80553
Users could try to circumnavigate the restrictions by setting their browser’s User-Agent to Googlebot – to prevent this a
reverse DNS lookup can be used to verify that the bot comes from googlebot.com and action taken accordingly.
The example given in the link above is:

> host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer
crawl-66-249-66-1.googlebot.com.

> host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1


5

2. Meta tag – noarchive
NoArchive Information: http://noarchive.net/meta/
It is also recommended to place a “noarchive” meta tag on the pages under the FCF regime. This metatag tells Google
not to keep a version of the webpage which could be accessed via a “cache:” command in Google to access page
content:

To prevent this, place this tag in each page to be available under FCF:
<meta name="robots" content="noarchive" />


6

3. Robots.txt
http://www.robotstxt.org/
Ensure that Googlebot is allowed in the robots.txt file for all FCF content – that is the User-agent in the file found at
ne.se/robots.txt does not block Googlebot – the current robots.txt file is:

User-agent: *
Disallow: /admin/
Disallow: /clusterjsp/
Disallow: /dev/
Disallow: /dns/
Disallow: /error/
Disallow: /modals/
Disallow: /resources/
Disallow: /temp/
Disallow: /xhp
Disallow: /static/xhp
Disallow: /google69eed29ee822c5b1.html
Disallow: /ip.jsp
Disallow: /newnormalpage.jsp
Disallow: /testads.html
Disallow: /testads.jsp
Disallow: /testanalytics.jsp
Disallow: /testlayout.jsp
Disallow: /user/
Disallow: /openschool/
Disallow: /sok/
Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=f&to=l
Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=l&to=q
Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&from=q
Sitemap: http://www.ne.se/rss/feed.jsp?option=googlebot&to=f


7

4. Cookies for FCF
Cookies are not strictly necessary for FCF, however may be needed if:
FCF content is available across paginated articles, and a user needs to be tracked across a session
The five (or more) articles a day per user implementation is required
In such cases, a first party cookie is needed to be set to track users within and between sessions.
For paginated articles a session cookie (expire=0) is all that is necessary
For the Five clicks a day the expiration should be 24 hrs and include a counter on article views.
Cookie Page Request Diagram


8

Implementing Flash LSO cookies
Using cookies for tracking users isn’t ideal as it can be circumnavigated quite easily – a user can use another browser,
flush their browser cookies etc., however if the cookie information can be moved to a Flash LSO the cookie information is
more rarely removed and can be more reliable.
Guava have a solution that can provide this extra robustness in the cookies, which also allows more advanced
implementation of Google Analytics tracking. How these files communicate between the Flash and the client javascript
has been included with this report – please review this code and the examples provided.
The example in the files (ne.se.example.html) shows very simple the methods of setting, retrieving and resetting a
variable in the LSO using Javascript.
Left out is the actual synchronisation with a possible browser cookie, but it should be fairly simple to build this upon the
methods in the example:
1) If a browser cookie exists: Store the cookie value in the LSO
2) If a browser cookie does not exist but the LSO exists: Create browser cookie with the value in the LSO
3) If neither exists: Create browser cookie and store the value in the LSO


9

USER ABUSE CONSIDERATIONS
Some web-savvy web users may look to work around the measures put in place for First Click Free. An example of
some techniques that are used against the New York Times is shown here
http://www.makeuseof.com/tag/5-ways-york-times-paywall/
In this section we look at common methods and what technical measures can be in place to help prevent them:
Disabling Javascript – if the referral or cookie setting scripts are done via JavaScript, simply turning JavaScript off will
enable users to carry on as if the script wasn’t there.
Likelihood: 5 out of 5
Prevention: Avoid security based on JavaScript.
Faking Referrals – with plugins or other methods, a user can make his browser look like it is a Google crawler when
surfing the site. This way any privileges given to Googlebot will also apply to that user, including access to FCF content.
To prevent this, only techniques such as reverse DNS lookups to verify the User Agent will work.
Prevention: reverse DNS lookup to verify user-agents
Blocking/Flushing Cookies – the most common method, this will allow a user to navigate restrictions based on browser
cookies, such as those tracking how many articles they have seen that session or day. If the Flash cookies LSO are
used this will go a long way to stopping this occurring, since Flash cookies are a lot harder to remove. Also, tracking the
user’s IP address as well and comparing them across cookies should flag up users who are holding multiple sessions.
Prevention: Compare cookie to IP address, LSO flash cookies
Using another Browser – this is related to the cookie issue above: if a user uses another browser such as Internet
Explorer vs Firefox, then a new set of cookies are needed. The only way to prevent this if they are using the same PC is
to also track the IP addresses and compare them across cookie sessions.
Prevention: Compare cookie to IP address, LSO flash cookies
Using Google search – a common method which prompted the Five Clicks a day policy, users could take advantage of
the first click free from Google domains to do searches in Google that would bring up the content they knew the user
could see. They could then click through and read the article at will. This would need to be prevented by tracking a user
across sessions, either via IP address or cookies.
Prevention: Cookies to limit access
Using Google cache – whilst Google crawls the web, it keeps a copy of every page available in its search results in the
Google cache (sitting in RAM on its numerous servers) – a user could access this information by using a “cache:www.urlto-look-at.com” command to get access to all content Google can see. The use of the “noarchive” meta-tag is meant to
prevent this.
Prevention: “noarchive” metatag


10

Printing out the web article – A user may look to take the article and print out a hard copy using the browsers “Print”
command. If this is not wanted for non-paying subscribers, look at using stylesheets to make the print page blank for
non-logged in users – i.e.

<style type="text/css" media="print">
body {visibility: hidden; display: none;}
</style>
Prevention: CSS Hacks
Copy-Paste – A user may also use their PC to copy-paste article information and place in their own format – an even
worse scenario will be for them to publish this information on the web giving away the content with no restrictions. Whilst
little can be done to prevent this against a determined user, steps such as disabling right click on a page will prevent the
casual user (Although create usability concerns) – these commands are usually still easily circumnavigated by turning
JavaScript off.
Other methods include making it harder by turning off the copy-paste highlighting for your page:

ondragstart="return false" onselectstart="return false"
Another solution is to place a transparent gif across the entire article that is copied instead when users try to.
A final and more marketing focused solution would be to use something like the Tynt service, which inserts links back to
the website when someone copy-pastes – in this way there may at least be a link back to your website.
http://www.tynt.com/publisher-tools
Prevention: Disabling right click?


11

3rd PARTY WEBSITES
rd

This section deals with how First Click Free affects other 3 party websites that are not Google, and suggested
strategies on how www.ne.se can best deal with them.

Other Search Engines
Whilst Google is the dominate search engine for the moment, other search engines such as Bing and Yahoo are still
used and should be catered for if at all possible. In many cases this will simply mean include those search engine User
agents in the same process as Google, including cookies and referrals.
The most pertinent user agents will be:
Google Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Full list including mobile - http://www.google.com/support/webmasters/bin/answer.py?answer=1061943
Bing msnbot/1.1 (+http://search.msn.com/msnbot.htm)
Yahoo Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Social Media websites
Getting mentions on other social media sites can be good for marketing and page purposes, and it may be considered
worthwhile payment in getting a social mention to give some free access to users who follow the link.
Again, this could be restricted on a per user basis, or per referral by only allowing 10 visits per day from that site etc.

SEO considerations with backlinks
To compete well in SEO, backlinks are part of the essential mix. If the site restricts content then naturally less people will
link to the content as it will be less useful for readers hitting a paywall. A good incentive to increase linking to the site
would be to offer users viewing the content from that page a free view of the page.
Factor in the cost of that link if it had to be bought for marketing and SEO purposes verses the revenue lost from
potential subscribers, and it may look to be worthwhile.
This could be coupled with only allowing users with a paid for subscription to be able to use the links for their own
websites, ensuring that at least one subscriber has paid per link and also providing additional incentives to become a
member of www.ne.se

Partner Websites / Content Syndication
Rather than let any website that links to www.ne.se gain free access, websites that are pre approved partners could
appear on a “white-list” of websites that allow its users to see www.ne.se content. Effectively this is what Google will be
receiving, and other websites will generally give better converting traffic than Google.


12

how_first_click_free_can_help_in_SEO

Recommandé

Recommandé

Contenu connexe

Plus de Indranil Bhattacharjee

Plus de Indranil Bhattacharjee (13)

Dernier

Dernier (20)

how_first_click_free_can_help_in_SEO