Presentation on Web Crawler, Search Engine and Web Optimization.

1. Web Crawler, Search Engine and Web Optimization
Dinesh Devkota

2. Web Crawling Algorithm.
Applications of Web Crawling.
Relation to Search Engines (Google).
General overview of Google’s Algorithms.
Online tools Demo.
Web Optimization.
SEO, Blackhat/Whitehat SEO.
Concerns for Web Developer.
Nested Topics within Crawler

3. ◇ Web is a Graph (Pages = Nodes, edges=links)
◇ A Crawler(Spider) performs graph traversal.
1. url= PriorityQueue.pop(); // A priority Queue is built with other algorithms
2. Html=get(url.host, url.page);
3. Store(url,html)
// can have some other procedure to built the queue
//sort queue based on PageRank.
//change the rank of backlink page.
4. For each url in html
PriorityQueue.push(url,time);
Web Crawler Algorithm

4. ◇Page Rank and relation to Priority Queue
￮ Considers the following factor
Back Links and reputation of backlinks
Broken Links , 404
Internal Server Errors 500,503
Bounce Rate
￮ Demo
Response Time/Page Speed
https://developers.google.com/speed/pagespeed/insights/?url=http%3A%2F%2
Fwww.selu.edu%2F&tab=desktop
SEO
https://www.woorank.com/en/www/selu.edu
https://www.woorank.com/en/www/dineshdevkota.com
Google Algorithms

5. ◇Black Hat SEO
￭ Polluted Search Engine Result
Notes to Developer
Write code using best practices.
Don’t just make something work.
Ethical/Moral Concerns

6. ◇S. Brin, L. Page, “The Anatomy of a LargeScale Hypertextual Web Search
Engine”, http://infolab.stanford.edu/~backrub/google.html
◇Bounce Rate, http://en.wikipedia.org/wiki/Bounce_rate
◇ Backlink, http://en.wikipedia.org/wiki/Backlink
◇ Web crawler, http://en.wikipedia.org/wiki/Web_crawler
◇Search engine optimization,
h​ttp://en.wikipedia.org/wiki/Search_engine_optimization
◇Bounce Rate, http://en.wikipedia.org/wiki/Bounce_rate
◇Google Developer Page Speed,
https://developers.google.com/speed/pagespeed/insights/?url=www.selu.edu&tab=d
esktop
◇SEO Test Page, https://www.woorank.com/en/www/selu.edu
◇Web Crawler, http://en.wikipedia.org/wiki/Web_crawler
◇404, 500,503 Status code, http://www.w3.org/Protocols/rfc2616/rfc2616-
sec10.html
◇Working of Web Crawler, https://www.youtube.com/watch?v=qbQ8ZSEwniM
◇Backhat SEO, http://www.webopedia.com/TERM/B/Black_Hat_SEO.html
◇Whitehat SEO, http://www.webopedia.com/TERM/W/White_Hat_SEO.html

7. Thanks!
Any questions?
You can find me at:
◇ www.dineshdevkota.com
◇ info@dineshdevkota.com