Crawling the Web (for fun and profit) discusses how web crawlers systematically browse the internet and how they allow search engines to index only a portion of the deep web. The document also outlines several potential uses for custom web crawlers, including backup archiving, social network analysis, trend tracking, and building one's own search engine. With a basic framework like Scrapy, even individuals can build crawlers to extract useful data and information from the internet.

1. Crawling the Web
(for fun and profit)
Federico Feroldi

2. “A Web crawler is a computer
program that browses the World
Wide Web in a methodical,
automated manner.”
Wikipedia
Picture greetings to photoholic1 --LennyB

6. Search engines only show you
what their crawlers can catch
Picture greetings to jimbrickett

7. The deep web contains a
lot of valuable information
e-commerce finance
transportation
yellow pages
medicine
government
opinions real estate
personal
intranets social
Picture greetings to tricky ™

8. Dig deeper with
your own crawler
Picture greetings to Super*Junk

9. Information
=
Competitive
Advantage
Picture greetings to mastrobiggo

10. B a cku p h i s t o r i c a l
data: web sites, blogs

11. Social network analysis: find
influencers and interests
based on “social circles”

12. Find what people like

13. Sentiment analysis: find
what people say about
your brand or product

14. Trending topics
and products

15. Competitor price tracking

16. Real estate

17. Personal data and
online reputation

18. Do It Yourself
Picture greetings to vic_206

19. Anybody can build
a search engine

20. Scrapy Scheduler Internet
architecture
Re
qu
es
Data
ts
Item Scrapy
Downloader
pipeline Requests Engine
es
Ite ns
ms po
R es
Spider

21. Twitter social graph crawler
with Scrapy in 150 LOC

22. The Web is much bigger
than what you can search
with Google

23. Thank you
federico@cloudify.me
twitter.com/cloudify