Presentation slide from my Talk on Python User Group Nepal Meetup #8. Demo code available on https://github.com/s2krish/dn-python-meetup-8

1. Web Scraping Technologies
KU IT MEET
Dhulikhel

2. Krishna Sunuwar
Founder & CEO, ontreat & Jaljale
@s2krish

3. Demo
• Google translation

4. What is scraping?
• Extract data form web
• Why extract?
– To structured
– For mining
– No APIs available
– Business Intelligence
– Abuse/Spam

5. Components
• Network
• HTML Parser
• Data Saving

6. HTTP/Network Libraries
• urllib2 – library for opening URLs
• requests – HTTP library for humans
• mechanize - Stateful programmatic web
browsing

7. Parsing HTML
• BeautifulSoup
• lxml (xpath, cssselect)
• HTMLParser (html.parser)
• regex

8. The Challenges
• At Server
– Throttle Limit
– IP Ban
– Authentication Required
– CAPTCHA
• At Client
– Broken HTML
– JavaScripts
– Badly structured

9. Tips
• Rotate User Agent
• Use different IPs (Use Proxy)
• Don’t go fast
• Break CAPTCHA - Deathbycaptcha

10. Applications
• Scrapy Framework

11. Thank you
Q & A