1. The Evolution Theory of Spider
逐浪@淘宝北京研发中心

2. Topic
• Simplest Spider
• Framework(Scrapy)
– Abstraction
– IO Model
• Evolution
– Architecture
– Module
• Simplify
• Do it

3. Simplest Spider
import urllib, lxml, MySQLdb
urls = [...]
for url in urls:
html = urllib.urlopen(url).read()
item = parse(html)
save(item)

4. Framework(scrapy)---Abstraction
• Workflow abstraction
– Work abstraction
– Flow abstraction
• Task abstraction
– Request/Response
– Task
• Platform abstraction
– Linux
– Windows

5. Framework(scrapy)---Abstraction
• Workflow abstraction
– Work abstraction
• Schedule
• Download
• Extract
• Pipeline
– Immutable and variable
• Scrapy perspective
• My perspective
– What is spider class?
• Variable works abstraction

6. Framework(scrapy)---Abstraction)
• Workflow abstraction
– Flow abstraction
• Apache vs Scrapy
– Why control center
• Control ability
– Error Retry
• Extensibility
• Module independency

7. Framework(scrapy)---Abstraction)
• Task abstraction
– Request/Response
– Task
• Platform abstraction
– Linux
– Windows

8. Framework(scrapy)---IO Model)
• Concepts
– Synchronous/Asynchronous(IO state consistency)
– Block/Nonblock(Process/Thread status)
• IO Model
– Synchronous Block(urllilb)
– Asynchronous Block(spynner, gevent, nginx_lua)
– Asynchronous
NonBlock(twisted, reactor, proactor)
– Synchronous NonBlock(mistery)

9. Evolution---Architecture
• Why
– Scrapy
• Single Process
– Etao Spider v1
Etao Spider v1

10. Evolution---Architecture
• Distributed on Processes
• Distributed on Machines
• How
– Thrift/HSF
– Interact
• Direction
– Dependent
• Task queue
– Stateless

11. Evolution---Module
• Downloader
– Render
• Webkit(Javascript)
• Webkit(AJAX):click simulation, event notify
• Webkit(CSS): css feature
– ADSL Proxy
• How to get
– Why scan by ourselves
• How to use
– Why nginx

12. Evolution---Module
• Extractor
– Wrapper induction
• Semi automation
– Firefox extensions
– How to improve
– Templates management
• Full automation
– Scrapy extract tool
• Cascade extraction supported

13. Evolution---Module
• Scheduler
– FIFO Queue
– Priority Queue
• Seed weight
• Smallest interval
• User Query distribution
• User Query importance
• Webpage change characteristics

14. Evolution---Module
• Processor
– Mysql
– Redis
– Hadoop

15. Simplify
• IO module
– Synchronous block
• No Middleware supported
• No Item Loader
• No Framework
• No …

16. Do it
• Time Estimation
– Basic 1-2 month
– Improve