Prezdev parsing & crawling libs
•
0 j'aime•555 vues
Parsing & crawling libs we don't use at Pricing Assistant
![Beautiful Soup
built on top of lxml and html5lib
higher levels commands
handles encoding itself
example :
frombs4importBeautifulSoup
soup=BeautifulSoup(html_doc)
soup.title
#<title>TheDormouse'sstory</title>
soup.title.name
#u'title'
soup.title.string
#u'TheDormouse'sstory'
soup.title.parent.name
#u'head'
soup.p
#<pclass="title"><b>TheDormouse'sstory</b></p>
soup.p['class']
#u'title'
soup.a
#<aclass="sister"href="http://example.com/elsie"id="link1">Elsie</a>
soup.find_all('a')
#[<aclass="sister"href="http://example.com/elsie"id="link1">Elsie</a>,
# <aclass="sister"href="http://example.com/lacie"id="link2">Lacie</a>,
# <aclass="sister"href="http://example.com/tillie"id="link3">Tillie</a>]
soup.find(id="link3")
#<aclass="sister"href="http://example.com/tillie"id="link3">Tillie</a> 2 / 13](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Contenu connexe
Tendances
Similaire à Prezdev parsing & crawling libs
Similaire à Prezdev parsing & crawling libs (12)
Dernier
Dernier (20)
Prezdev parsing & crawling libs
- 1. Parsing & Crawling libs WE DON'T USE 1 / 13
- 2. Beautiful Soup built on top of lxml and html5lib higher levels commands handles encoding itself example : frombs4importBeautifulSoup soup=BeautifulSoup(html_doc) soup.title #<title>TheDormouse'sstory</title> soup.title.name #u'title' soup.title.string #u'TheDormouse'sstory' soup.title.parent.name #u'head' soup.p #<pclass="title"><b>TheDormouse'sstory</b></p> soup.p['class'] #u'title' soup.a #<aclass="sister"href="http://example.com/elsie"id="link1">Elsie</a> soup.find_all('a') #[<aclass="sister"href="http://example.com/elsie"id="link1">Elsie</a>, # <aclass="sister"href="http://example.com/lacie"id="link2">Lacie</a>, # <aclass="sister"href="http://example.com/tillie"id="link3">Tillie</a>] soup.find(id="link3") #<aclass="sister"href="http://example.com/tillie"id="link3">Tillie</a> 2 / 13
- 3. Beautiful Soup - Y U no use me ? yet a new kind of soup gotta go a step lower crappy acronym 3 / 13
- 4. html5lib implements the WHATWG HTML5 specification. will inject tbodyand such is actually usable directly in lxml, we could use it 4 / 13
- 5. html5lib - Y U no use me ? Y would I ? uuhh 5 / 13
- 6. Scrapy write rules built-in handling of compression, cache, cookies, authentication, user- agent spoofing, robots.txt handling, statistics, crawl depth restriction, etc extendable : middlewares, extensions, and pipelines Web management console for monitoring and controlling your bot Telnet console for low-level access to the Scrapy process fromscrapy.itemimportItem,Field classTorrentItem(Item): url=Field() name=Field() description=Field() size=Field() fromscrapy.contrib.spidersimportCrawlSpider,Rule fromscrapy.contrib.linkextractors.sgmlimportSgmlLinkExtractor fromscrapy.selectorimportSelector classMininovaSpider(CrawlSpider): name='mininova' allowed_domains=['mininova.org'] start_urls=['http://www.mininova.org/today'] rules=[Rule(SgmlLinkExtractor(allow=['/tor/d+']),'parse_torrent')] defparse_torrent(self,response): sel=Selector(response) torrent=TorrentItem() torrent['url']=response.url torrent['name']=sel.xpath("//h1/text()").extract() torrent['description']=sel.xpath("//div[@id='description']").extract() torrent['size']=sel.xpath("//div[@id='info-left']/p[2]/text()[2]").extract() returntorrent 6 / 13
- 7. Scrapy Shell scrapyshell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/" [s]AvailableScrapyobjects: [s] crawler <scrapy.crawler.Crawlerobjectat0x1e16b50> [s] item {} [s] request <GEThttp://scrapy.org> [s] response <200http://scrapy.org> [s] sel <Selectorxpath=Nonedata=u'<html>n <head>n <metacharset="utf-8'> [s] settings <CrawlerSettingsmodule=None> [s] spider <Spider'default'at0x20c6f50> [s]Usefulshortcuts: [s] shelp() Shellhelp(printthishelp) [s] fetch(req_or_url)Fetchrequest(orURL)andupdatelocalobjects [s] view(response) Viewresponseinabrowser In[1]:sel.xpath('//title') Out[1]:[<Selectorxpath='//title'data=u'<title>OpenDirectory-Computers:Progr'>] 7 / 13
- 9. Scrappy - Y U no use me ? I want to ! how to integrate scrapy daemon with MRQ ? have to implement a proxies rotating middleware 9 / 13
- 10. Scrapy - Scrapinghub 10 / 13
- 11. Did I miss something ? mechanize, twill => shitty deprecated crawling modules i forgot their names => black boxes paid services 11 / 13
- 12. Did I miss something ? GET LARGE 12 / 13
- 13. Adrien Di Pasquale 16/05/2014 13 / 13