Introduction to web scraping from static and Ajax generated web pages with Python, using urllib, BeautifulSoup, and Selenium. The slides are from a talk given at Vancouver PyLadies meetup on March 7, 2016.

1. Intro to Web Scraping with Python
Vancouver PyLadies Meetup
March 7, 2016
Maris Lemba

2. 2
Before we begin
● Do consider whether the site you are interested in allows scraping by
examining its robots.txt file
– This robots exclusion standard is used by web servers to communicate with web
crawlers and other robots
– Located in the top-level directory of web server
– Can specify which directories the site owner does not want robots to access
– Can have different access rules for different robots
● If you intend to publish the output from your scraping, consider whether it
is legal to do so
– It is generally ok to re-publish “factual data”
#Sample robots.txt file
User-agent: Googlebot
Disallow: /folder/
User-agent: *
Disallow: /photos/

3. 3
Basic components of a web page
● HTML – a set of markup tags for describing web documents
● CSS – style sheet language for describing the presentation of an HTML document
● JavaScript – popular client-side programming language used for web applications
Sample html source
This is what it would look like in the
browser
If we wanted to scrape information from the “Day of the week” table, we can locate it in the
html inside the <table> ... </table> element.
If there are multiple tables, we can use attributes such as id to identify the right table.

4. 4
How the web works
user interface
web server
web browser
database, data
handling, etc
server-based system
user interface
Ajax “engine”
(javascript)
web browser
HTTP request
HTTP request
HTML +
CSS data
web server
database, data
handling, etc
server-based system
XML/JSON or
HTML/CSS or
Javascript data
Schematic adapted from https://en.wikipedia.org/wiki/Ajax_%28programming%29
ClientServer
Javascript
call
HTML+CSS
data
Conventional model of web application Ajax model of web application

5. 5
Scraping ad hoc vs at scale
● Use case 1 (basic scraping): I need to scrape a few pages,
maybe crawl a small site. I can wait to have all data retrievals
done sequentially.
– This is what we will focus on in this talk
– Two scenarios:
● All the data you need is in the HTML files you get from the server
● The page is dynamically generated using Ajax
● Use case 2: I am crawling large sites periodically. I need task
scheduling, asynchronous retrieval, pipeline automation, and
lots of other tools to make this easier.
– You may want to look into a crawling/scraping framework like Scrapy

6. 6
Agenda
● We'll walk through examples for the two
scenarios under “use case 1”
– We will use two web sites,
pacificroadrunners.ca/firsthalf/ and
ultrasignup.com, to scrape the results for all
years of two running races
● Pacific Road Runners First Half Marathon
● Chuckanut 50K

7. 7
Basic scraping: static HTML
● Tasks:
– Inspect the HTML source to locate where in the Document Object Model (DOM) the data of
interest resides
● Right-click on the page in browser window and select “View Page Source”
– Retrieve the HTML file using built-in urllib library (note that the interface to this library has
changed between Python 2 and 3)
– Parse the HTML file and extract the data of interest, using either a built-in parser like html.parser,
or third-party libraries such as lxml, html5lib or BeautifulSoup
● Which library you choose for parsing depends on
– how sloppy your html is: if your html is “incorrect”, the parsers may handle it differently
– whether you want to maximize speed (choose lxml)
– what external dependencies you can live with (lxml is a binding to popular C libraries libxml2
and libxslt)
– how “easy” you want the interface to be (BeautifulSoup has an easy-to-get-started API and great
documentation with examples)

8. 8
urllib library
● High level standard library for retrieving data across the Web
(not just html files)
● urllib library was split into several submodules in Python
3 for working with URLs:
– urllib.request, urllib.parse, urllib.error,
urllib.robotparser
● The urllib.request provides methods for opening URLs
● The urllib.request.urlopen() method is similar to
the built-in function open() for opening files

9. 9
BeautifulSoup library
● Python third-party library for extracting data from html and
xml files
● Works with html.parser, lxml, html5lib
● Provides ways to navigate, search and modify the parse
tree based on the position in the parse tree, tag name, tag
attributes, css classes using regular expressions, user-
defined functions etc
● Excellent tutorial with examples at
http://www.crummy.com/software/BeautifulSoup/bs4/doc/
● Supports Python 2.7 and 3

10. 10
Quick intro to BeautifulSoup
● BeautifulSoup constructor creates an object that
represents original documents as a nested data
structure
●
The simplest way to address a tag is to use its name
●
find() and find_all() methods allow to search for an
element; find() returns the first element found, and
find_all() a list of elements
●
You can access a tag's attributes by treating the tag
like a dictionary
●
You can extract text from the element
Parsing samplepage.html with
BeautifulSoup

11. 11
Example: Results tab of Vancouver First Half race
The main Results page
has links to yearly results

12. 12
HTML source inspection: front page of results
Find the links in the HTML source file corresponding to the rows of yearly results you
see in the browser
Note that the links we are interested
in are all inside
<div class=”entry-content”> ... </div>
The links to each year's page
are inside an <a> tag, as a
value to attribute named href
The text field of the <a> ... </a> element is a
string containing the year of the race

13. 13
Extract all links to individual year's results
Note that the collected links are relative; we can append
them to the main url to get absolute links

14. 14
HTML source inspection: individual year
Find the table of results in the DOM This is the table we want, each
runner's info in <tr> ... </tr>
Each participant's entry is a table
row <tr> ... </tr>.
The text (participant's placement,
name, finishing time, etc) are
inside <td> ... </td> elements
nested in the table row.

15. 15
Scrape results for all years
First five finishers of the 2016 race on
the web page
Let's print out the first five of the
2016 entry in our dictionary

16. 16
What is Ajax
● Name comes from Asynchronous JavaScript and XML, even
though XML is no longer the only format for data exchange
● Refers to a group of web technologies used to implement web
applications that communicate with the server in the background
without interfering with the display of the page or reloading the
entire page
● For web scraping, this means that the data you are interested in
may not be in the source HTML file you receive from the server
but is generated on the client side
● You can use browser development tools (“inspect element”) to
inspect the Ajax-generated HTML that your browser displays

17. 17
Second example: ultrasignup.com
So far, web site looks similar to previous example. We are interested in
scraping results for all years of the race.

18. 18
Results page for an individual year
This is a sample page for 2014 results. It looks like the results
are in a neat table.

19. 19
Inspecting the DOM: the result table is not
filled out in the source HTML
However, the <table id=”list”> element is empty
when we view source html file.
This means we can't simply extract the result table
from the HTML file that the server sent to us like we
did in the previous example.
We see all result rows in
<table id=”list”> </table>
when inspecting the element
in browser developer tools
such as Mozilla Firebug
(which shows the generated
HTML)
Again, the details of the
results are within the
<td> ... </td> element inside
each <tr> ... </tr> element

20. 20
Scraping dynamically generated HTML
● Your scraping system needs to be able to execute JavaScript and
either retrieve the results directly, or generate the HTML as seen in
the browser and scrape data from that
● A popular choice for the latter is to use Selenium, a suite of tools for
browser automation
– Python's binding to Selenium suite is the library selenium
– Selenium supports several browsers (Firefox, Chrome, etc), but for scraping
a headless non-graphical browser like PhantomJS is a good choice
– Once Selenium has generated the HTML after running JavaScript, you can
either access it as a string and use this as input to BeautifulSoup for data
extraction, or you can navigate the DOM and find elements within Selenium

21. 21
Quick intro to selenium
● Selenium WebDriver accepts commands, sends them to the browser, and retrieves results
● You can navigate and search the DOM
– find_element_by_id('my_id')
– find_elements_by_tag_name('table')
– find_elements_by_class_name('my_class_name')
– find_elements_by_xpath()
● For example, find_elements_by_xpath(“//table[@id='list']”) will find a table with the id = “list”
● Tutorial on xpath syntax: www.w3schools.com/xsl/xpath_intro.asp
– and other methods (using css selectors, link text, etc)
● You can click on links using WebDriver's click() method
● You can retrieve the generated html source by calling .page_source on webdriver
instance and parse it outside of selenium
● More on Selenium with Python:
http://selenium-python.readthedocs.org/index.html

22. 22
Scraping ultrasignup race results using
Selenium and PhantomJS

23. 23
And it works!
Note that the first two lists (rows in web page table) in our dictionary corresponding to
a year do not actually contain finisher results.
Let's ignore those, and print out the subsequent results for the first 12 finishers.
This table is from the 2014 results page for
comparison: the results match

24. 24
Additional resources for scraping projects
Additional third-party libraries you may want to look at:
● requests library
– Filling out forms/login information before accessing data and managing cookies
● mechanize library
– Navigating through forms, session and cookie handling
– Does not yet support Python 3
– Does not run JavaScript
● scrapy library
– Framework for web crawling and scraping: managing crawling, asynchronous
scheduling and processing of requests, cookie and session handling, built-in
support for data extraction etc
– Does not yet support Python 3

25. 25
References for further reading
● BeautifulSoup
– http://www.crummy.com/software/Bea
utifulSoup/bs4/doc/
● Selenium with Python
– http://selenium-python.readthedocs.org/
● General web scraping
– Web Scraping with Python by Ryan Mitchell,
O'Reilly 2015